Affiliation
American Association of Variable Star Observers (AAVSO)
Fri, 09/26/2014 - 17:06
If you have been waitng for a tool to make it easy to apply transforms to your data, its ready!
All you need are the transform coefficients that characterize your system. If you do not have these yet, just take some observations of a standard field (M67 or NGC 7790) and process them through Gordon Myer's TG program. The TransformApplier application (TA) will apply these coefficients to the WebObs data that you have ready to submit and create a new ubmission with that data all transformed and ready for WebObs.
Details and download of the TA are available at http://www.aavso.org/transformapplier
George Silvis
The AAVSO has pushed out a suite of tools to make the transformation process much easier. I'm wondering if we are using it now. I'd love to here some feedback on issues and questions about the process.
The first step is characterizing your telescope/imager with observations of a standard field. M67 is available now by about midnight. Take a handful of images in each filter. Next step is to process that image data through Gordon's TG program. Using VPHOT makes it pretty easy to prep the images. After that, process your target star data as you normally do to prepare a WebObs submission, but next run it through the TA application to perform the transform, a 5 minute process. That's all it takes!
TranformApplier (TA) has been updated with features to help give you confidence that it was done right: a detailed report showing you what went in, what was done, and what came out. There's also a feature (TC Test) that will run the same process over your check stars; if that result matches the reference values then you can be confident that your target star results are on the money.
So, tell me:
Are you using it?
What could be added/fixed to make it better?
Are there questions about the process? Please ask! Gordon and I are looking to help.
Cheers,
George
Hello George and Gordon
I use TA every clear night/next morning when I reduce data. Been a little cloudy lately. I used both my own spreadsheet and PTGP to determine the coeficients--using NGC7790. I plan to use M67 soon to determine them again.
Thanks again for your hard work. No suggestions at this time.
Gary
WGR
Next item in the maturation of my program. Will be a month or two yet though, when I get to the tail end of my season. Thanks for the work, looks like it will make this much easier.
I'm just waiting on some quality skies to take some images of M67.
Many thanks to those who have put this all together and given us the opportunity to present our observations with even more precision.
Douglas.
Hey, George:
I have used TA a number of times before but today when applying it to observations of S5 1803+78 with Comp 000-BBZ-984 (I used the AUID but the label is 145 on the chart). I first tried Seq table ID X15362BOV and then tried the actual chart ID X15362BON, but in both attempts TA returned my measurements untransformed with a message beneath each entry
"# No CREFMAG available. Possibly bad chart reference No KREFMAG available. Possibly bad chart reference"
I triple checked the table and chart references I entered. In both cases they were the correct IDs (table and chart).
Any idea what is going on?
My input file is attached.
Brad Walter, WBY
I had the same problem. Suspect something amiss in the Seq file.
Gary
Hi Brad,
I tried your submitted file and it appeared to work with the current version of TA which is 2.43.
Is that the version you are using?
George
Try Three for this message. Drupal keeps crashing when I open spell check.
Curiouser and curiouser: I tried several times with both version 2.38 and 2.43. neither worked. however, I just tried 2.43 again and everything worked normally. The program must not have been able to communicate with AAVSO earlier.
Attached is a little Excel file format I use to record all of the inputs and outputs to TA. I have found that when you save the Submit page as a txt file, you have to edit out extra tabs in the header comment lines and the final CR/LF that Excel adds after the line of data.
It is too bad that WebObs limits the notes field to 100 characters. I think it would be great to store the transformation information with each record.
George, a question of clarification: If you are doing multicolor transformations for a particular object, say B, V and I, how do you insert "#CREFMAG=" lines? Do group the observations for each filter together and insert a new "#CREFMAG=" line before each filter group?
Brad Walter, WBY
Hi Brad,
It looks like TA 2.43 is working for you. A couple notes:
- Yes, editting and saving from the un-transformed window in TA introduces spurious cr/lf's. Drives me nuts too. I'll try again to fix that.
- AID now accepts a super-log comment field. You can see that in the TA output: All the details of the transformation are recorded. If you submit as a file upload these get recorded. If there is a problem using WebObs's individual submit page (I see that it says "keep it brief" under the comment field), then please submit a bug report to HQ. The comment field can now be a mile long.
- The #CREFMAG and #KREFMAG lines are inserted by hand. They apply to the next observation line and then are remembered. Eg. If the line precedes a obs in V with comp 145, then that information will be applied to all the following obs records that are using comp 145 in V. Check this, but that's how its supposed to work.
Cheers,
George
Thanks, George.
I am glad to know that the WebObse Extended format takes idefinitely long comment fields. I won't chop off the comment fields from TA from now on.
From your comments about #CREFMAG it seems that I don't have to group by filter I just put in the comment line before the first observation in a given filter and it remember the filter used in the next following data line. and applies it to that filter for that object for the rest of the file.
Another questionn concerning groups. I often take symmetrically nested sets of data, VBBV, for example. When I submit a file all of the images in the first nested set will be group 1, the next set, group 2, etc.. In the output file the observations have different numbering The first V and B in the first set will all be given 1101,the second half of the set will have designation 1102 for B and V. The next set willbe assigned2103 for the first B and V and 2104 for the next set, and so forth for subsequent sets. That makes me think that when caculating transformations, the program is using B observation and one V observation at a time it isn't using both Bs and both Vs I have labeled as group 1. Is this a correct conclusion? I take the nested pattern deliberately to minimize second order extinction effects. If TA doesn't use all of the observations that are given the same group number, then I need to average the flux of my pairs of observations in each filter in a set, and convert back to magnitudes before submitting to TA to achieve the same result. Is my conclusion corredt that the program only uses one single B and V observations paired together at a time ?
Brad Walter, WBY
Brad
"Another questionn concerning groups. I often take symmetrically nested sets of data, VBBV, for example. When I submit a file all of the images in the first nested set will be group 1, the next set, group 2, etc.. In the output file the observations have different numbering The first V and B in the first set will all be given 1101,the second half of the set will have designation 1102 for B and V. The next set willbe assigned2103 for the first B and V and 2104 for the next set, and so forth for subsequent sets. That makes me think that when caculating transformations, the program is using B observation and one V observation at a time it isn't using both Bs and both Vs I have labeled as group 1. Is this a correct conclusion? I take the nested pattern deliberately to minimize second order extinction effects. If TA doesn't use all of the observations that are given the same group number, then I need to average the flux of my pairs of observations in each filter in a set, and convert back to magnitudes before submitting to TA to achieve the same result. Is my conclusion corredt that the program only uses one single B and V observations paired together at a time ?"
You are correct in your guess: TA is building groups as it works its way top to bottom. So a VBBV set will be transformed as VB and then a BV.
What you can do if you want to combine the V's and B's before transform is use TA's aggregate function. This will combine all the records with common target/filter/comp/check. In your case this will yield 2 combined record and a single BV transform group.
If you want to do it by your selected groups (1,3 and 4 (your group 2 has only one record)) you would need to run TA 3 times, once with each group of records submitted. That is, you would first include just the 4 records from group 1 and process with aggregation turned on.
I might add a feature where you can include Group in the aggregation criteria (ie you could aggregate records with common target/filter/comp/check AND group.) That would for this data set generate 3 BV transform pairs and then one record untransformed (that singleton group 2 record)
George
Brad raised a good point. If an observer is going to include grouping information in his observation records, then TA should use this when aggregating.
Version 2.44 now aggregates, when told to do so, all records with common target/filter/comp/check and group.
Note that TA will still rewrite the group name after transform, adding a serializing suffix. This is to avoid cases where the submitted records overload a group (eg, includes 2 V observations). If you want TA to aggregate, click the aggregate check box. If you don't want the records aggregated by group, remove the group number from the records.
George
If I understand
George,
If I understand correctly version 2.44 will take care of my problem. Using my input file in the string above as an illustration, with this new version that has groupings, if I click aggregate. It won't aggregate all six B observations into one and all seven V observations into one. I would end up with 3 B observations and 4 V observations.
If I have it right, that is great. I can't wait for 2.44 to be posted. Thanks.
Brad Walter, WBY
Brad
Yep, this version will do what you want. It's available now. Your current version can download it by clicking on the message near the top of the application.
George
You're the Man, George. I looked for a new download notice in the Ap right after seeing your e-mail but I guess it wasn't posted yet. I just tried again and there it was and I have downloaded it. That is FAST application upgrading and customer responsiveness. I am really impressed.
Thank you.
Brad Walter, WBY
I applied V 2.44 to the same S5 1803+78 file. It aggregated the pairs in groups 1, 3, and 4 and did not transform the single V observations in group 2. After a moment's thought, that made sense. When I applied V2.44 to the same file and did not aggregate the results, the single V measurement in group 2 was transformed.
However, I see problems that argue against using the aggregate feature for my purposes of combining a very small number of measurements – usually two. Please refer to the attached spreadsheet (or the PDF for those who can’t open an .xlsx file).
The primary issue is that TA derives the error value of the aggregate empirically from the individual magnitude values using the formula for the standard deviation of a population (N rather than N-1 in the denominator, at least for two measurements. I don’t know if you use N or N-1 in the divisor for three or more measurements). While I think this is appropriate for, say, five or more measurements (as long as there is no variability of interest in the signal over the aggregate time span) it can lead to very unrealistic results when only using a very small number of measurements. For example, TA V2.44 and both STDEV.S (standard deviation of a sample) and STDEV.P (standard deviation of a population) functions in Excel give an error value of 0.001 for the group 3 aggregated B filter inputs when the two individual inputs have estimated uncertainty of 0.015 each. The two values in the sample simply by chance are very close to each other. It seems to me that a more realistic approach for sample sizes of four or less is to use the error propagation formula for an average with the squares of intrinsic errors of the individual standardized magnitudes as the variances for the quantities being averaged.
A less important issue is that it appears the program averages magnitudes rather than fluxes. I say appears because Group 1 and 3 aggregates correspond to magnitude averages done in Excel, but group 4 seems to correspond to averaged fluxes, but the differences could be a result of differences in rounding in TA vs. Excel. I agree that in most cases the difference between averaging magnitudes vs fluxes is not significant in relation to the error of the measurements. However, on occasion it can be, and for Group 3 in the spreadsheet it is significant in relation to the too-small error value of 0.001 provided by TA. Gary Walker has convinced me that I should average fluxes instead of magnitudes if only as a matter of rigor for data that may be used by someone at some time for publication.
Please don’t think I am complaining. I really appreciate the program. It saves a lot of time and calculation. The only reason I made this post is that users need to know limitations of any software before applying it. It is very tempting to treat software, particularly when it comes from AAVSO, as a black box that you can apply universally. I use TA and will continue to use it, but for now, at least, I will aggregate my pairs of observations within “nested” sets (e.g. VBBV or in the extreme IRVBBVRI) before plugging into TA.
Brad Walter, WBY