Creating machine readable records from VSP tables

Affiliation
American Association of Variable Star Observers (AAVSO)
Sun, 12/11/2016 - 21:03

If you are familiar with spreadsheet string functions or are at ease with writing scripts what follows will be trivial. If you aren't good at either of those things, then you may find it helpful.

When you generate a sequence table for a VSP chart the only methods I have found for creating a local copy of the information is to print it (for me, that usually means to a PDF), download the web page, or copy and past the table into a spreadsheet. 

The problem with either of the two methods above for obtaining the table as data is that multiple pieces of information are contained in a single field (like sexagesimal and decimal degree coordinates). If you want to use the sequence as a database for automating photometry, multiple pieces of information in a field won't work. You have to extract the individual pieces of information into individual fields. For those who are good with spreadsheet string functions or are good at writing scripts, this is easy. For others without those skills, it means a manual cut and paste process to create a table with only one piece of information in each field of the record (column of the spreadsheet). That is a pain, but is still bearable if you only have a few stars in a sequence. If there are a lot of stars, as in a standard field chart, it becomes very time consuming and painful. 

Attached is a spreadsheet that takes the table for NGC 7790 and creates additional columns with the multiple pieces of information from fields of the sequence table extracted into individual columns. This spreadsheet can be used as a template for any sequence simply by cutting and pasting the formulas in additional columns of a spreadsheet created by cutting and pasting any UBVRI sequence table. If the sequence table doesn't have all of the columns of a UBVRI sequence table, you can still use the formulas in this spreadsheet by placing them in the same RELATIVE position to the columns that are present in the data you cut and past from the sequence table. Then you can simple delete any empty intervening columns. The cell references in the formulas automatically adjust and remain valid as is normal for spreadsheets. There are no macros or scripts involved that need to be modified. Even if you have spreadsheet skills the attachment for NGC 7790 will save the time involved in finding and applying the necessary string functions if you don't have them memorized.  When you save the spreadsheet as a .csv or tab delimited .txt file, which many programs require for import, all of the formulas are saved as the values you see. You may have to delete or "comment out" header rows if required by the program into which you want to import a sequence.

When you cut and paste from the online sequence table it will look a bit different from the attached. I deleted any borders or backgrounds from the formatting. Whether you leave this formatting in place or not makes no difference to the process of separating the information into separate fields. I simply find it easier to read the spreadsheet without the formatting that cutting and pasting brings along with the data. 

Brad Walter, WBY

 

 

 

Affiliation
American Association of Variable Star Observers (AAVSO)
Grabbing VSP photometry

Greetings Walter,

I see you're "screen scrapping" the vsp photometry. It's hard work. The first versions of my PhotomCap application which would grab the phtometry page and create a .STAR file for AIPWin did it this way.

But this method is problematic as it is dependent on the page formatting, something for which there is no guarentee. The preferred, programmatic way to get the photometry data from VSP is to request the XML version of the data. If you have the chartID (eg  2164EAF) you can request it like so:
http://www.aavso.org/apps/vsp/api/chart/2164EA/?format=xml

Now dealing with an XML file presents its own challenges. You can load XML into Excel and Excel has tools for manipulating the XML. There is a thread in the Software Development forum ( https://www.aavso.org/apis-aavso-resources ) which goes into some detail on how to parse XML files.

It would be nice if there was a worksheet output from VSP, an API that would output a xls file.

George 

 

 

Affiliation
American Association of Variable Star Observers (AAVSO)
"Screen Scraping"

Yes. I definitely am screen scraping because I code almost as well as I "jump tall buildings with a single bound", but I know my way around a spreadsheet fairly well and it is easy to ajust the cell references for different table formats.

Thanks for the references in how to interrogate VSD. I didn't know how to do that. To further demonstrate my ignorance of coding how do you include the "Accept: Application/XML" header into a request such as your example 
https://www.aavso.org/apps/vsp/api/chart/?star=SS+Cyg&fov=60&maglimit=14.5  ?

If I am going to parse XML would Python be a good language to start learning? It has been very high on the "most used" charts for several years. From the littlle I have done with it, it seems easier to learn than, say, JAVA. Beside there are now Pyraf and AstroPy. I Suppose you can also parse it in "R" in which I can do simple things, primarlily working with .CSV or tab delimited data tables, but from my very limited knowledge it seems a more general programming language would be better for working with XML. 

So If I am going to become conversant with one language to do astronomy "Stuff" what would be your recommendation?

Brad Walter

Affiliation
American Association of Variable Star Observers (AAVSO)
Photomcap - "is the bomb"

George,

Photomcap works really well. See the attached files which should look familiar. Once I opened up the window far enough that I could see the menu at the top, it worked perfectly see attached csv with txt suffix (since our stupid Drupal implementation won't allow you to upload a .csv or a .xlsx file)  which imported into an Excel spreadsheet perfectly. With a little modulo arithmetic magic, I easily added sexagesimal columns that the photometry catalog import function in Mira-Pro requires. See attached. 

It's the Bomb for getting machine readable data out of VSP. 

Thanks

Brad Walter, WBY