Automated image prossesing

Announcement: New Forums

We are excited to announce the launch of our new forums! You can access it forums.aavso.org. For questions, please see our blog post. The forums at aavso.org/forum have become read-only.

Pierre,

I can't help much with the current pipeline processing of the data taken with APASS but I can assist in the automation of the observatory from a hardware and environmental telemetry standpoint. I think it is a very good suggestion that we start discussing the data pipeline process and that we get onboard with the current processing system that is being used in the AAVSONet systems.

If there is anything that I can do to assist in the hardware automation or integration please let me know.

Regards,

Thomas C. Smith

Dark Ridge Observatory

Well, we have VPHOT to do photometry.  I wouldn't call it a pipeline yet but it has features that can speed up processing a lot.  Geir has implemented a procedure whereby with the correct FITS header items (especially one identifying which sequence to use) images are processed upon receipt and sent directly to the Analysis Log.  From there the results can be examined and the AAVSO report generated with a few clicks.  It is a feature I requested some time back and I have been "testing" it ever since (I wouldn't tackle the number of targets I do without it).  You'll have to ask Geir if he is ready to release the information for general use.

There are several manual steps that are necessary, some of them one-time setup type things.

• The proper items of the FITS header have to be set up (I use a script to do this for each image)
• The images have to be calibrated (BDF) (and that can be automated to an extent)
• The telescope parameters have to be set up in VPHOT with an identifying entry in the FITS header
• The correct sequences have to be set up in VPHOT (every time you add a new target a new sequence has to be set up for it)
• The results VPHOT comes up with have to be reviewed carefully as spurious results can happen
•  The results to be reported to AAVSO have to be selected
• The VPHOT output is a text file in the AAVSO format which must be manually uploaded to WebObs

The worst part for me is the images must be uploaded, there is no pre-processing (yet).  For those with a big data pipe that may not be a concern.  For those with slower connections or data quotas, maybe more of a problem.

We have discussed this topic before and It seems to be just another item on already full plates.  The description above is my adaptation until something else comes along.

Jim Roe

[quote=akjalam2]

I have looked into just IRAF and Sextractor using my Mac and after a great deal of agony over installationproblems i finally got them running.

IRAF is a fantastic tool .. and when you learn to type again :) (command line ) it is real fun.

[/quote]

I'm experimenting with SExtractor as well, I found it easiest to just take the SciSoft collection from ESO, and install it inside its recommended habitat, Fedora Linux. All of it runs happily inside a VirtualBox VM on my Mac, my Windows notebook and a Linux host. Scisoft has Sextractor, Scamp (for plate solving), IRAF (which I'm not using) and tons of other useful stuff.  

I'm doing DSLR photometry, producing wide fields which are tedious to reduce, so an automatic pipeline (w/o image calibration, tho: the inputs are calibrated images in FITS) is what I have in mind as well. It works reasonably well so far but needs more work to make it robust and easy to use.

CS

HBE

The current processing pipeline, used by both APASS and AAVSOnet is a mixture of IRAF, plus Fortran programs based on DAOPHOT aperture photometry routines and SLALIB astrometric libraries, cobbled together with scripts.  I've developed this over a couple of decades, which is why it is non-homogeneous.  However, we need to update this to a system that others can support; I won't be around forever!

Because APASS is run by the same pipeline, and I need better deblending in the APASS images, I am switching over its processing to either Sextractor or PSFextractor this summer.  At that time, we will be removing the IRAF portion by recoding the image processing in Python, and adding in Astrometry.net WCS routines instead of the method currently used.  Then we get to reprocess about a thousand nights of APASS imaging. :-)

For AAVSOnet, my wish is for everything to be platform independent, probably Python based.  That way, the Windows machines can process data during the daylight hours, or transfer the images over to the linux machine for parallel processing while imaging is underway.  I think such a pipeline could then be fruitfully used by others for their own robotic setups.  If some programmers want to give Matt and I a hand at this, we can generate a design document and assign programming tasks.  My understanding of Windows environment is pretty limited, so help there would be appreciated!  However, I'd then request that this thread resume in the software forum.

I want to stay clear of the image acquisition/observatory automation loop, as I feel there are vendors who are actively supporting that aspect.

Arne