Sat, 10/12/2019 - 13:07
Dear colleagues,
my VV Cep spectra, which have been processed with VSpec, have some typical settings in the fits-header, which are not in agreement with the fit-convention at AVSpec. If I want to upload several hundred of these spectra, a manual correction of each VSpec fits-header would be necessary.
Is here someone, who have experience with this fit-header issues?
Ernst Pollmann
Hi Ernst,
I am familar with the BeSS standard and its implementation in BeSS and the BAA database. This is worrying as AAVSO claim their fits format to be compatible with BeSS (though it is not a full implementation) and VSpec produces spectra to BeSS specification. Are your spectra accepted by BeSS and the BAA database? If so there would seem to be an issue with the AAVSO database which should be resolved rather than modifying your headers. Can you describe exactly what the problem is ?
If all else fails you can add them to the BAA database for similar long term security
Cheers
Robin
Hello Robin,
Thanks for your response and your concern. I have been helping Ernst with this so let me provide a little more information. The fits header itself is fine in the sense that all the keywords are there and are compatible. However, the values of the keywords themselves, at least in Ernst case, have a couple issues which our cleaning process has an issue with.
1.) Most of the fits keywords have a bunch of extra spaces in them which confuses the system. When it tries to match for instance ' VV Cep' with 'VV Cep' it fails to find the star.
2. The CRPIX1 value is '1.' instead of '1'. This may not seem significant, but since the reference pixel must by definition be an integer the system is insisting that there is a problem as that dot makes it a float.
I am unsure why these things are happening as I have not used the VSpec software. While it is easy enough to correct the headers, it doesn't answer the main problem which is why VSpec is adding it's header keyword values with all of these unnecessary characters. I was hoping someone more familiar with VSpec could help Ernst (and by extension us at the AAVSO) figure out why this is happening.
Thanks,
Bert Pablo
Staff Astronomer, AAVSO
Hi Bert,
Yes It looks like Visual Spec pads out the header entries with leading spaces so that the length of the string matches the maximum allowable in the BeSS specfication. Not sure why but I suggest asking the Author Valerie Desnoux who is also a long standing contributor to and validator for spectra submitted to BeSS so will be able to answer your questions. You can find her contact details on her Visual Spec website here
http://astrosurf.com/vdesnoux/download.html
However I think perhaps the AAVSO database should be able to manage this situation. For example I was able to search VSX, BeSS and SIMBAD successfully even with leading spaces added to the name
Cheers
Robin
In the mean time if the spectra are opened in ISIS and then resaved, ISIS will remove the leading spaces
Robin
I have just double checked and can confirm that the BAA database ignores leading spaces and identifies the object name correctly so spectra conforming to BeSS standard and produced using Visual Spec would be accepted there
Cheers
Robin
"1." is a valid value for CRPIX1 in the BeSS standard. Extract from the standard:-
Reference pixel for the reference wavelength ● Keyword = CRPIX1 (mandatory). This keyword defines the pixel where reference wavelength (CRVAL1) is given. It is usually the first pixel of the image – in this case, value is 1. ● Format: floating point ● Units: none ● Valid Range: no limit ● Example: 1.0
Hello Robin,
Thanks for your input. I will look into ignoring leading spaces I wish I understood why that was done, though, because it seems unecessary. I can also change CRPIX1, however, in the definition you mention up above it states that it is the reference pixel. Pixel is a finite unit. You can't have a reference pixel of 1.5 or 2.7 can you? Your reference pixel should be an integer not a float simply by definition of what it is. Am I missing something? Why would this be defined as a floating point number?
Thanks,
Bert Pablo
Staff Astronomer, AAVSO
Hello Bert,
this spring I had to dig deep into the topic of FITS standards. It is quite well defined, and there exists a rather exhaustive document about that, the latest one is: https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf In general https://fits.gsfc.nasa.gov/fits_home.html contains a lot of useful information about FITS-related things, including list of software and reference to a validator.
About coordinate systems in FITS, pages 29-32 are probably most useful. Standard defines that reference pixel values and other related keywords shall be given as floating point numbers (see p 29 below the figure, p 31 in general), with defaults to 1.0 or 0.0. There is also good short indication about spectral coordinate systems in paragraph 8.4 (p 34).
I'm a bit curious, how are photometric observations uploaded to VPHOT and spectra submitted into AAVSO spectra database validated - what tools are used?
About CRPIX# values - it is definitely possible to measure something with higher accuracy than pixel size and it is rather common to e.g. linearize a spectrum with smaller (or larger if needed) steps than one physical pixel. E.g. if I reduce long slit spectra from our telescope, average uncertainty of wavelength scale is in the order of 1/15 pixels. Now it depends on the software or just practicality if I rebin original data to linear scale (usually defined by pixels) or work in wavelength scale coordinate system with some linear or non-linear step.
CRPIX is representing a coordinate system reference pixel, that is not necessarily a physical one.
As it is shown in FITS home, in FITS keyword dictionary:
KEYWORD: CRPIXn
REFERENCE: FITS Standard
STATUS: reserved
HDU: image
VALUE: real
COMMENT: coordinate system reference pixel
DEFINITION: The value field shall contain a floating point number,
identifying the location of a reference point along axis n, in units of
the axis index. This value is based upon a counter that runs from 1 to
NAXISn with an increment of 1 per pixel. The reference point value
need not be that for the center of a pixel nor lie within the actual
data array. Use comments to indicate the location of the index point
relative to the pixel.
There is an interesting question that IMHO has been solved differently by e.g. NOAO and ESO - what are the coordinates of pixel corners ;-)
However, I think that when creating a new infosystem, it is safer to stick to FITS standards as much as possible, especially in the case of mandatory FITS keywords.
Best wishes,
Tõnis Eenmäe
Hi Tõnis,
Yes indeed. The BeSS standard complies with the universal definition of CRPIX where it can be non integer. There are fits formats for spectra where the coeficients describing the full non linear mapping between the original pixels and wavelength (not just CRPIX1) are included in the fits header. In the BeSS standard the spectrum is linearised using equally spaced bins and the mapping between pixels in the image and the final spectrum is lost. The final spectrum bins are therefore not the same as the original pixels so the "pixels" in the BeSS standard here become the central wavelengths for each wavelength bin. CRPIX1 is therefore always integer in this case.
Cheers
Robin
Hey Tõnis,
Thanks for your comments. They are very useful. It is an interesting idea that the reference pixel could be at a corner or center of a pixel. I had not really considered this. As to your question:
"I'm a bit curious, how are photometric observations uploaded to VPHOT and spectra submitted into AAVSO spectra database validated - what tools are used?"
I'm not really sure what answer you are looking for. VPHOT and AVSpec are very different in how they handle things. I'm not an expert on VPHOT so I cannot tell you that off the top of my head. AVSpec however, I can give you a general overview. It is a django tool which is python based. It uses astropy for a lot of the opening and checking of fits files. There is a cleaning routine that checks that all the necessary keywords are included, that their values match the correct type, and a few other things along the way. There is also post upload validation, where each spectra is checked individually to make sure that the spectra themselves have no obvious issues (e.g. wavelength calibration, right star etc.). Does this answer your question?
Thanks,
Bert Pablo
Staff Astronomer, AAVSO
Thank you, Bert for very good answer!
Tõnis
I can't help you with that as I didn't write Visual Spec or define the BeSS standard (That was done by professional astronomers at Observatoire Meudon France). The contacts are there on the BeSS database website though and most in the amateur community doing research grade spectroscopy (Including Ernst), will know Valerie Desnoux. AAVSO is not working in a vaccuum here. There has been a well established community doing this sort of work for well over a decade.
Cheers
Robin
Thanks for your response Robin. I do not wish to change the community standards. I was just trying to understand why things are the way they are, because I had not run across any issues like this with other spectroscopy software. Thank you very much for all your helpful comments. I will do my best to address these issues as quickly as possible.
Thanks,
Bert Pablo
Staff Astronomer, AAVSO
Hey,
FITS keywords (even the so-called extension for the file) can be vexing.
Sub-pixel coordinates, expecially during resampling while combining spectra, is vital to spectral WCS.
Below, from the spec...
From fits_standard40aa-le.pdf
https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf
https://fits.gsfc.nasa.gov/fits_standard.html
Version 4.0 (13 August 2018) (ish)
~page 31
"CRPIXj – [floating point; indexed; default: 0.0]. Location of the
reference point in the image for axis j corresponding to r j in
Eq. (9). Note that the reference point may lie outside the image and
that the first pixel in the image has pixel coordinates (1.0, 1.0,
. . .)."
--Wayne Green
Ernst, as you know I process all my Spectra for BeSS compliance in Bass 1.9.9, once the basic processing is finished in Isis5.9.3; I still use Is if any spectra need a few A of shifting. Using BaSS to populate the spectra for BeSS compliance doesn't present an issue to submitting to AASVO. Only thing I have to remember is a quick edit on my spectra's BSS_INST field because AASVO's is slightly diffferent.
James
Hi James,
Are you saying that the AAVSO database will not accept spectra reduced using ISIS?
ISIS is fully BeSS compliant of course. There should no need to modify headers or pick specific software. In my view if AAVSO have ambitions to become a recognised participant in this area they should be sorting this out and bring their database fully into line with FITS specifications and the BeSS standard that they have already claimed they are compliant with.
Cheers
Robin
Hello,
We make a point of being fully compliant if possible with all previous standards. However, it is possible we have missed something or there is a bug. If you let me know what the exact issue is James I will look into it.
Thanks,
Bert
Has this issue been identified, and if so, has it been resolved? Potential incompatibility of the AAVSO database with well-established standards is a serious issue that needs prompt attention. The thread ended abruptly but the issue may have addressed in direct communications outside the forum.
Brad Walter, WBY
Hello Brad,
I had multiple conversations with Ernst offline. All of his issues have been addressed as have all other issues that I have been made aware of. To the best of my knowledge everything mentioned in this forum has been addressed and implemented. While I cannot say there are no bugs (as those exist in any piece of software), I have addressed all the significant ones. If there are any more specific issues please contact me directly through the contact form: https://aavso.org/user/61321/contact
Thanks for bringing my attention back to this forum. Apologies for leaving it unfinished.
Thanks,
Bert Pablo
Staff Astronomer, AAVSO