›

Multi-night imaging, what do you save?

Linwood
in PixInsight
I had an issue with the new Pixinsight and it triggered a discussion with Juan (gently) chastising me for not saving my originals.

When I am collecting data over weeks or months (maybe one day years), what I do is calibrate with WBPP and stop.  I do include a light cosmetic correction, but with a very light touch.  I review the _C_CC files, do a cull of any that are obviously bad.  The calibration is against a dark library of masters, which I redo every 6 months or so to pick up new hot pixels (and the dark library includes dark flats, no bias used; ASI6200MM Pro), and of course against flats taken every night as it is a portable setup that may change a bit.

So.... 

In this discussion Juan said that one should save the originals.  Of everything: darks, flats, dark-flats, lights.  Not integrated masters, everything, and with a new version of PI be prepared to run though all the calibration steps from the beginning.

So with 50 darks each, 25 flats, let's say 10 nights of imaging, MAYBE a new dark library at some point in those nights... that's a fair amount of stuff to save, but perhaps more importantly it means maintaining the association between the darks used for each flat set, the flats and darks used for each light.  Rather than a collection of calibrated lights per filter, there is a derivation tree of input images one has to keep track of.  Darks in particular either get duplicated in each day's files, or you need some kind of pointer to which darks were used (and save them). 

So... my question for those doing multi-night image, especially over a longer period.  What do you do?   And if not saving it all, will you change? 

On a related note, I had some data from the prior PI version (this started with 1.8.8-11 coming out) and went back and re-calibrated the same images with the same flats, and same master dark (I do not have the original darks).   So exact same import into the calibration process, same parameters, the only difference was the new PI version.  The results were NOT the same.  Very close, visually the same, but a pixelmath subtract showed 0.3% of the pixels were different and the difference pixels had fairly large values (seemed to be in/around stars). . 

A difference that mattered?  I have no idea.  But different on an aspect (simple image calibration) that I would have thought mathematically simple and unchanging.

So Juan is right in that I get different results.  Of course, since he wrote the code. 

But... it's going to be really painful to keep track of those input files over months.  And costly in terms of size (though to be far raw images are about half the size of processed due to the 16 vs 32 bit storage). 

I'm torn.  What I do now is simple. Is it really worth making it that complicated? 

Linwood

Comments

  • Wade Hilmo
    There is no, single, right answer for file management.  Here is what I do.  Other folks might do something completely different:

    I image automated.  Each morning after an imaging run, I collect all of the log files from each piece of involved software into a single log folder.  I create a zip file named "2021-11-30.zip" (or whatever the date of the start of the session).  I copy the log folder and all of the image folders from the session into the zip file.  That file is archived in a location on my NAS.

    Raw calibration files are treated like sessions.  For example, if I took bias and flats yesterday, I would have a zip file called "2021-11-30_calibration.zip".

    When I want to make a master calibration file, I copy the raw files from the zip and do the work in PixInsight.  I keep a copy of the resulting master calibration file on each of the machines I use to process images.  I have a folder for each camera where I keep every calibration master that I've made.  Since I use a library for my calibration frames, I don't take calibration files for every imaging session, so keeping all of them on my working machines does not take a huge amount of space.

    When I want to process an image, I copy the raw light files from the zip.  I do all of the calibration and integration to get a master light frame for each channel.  If I am happy with the master files, I delete the raw files and all the interim files created during creation of the masters.  Since I still have all of the raw files in the original zips, I can always recreate everything, including the calibration masters, needed to process the data.  It can be a lot of work to recreate them, but it's at least possible (and I have done it a few times when I wanted to improve the calibration and integration results).

    Once I have the channel masters, I create a PixInsight project for the target object.  The project exists only on the specific machine where I am doing the processing.  When I have completed the processing, I keep the result in my collection of finished images and move the PixInsight project over the the NAS for archival.

    Regarding your comment about the different results between PixInsight 1.8.8.11 and earlier versions, that is expected.  Version 1.8.8.11 completely changed star detection and normalization, which affects pretty much everything in subtle ways.

    It's worth noting that the new way that 1.8.8.11 does it, is far superior to earlier versions.  Also, I am pretty sure that this change is ultimately a result of Adam's work with the NSG script and it's effectiveness at weighting, versus other mechanisms in earlier PI versions.  Great stuff!
  • Wade Hilmo
    I notice that I didn't acknowledge the "multi-night" part in the thread title.

    Most of my objects are multi-night projects.  In that case, I still create the nightly session zip files as above.  In that case, there will be multiple zip files with data that I've collected.  They are not necessarily consecutive nights in many (most?) cases, due to weather and other factors.  That's ok.

    When I have all the data that I intend to collect, I then copy the raw files from multiple zips and process as above.

    Also, I store the zip files by date alone.  The downside to this, is that it can be difficult to know which zip files contain which targets.  For that, I keep an "info.txt" stored, along with each PixInsight project.  The info file tells me about the target and the capture data (dates, location where I imaged, the number of raw subs, any notes, etc.)  The finished PixInsight projects are archived on the NAS by year and target object.  That way, I can easily see an overview of each of my projects.  And if I need to go back to the raw files for some reason, I can get their location in the NAS from the info file.

    It might sound complicated, but it's just a habit at this point, so I don't think much about it anymore.
  • Linwood
    Interesting, and thank you.  So you do, indeed, keep originals of every single thing, as Juan recommended.
  • BlockHeadBlockHead
    I keep all originals as well.
    Calibration is a process that adds noise and artifacts. 
    The original data with its associated calibration files is the only way to recreate/use and past data with some degree of assurance in the quality of the result. If you are doing science of any kind- and your old calibrated data shows there isn't a star in a galaxy and your new data DOES SHOW IT...you therefore think you have discovered a supernova explosion! Great. But... you need to assure yourself there wasn't anything introduced in the original calibration (processing) of the data that might be cause for the star to *not* be there. Lots of reasons. If you only save the calibrated data..or worse the integrated data (masters)- you will not have a complete provenance of the signal. 

    If you are an instructor (as I am) or want to ever demonstrate something- having that original data is truly useful as well. 

    So... as for me...I save everything. 
    You only get so many seconds in life... some fraction of those seconds were spent acquiring darks, biases, and flats. Yep, they do have value.

    -the Blockhead
  • Linwood
    Well, I guess it's time to change.  Thank you both for the advice. 
Sign In or Register to comment.