›

What Files are Worth Keeping??

James Glucksman
in PixInsight
Hi everyone! I have been engaged in astrophotography for around three years now, and am starting to wonder if my file retention policy is maximising my storage space, and whether I am needlessly retaining files that are either unnecessary or can be more effectively recreated if they should ever be needed.

Here's my practice:  After I check my raw files out of the camera, they go in a folder called "RAW". In here, I have folders for each of my targets, with each folder named for the target, the camera I used, and the date (e.g. NGC2070 (ASI1600 27 April 2023) for the files I shot last night). With the folder I have a separate folder for the lights, the flats and the dark flats (though as I amass dark flats at various exposure lengths, if I have recent dark flats at those exposures I will link via alias to them). 

Then there is a separate folder for each year, and within that folder I have subfolders for each target I process during the year. That can include reprocessing of earlier data, and the titles of the folders always indicate what raw files were used. It's in this folder that I think I have the most unnecessary gunk, since every time Pixinsight processed a session using WBPP (which is what I often do) it generates cosmetised files, registered files, debayered files (when using an OSC), calibrated files and of course masters. Up until now, I have been keeping all of these--and in fact it's only quite recently that I realised that within the master files folder are ready-to-use master lights that I could have been processing, rather than using the registered files to re-integrate into masters--that's on me!).

So is there a good reason to retain those debayered, cosmetised, calibrated and other files that precede the generation of masters, and can I free up quite a lot of disk space without concern?

Thanks in advance!

Comments

  • Gordon Aspin
    I keep the calibrated files (as these are, I think, unlikely to be improved by future changes to the calibration processes) but dump the debayered, cosmeticised, and registered files as these can relatively easily be regenerated if needed, perhaps with improved processes. That saves me quite a lot of disk space.
  • Mark Mayer
    I agree that dumping the debayered, cosmeticised, and registered files is sensible. An alternative strategy is to get enough storage to free you up from worrying about disk space and just keep everything. 

    I have a Synology 6 bay NAS configured as RAID 6, with 6 x 4 GB drives. This will easily fill my needs for a few years, but if you have the opportunity to image frequently, you could purchase 8 GB drives or even high for mind boggling capacity. 

    Considering the cost of the mounts, telescopes, camera's etc used for astrophotography, data storage is relatively 'cheap'.
Sign In or Register to comment.