Low noise cameras and darks

Linwood

I have heard a few pretty smart people make a case that the new CMOS sensors, like the ASI6200MM Pro, are sufficiently low noise that it is not necessary, and may actually make things worse subtracting very low noise from very low noise (and virtually zero amp glow).  So they recommend no darks at all. 

Adam, do you have an opinion or recommendation? 

BlockHead

Linwood,

Define "sufficiently low."

I suspect the answer is not as simple as you present it. 

It isn't that I do not have an opinion- what is important is that I do not have the NUMBERS that show what you purport to be true (or possible).

Until I have numbers that answer a quantitive question such as this- I will not opine.

-the Blockhead

Linwood

Ah, well, "yes" or "no" are simple.  It's the justification is complex.  :)  

I will be very interested if you get data and have an answer.  If you have any interest in ASI6200MM data I'd be happy to oblige (transfer logistics to be worked out, they are big subs). 

Wade Hilmo

One way to approach this question objectively would be to do the following exercise with your camera:

1. Create a master bias from at least 15 raw bias frames.
2. Create a master dark from at least 15 dark frames.  The duration of the dark should be something that you use typically.  Personally, I like to use 10 minute raw darks for analysis of dark current behavior.
3. Use PixelMath to add 0.5 to the master bias from step 1.
4. Use PixelMath to subtract the master dark from the altered master bias.

The resulting image will be a measure of your specific camera's dark current (for the exposure length you chose).  The pixel values will be centered on 0.5, so the image will probably be grey.  The reason that we want to add the 0.5 in step 3 above is to prevent the subtraction in step 4 from black clipping pixels.

Oh, and the reason that I subtract the master dark from the master bias is that I want to prevent clipping pixels as much as possible in the experiment.  Intuitively, the simplest thing would be to just subtract the master bias from the master dark.  The reason I don't suggest that is that, for a camera with very low dark current, you might end up with outlier pixels where the bias might actually be brighter than the dark.  In that case, a simple subtraction would result in black clipped pixel.  Similarly, adding 0.5 to the master dark has a higher chance of white clipping a pixel, than adding it to the bias.  Hence, I add 0.5 to the bias, and then subtract the dark.  The variance of any pixel in the result from 0.5 is your dark current (subject to noise).

If your dark current is sufficiently low, then skipping darks may make sense.  There is probably an interesting discussion around what is "sufficiently low".

Linwood

Yes, "sufficiently low" is a question, and not quite sure how to measure the thing (after doing this) to compare to whatever "sufficiently low" means.  

Wade Hilmo

Well, the exercise that I described will give you the dark current information for your camera, expressed in ADU's.

A couple of obvious comparisons would be to the read noise for your camera, or the shot noise for the master dark you used in the test.  If the dark current is significantly less than both of those, then you are probably not benefiting a huge amount by doing dark calibration.

Linwood

I had some downtime and decided to pursue this more aggressively. 

First, to the experiment above, I did exactly as described, with a 500 image master bias, and a 50 image master dark at gain 0 and 240s, which is my most frequent broadband exposure. 

I am still unclear how to interpret this. The statistics looked like this: 

            K

count (%)   99.99999

count (px)  61171481

mean        0.4999950

median      0.4999964

avgDev      0.0000146

MAD         0.0000121

minimum     0.1026217

maximum     0.5004949

I think the mean being almost precisely 0.5 still (32767.170) implies a very low dark current. But I still am unclear whether that low average dark current means dark calibration is unuseful, so I decided to do a real world test.  

I am building time on M82, so I took 7 nights' worth and ran through WBPP through cosmetic correction, then manually did SFS, Star Alignment, and integration using the same settings for these latter steps, but once for bias calibration, once with dark calibration (of both flats and lights).  I wanted to see how they differed. 

This was a rather tortuous path as some weird things happened.

First, by habit I usually do thin plate alignment with local distortion, not because I need it all the time, but because I need it sometimes and had the icon handy. I was doing LRGB + Ha, and the Ha star alignment failed miserably with the bias calibrated lights, but worked nearly perfectly for the dark calibrated ones (one failed). 

Aside: There is a "feature" in PI that star alignment that cannot produce a valid distortion model SUCCEEDS the star alignment, but kicks out an unaligned image.  Beware.  So I did not even notice, integrated, and it was a very interesting high rejection map.  

So I started over and redid alignment without local distortion for Bias calibrated, and that worked.  Why did the other fail?  My hypothesis (i.e. no real facts involved) is that hot pixels were throwing it off.  More later. 

So I now did the integration, and compared images, specifically compared Luminance and Ha.

Luminance (gain 0, 120s) was visually identical when zoomed in, the noise evaluation gave very similar numbers, 0.9884 / 87.1% for dark calibrated, 1.006 / 88.65 for bias calibrated. 

Ha (at gain 100, 300s) was another story.  The numbers were still similar though not as close (1.076 / 88.84% dark and 1.104 / 90.68% bias), but the images were visually different.  At a close zoom the noise pattern was different when blinked, but neither looked worse or better - just different, like fog that shifted.  When I looked at individual subs, however, the difference was significant -- there were a LOT of warm pixels, i.e. in a preview stretch very light, but not saturated, just too warm.  A lot.  These were mostly mitigated on the dark calibrated subs.  This is after running through cosmetic correction with auto-correct at 3 in both cases. I ran the real time preview and increased this substantially without getting a good result -- too high caused other problems before it cleaned up the warm pixels. 

Because these images were dithered, this did not show up in the final stack (150 or so) but I think did contribute to the change in the apparent noise visually.  My guess is it is also the reason that the star alignment did not go well with local distortion, though again, zero data to support that, just supposition.  But fact is that it did not work well, I did it again manually to be sure. 

The final "feature" noticed in this test is also a bit odd -- the Ha images were flatter using the bias calibration (for both flats and lights) than the dark.  The dark calibrated ones had a very slight vignetting at the corner -- not enough really to matter, it would come out easily in a DBE, but if you blink back and forth, visually different.

The end result of all of this for me is this: 

There is virtually no noise difference in using bias vs darks, but the slight difference that remains is in favor of darks (admittedly with one data pair, hardly statistically significant). 

The higher gain images have a lot of warm pixels that darks correct very nicely, and cosmetic correction does not.

And whether or not caused by the warm pixels, star alignment struggled with the bias corrected, and I do have some images (notably when using a focal reducer) where I think local distortion helps.

The slight benefit on the flat correction of vignetting is a benefit for bias.  But these other issues are a slight, but to me enough reason to keep using darks. 

But... it's a tough call.  Maintaining an updated dark library for all exposures (and I do flats as well), and refreshing it periodically, is a pain.  I can see a good argument, given the mild differences, for not doing so. 

Attached are two animated GIF's that show the corner differences, and what I'm calling warm pixels.  Well, at least I hope this web site can do animated GIF's.