›

A Story about Hot Pixels - beginner question

Woodbine
in PixInsight
I'm working through Fundamentals and have a question about dark frame best practice - my apologies if already asked and answered (if so, please just point me to the right thread).

In Cosmetic Correction > A Story about Hot Pixels, Adam seems to imply that it is better to take a long "high quality" dark frame and not to worry about "hot" pixels (dithering and well behaved sensor is assumed). Rather than take shorter Dark Frames to match the Light Exposure time. My questions then are;

1) Am I understanding the content correctly?
2) If dark current is linear, why would, say, a 30 minute dark frame be any better quality than the integration of 6 x 5 minute dark frames?
3) Adam doesn't describe how you would use a 30 minute dark frame to calibrate a 5 minute light frame in this video - but does drive home the point that the dark current is linear - so I would assume that it should be possible, for example, to simply scale a 30 min dark frame in pixelmath with something along the lines $T/6 to give a 5 min dark frame - is that true?
4) In the following section on WBPP > Flat Darks and Dark Frame Optimization, Adam partially answers 3) above by describing how WBPP uses some "black magic" iterative process to scale an unmatched Dark Frame to calibrate a light. But, if the Dark Current is linear, why does the scaling require iteration and black magic rather than a simple ratio of the exposure times?

I ask this as I'm just getting started with a cooled CMOS camera and I'd really love to simply pick a fixed temperature, take one long master dark and be able to use it to calibrate all my images for months at a time.

Many thanks - Paul 



Comments

  • BlockHeadBlockHead
    Hi Paul,

    Hopefully I am consistent... and I will assume so for the moment (can't look back at everything).

    Here are some comments:

    1. My implication about hot pixels is that many novice imagers believe that a bad frame is determined by the number of hot pixels they see. In the case of dark current scaled or optimized frames, this generalization is completely incorrect. Hot pixels will not be characterized by scaled/optimized darks- they are expected and require cosmetic correction and or dithering to take care of. 

    2. The use of longer master darks *can* be more efficient if a person completely understands what they are getting  in terms of calibration results. Using a single dark to calibrate frames of times less than it is a cool simplification. For many CCD sensors this is a real thing. BUT- there are so many pitfalls, that I DO NOT recommend this for anyone who has not mastered matching darks and has absolutely no calibration issues. To many beginners use scaling/optimization of darks does them a disservice. I think early on, I gave more credit to novices than I should have. The use of optimization for flat field images in particular is a big no-no. Many people are self-imposing flat field issues with the use of optimization/scaling- and they have not gone through route of normal calibration and mastered this...so it only adds to their confusion.

    3. I did not say  that 30 minute darks are better than 6x5 minute dark frames other than READ NOISE. Every time you download an image you add noise. You will need to take more 6x5 minute exposures to get the same reduction of noise statistically. Please see Berry and Burnell's explanation of this in the book I mention. I believe you can find the analysis on line. 

    4. I do describe how to use optimized dark frames in a number of examples and as you note in the WBPP section as well. 

    5.Scaling linearly is absolutely correct. It is a function of the material. At a given temperature you will get a certain rate of dark current... period. BUT... temperature does change and drift for any sensor. In addition other electronic issues can play a role. So what PixInsight does is it searches for the scaling factor that minimizes the noise. It does this iteratively by making a guess (perhaps from the ratio of exposure time) and subtracts the thermal frame... recording each result. The factor that minimizes the noise is the one that is used..and this can be different than the ratio of exposure. In most cases the difference is small.. but for some detectors this is a optimization (hence the name).

    Most of the above is inherited from CCD sensors and electonics. CMOS is a little different...but the general principles are still valid.

    -the Blockhead

    P.S. Did you know that sensors can change their temperatures by downloading images? Yeah..that is a thing. See the implications?
  • Woodbine
    Hi Adam,

    Thank you so much for taking the time to answer my question so thoroughly and carefully. I think I understand the video more clearly now.

    Also, sincere thanks for creating this course and sharing your knowledge. I am quite new to Astrophotography and you are teaching me a huge amount about astrophotography and image processing, in addition to Pixinsight - I'm drinking from the proverbial fire hose! 

    As a novice, I am working my way through Quickstart and Fundamentals from front to back and so I guess I'm a fairly typical reader. Following the course in this way then, the "Story about Hot Pixels" video is the first time that you introduce the concept of scaling long dark frames.

    With that background please let me explain why I think I was getting confused on first viewing in the hope that it is helpful feedback. To respond to your points;

    1.My implication about hot pixels; Yes, this came across very clearly. Hot pixels are easy to remove with cosmetic correction and are not something to worry about in raw data. Dithering is good. I think these are the key CC takeaways as a beginner?

    2. The use of longer master darks *can* be more efficient; That you do not recommend beginners (like me) to scale/optimize master darks didn't really come across to me in this lesson - I came away with the opposite impression, even on second viewing. I agree that this seems quite an advanced topic to introduce at this stage in the course and I suppose you might argue that it shouldn't be introduced this early to beginners. However, I'm glad it is there as I've now learned something very interesting that I will be able to experiment with my imaging when I'm ready.

    3. I did not say  that 30 minute darks are better than 6x5 minute dark frames other than READ NOISE; Ah! now I see (and thank you for the reference, I'll follow it up). I think you are saying this around 12:30 onwards, but you don't explicitly mention READ NOISE - at this stage in the video you use the mouse to point to "this part" (hot pixels), "this part adds a certain degree of noise" (dark current?) and "this stuff you don't see, it just adds a kind of graininess to the image" (??). 

    4. I do describe how to use optimized dark frames in a number of examples; I watched it again and I'm fairly sure that the optimization/scaling technique isn't explained in *this* video where (for a beginner working through the lessons in order) you first introduce the concept of scaling long master darks. At the time of viewing I found myself thinking "so it is best to use scaled long master darks - but how??". As you say, the "how" is introduced a few videos later. I would have found it helpful if there had been a brief mention in the Hot Pixel Story that the "how" would be introduced shortly.

    5. > BUT... temperature does change and drift; This was a major reason I moved to a cooled camera - you get to set and measure the sensor temperature to a high degree of accuracy to minimize this error. However, if this is a significant problem (is it?) then isn't this an argument always to use optimized dark frames even when using matched dark exposures?

    other electronic issues can play a role; OK you can't leave me with that cliff hangar :) can you point to a reference on the specifics here?

    PS - Did you know...; I did - it was one of many frustrations with a DSLR shortly after starting out on this slippery slope, sorry, hobby. There was no way of knowing the sensor temperature and it was changing all night, so I couldn't properly calibrate (I did learn to turn off the LCD screen to stop amp glow). To answer your question; At least one implication, I think, is that you need to allow the camera to "cool off" between frames following a download for Lights and Calibration Frames if you want to match exposure temperatures. Did I get it right? Are there other implications?


  • Woodbine
    Now I have my cooled camera up and running, I took a look at the difference between 20 x 3 min exposures and 3  x 30 min exposures for dark frames. The result was interesting and surprising to me and I thought I'd share. The shots below are what I think are the thermal frames. I integrated each set of dark frames and subtracted masterbias from each masterdark. Then I used pixelmath to divide the 30 minute exposure values by 10 and compared them. Looking "up close" the warm pixel values matched very closely, so I think I got the pixelmaths correct.

    The stats deviation on the scaled long dark is significantly lower than on the integrated shorter frames - less read noise?

    The thing that really jumps out though is that the integration of many shorter frames has noticeable "amp glow" which is almost invisible on the scaled long darks. I guess this is the effect on the sensor of the camera downloading images you described Adam?!

    The thing is, which is the best dark frame here? The scaled long dark looks cleaner - but if I ran the Light sequence with the same delay between frames as the darks used here then presumably I'd need to use the stack of shorter darks to match? Or, I need to figure out how much delay I need to allow the sensor to cool off following an image download (which would lose imaging time).

    Pretty interesting stuff (it is raining, so...)
    Screenshot 2022-10-26 211419.jpg
    1372 x 484 - 265K
Sign In or Register to comment.