Adding maximal pedestal stilldoes not resolve all clipped values after calibration (using WBPP2.3.1)

Comments

• BlockHead

  November 2021

  I am not certain I am following...

  Black spots, in general, are caused by hot pixels in your darks that oversubtract from your data. (Taking *new* darks and applying to old data does not help. In your post you did not indicate you took new data that was taken on the same night as the darks...).

  
  

  You should be able to look at your darks...zoom in and find the offending pixels. If there isn't a hot/warm pixel in the same place in your lights then there is a calibration issue- or the calibration frames simply do not match your light frame data.

  
  

  There are so many ways to achieve this result... likely the way forward is to look in detail at the data itself.

  You know one funny way people sometimes have this issue? (I am *not* saying this is your issue... but an example)... Sometimes people rotate their light frames. Some acquisition software does this (for meridian flips)...then when you subtract the darks- t h e y  d o  n o t  m a t c h... lol. The noise of the dark current will subtract but the hot pixels will not ... leaving lots of dark spots (and uncalibrated hot pixels in the lights). Salt and pepper. But the person will claim new darks, same temp, same everything... which is true!

  
  

  Anyway... just an anecdote.

  -the Blockhead

  
  

  
  
• Shlomi Vaknin

  November 2021 edited November 2021

  Thanks for answering so quickly! I didn't mean that I have a problem with dark spots, I don't, I was just inspired by that episode to examine my own calibrated frames, so thanks for that!

  
  

  I also should have mentioned that I took all shots yesterday, both lights as well as all calibration shots. The table I added to the original post showed exactly what you suggested, I zoomed in on the dark (that's the column titled "dark"), and on the same pixel in each calibration attempt (each row is one pixel), one with at no pedestal, with pedestal of 200, 900 and 999, and noted the 12 bit value of that pixel in each frame.

  
  

  Also, lights and flats are perfectly aligned (no meridian flip or any camera movement) at all.

  
  

  If we look at the table, all the numbers make sense. The darks mean of 90DN is subtracted from the mean of the uncalibrated image, to produce a value of 40 (131-90 ~= 40) in the "no pedestal" calibration attempt. With pedestal of 900DN in 16 bit = 56.25 in 12 bits, we see the mean becomes 40+56 = ~97DN, as the table shows. So all the numbers make sense. 

  
  

  You are also right that the offending pixel is a hot-pixel, which we can see in the dark frame, it has a value of 1269. That same pixel in the uncalibrated frame has the value of 1204, so the difference is 65DN in 12 bits. However, that number in 16 bits is 65*16=1040, which is greater than the allowed 999 in the dialog box of the WBPP tool.

  
  

  I guess my question is: should this limit be raised in the tool, because my case makes sense? or, is there something wrong with my workflow if I need a pedestal larger than 999 16bit?

  
  

  Or maybe something different all together?...

  
  

  Thank you for helping :) 

  
  
• BlockHead

  November 2021

  I guess what I am saying is this... the pedestal that is used to deal with Narrowband images (to prevent negative values due to oversubtraction) is necessary because the values of the sky (virtually nothing) are already so close to zero- *noise* ... normal fluctuations is what causes the issue. Some pixel values become slightly larger. So for Narrowband images you need your pedestal above your noise floor so no fluctuation is large enough to go negative after calibration.

  
  

  In your case the difference between your calibration frame and light frame, with respect to the hot pixel, is NOT a readnoise (or other small fluctuation) issue. Instead, it is a true mismatch between the darks and lights.

  This is NOT good. The difference you are quoting is large. Raising all values up (by adding a pedestal), although solves the issue of negatives (and zeros)... but it will *still leave holes/spots* in your data. That oversubtraction is real. The hot pixels in your calibration data, especially contemporaneous calibration data, should match nearly exactly. The differences you have indicate there is a problem. If you swear there isn't a mismatch in settings with gain, offset (camera, not processing), temperature, exposure time.... Then I can't explain it. If you take a light frame...and then a dark (assuming no image persistance)- those hot pixels should match.

  
  

  There is a very real chance there is a camera issue here. But checking off the boxes for all the things it could be is necessary.

  
  

  -the Blockhead
• Shlomi Vaknin

  November 2021

  I see.. Absolutely no rotation or any camera movement between the lights, flats and even darks, all were taken in one go without a meridian flip (I started imaging when the object was already in the west), and I checked the FITS headers, all gain, offset and duration match :| 

  
  

  That's a bit troubling.. I will try to rerun the steps again to triple check I did not miss anything silly (I must have, right?..)

  
  

  Thanks again for the answer, I really appreciate it. Happy Thanksgiving to you!
• Shlomi Vaknin

  November 2021 edited November 2021

  I took a closer look at the data, this time using python and google sheets. 

  
  

  I ran the following code to closely examine the dark stack:

  
  

  from astropy.io import fits

  import glob

  
  

  for dark in glob.glob('E:/Astrophotography/ASI1600/2021-11-24 - New Darks and FlatDarks/* DARK *.fits'):

     with fits.open(dark) as o:

       data = o[0].data

       h = o[0].header

       HotPixel_y, HotPixel_x = 2623, 2648 // Location of the hot-pixel

       Random_y, Random_x = 1023, 1023     // Just a random pixel

       print(data[HotPixel_y, HotPixel_x], data[Random_y, Random_x], h["EXPTIME"],h["GAIN"], h["OFFSET"], h["CCD-TEMP"])

  
  

  I took the data and plotted it in google sheets. I attached two plots:

   - one for the hotpixel, where exp-time, gain, offset and temp are relative to the right axis (red) and the pixel values are on the left (blue) axis

   - the other for the random pixel, where exp-time, gain, offset and temp are relative to the LEFT axis and the pixel values are on the right axis

  
  

  Easy to see that all values (beside the pixel value) are constant throughout the sequence in both plots.

  
  

  For the hot-pixel, the pixel value has an enormous error range (all values are in 16bits): 
  

  Average	20305.90099
  StdDev	878.4809788
  Min	18288
  Max	22752

  
  

  Looking at the random pixel (second attached file), I also get quite a large range: 
  

  Average	1410.534653
  StdDev	42.37630573
  Min	1312
  Max	1552

  
  

  
  

  These are ONLY the dark frames, all taken when the scope was parked, and I am asleep, so I expect the values for each pixel to be relatively steady (up to read-noise). 

  
  

  Is this an indication of the camera malfunction? Or still anything else I can try?

  
  

  Thanks,

  Shlomi

  

  

  Darks-hot-pixel.PNG

  1356 x 842 - 126K

  

  

  Darks-random-pixel.PNG

  1362 x 844 - 98K
• BlockHead

  November 2021

  Indeed...that does not make sense to me. 

  In my dark frames (with a CCD camera), my hot pixels vary by hundreds of counts... not thousands.

  The dark current has "noise" as does the readnoise...so there certainly will be variation...but it should be small. A absolute variation of 4,000 seems high to me.

  
  

  BUT...the number does not matter. Fundamentally, if you take a 300 second frame once your camera is at temperature and you take a light frame (or even dark frame) of the same exposure time. When you subtract the dark from the light the hot pixel should "go away" (not much leftover). 

  
  

  This is why it used to be people would argue over whether creating a library of darks is "bad." Because, they argued, a contemporaneous dark frame taken on the night (even immediately afterwards) will be an exact match. What is neglected in this argument is that subtracting a dark *adds noise* to images. This is why we take lots of dark frames...to minimize the added noise calibration itself adds in.

  
  

  So....it is not surprising the hot pixel has a greater variance then a "random" pixel...but it is surprising to see that much variation (based on my eperience). It could be CMOS detectors are different creatures and the fact they have greater dark current gives greater variation in hot pixels.

  
  

  But... again, it should subtract from light frames in such a way that very little residual value is left. If I understand what you are saying... this is not happening for you.

  
  

  -the Blockhead
• Shlomi Vaknin

  November 2021

  Yes, I assumed a normal "sqrt(mean)" to be an acceptable noise level, which should be ~142(16bit) for the hot pixel, and ~37(16bit) on a standard pixel, surely not ~4000 and ~240 respectively..

  
  

  In any case, I sent an email to zwo with all these details, lets see where this takes me.

  
  

  Thank you again for helping out, both in terms of your wonderful videos as well as your responses here 

  :x

  
  

  Shlomi
• Shlomi Vaknin

  December 2021 edited December 2021

  Finally got an answer from ZWO, they say this is within normal range.. 

  Hi Shlomi,

  Sorry for the delay.

  Our tech analyzed this, you mean that when the camera with a long exposure time, the value of hot pixels has a fluctuation, right?

  The hot pixels are caused by the internal heating of the Sensor as the exposure time increases, generally,  it can't respond to the light signal,  cannot be effectively compared with other normal pixels.

  About your ASI1600MM PRO's pixel value has fluctuated at gain 139 exposure 120S (the fluctuation range is roughly 100ADU), this is normal and this is caused by noise. The read noise and dark current noise are around 4-5e(64-80 ADU) with unity gain, 120s, 12 bit, plus other weak noises, the fluctuation range is 100 ADU, and the range will expand as the gain value increase.

  About fluctuated range of the hot pixel is around 1100ADU. since the hot pixel is greatly affected by the thermal motion of the Sensor itself, the hot electrons are more than the electrons collected by normal pixels,  the fluctuated range will be expanded after the analog gain and digital gains zoom in, the specific degree of expansion depends on the degree to which this hot pixel is affected by the thermal motion of the Sensor.

  I wonder what else I can do at this point. I have not been using cosmetic corrections in my "calibration" process so far, I think it could be useful with this problem, right?

  
  

  
  

  In any case, the original data that got me aware of this problem did turn out to be a nice image :) (would love to hear any critiques you may have on it..)
  
  Thanks,

  Shlomi
• BlockHead

  December 2021

  Yes, this is the purpose of Cosmetic Correction.

  However, the ZWO tech did not, if I understand the response, address why the hot pixels do not calibrate (subtract well). The tech is explaining why the hot pixels have the values they do...but from exposure to exposure... those hot pixels will be doing the *same* thing each time. I understood you to say the differences (after subtraction/calibration) were on the order of 1,000s of counts. This doesn't sound right to me.
  

  -the Blockhead
• Shlomi Vaknin

  December 2021

  Yes, they only addressed that part. I responded with another email detailing exactly what you said, that I cannot calibrate these out properly, but I doubt they will have a different answer for me.. 

  
  

  Will give it a try, as well as adding CC to my workflow. Perhaps I should also lower my gain, maybe that also contributes to exaggerating the values of the hot pixels.. 

  
  

  Thanks!