›

Dark Frame Calibration in WBPP Part #9

Samuel Cartinhour
in PixInsight
Hi,

I'm greatly enjoying (and benefiting from) your new WBPP series, especially the content that focuses on CMOS cameras.

In WBPP Part 9, you go through a CMOS (monochrome) example with flats, flat darks, darks, and lights. At 07:34, you note that "Calibrate master darks" should be checked, but you also state that it doesn't matter.

Is the master dark that is generated by WBPP in this example calibrated in some way, even in the absence of bias frames?

Thank you,

Sam Cartinhour

Comments

  • BlockHeadBlockHead
    Hi Sam,

    Wait wait wait! I am talking about the "Calibrate Flat Darks" check box. NOT the "Calibrate master darks." 
    I think you are referring to what I say at 8:56. I am correct (*laugh*) that it would not matter in this case if I checked it or not because I have matching Flat Darks. The logic of having it checked means, it looks in the Darks field and finds an exact time matching Dark frame... or uses a Bias (bad for CMOS) ...or does nothing. The logic for unchecked "Calibrate Flat Darks" is to find the dark with the closest matching time.

    Since a matching Flat Dark is being provided, it will find it in both cases of the logic above...so it does not matter if I have it checked or not. 

    If I have interpreted your question wrongly... please try again and I will look more closely. 

    -the Blockhead
  • Samuel Cartinhour
    I listened again just to confirm -- between 7:25 and 7:35, this is what I'm referring to  (very roughly transcribed):

    "I should tell you, though, that I do have this -- it doesn't matter -- I have this calibrate master darks thing, but there's no bias, so it doesn't matter, but that should be checked, right?"


  • BlockHeadBlockHead
    edited May 2020
    Ah. I see.
    Once again I still agree with me.. 
    In THIS example I am processing CMOS data and there isn't a master bias. Thus calibrating master darks is completely off the table (it doesn't really matter). Calibrating master darks (what WBPP normally does for dark frame scaling/optimization) requires a bias.You do not scale darks with CMOS data.  But, yes, as I said, that box should always be checked (even if we are not dealing with it right now anyway). 

    Is this ok? I probably should have kept that little bit of innelogue to myself. :) 

    -tte Blockhead


  • Samuel Cartinhour
    I think I understand part of what you're saying. Since there are no bias frames in your example, the master dark can't/won't be calibrated. That checkbox is in fact greyed out in your video. I found that it can't be modified unless you first check the "Use master dark" box on the right of the script window.

    This brings up a related question. What if you actually had a master dark that has been prepared by simple integration (i.e., in the absence of bias frames). This is described for a cooled CMOS camera with amp glow (such as my ZWO ASI294 MC Pro) in "For beginners: Guide to PI's ImageCalibration" on the PixInsight forum (the post is by the user "bulrichl"). The master dark is never calibrated or optimized.

    If I were to load such a master into WBPP (along with a master flat dark, master flat, and lights) how would I set the "Calibrate master darks" checkbox? It seems to me that unchecked is what you would want in this case. Is this correct? Apologies for the confusion --
  • BlockHeadBlockHead
    All darks are prepared by simple integration. You have to purposefully do something weird and pre-calibrate your dark otherwise. I really wish this checkbox was not there!  You keep it checked. 
    For CMOS, they have variable biases... so instead of calling it a bias- call it an electronic signature that characterizes the sensor for a given exposure time. This is why you match darks with data (flats and lights). You will not be scaling darks (not optimized). So you always keep that checkbox checked. 

    That checkbox is saying... In the case I need to subtract a bias from a dark, I, WBPP, will assume there is a bias there to begin with (meaning I will calibrate the dark). But... WBPP will not calibrate darks if there is no optimization. And you are not Optimizing for CMOS. 

    The only reason to uncheck that box... if you purposely manipulate a master dark.  That is YOU screw with it in PI. If you are not adjusting your dark frame (which I really think is true for 99% of people) you keep that box checked! lol

    -the Blockhead
  • BlockHeadBlockHead
    Hi again Sam:

    Here is what I wrong on the PI Forum... I am waiting for some responses:

    Hi,

    This crazy box is an endless source of confusion for a vast majority of PI (WBPP script) users:

    1. Most people do not know what WBPP is doing. WBPP is going to calibrate darks (that is subtract the bias) for purposes of dark current optimization/scaling. Most users do not make that connection that a normal (simply integrated) dark contains the electronic signature (bias or bias+amp glow+fixed pattern, CMOS stuff). So when the check box says "Calibrate Master Darks" this sounds out of the ordinary. Usually when you check something, it is to turn on something that is optional or a *different* mode of operation. You generally do not check something that is the norm 99% of the time. Just look at the other checkboxes in this list. They are all of this sort except Calibrate Master Darks.
    2. The checkbox could read "I am providing Calibrated Darks" and you are forced to check the box. The tool tip would explain that it means the darks are already bias subtracted and WBPP will not subtract the bias as it normally would in the course of calibrating data. This is unusual... but you must know what you are doing.
    3. Isn't this checkbox conditional with Optimize Dark Frames? In other words, what does it mean to uncheck Dark Optimization *and* uncheck Calibrate Master Dark. Who is doing this?? It seems like this checkbox should only be available with Optimize Dark Frames checked. This might prevent some of the issues by removing the flag from consideration by one-step.
    4. WBPP is basically a mini-pipe line it really doesn't have to be inclusive of every boundary case. The "manual" calibration processes are still there and fully functional. Who is pre-calibrating darks anyway (and then coming to WBPP)?

    Let me know what you think. I might have something fundamentally wrong in my logic above.
    -adam
  • Samuel Cartinhour
    Hi Adam,

    I initially interpreted "Calibrate Master Dark" to mean: please subtract a master bias from the master dark that I am providing, or subtract it from the master dark that WBPP will construct for me from the dark files I provide.

    But since I am not providing any bias files, or doing any optimization, I became completely confused.

    Thank you for your explanations!!!

    Sam


  • BlockHeadBlockHead
    edited May 2020
    That is what the Checkbox means. 

    Please go back to your statement here:

    "If I were to load such a master into WBPP (along with a master flat dark, master flat, and lights) how would I set the "Calibrate master darks" checkbox? It seems to me that unchecked is what you would want in this case. Is this correct? Apologies for the confusion --"

    If you have a master bias, master flat darks and master light darks (simply master darks) and lights.... and you CHECK the box. WBPP will say OK... I will subtract the bias when I need to. So WBPP will directly subtract your flat darks from your flats. It will directly subtract your other matching dark from you lights. And what? Nothing else. There is no optimizing of darks- so it does not "calibrate" your master darks at any point. Yes, you gave it a bias..but it doesn't use it. So leaving it checked is fine. This is the normal way WBPP behaves. If you did want to optimize (which you are not for CMOS) WBPP will handle that fine too. 

    If you Uncheck it... here is the logic... WBPP will say OK... I will NOT subtract the bias from master darks. So WBPP will directly subtract your flat darks from your flats. It will directly subtract your other matching dark from you lights. And you end up with the same result as the above. If you did want to optimize, that is OK- but it will not subtract the bias, because by unchecking you are telling WBPP you gave it an already calibrated master dark.

    That is why I said it doesn't matter in this case... and that is why I say...just keep the damn thing checked. lol

    So this is another go at this... if you agree with me... I need to find a simpler way of saying this!

    Other better labels for this checkbox could be:

    "Use Pre-Calibrated Darks" 
    or
    "Are you giving me a Dark with the Bias already removed? I normally do this for you, but I will not if you have already done it for me." 


    -the Blockhead

  • Samuel Cartinhour
    Hi Adam,

    I think I can be more specific now about my confusion -- perhaps shared by others --

    A check mark next to "Calibrate Master Dark" is a request to the script to do a specific operation (subtract the bias).

    The absence of a check mark means: I have already done the bias subtraction.

    So this checkbox is really about bias frames: have you already used them to calibrate your darks? Or, do you want WBPP to do it for you?

    Unfortunately, if you are a user who wants to process without bias frames and does not want to calibrate or optimize the darks, the first reaction to the label next to the checkbox might be: I absolutely must uncheck this box.

    The tool tip that appears when you hover over the checkbox does not address this contingency.

    Currently, the correct (if counter-intuitive) thing to do is to leave the box checked because WBPP will be unable to perform the requested operation without bias frames anyway. Am I right about this?

    Your solution (the label should be "I am providing Calibrated Darks") seems far less confusing to me.

    Sam



    Sam




  • BlockHeadBlockHead
    What I just wrote on the forum is:

    "WBPP normally subtracts the bias from master dark frames when optimizing darks. If you are providing darks that already have the bias removed WBPP needs to know this so that it does not do an unneeded subtraction. Only uncheck this box when inputting pre-calibrated darks."

    I think this could be the first line of the tool tip.

    -adam
  • BlockHeadBlockHead
    edited May 2020
    Here is the thing..yes you are correct (which is why I said out loud what I should have maybe kept to myself)
    . But there is more.
    Juan explains apparently there are situations (remote observing services) that supply calibrated darks. That is the strangest bleeping thing I have heard of...and he says it comprises a large number of users. So you leave the box checked because you are not supplying calibrated darks... 
    not because it will be unable to perform the requested operation without bias frames anyway.. 
    There is a subtle distinction.
    -adam

    In the one moment of speaking... you will hear me pause- I ran through all of this logic in my head in THAT moment. LOL  I hope that impresses someone out there..even if it does confuse them too. hehe
  • Samuel Cartinhour
    Adam,

    I think the problem is really entirely with the label and the checkbox location. It presents a binary choice (please subtract bias vs. I've already subtracted bias) at the global level. Neither choice is relevant to the user who has a bias-less workflow.

    The tool tip is also unhelpful, and it has a warning that implies that you'd better make the right choice!

    So one solution might be to add a bit of additional info to the tool tip, e.g. "If you don't have pre-calibrated darks, or have not loaded any bias frames, or you are not optimizing, you can safely ignore the setting of this checkbox."

    Alternatively, maybe the checkbox belongs on the darks tab? The essential question seems to be "are these darks pre-calibrated? Yes or no."

    Sam
  • Samuel Cartinhour
    Our posts are crossing -- yes, I completely agree with your proposed addition to the tool tips.
  • BlockHeadBlockHead
    I believe in essence I already made this exact suggestions (including putting it in the darks panel). 
    But I went a step further. This checkbox represents inaction for the most part. Generally when checking something you are doing it because it is optional or a different than normal mode. 

    My opinion is... have inverse logic and have it unchecked by default. 
    It might be labeled "Use Pre-Calibrated Darks." Then users might pause and think... "Well, I don't know what that is..but I don't think I have them" (and they would be correct).

    -adam
  • Samuel Cartinhour
    I believe that moving the checkbox (with the logic inverted) to the Darks panel is the best solution. My suggested label would be "Using pre-calibrated Darks".

    I hope the script developer considers these changes. But even if they don't, I want to thank you for your patient explanations. Your website has greatly increased my confidence as a PixInsight user.

    Sam
Sign In or Register to comment.