WBPP : Don't understand what's wrong with calibration

Jean-François HERVIER

Hi,

Last summer, I shooted images of M31 in LRGB (180 sec for each filter + 300 sec for luminance) with flat and darkflat.

When processing in WBPP the results are over exposed or invert video images, except for the 300 seconds Luminance (see images below). 

I probably did something wrong somewhere, but I don't see where...

Any help or suggestion are welcome because I'm totally lost

Jean-Francois

BlockHead

Hi Jean-Francois,

I would like to make a series of videos on how to approach solving issues like this. I think there is a standard approach to doing so that will be helpful to people. Can you make your raw data available to me?

I don't know if a subset of files would work because the problem might be with information you did not give me. So... is it possible to give me the files?

In the meantime... the first step is to blink your raw calibration frames. Then yo blink your light frames. You compare your dark level with your lights and make certain the darks are correct. You check your offset/gain settings and make certain you are not mixing them. So on and so on...

-the Blockhead 

Jean-François HERVIER

Hi,

Yes of course. I will be pleased to provide you my raw files.

I have :

L (180 sec) : 45 Light + 30 Flat + 30 DarkFlat

L (300 sec) : 47 Light + 30 Flat + 30 DarkFlat 

R (180 sec) : 73 Light + 30 Flat + 30 DarkFlat 

B (180  sec) : 33 Light + 60 Flat + 60 DarkFlat (2 sessions) 

G (180 sec) : 120 Light + 30 Flat + 30 DarkFlat   

Add to this : 30 dark@180 + 30 dark@300

All is about 65 Go.

Do you need all the files ?

How should I send them ?

Thanks

Jean-Francois

BlockHead

I know it is huge..but yeah, I guess I need a REAL world example of all of the files. 

Here is a way to minimize the number of files.

I would want 5 Flats (per filter)

I would want 15 Darks at each exposure time. Choose every other dark.

I would want 15 Darks for Flats ...choose every other dark.

I would want 15 Lights at each L ... choose 15 random files. So this 15 per exposure

I would want 15 Lights in each of the other filters ..choose randomly

This should lessen the total size considerably.

You would need to upload them somewhere for me to download. 

You would need to set the permission to be "Anyone that has the link" so I can easily download the files.

This is exciting...

-the Blockhead

Jean-François HERVIER

OK.

Could you please give me an email address to send you the links to download the files ?

Jean-Francois

BlockHead

ngc1535 @@ caelumobservatory dot com

-the Blockhead

Jean-François HERVIER

I found a way to send you all the images in 3 zip files.

Let me know if you have any issue when unziping.

Jean-Francois

BlockHead

Thanks.

I see the file extension is *.z01

I am not familiar with this extension. What program did you use to compress the files?

I do have 7Zip... so it might figure it out.

-the Blockhead

Jean-François HERVIER

You have to wait to receive the third file.

It's a zip file.

Just double click on this file and all the files will be automatically extracted.

Jean-Francois

BlockHead

Thank you for the data.

In 5 seconds I already found at least one issue.

It is what I suspected in my initial answer to you.

I wrote:

"In the meantime... the first step is to blink your raw calibration frames. Then yo blink your light frames. You compare your dark level with your lights and make certain the darks are correct. You check your offset/gain settings and make certain you are not mixing them. So on and so on..."

When I looked at your file headers. Your Red Data has an offset of 0 but your 180sec Dark frame has an offset of 30. So your darks will obliterate your light frame data. 

This also explains why the 300sec L frames worked. 

Your 300sec Dark and 300sec Light frames both have an Offset of 30. 

Do you agree with me? I am not going to move forward with processing since this appears to be a showstopper.

-the Blockhead

Jean-François HERVIER

Yes, you're right.

I agree with you.

I'm going to redo a serie of dark 180 at gain 0 and offset 0 and process again.

Jean-Francois

Jean-François HERVIER

So, the new process in WBPP with the right darks@180 was a success ! Even if the masterdark@180 seems weird...

I didn't imagine that this type of error could have such a strong impact on the result.

I have to thank you again because I wouldn't found out what was the issue without your help.

Also, I have 2 suggestions regarding WBPP :

- Wouldn't it be useful to be able to display the gain and offset of each image in the "Dark", "Flat" and "Light" tabs by adding two additional columns?

- Would it be possible when we launch the "Diagnostics" that WBPP also checks the consistency of the different gains and offsets with each other and, in the event of a gap, an alert message is displayed in the report?

The new masterdark@180 looks very "grainy".

So, I tried to integrate myself the dark raw files and got a "smoother" image.

I forgot to mention, but I also have a serie of Ha images of M31. The new processing with WBPP (with the grainy masterdark) provide a MasterLight_Ha with a remaining Amp glow.

Then, I processed the Ha images with the masterdark I integrated myself and get a better result (see all images below).

So, two more questions :

- How explain the difference between the 2 masterdark@180 ?

- Do I have to process again all the LRGB images with the masterdark I integrated myself to have a better result ?

Jean-Francois

BlockHead

Concerning GAIN and OFFSET ...I agree that it is a nice thing to make visible and match on. I am not certain what the complexity is though... I have asked this question to the developers... I will write back when I have a response.

Concerning the integration of the MasterDark... whether you do it yourself or you let WBPP do it... they will be EXACTLY the same. Do not compare the appearance.. you must look at the values. If the values are different you have likely made a mistake. 

I demonstrate this in my videos on Manual processing. I actually show how the answers are EXACTLY the same. There is no difference. Please see my Manual Calibration videos at:

https://www.adamblockstudios.com/categories/ManualCalibration

Note the fourth section is where I compare..but you need to go through the first sections to get there. 

-the Blockhead

BlockHead

Ok... got a response. I agree with the suggestion. 

You can use the GAIN and OFFSET Keyword to make certain files are matched properly. You will then see the values for GAIN and OFFSET directly. Nothing has to change in WBPP. 

The developer will be putting a "Save Default Keywords"... which means everytime you use WBPP these will be loaded and used to prevent exactly the issue you had.

-the Blockhead

Jean-François HERVIER

I watched again your video about manual calibration and compared the WBPP Masterdark and mine with Pixelmath (difference between two darkframe + 0.5).

The result is a grey image, but the values are not always 0.5. They are between 0.4998 and 0.5, so very close.

My Masterdark was built with the same parameters as described in your video (“Average”, “No normalization”, “Don’t care weight”), except for “Linear Fit clipping”  (I have 30 images) and noise evaluation (I have unchecked).

I processed again all my files (Ha included) with WBPP and my Masterdark. Then I compared the final Masterlight with the previous ones (Pixelmath again).

For LRGB Masterlight, visually no difference and here again a very small gap between point values (between 0.5 and 0,4997).

The only exception is Ha. Visually, the difference is obvious. But Pixelmath reports values between 0.5 and 0,4999.

So, I don't understand why this specific difference with the Ha when it does not appear with the other filters.

Why is this difference visible in Ha when the values of the points are almost the same?

Should I not believe what I see ?

BlockHead

Does your calibrated Ha have zeros?

Did you study the video on Pedestals and narrowband imagery?

-the Blockhead

Jean-François HERVIER

I watched the video and checked with my calibrated Ha files and I have a lot of zero points (in both case).

So I processed again all my Ha raw files in WBPP with a pedestal (100), twice, and checked that I had no more zero points.

But unfortunately, the result remains the same.

Visually, the two Ha Masterlight are slightly different.

Jean-Francois

BlockHead

Why do you think 100 is enough?

-the Blockhead

Jean-François HERVIER

Because I have no more red point when using you pixelmath formula

That's what I understood from your video

BlockHead

Sorry, I did not understand. You said the result remained the same. However, if you solved the zero issue there result is definitely NOT the same. You made a significant improvement. Zeroes in your data means you are literally throwing away information. 

So we need to revisit, what do you believe is the difference. Can you show the difference you are seeing again? Are the values different between the two versions? How much. Pick a pixel... 

Provide the two images for download.

-the Blockhead

Jean-François HERVIER

I sent you my files.

Hope this xill clarify...

Jean-Francois