SPCC Graphs following along FastTrack are poor

Brent Ewers

I have been following along with the FastTrack lessons on the way to doing Fundamentals. In my own post-processed images, I have noticed that SPCC provides a blue tinged corrected image. I was hoping to find out what error I may be causing by learning from the FastTrack and then Fundamentals lessons.

I have been able to replicate everything in FastTrack except that SPCC also creates blue tinged final image of the Flaming Star nebula. Essentially I get the same error in both my own images and from the data I downloaded to replicate the FastTrack lessons. The graphs from SPCC also show that the fit for R/G is poor and the white balance ratio for blue is around 0.4, much lower than what is in the video. I have attached the pdf of the graphs. I look forward to troubelshooting suggestions.

BlockHead

The graph you show looks OK to me. There is some scatter... but the fit is likely OK. 

Tell me more about the "blue tinge."

1. Did you choose an appropriate background neutralization area?

2. Did you select the sensor AND the filters for the Sony stuff? Note, you do not do this for OSC. The filters are quite variable. The QE curve is effectively a combined system/camera result. You would keep the "Ideal QE" (basically 1) for the sensor.

The images you may have tried to attach did not show up. (broken)

-the Blockhead

Brent Ewers

1. I chose approximately the same preview area as in the SPCC Color Calibration video for background neutralization.

2. I used the same Sony information used in the video. I did try using "Ideal QE" and other sensors such as ZWO as well as different fillters with the Sony just to see the effect and troubleshoot. They made almost no difference--the graphs were nearly the same and the image took on a blue tinge in each case after correction. The white balance factors are very different from the results in the video even though this is the same data--0.4, 0.3, and 1.0 seem very different from 0.7 and 0.9, and 1.0.

I did not try to attach any other images, so those image failures are likely from trying to figure out how to get the pdf to upload.

The blue tinge is throughout the image so the stars are more blue than both the unlinked STF as well as the final version of the image after SPCC in the video. The nebulosity is also less than either unlinked STF or the final image after SPCC in the video..

BlockHead

So... I do not know.

You have the same numbers... the numbers should come out the same.

Computers obviously will not give you different answers for the same input.

However, we need to definitely agree on something- because you kinda called me a liar. :)

In this video:

https://www.adamblockstudios.com/articles/fasttrack-2022-spectrophotometric-color-calibration

At a little after minute 10 I specifically point out and explain to NOT select the sensor and I explain why.

It may not make a difference...but we have to agree on my video!

There may be another issue upstream that you are having that is affecting the color balance.

I would need to go through step-by-step with you. You might be able to look at some of my previous numbers that I incidentally show in the video and make certain you agree with those as well.

-the Blockhead

Brent Ewers

Certainly didn't mean to kinda or even slightly imply that any lying was taking place.. I made many permutations of cameras and filters to see what effect they had on the blue tinge and graphs, and then misstated the instructions in the video. Mea culpa. I just replicated exactly the Ideal QE and three Sony Color IV/IR cut filters as well as the rest of the settings for SPCC that were used at around the 17 minute mark in the video. I still get the same results of blue tinge lower white balance numbers.

I think that everything prior to this final step of actually executing SPCC agreed with the videos. However, I will carefully check again. Because I get the same blue tinge in my own SPCC corrected images, the evidence points to something upstream of SPCC or a setting in SPCC that I am doing incorrectly.

BlockHead

Yeah... you might need to post a series of screenshots... 

The previous step is DBE... 

1. It is possible that you introduced bias..but I find that hard to believe.

2. You forgot to reset your STF. Remember after DBE (and carrying through to SPCC) you need to have a linked STF. If you still have it unlinked.. indeed you will certainly see a bias.

-the Blockhead

Brent Ewers

First, I have attached two screenshots using the Flaming Star nebula from Fast Track including the settings I used for SPCC after DBE. The fit of Image R/G to Catalog R/G looks very similar to the video. However, the fit of Image B/G to Catalog B/G is worse, more scatter, and the slope is much less. In one screenshot I show the STF linked, which looks too blue to my eye comparing to the video, and the second STF is unlinked and looks better to my eye, less blue and more color balance.

Second, here are two screenshots from working through my own data of M92. These are taken with a ZWO 533MC with a Astronomik L-2 UV-IR. The scope is a Redcat51.

I have added two more screenshots showing the settings for SPCC and the resulting graphs and regressions analyses. In one screenshot I use an unlinked STF. In the other, I use a linked STF. The linked STF looks overly blue to my eye.

The graph of Image R/G to Catalog R/G looks fine. The graph of Image B/G to catalog B/G indicates a problem. The Image B/G has a very small range of variation, mostly between .35 and .45 compared to the Catalog variation of .35 to .65. Moreover, there is essentially no relationship, i.e. the slope is flat, ~ 0.022.

I find it interesting that I am getting poorer B/G fits to catalog in both the Fast Track data and my own, although my own data fits are substantially worse for B/G than the video.

Brent Ewers

I had to add the screenshots from my own M92 data separately. They are attached here.

BlockHead

I am not convinced your debayer pattern is correct for the M92 data. Is it?

Regarding the FastTrack images- did you do any other extra processing before doing SPCC? (anything I did not demonstrate in my video) The more blue you see in your image...I wonder about the background. Can you please select my same region?

-the Blockhead

Brent Ewers

I don't believe I did anything extra to FastTrack images processing beyond what is in the videos except trying different background regions. I don't remember doing anything different; however, I did the work prior to watching Fundamentals where I learned about Saving Projects and not files. 

Here's a screenshot using a preview very close to the same one in the video. The regression slope is now more similar, but my scatterplot has more scatter and my white balance factors are different.

I have used both Auto and RGGB in WBPP for debayer of my own images. I have confirmed RGGB is the Debayer for the 533MC. I did learn that the matrix can be read in different orientations, but I don't see the FITS keyword ROWORDER in my FITS header. So, I processed images as GBRG to reflect reading the matrix from the opposite corner. Now, I get a very strong purplish cast for linked initial STF instead of bluishgreen, and no color in unlinked STF. SPCC after DBE is much better statistically but now the slope is negative for R/G and B/G looks great with a positive slope. There was a green cast, so I used SCNR. Linked and unlinked STF images after SPCC and SCNR have no color in them.

Brent Ewers

Here's the screenshot from M92 with the GBRG processing described above.

BlockHead

Again, that cannot be correct. Note that one of the slopes is negative. Ignore what it says in the header or what you think it should be... only one of the combinations is the correct one. You need to try all of them.

Indeed the roworder and mirroring will definitely cause an issue.

So you will want to debayer in each combination. Do not use an unlinked autostretch...just ignore the bias and note the signal (stars). A nebula is a better way since it is clearly red. Stars with OSC are more difficult since they are already nearly white. 

You might just want to take a 5 minute exposure of a nebula...something for which the color you are certain. 

-the Blockhead

Brent Ewers

After running through all Debayer matrix options and checking both my eye and SPCC, it seems clear the camera is RGGB. The scatter is a bit noisier but this is just one sub of IC 1805 Heart Nebula, see attached screen shot. After using image solver and then SPCC, the nebula is red with a linked STF. The scatterplot of both R/G and B/G look much more reasonable.

Since SPCC is working for this one sub and not for fully integrated image, I think this means I need to trouble shoot through WBPP set up and potentially work through a full set of preprocessing of my images without WBPP, i.e. manually, to inform WBPP. I'll work on this next.

Thanks for your continued troubleshooting suggestions.

Brent Ewers

I wanted to give an update on my troubleshooting success of the above problem. Because the SPCC graph of B/G appeared to just be noise from my image, I separated the OSC into three channels. The blue was clearly problematic with weird structures in only this channel, the other two looked fine. I went back into the raw subs and debayed them and the blue channel looked fine. I went through all the stages of WBPP processing manually and found that the blue channel issues happened after cosmetic correction. If I turned off cosmetic correction completely, the blue channel was fine and I could fully process the images with appropriate SPCC R/G and B/G fits. After some searching I find some posts on cloudy nights that said the 533mc camera sometimes defaults to 55 red and 85 blue in the driver acquisition setting. I looked in Ekos and indeed that was the case.

My images had to high of blue acquisition which was algorithmically interpreted by cosmetic correction as pixels to be eliminated. The loss of whole groups of blue pixels led to the problems. Thus, the solution to the problem was to turn off cosmetic correction for the old images. I have now updated my driver for the camera and set it to 50 red and blue, so I can do normal cosmetic correction on images acquired going forward.

I could not have figured this out without your excellent fundamentals path in which I learned how to trouble shoot all stages of image pre and post-processing. I fiound the issue while following along to your NGC1491 workflow. I then acquired my own set of NGC1491 and processed them using fundamentals.

[URL=https://astrob.in/8jfp62/0/][IMG]https://astrob.in/8jfp62/0/rawthumb/hd/get.jpg?insecure[/IMG][/URL]

BlockHead

Great job on that! Wow.

One more thing... the dark frame method (new CC in WBPP) would have avoided this issue.

You might consider using this method going forward with your sensor.

-the Blockhead

Brent Ewers

Thanks for suggestion to use the dark frame method; I will definitely do so for all the images with the blue channel set to high in the driver for the camera. I used the dark frame method on several images based as I followed along with Fundamentals. I also used it to look at high many pixels were being removed at various sigma levels.