›

Gradient and Debayering Issue after Calibration in WBPP

RichardM
in PixInsight
Last night I took about 90 30-second frames of M95/96 with my system and used the Fasttrack process to calibrate in WBPP.  I used dark flats in WBPP (vs bias) since I have a cmos camera.  Experiencing the following issues.

1)  Received error message saying frames 66-90 were not able to be debayered.  I reviewed the debayered images in Blink but not see anything unusual going on for those particular frames.  

2) However, there WAS a consistent gradient issue on the left-hand side in ALL the debayered frames that I did not see in the calibrated frames via BLK.  I looked at the master dark, flat dark, and flat and did not see anything unusual.  I aslo noticed that the gradient did not change position in the frame even though some of the final frames were rotated due to a meridian flip.

Can't see to attached files.   

Any help is greatly appreciated.



Comments

  • BlockHeadBlockHead
    Sounds like corrupt files.
    You can make one of the files that couldn't be debayered available by a cloud drive (Google Drive, OneDrive..etc). Be certain to make the link public... so that anyone with the link can download the image.

    -the Blockhead
  • RichardM
    https://drive.google.com/drive/folders/1-TFEuDl7bzKc-tRS-h6sPZ9xKMueXLvN?usp=sharing

    Here's the link to my M95_96 files.  Calibrated, Cosmetized, Debayered, MasterDarkFlat, MasterFlat, MasterDark
  • BlockHeadBlockHead
    My first observation is that your 5 second MasterDark has LIGHT in it. This is bad. :)
    The 30 sec dark looks OK. However... it appears there is probably a more fundamental processing error going on. In FastTrack I suggested you look at all of the data and make certain it makes sense.... the light the dark frame is not good... so I am concerned there are other errors at play.

    My suggestion is to star from scratch. Remove all files generated by WBPP. 
    Make absolutely certain that all light frame data is raw, all calibration files are free from light leaks or other issues... and provide the screenshots of your WBPP set up. The diagnostics tab can be used to generate the screenshots for you (at least most of them) in a single click.

    -the Blockhead
  • RichardM
    Thanks! 

    I did look at all the masters but I thought my master dark flat was "ok". I'll take another look.  Maybe I didn't cover or seal up the scope well enough when I finished the regular flats and some light leaked in.  When I create a flat I put a light panel on top of my due shield and play with the exposure rates and panel volume until I get a curve that's about a third of the way in the histogram.  This was the first time creating dark flats so I turned off the panel and covered up the entire thing with a black plastic trash bag.  Thinking about that now I probably should have just taken the light panel off before the plastic bag. Camera probably picked up light off the panel.  


     I'll check the format of my light frames, but aren't "fit" files in RAW format?  

    Also, attached is one of my screenshots that I did take in WBPP.  (pending, not sure where PI dropped them)

    Also, should I just try WBPP using one of my Master Bias files instead of a dark flat since my dark flats are no good? Or should I try taking the dark flats again since my scope is outside and nothing has changed in the optical train except maybe the position of the cords from the camera?

    Thanks Much


  • RichardM
    Here are the screenshots of my WBPP board for the above.  
    00_Bias.jpg
    2880 x 1620 - 286K
    01_Darks.jpg
    2880 x 1620 - 285K
    02_Flats.jpg
    2880 x 1620 - 349K
    03_Lights.jpg
    2880 x 1620 - 550K
    04_Calibration.jpg
    2880 x 1620 - 316K
    05_Post-Calibration.jpg
    2880 x 1620 - 289K
  • RichardM
    Any help is greatly appreciated relative to these WBPP screen shots.

    Thanks!
  • BlockHeadBlockHead
    Hi Richard,

    There is nothing wrong with the setup.... but this is after you have already run through WBPP once (since the masters are loaded). My best suggestion- is that you should start from scratch and check everything before you tell WBPP to start. 

    You should include some frames that worked and some frames that did not work. No need to do a thousand frames... you need to figure what went wrong quickly. 

    Then you need to *prove* (demonstrate) that you did everything correctly. The calibration files are clean and have the expected values. The light frames are all the same size and acquired with the same camera. You know the correct debayer pattern and can demonstrate it works on any raw image. These kinds of things. Then we can converge on the issue. 

    I would recommend getting new light frames that match new calibration frames. Many times people mix sets of data expecting things to work... only to have another subtle error creep in and cause frustration. 

    -the Blockhead
  • RichardM
    Thanks, I'll take another shot at it!  

    I did complete the Fasttrack training.  Excellent course!  I know SO much more now and feel confident I can start to create some nice photos.  I have data on about a dozen objects that I will use to practice these steps. Then, I'll sign up for the Fundamentals in a couple of weeks.

    I did notice that when I finished the Bubble Nebula image it was reversed vs your image.  Used the 180 degree command to revise it so it now is oriented to your version.  (My final version is in the link below.) 

    In addition, when I used "SCNR" to take the green out I got an image that was WAY too red in the nebula.  I backed out of that and then tried Red in the SCNR and the nebula was ugly green.  So I wasn't successful using SCNR.  

    Overall, I'm happy with the final image.  

  • RichardM
    I'm going back thru all my frames and reviewing the k values.  Additionally, I'm running Blink again.  Hoping to resolve the issues with M95_96 that I'm having.  

    The average k values are as follows:

    Lights  0.07-0.08  30 sec exposures
    Darks  0.0297       30 sec exposures
    Bias    0.0292
    Dark_Flats 0.25    shot at 5 sec exposure after light panel was turned off and removed and lens was                                         covered in several blankets.  I would think that the dark-flats would be closer to the bias or                               darks.
    Flats   0.27-0.38   shot at 5 seconds under light panel and histogram was about 1/3 of the way

    To me everything looks "ok" in Blink.   BUT 0.25 for my 5 second dark_flat seem high! 

    I want to resolve this before moving on.  

    Thanks!!
  • BlockHeadBlockHead
    Oh yes you are correct... 0.25 is an absolutely crazy number and it should match the bias. That ain't right. 
    So unfortunately this is one problem. 
    If you want to continue to poke at your current mystery- you will need to look at the log file that WBPP generates. It will indicate where the problem actually begins. 

    It is hard for me to believe that when you blink your calibrated (_c) images that there is nothing wrong with some of them. You began by saying you didn't see anything wrong in the flat dark frame... but it appears there certainly is an issue. 

    So, you can either play the game of generating a new flat dark.and seeing if this improves things- but as I said you run the risk of compounding errors (and getting frustrated). But if you have the patience... it is certainly easy enough to do. Remember, you need to start fresh in WBPP. Do not load any masters... remake them. Remove all generated WBPP files from the previous session.

    -the Blockhead
  • RichardM
    Got it!  

    Relative to the log file what would I need to look for? 

    Thanks Much!

  • BlockHeadBlockHead
    *laugh*
    Those log files are pretty long! I can't say in advance- you need to read it!
    I believe you can attach it to your message on this forum. 

    -the Blockhead
  • RichardM
    I went back a reshot my dark flats again and got good k values for them.  I took another look at all the frames again in Blink.  I then reprocessed everything in WBPP without using any prior masters whatsoever.  The master are in the link below.  

    All the calibrated and cozmetized frames looked good in Blink till I looked at the debayered images.  You can see a well-defined gradient on the left-hand side of the frames.  I went ahead and performed Image Integration on the frames and except for the gradient I liked the outcome of the master light frame, named "M95_96 Galaxies April 2022..."  in the below link. The details of the galaxies were starting to appear and the brightness and color of the stars improved. 

    I then went ahead and tried DBE on the final frame using several variations/guesses of tolerance and smoothing, creating about 30 samples.  But I could never get rid of the da#m gradient and it was very grainy.  That said, the details of the galaxies and the stars improved yet again.  See the frames with DBE added as suffix. I also tried ABE but got the same result.  

    Can you look at these and perhaps advise as to what DBE settings to use or do you think there is still something upstream that is a serious problem?    



    Thanks Much!!


  • BlockHeadBlockHead
    Something is wrong (in my opinion). The shadow feature on the left side looks like an obstruction. Are you absolutely certain you didn't track into the line of sight of anything (like a wall, a building...etc)? if not... then I think something is obstructing your camera. This isn't the kind of thing to fix in post-processing.

    When you blink your debayered frames...does the darkness on the left slowly creep in>? Does it appear all of the sudden??
    -the Blockhead
  • RichardM
    I think I have figured out the issue this am. (And you're right!) My dew shield is very flexible and when it's on the scope all the way it's shape is not a perfect circle.  It's oblong.  So it's blocking light on one side of the lens.  Not sure how to fix this unless I purchase a more rigid one.  Because I'm using a HyperStar I need the sheild not only for dew prevention but also to lay my light panel down flat for making the flats.  I'm seeing this issue in all of my images the past few months.   
  • RichardM
    I might be able to wrap a belt around the shield while on the scope to prevent the material from bending in an irregular manner.
  • BlockHeadBlockHead
    A nice message to see this morning... cluing in on the statement that I am right. :)
    -the Blockhead
  • RichardM
    Well, a few weeks ago I thought I had figured out what was causing the gradient on my images, specifically, either issues with the dew shield when creating flats, or light from neighboring windows in our condo complex.  But while on vacation I took a new images using a different lens (e.g. a Sigma telephoto at 400 mm) coupled with my same camera (ZWO 294 MC Pro) AND in a different location altogether.  I created new flats and darks and still get a steep gradient in the final frames.  I still get basically the same gradient in the same location in  the frame.  I was able to somewhat correct the gradients in each of these cases with DBE and actually like some of the results but I'd still like to figure out what's happening.

    In the link below are: 
    IC4665, M5, and M88 with their associated pre-DBE image and then post DBE.
    The first two were processed the same night in my home location using my ZWO camera with 30 sec frames, 30 sec darks, and new set of flats with my Celestron scope.
    M88 was shot on vacation with same camera and camera settings but with a Sigma lens (set at 400 mm). I used the same camera settings, uses the same darks to process, but created new flats since it was a different optical train.
    As you can see I'm getting a very similar gradient regardless of the lens I used and regardless of where I took the shots.  

    Any thoughts are appreciated.  


    RichardM
Sign In or Register to comment.