Satellite trails still present in Integrated image (after NSG or WBPP)

Nihal Mirza

Hello

Looking for some advice, I've got a case where after calibration, integration I can still see satellite trails in my image, they are rather faint but notice  while I was experimenting with NSG script.

I've got a set of lights for M51 two exposures 45s and 120s, calibrated and integrated via two methods:

1. WBPP v2.2.0
2. NormalizeScaleGradient script v1.4.2

WBPP generated a master for 45s and another for 120s,

• 45s exposure - "No" trails
• 120s exposure - trails visible but are faint

NSG

• 45s exposure "No" trails visible
• 45s and 120s exposure combined into single master file at end of NSG - trails visible (45s frame used as reference)
• 120s trails visible
• 120s and 45s exposure combined into single master file at end of NSG - trails visible (120s frame used as reference)
  

I can only conclude that 120s exposure is the cause and for some reason master generated after image integration still have the reminisce of the satellite trails.

Am I being picky, may be but am intrigued as to why these trails are not getting rejected.

Attaching a screenshot. I've also attached link to drop box files.

https://www.dropbox.com/sh/drinyil87f24od8/AADwE6lgQXn7p19uEjRPkiORa?dl=0

Thanks in advance,

Nihal.

ps: I've also tried Average Sigma clipping when running Image integration after NSG

BlockHead

I assume you combined many frames (you didn't say)- at least more than 20. 

So... this has all of the hallmarks of rejection thresholds not being correct.

You can certainly use selective rejection to absolutely correct for this...but without seeing your data I can't say.

You can 20 frames of the NSG output available from the 120s.

-the BLockhead

Nihal Mirza

Ahh yes, there were 195 frames, 

• 162 were 45s exposure and 33 were 120s exposure
• reference frame chosen were "w100_Light_M51_120.0s_Bin1_Session_9a_L_gain50_20210419-011708_-14.1C_0002_c_cc_r_nsg.xisf" (the weighting in the file name makes it obvious, I guess)

As mentioned before, If I did the same but chose 45s exposure as reference frame then the effect of the trails are much much fainter but still present..

I've provided link for these via google drive

https://drive.google.com/drive/folders/1pu61rXq7RW-3BB4AjZ-5jEz1Aa8vRh6v?usp=sharing

ps: there is some artefact that is visible on the top left of the frame it only appears in the Luminance filter (120s) & Baader Moon & Skyglow filter (180s), I think its light leak in the filter wheel.

BlockHead

I am sorry, I do not get the same result you do with these files. See the image below. 

Create screenshots of your settings in ImageIntegration.

Many of the satellite trails are quite faint. So even simple averaging should make them almost disappear. 

Faint satellite trails will not necessarily reject- since the values are not really strong outliers for a typical sigma threshold. You can either lower the threshold (sigma high) or you can use selective rejection to absolutely reject the offenders. 

Do you agree that the result obtained is different/OK?

-the Blockhead

BlockHead

By the way, if you look at the rejection maps shown in this picture... GESD does a fantastic job of rejecting the satellite trails. The integrated result is even *better* than I showed...there is no trace of the trails.

(GESD is the rightmost rejection map, Winsorized is on the left)

-the Blockhead

Nihal Mirza

Hi Adam, 

Thanks, yes I do agree the 120s exposure on their own are better for satellite trails. However your results are much better than what I have got up until now.

Mind you I've used defaults settings up until now (i.e. the setting that automatically pop up with Image Integration after NSG)

When 120s are combined with 45s exposures, that's were they trails are a bit more prominent.

45s data (just a few files)

https://drive.google.com/drive/folders/1cEe_tSCgdkxDIdl6pOGmLgRnXSpOaDrJ?usp=sharing

BlockHead

I believe I understand the issue. 

First of all there are *no default* settings. (Never!)

You  need to understand the settings for most processes in Pixinsight. 

(I even have a PixTV episode titled "Defaults are an Illusion")

In this case NSG will properly configure the Normalization settings ("None") and the use of the NWEIGHT keyword for weighting. BUT- NSG will not configure the rejection thresholds for you. The rejection thresholds by "default" are too conservative. They will not reject exactly these things you show in your images at their "default" (it isn't a default) setting. For example a sigma high setting for WInsorized Sigma Clipping of 4 is way too high and will likely not reject anything at all that is of interest to you. Switching to another method (Average Sigma Clipping) would not have helped. The sigma threshold is your issue.

I explain in the ImageIntegration Primer and ImageIntegration sections how to find the best values. Indeed I also suggest a working value of 2.2 Sigma (for Sigma Threshold algorithms). You need to understand the justification I gave for this particular value (I demonstrate for a normal distribution, you will reject about 1-2% which is in practice a very good amount.)

GESD is different. This algorithm is not a sigma threshold method- and I have a section dedicated to it! 

However, again you need to adjust it to get the rejection that you want (as I showed).

May I use this example as a YouTube video? I think this exercise is a common one for many who are just starting to dive into the nuances of this. (I tried to give you a call to ask permission.)

I would present the problem as you presented it to me...and then solve it much like I did with you above.

Please let me know!

Thanks,

-the Blockhead

Nihal Mirza

Hi Adam,

I've been going through your videos (Fundamentals, working down the list and yet to complete all) and am at the WBPP stage. Having watched the NSG video on YT as well as your videos in Fundamental, I thought it would be good to use WBPP up until "Register" and then NSG...hence was experimenting.

In previous set of data I didn't have that many satellite trails, may be that's why I got away with "defaults"...its just since Feb/March 21 that quite a few frames have regular and frequent of trials...hence I've fallen into this trap. Its only because I was closely inspecting the Master frame (I generally do to see result) that I noticed the trails.

I'll watch the PixTV episode, and might have to jump to the Image integration primer which I believe is under Processes.

I'd be happy for you to use this example in the YouTube video, let me know if you need more data or still want to have a call (ahh yes I was driving back home, I do remember getting a missed call).

BlockHead

You may have seen it...but I did release a video that highlights your data.

I hope you get a chance to review it... and I hope struck the right balance with 

my "tone" in the presentation of the problem and the solution.

Thanks again very much for letting me use your data...

-The Blockhead

Nihal Mirza

Hi Adam,

Indeed, I got a notification for a new video. I have watched it, thank you for the further explanation and insights in the YT video.

Wade Hilmo

Hi Adam,

I went to download this video and the option to "save as" is not available when I play it.

BlockHead

Sorry... this should be fixed now.

-the Blockhead

jeffrey merrill

Hi:

Thanks for the video reminding us to tweak default settings.  Always a good message.

Curious if anyone tried large scale pixel rejection? I used to do a similar manual removal method suggested by Rick Stevenson, but when large scale pixel rejection became available it was amazing how well it worked with the trails.

Also, is there a way to get email notifications of new content?

jeff

BlockHead

I haven't figure out a good way to do that.

I am generating enough content, I am not convinced that people want to see an e-mail every other day?

I did establish the mailChimp e-mail list- so I certainly could start that practice.

Right now I post on Facebook at the AdamBlockStudios page as well as update the member page on the site. (So login every once and a while and look at the bottom of the list.)

-the Blockhead

Don Parkes

To avoid a new thread as I'm experiencing a similar issue, just can't get all the trails out. Adam, as you pointed out on YT the witch head is in a heavily trafficked part of the sky and I'm getting blitzed by trails in over 90% of my subs.

I'm using ESG and then ESD and working with the tolerances. One thing that I can't sort out is there are subs with sat trails that are getting higher weights than my clean reference image in ESG. IIRC there is a way to prohibit that, or am I just picking the wrong reference??

BlockHead

Don,

I do not think NSG has anything to do with the issue. 

With regards to the reference frame in NSG, it shouldn't matter very much which image you choose (as long as it isn't the very worst one anyway). Normalization is a image matching process and the resulting weights are relative within that set. If you have images with a higher weight- that just means by signal and noise metrics they are better..but it does not mean your reference is "wrong."

So if you have many frames I just would not expect the rejection to be poor in general. And if the rejection was not complete by adjusting tolerances of the given algorithm- you can next use Large Scale Rejection which will broaden the rejected pixels if the faint margins of ALREADY DETECTED trails are an issue. And if that doesn't work- you can absolutely reject by Selective Rejection.

So, if you want to pursue this- make your registered images available to download. If you do, be certain to make the link public.

If you look at this article... you can clearly see rejection of GEOsats is usually complete and without issue.

https://www.syfy.com/syfywire/earths-equatorial-belt-of-satellites

(scroll down to where my image is that shows the satellites)

-the Blockhead

Don Parkes

Adam, thanks for the syfy link (Phil is a "neighbor" of mine). 

What you were able to do with M42 gives me hope but I'm minor league with my processing abilities. Trying different tolerances + large scale redux but still not quite there. 

Would love to throw the subs up but it's a bit chunky (35GB) so I'll wrestle with things a bit more on my own. 

Simple stretch of integrated image attached using following parameters:

- ESD outliers = 0.30 [do I just keep cranking this higher? I've worked my up way to this value..]

- ESD significance = 0.10 [same here, just crank it up?]

- ESD low relax = 1.50 [frankly not sure what to do with this variable]

- Reject high large-scale enabled with 2 layers/growth. 

BlockHead

If you pick 20 sub frames with lots of satellites... you can upload a subset and we can see.

Sigma Thresholds will be a bit more brute force with the rejection (as I demonstrated in my video). So you can try WInsorized Sigma Clipping with 1.8 high sigma.

Yes, you can also increase the ESD significance to 0.2 or so.

The outliers of 0.3 (30%) is very likely good enough. 30% outliers per pixel stack is quite a few for large data sets (many frames).

The ESD relax... putting it at 1 (the lowest value) can also help. 

I explain this in my ESD video.

At a minimum you can show some screenshots..but making the files available is the ultimate way.

-the Blockhead

Don Parkes

Threw a few subs up, hopefully this link works:

https://drive.google.com/drive/folders/1hfgf_fnt-wFLkTGXbXHPVLv_socefQM_?usp=sharing

- I ran these using pretty aggressive ESD integration settings and still trails. 

- You were right, WSC rejection algorithm @ 1.8 does the job, maybe a bit too good.

Not the end of the world if I can't get this data to process 100%. It was collected with lots of moon stuff in the sky so I'll come back to the target under better conditions (and better armed for sat battle) so this is purely a learning exercise.

Thanks for the help.

- Don

BlockHead

So... I looked at your data- and this is a very special case. When you are looking to the Geobelt- the satellite trails run over the *same* pixels multiple times. This is why the rejection isn't working as well. And paradoxically, if you take more exposures it probably will not help since you will increase the chances of  them running through the same place!

I even tried using just the MEDIAN... meaning it should pick the middle value...and still the satellite trails show up which mean for the trails we see leftover- the satellite trail was in more than half of the frames. No rejection can do much with this!

So...there is a way to minimize the trails a little bit more..but not certain if it is worth it. For the first run of image integration, you use WSC at a high sigma of 2.0. 

Then you create another image using Min/Max as the rejection. And you reject all but 5-6 frames in your stack. So if you have 50 images... you make the Min 0 and the Max 45!! (The more images you can average the better). Then you take this image, for which the trails should just about be gone... you use a mask and insert this image into the area of the first image where the trails are not completely rejected.

This is the best I could come up with! 

-the Blockhead

Don Parkes

Adam, thanks for digging into this. I'll report back if I give this method a go.