›

NSG Causing PI to Crash

Bruce Donzanti
edited February 2023 in PixInsight

I've been using NSG for several months with no problems.  Tonight, I tried running 155 Ha registered frames through it but at the 100th frame, it stops and PI completely shuts down.  When I reopen PI, I can see the 99 files in the NSG directory and the 100th one was 84% being normalized before it crashed.  This happened 3 times and I di shutdown and reboot my computer.  I can't find anything wrong.   I attached a copy of the log where the program stops.  Any ideas what would cause this?


Bruce

Screenshot (55).png
1920 x 1080 - 192K

Comments

  • Bruce Donzanti
    edited February 2023
    I ran it a fourth time but it gave me an error before shutting down...........

    There is nothing strange about the 100th frame as they all looked the same when I blinked them.  The error is saying insufficient resources but not sure why that is the case.  I have plenty of disk space and memory.  These are 155 full frame images. 
    Screenshot (56).png
    1920 x 1080 - 261K
  • BlockHeadBlockHead
    This is probably best address to John Murphy for a good answer....

    -the Blockhead
  • Bruce Donzanti
    I just got an answer from John.  Just so you and others know, it is a current bug he is working on.  After so many files are entered, it will crash and shut PI down.  
  • Bruce Donzanti
    And it only occurs using Windows.  
  • BlockHeadBlockHead
    Thank you for that. Indeed... John was the best source in this case.
    Thanks for letting everyone know!
    -the Blockhead
  • Bruce Donzanti
    Quick update:  I am still testing this issue at John's request.  He asked I lower the number of cores being used on my desktop to use NSG.  I have a 32-core desktop and he asked I lower it to 8.  I did so and ran my large dataset and it worked.  I am going to test it using 24-cores and see if it runs properly.  This is not a fix, just testing to help identify the problem for a permanent fix. 

    To reiterate:
    1- issue is only when using Windows
    2- issue only seems to occur using a very large dataset 
  • John Murphy
    edited February 2023
    Yes, I have been working very long hours during the whole of January to try to find a solution to this problem. It only appears to happen on top end machines (which I can't afford!) which has been an extra challenge.

    The error is usually reported as File I/O related, so the first thing I did was to completely rewrite NSG's file I/O. I switched from using PixInsight's ImageWindow object, to their lower level FileFormatInstance object. On my test dataset, this provided a 10% improvement in performance :)

    The error generally reported insufficient resources for the File I/O action. NSG has never used much memory, but I have made it even more memory efficient. I also ask the JavaScript memory garbage collector to be more active during processes that are single threaded (there should be very little performance hit from doing this since the garbage collection will occur is a different thread).

    The same error is also very occasionally reported from WBPP. See this post: 

    In case the error is in the PixInsight code that I rely on, I have also made changes to NSG so that it can handle a PixInsight crash gracefully. In a new version, which is in the final stages of testing, you can continue a previous run. This will be available even if PixInsight completely crashed.

    I have never heard of it crashing with 8 logical cores. But with a large number of cores, and if you are processing hundreds of files, you might be unlucky.

    There is a useful test that would help narrow down the problem:
    • Use all logical processors
    • Set Gradient smoothness to maximum smoothness (+4.0)
    At maximum smoothness, NSG uses its own algorithm to fit a flat plane to the gradient surface (a 2D linear fit to the differences between the reference and target). At all other levels of smoothness, it uses PixInsight's surface spline algorithm.

    The next release is coming very soon. It also has improved normalization for comet processing.

    Regards, John Murphy, author of NSG and PhotometricMosaic.
  • Bruce Donzanti
    I just ran my full dataset again, only changing the gradient smoothness to +4 as John suggested.  NSG ran perfectly, at least in my case.  Hopefully, as more test data roll in, John can isolate the problem and make the fix. 
  • Bruce Donzanti
    FYI - 

    I've been the beta tester for John's next version of NSG, that is, v3.  It works extremely well and is to be released soon.
Sign In or Register to comment.