Keyword grouping help--watched vids on multi-night and grouping but each frame listed individually

Jonathan Anderson · January 21

I watched the videos on wbpp and multiple nights. NIGHTS keyword works great for pulling in flats and lights together. I do see that there is a batchfitskeywordedit script, but I'm struggling to figure out what exactly I need to change what to and where . . . if that makes any sense. . .

As in Adam's video, I am shooting flats nightly . . . and since I am shooting pre and post midnight, a single session of m65 will have 2 date stamps (and a bunch of individual time stamps). I tried using my mac finder window to just rename all dates to the same date, but then when I pull data in, it all gets there, but each light and flat is treated as a single file because of that time stamp (I think). Flats are grouped by exposure time . . . attaching screenshots:

First one shows calibration tab . . . I cannot get the Keyword window to show all of the keywords (it won't expand), but it shows LEO, and each of the 4 session dates. I have NIGHTS keyword disabled.

Second one shows Lights tab with each individual light treated as a single frame rather than grouped by date (probably bc of the time stamp?).

Third one shows Flats tab with flats grouped by exposure length, AND showing each flat as a single frame rather than grouped together by date.

Last one shows my folder/file structure and names (I had already changed all dates to have the same YYYYMMDD for each session--this is the DATE-OBS piece I'd imagine).

Am I needing to use the batchfitskeywordedit script for everything? Or do I need to rename in my mac finder window like I did to get all night's dates to a single date? I hesitate to completely remove the DATE-OBS keyword from the fits info bc i am guessing the timestamp is used for things like ImageSolver script. [note: i thought that it might be that there is a dash instead of underscore between the date and time, so I renamed all dashes to underscores in my finder window, but this yielded the same result]

I believe I am going to have to find a way to assign keywords for each night that I shoot, each target that I shoot, etc., which is fine, I just need to know how to set things up--so far after hours of trying, the screenshots show you my results. Closer, but not what it looks like in the video where everything is nicely chunked together.

Jonathan Anderson · January 21

(I've changed all the dates back to their original YYYYMMDD since changing the day could mess with important information needed by scripts) . . .so it's back to each night having 2 dates for one target.

BlockHead · January 21

Hi Jonathan,

Please repost your screenshots with your new configuration.

I got a little lost in your explanation.

I do know that what you showed will not work.

Remember the full path is what is looked at when WBPP searches for keywords and their values.

You have "LEO" multiple folder names. This is no good.

There are some other issues as well.

You should use the diagnostics button to generate the screen shots so they will be full screen... or you need to stretch (lengthen) your WBPP window. There are more columns you are not showing...including the values for the keywords. This will tell you exactly what WBPP is doing...and how you can get it to do what you want.

You do not need to use the batchfitskeywordeditor to do what you want I think.

I would probably avoid using dates as values for keywords for exactly the reasons you are having issues.

1. The rollover at midnight... this is a single grouping of files you want to calibrate with a set of flats... but it falls into two different folders.

2. You were, I think, getting hung up on dashes. Remember both underscores and dashes are stop characters.

Here is what you need to do. You need to make your folders have a common word. Lets say it is LeoTp.

Forget about the date for the moment. That does not matter. What if you took three nights of data...but you only need one set of flats?

So make the first folder called LeoTp_Kermit

Inside of LeoTp_Kermit should be all of the data you acquired (on one night or multiple nights) that needs the same flats (and biases or darks).

The next folder should be LeoTp_Piggy.

The keyword (just one!) you would put in the Pre for the WBPP is LeoTp.

Now if you keep your current method of having two folders (or multiple folders) per night. That is OK. You just need to find a way to put those folders that need to be grouped together under LeoTp_Kermit.

LeoTp_Kermit has two folders called "1-10-25.... " and "1-11-25... " The flats would be outside of these two folders but within LeoTp_Kermit .

This will basically do what you want.

-adam

Jonathan Anderson · January 27

Yes, I am getting confused as to what PI looks at: Folder names/File names and/or FITS keywords embedded in the file (ie you don't see them in the finder window on mac). It looks like it is both.

What I have been trying is using the batchkeywordedit to add DAY, setting the value as '2025-01-01' (for example), then doing it again adding the FILTER keyword as LENH (for L enhance) (since I don't use a filter wheel, the asiair doesn't add that keyword automatically, but I'll see if I am able to specify in that tab even if not using a wheel).

That method has fixed the issue, but boy howdy is it tedious to go back and go thru that script since you cannot add multiple keywords at once!

If I am hearing you correctly, you're saying that folder naming ought to do the trick? I store my flats in the same folder as my lights (in a Nights_01-01-2025 folder, for example) . . . for current wbpp, I have deactivated the Nights keyword, and activated Day and Filter (although since I'm not trying to combine filters here, I'm not sure the filter one is needed).

Apologies for the delayed response...I appreciate the help. Attaching screenshot of current folder hierarchy and naming, and my wbpp using the batchfitskeywordeditor script.

Screenshots: current working method using tedious batchkeywordedit script, correctly sorted flats (lights correct too), warning on calibration if individual bias and darks deleted, folder hierarchy etc.

Jon

PS . . . odd occurance . . . if I delete the individual darks and bias files, leaving only the masters, I get the warning seen in the screenshot. It's not a huge deal to just leave the individual files in there with the master, but it is a curious thing that I have not seen before--I have not done any batchkeyword editing on those.

BlockHead · January 27

Keywords are found in two places... the PATH statement or in the FITs header.

Yes, PI (WBPP) looks at both.

The PATH statement is a STRING (a long list of characters) that includes BOTH the folders and filenames. This is searched by WBPP for Keywords.

Custom Keywords you put in a path take precedence over those in the FITs header. YOU put them there..so they will override the original ones if they exist in both places.

I strongly recommend you do not have values such as '2025-01-01' ... hyphens and underscores are stop characters. Better is is 20250101 ... this is a good value.

The logic you have is still not quite right.

But it is hard to tell since you are mixing keywords in the header with keywords in the PATH.

My suggestion is to just use the PATH and do not use the HEADER.

I assume you used "LENH" because you are doing dual band imagery?

I don't understand this... what is the original filter name?

-the Blockhead

Jonathan Anderson · January 28

LENH is just my code for LeNhance filter. I'm specifying so I can start to figure out how to combine filters' data, esp if I ever decide to tackle mono. I do have some m31 data with a few different narrowband filters that I was just experimenting with and could play with combining, but would need the flats to calibrate correctly with the lights . . . if I try mono, it'll be Ha, oiii, etc filter names instead).

I definitely want to only use one or the other. I'll get rid of the added keywords in the fits headers via batchfitskeywordedit script and figure out how to name the folders. I put SO much info in one of those folder names so I can see at a glance where things are located instead of having to go open a file to see what camera was used, what filter was used, and what scope was used. One long folder name seemed (at the time) simpler than having:

Date Folder > Target subfolder > Scope sub-subfolder > camera sub-sub-subfolder > filter sub-sub-sub-subfolder > Seconds Gain Temp sub-sub-sub-sub subfolder.

I need to get rid of the dashes except where needed to seperate keywords.

BlockHead · January 28

Filters are automatically discriminated in WBPP (automatic keywording)...so you do not need to specify that as long as your acquisition software puts the information in the Header (as all software will do).

Long folder paths will give you another issue (in Windows) if you do not adjust the registry and allow for long path names. Be certain you saw this video and made the adjustment.

You only need to get rid of dashes and hyphens when dealing with a keyword or its value... it is OK to have them in the file name else where. WBPP first looks for the keyword.. then looks for the stop character... then reads in what follows as the value once it reaches the second stop character.

-the Blockhead

Jonathan Anderson · January 28

I am using a mac...max_path is 1024, so I'm good there (although I will now be shortening things up anyway), but I did check that when I saw that video title.

I am using the asiair plus, and since I do not use a filter wheel, that tab is not turned on (and cannot be turned on if there's no wheel attached) and will not allow me to enter the Filter name. When I look at Fits Headers of lights (and flats), there is no Filter keyword for this reason. See screenshot.

So, Keyword can be the target (LeoTp, for example), then the value can be the date (20250101, for example), so the folder would be LeoTp_20250101, then the next folder would be Filter_LENH (since I'm not using a wheel, so asiair software will not automatically add a Filter Keyword in the FITS headers).

I very much appreciate your help and patience.

BlockHead · January 28

Well.. my point is... if you are not using a filter... your filter name will be "No Filter" (this is the default in WBPP). You do not have to make a filter name if you do not want to. If, however, you are indeed using an OSC and you put a LP filter in front for some exposures and no LP filter for others...this would be the case where you need to distinguish. I think, if I understand, you literally do not care what the filter is called because you are not discriminating between any.

When you start using mono...you also will not need to do anything as well. As you note, once your software has a filter wheel...it presumably will write the VALUE (filter name) for the keyword FILTER that in your FITS header.

-the Blockhead

Jonathan Anderson · February 7

Ok, I'm VERY close to getting my method down . . . there's another user on CN who does mosiacs and his brain seems to work very similar to mine in terms of how to structure things . . . I've got it 99% working! Thank you for your help!

One slight detail: in the screenshots, you can see that if my folder is named "nights_20250121 scope_RC cam_2600 fil_Lpro 120s 0g 0c" (I had dashes between each KEYWORD_value, but thought that might be the problem, so I removed them, but this still persists) . . . with that folder name, values for things in the WBPP file lists and calibration screen have the next keyword as part of the previous value. Since dashes are stop characters, I had thought that they would break things up nicely, but that didn't work.

Not really the end of the world, but something small like this could lead to bigger issues later, so if I can understand what I'm missing, that would be ideal.

At this point, I'm starting to feel more ready to start doing mono in the next few months to learn the process . . . so time to start saving for some good filters, a wheel and a mono cam!

Thanks!

Jon

BlockHead · February 7

You are still fighting the convention.

Just put underscore characters between any words. No exceptions. Do this going forward and your life will be easier. The issue is that spaces are not stop characters. You have actual spaces in your folder names.

Secondly why do you need to specify the camera? Are you really going to be using multiple cameras for a single target? You do not need keywords for filters or exposure times (use the tolerance setting). Again for a single target are you really going to have multiple temperatures for darks?

Finally, you need to specify registration by keyword. You are currently attempting to register images of M87 with images of M42. This will fail I think you will agree.

It appears to me you are making this much more complicated than necessary.

-the Blockhead

Jonathan Anderson · February 7

(I'm not sure I'm seeing m42 anywhere, but if I'm missing that, I certainly need to address that . . . i haven't imaged that this year at all. Could you please point me to where this is? I'm just not seeing it)

re: spaces, dashes, underscores:

I am so sorry that I misunderstood...on 1/21 you said,

"2. You were, I think, getting hung up on dashes. Remember both underscores and dashes are stop characters." I
thought that both underscores and dashes were stop characters, and I did
try dashes between everything, but I'll try again and attach screenshots of the results, then try only using
underscores. I wanted to use dashes to show a clear distinction between
keywordA_valueA and keywordB_valueB. (just added screenshots...I label what they are at the bottom of this post)

re: CAM specification:

I do have images using different cameras on the same target (and same filter and settings). I don't want to accidentally combine data of m84 (etc) from a different camera even if it has same filter/settings.
It makes it very easy for me to see exactly what settings/equipment to use when I decide to shoot a target that I've shot before and liked the results--so it's almost a log for me.

Screenshots:

Using all underscores (Works!) . . . apparently even though dashes and underscores are both stop characters, they cannot be used together if you want clear info in wbpp columns (also cannot use just dashes to get clear info---see next 2 items)
Using a mix of dashes and underscores (small mixing of keywords/values in wbpp columns)
Using only dashes (BIG mixing of keywords/values in wbpp columns)

BlockHead · February 8

Do not forget what I mentioned about registration!

I was just giving an example of M42. If you run two different targets through WBPP (which you can do now)- you need to distinguish between them for registration (which is post processing) by changing registration from "Auto" to "Register by Keyword."

-the Blockhead

Jonathan Anderson · February 8

Oh, you were giving an If/then scenario . . . apologies, that wasn't clear. I was hunting for where I had dropped in m42 images.

But yes, I definitely will be trying to integrate multiple targets overnight at some point, so I will need to be able to register by keyword (the Target keyword in my case, or Object if I use what is filled in the fits headers, but not visible in my folder name).

Any idea about why Dashes are stop characters that apparently do not act like Underscore stop characters (see the 3 screenshots with entirely different results)

BlockHead · February 8

I can't remember what people wanted for dashes... it might be they are no longer stop characters.

The underscores are definite. I was never a fan of having multiple stop characters. Just have one... no confusion.

-the Blockhead

Jonathan Anderson · February 8

Thanks! I agree 100% with only 1 type of stop character . . . no confusion.

J

Jonathan Anderson · February 8

Just a at-a-glance summary:

Only use Underscore ( _ ) as stop character. Put it between all words. Avoid using spaces or dashes (apparently it can cause extraneous words from your file/folder names to appear in the wbpp columns--it'll probably still work, but looks a little confusing)
You'll define which of the words in your folder/file names are keywords by assigning them in the Grouping Keywords box in WBPP. Watch Adam's videos on WBPP keyword grouping for how the little icons work. Words that you don't define as keywords in the Grouping box won't impact anything. When you assign a keyword in the Grouping Keywords box, whatever is after the underscore in your file/folder name is the value it'll use for sorting.

Regarding number 2, depending on what you are doing, you may not need all of the keywords that you have defined . . . you can toggle between using them for pre-, post-, pre- AND post-, and nothing (off) by clicking through the gears icon beside the list of keywords. For example, if you are just processing 2 nights of non-mosiac data that have their own flats, you wouldn't need keywords that you have created for mosiac panels.

I think that basically sums it up. I'm looking into the multiple target registration thing . . . I see where you can select Auto or Manual in a dropdown menu in wbpp (Registration Reference Image section), but need to read more about how that actually works. You said it's a post-processsing thing, so I also looked in the post-calibration tab but didn't see anything about "register by keyword" (checked the StarAlignment process as well and don't see it there). I'm sure it's in a video.

BlockHead · February 9

Since registration is a post-processing option- you need to make one or both keywords be for BOTH pre and post processing. Then the option will be available.

-the Blockhead

Jonathan Anderson · February 11

AH HA!! Thanks!!

J

Jonathan Anderson · February 13

Somebody on CN just pointed something out that I had not considered with respect to using Dashes . . .now i rename my MasterLights without any dashes anyway, but I could see there being an accidental issue had I not seen this: PixelMath sees Dashes as a function (like subtraction) . . . if there is a dash in the filename, that could present an issue . . . pixelmath may have something in there that can tell the difference between a function and a file name, but better safe than sorry!

BlockHead · February 14

Indeed..that is also a good reason...

-the Blockhead

Howdy, Stranger!

Categories

In this Discussion

Keyword grouping help--watched vids on multi-night and grouping but each frame listed individually

Comments