0.7.288 : in .ini file SCIENCE_TARGETS requires *all* listed objects to be in the astrometric db ?

[larnoldgithub]

Hi Neil
Follow up after the email I sent you - I'm opening this ticket

07:10:46.034-!!|PROC| Could not find all objects in SCIENCE_TARGETS in astrometric database. Please
07:10:46.034-!!|PROC|   check SCIENCE_TARGETS, add missing objects to astrometric database or remove
07:10:46.034-!!|PROC|   from SCIENCE_TARGETS and try again.
07:10:46.037-!!|PROC| objnames=blablabla
07:10:46.314-**|PROC| ***************************************************************************
07:10:46.334-@!|PROC| W[40-003-00005]: Recipe apero_processing has NOT been successfully completed
07:10:46.345-**|PROC| ***************************************************************************

Its seems that with SCIENCE_TARGETS= obj1, obj2, obj3 and with obj3 not in the astrometric db, the processing fails to start.
How to handle a list of objects with some not worth being in the astrometric db ?

I do want to process only a limited number of objects, with a bunch not in the astrometric db (too faint objects).

[larnoldgithub]

additional question: does SCIENCE_TARGETS accept DRSOBJN ?

[njcuk9999]

objects with some not worth being in the astrometric db

They should all be in the astrometric database - for lbl and air if they have no proper motion you can use one of the arguments (see the --help) please comment of these for now and I'll add a tag to them for NO_PM - in v0.8 we'll use this tag

SCIENCE_TARGETS crossmatches against the database - the preferred name is the DRSOBJN but any alias will work.

[larnoldgithub]

I have 150 stars to process for 24A, some of them aren't in Simbad. How can I proceed then ?

[larnoldgithub]

those are 'custom targets' entered by PI in our Kealahou system.

[larnoldgithub]

I'm running another Apero for the quicklook, just for one night, with SCIENCE_TARGETS=All and I get

00:42:11.031-@!|PP_SCI[00159]| W[10-010-00005]: Object LP_695M351 is not in the astrometric database. Using
00:42:11.031-@!|PP_SCI[00159]|  header values for astrometric parameters

This is the usual warning, so just a warning, not an error as with the processing mentioned above.

It seems that SCIENCE_TARGETS=All and SCIENCE_TARGETS=obj1, obj2,obj3 aren'n processed the same way ?

[njcuk9999]

I have 150 stars to process for 24A, some of them aren't in Simbad. How can I proceed then ?

Can you not process all targets in 24A? Why do you need such a big list, how many targets are not in your list?

It seems that SCIENCE_TARGETS=All and SCIENCE_TARGETS=obj1, obj2,obj3 aren'n processed the same way ?

I'm not sure what you mean here. SCIENCE_TARGETS=All does not filter by target name so just gets all .fits for the raw directory. For SCIENCE_TARGETS=obj1,obj2,obj3 I have to look in the database for aliases hence why they have to be there. How does APERO know obj1 is not an alias to another target? It is dangerous to process objects not in the database because it means no one has checked for an entry already existing under another name. That being said I should be able to just try to find any object name you have on disk - right now I don't know how much code change this would be, I'll have to check.

[larnoldgithub]

I have 150 stars to process for 24A, some of them aren't in Simbad. How can I proceed then ?
Can you not process all targets in 24A? Why do you need such a big list, how many targets are not in your list?

I only want to process the objects of 24A (new or already observed before); I'll recalculate the template for those, so I set INCLUDE_OBS_DIRS =All. If I do SCIENCE_TARGETS=All too, then I'll repreocess all SPI data, which I don't want. I only want to reprocess the 24A objects and update the template for those.

Is there better way to proceed for that ?

[larnoldgithub]

It seems that SCIENCE_TARGETS=All and SCIENCE_TARGETS=obj1, obj2,obj3 aren'n processed the same way ?
I'm not sure what you mean here. SCIENCE_TARGETS=All does not filter by target name so just gets all .fits for the raw directory. For SCIENCE_TARGETS=obj1,obj2,obj3 I have to look in the database for aliases hence why they have to be there. How does APERO know obj1 is not an alias to another target? It is dangerous to process objects not in the database because it means no one has checked for an entry already existing under another name. That being said I should be able to just try to find any object name you have on disk - right now I don't know how much code change this would be, I'll have to check.

What I meant is what you're explaining: SCIENCE_TARGETS=All doesn't filter, ='obj1,obj2,obj3' does filter. So I understand that if I would set SCIENCE_TARGETS=All instead of the long list of objects, I would have no issues reducing all objects, but as I said this is not what I want to do.

Let me know if I'll still unclear!

[njcuk9999]

Is there better way to proceed for that ?

Currently no objects have to be in the astrometric database.

[larnoldgithub]

Hi @njcuk9999 good to hear this morning that fixing this issue is in your immediate todo list ;)

A full reprocessing may be worth with the new CCF of the .7.290. Do you need a test of the .290 with the minidata2 ? I may be able to run a test in the coming days. Or could I run the CCF on the existing .288 files ?

[njcuk9999]

Urm there were some changes to the background which start with the preprocessing so probably best to run the mini data fresh.

You can of course try to run existing files but I can't promise you wont get errors.

[njcuk9999]

So I've been looking into doing this, so I can do this but:

• These targets will not work with LBL
• These targets will not use the right mask in CCF
• These targets will not be combined correctly in templates if more than one object name is used
• These targets will not have a good BERV

In the future, there may be other reasons why these won't work as having an object database is being used in multiple places (not just for the BERV).

I would still strongly recommend adding all objects to the object database.

It will also be much slower to find the targets as instead of currently just looking in the object database I would have to do the following:

1. Make sure the file index database for all raw observations is up-to-date
2. Get the reject list
3. Check all objects against the reject list
4. Get a list of unique objects
5. Check against the database and all aliases

This takes a lot of time every time I want to find "unique" objects when one could just get a list of objects from the object database very quickly.

Also there is no way to know whether the user made a typo or really wants these targets - I've found it very useful to have a "catch" when writing the wrong target name in SCIENCE_TARGETS (i.e. WASP-90 instead of WASP-80)

[njcuk9999]

okay one thing I could do is when checking whether they are all in the database ask the user to confirm if targets that are missing from the database are okay to be added to processing... you'll get a pause and have to confirm each one that was found to be missing but otherwise, I see no way to "check" these are the objects you want.

[larnoldgithub]

Neil, I have of the order of 400 objects at least that are too faint, or are for prgm different from RV at all. If I do a full reprocessing, do you mean Apero will pause 400 times and ask mee to confirm each one ? That doesn't sound very convenient.
I'll try to put a maximum of targets in the objdb. Do you think apero_astrometrics could have an option -simple where I could put the target name and enter a minimal number of things, RA DEC Teff ? and not be asked for PM or more ?

[larnoldgithub]

I'll use the option --nopmrequired

[njcuk9999]

We could add a "simple" mode I guess this would have to be in v0.8 as I would want to add a flag in the database (that is not possible in v0.7 but is in v0.8). Could you tell me which ones of your new targets have nopm (by email), I'll add a tag that they are non proper motion targets (this will be a flag you can add in v0.8 but not v0.7).

[larnoldgithub]

I think I can manage all the targets for 24A, but for 24B we have a prgm with 201 faint objects (we are not gonna observe all though) and it apparently will continue in 25A with 165. Maybe the 'simple mode could readn from a file targetname, RA, DEC, Teff. I'm not sure if it's the best solution.

If I set the PI_NAME for that PI and set SCIENCE_TARGETS=All it may only process the objects of that PI? I will do a test with --test=True.

[njcuk9999]

It should work with the PI_NAME, I've had some problems with PI_NAME and white spaces but that could be a work-around.

We have many targets in the database not from SPIRou (we use it a lot with LBL for example, and for NIRPS), we really need a human checking that target name is resolved in simbad.

The whole point of the database is to avoid getting into an "automation mess" and avoiding duplicates that will break templates etc.

Maybe we could get a few students here to enter the list of targets? We did this with a few hundred target for NIRPS.

Are they all going to be unresolvable with Gaia?