DRS Realtime Trigger and Processing Tools

In the current trigger, there is a listener process which is triggered by endgo. Currently the listener is responsible for firing off all parts of the reduction for each file it is passed. This will be changed so that the listener will only run pre-processing on the file. The pre-processing updates the index.fits file which can be leveraged to keep track of what files already exist, removing the need for extra book-keeping logic in the trigger itself.

Shared functionality

Index Load

1. Read index.fits
2. Make sure ordered by date/time
3. Split up exposures by sequences
4. Split up sequences by DPRTYPE (for calibrations at least, what about for objects?)

Do we want to do the pre-night calibrations separate from the post-night? This might imply having parameters to ignore entries before/after specified timestamps. Also important to consider that science exposures are useful (ready to be extracted) without waiting for full sequence.

Index Check

This should take in one or more DPRTYPEs, and verify that the currently loaded index has a sequence of each. This check should have a realtime flag that causes it to re-run the Index Load then wait and retry until the check succeeds. If the realtime flag is false, it should probably throw an error on failure.

Calibrations

Before each recipe, run the Index Check for the necessary [DPRTYPEs].

1. cal_DARK [DARK_DARK]
2. cal_BADPIX [DARK_DARK, FLAT_FLAT]
3. cal_loc_RAW [DARK_FLAT]
4. cal_loc_RAW [FLAT_DARK]
5. cal_SLIT [FP_FP]
6. cal_SHAPE [FP_FP, HCONE_HCONE]
7. cal_FF [FLAT_FLAT]
8. cal_HC [FP_FP, HCONE_HCONE]
9. cal_WAVE [FP_FP, HCONE_HCONE]

Will need to run twice per night (calibrations at start and end of night).

Science Objects

TODO: gotta think about how to best handle this, including handling things like:

• Polar sequences
• Telluric standards
• Sky exposures
• Faster extraction for quicklook
• Knowing when science starts and ends from index file for realtime processing?

Realtime Trigger

Listener

Similar to current listener, but only run pre-processing.

Quicklook

1. Calibrations (pre-night, realtime = true)
2. Science Objects (quicklook mode) plus database updates
3. Calibrations (post-night, realtime = true)
4. Wait for next night.

Reprocessing Tool

Night Reprocessing

1. Load Index
2. Calibrations (pre-night)
3. Calibrations (post-night)
4. Pre-science drift stuff?
5. Science Objects

Long-term Re-processing

Re-run mk_template and fit_tellu on all objects?

TODO

• Re-processing from archive
• Useful stuff in current trigger that should be carried over
• Add table with DPRTYPE rules based on Claire's excel sheet