no drift.rdb file after the LBL processing in APERO

[larnoldgithub]

After the completion of a full reprocessing with APERO, I ran the LBL for a set of targets.
APERO 0.290/291

here is how the lbl was set in the ini file:

Define what to run
Run the lbl recipes
RUN_LBLREF = True
RUN_LBLMASK_FP = True
RUN_LBLCOMPUTE_FP = True
RUN_LBLCOMPILE_FP = True
RUN_LBLMASK_SCI = True
RUN_LBLCOMPUTE_SCI = True
RUN_LBLCOMPILE_SCI = True

Define what to skip (if files found)
Skip the lbl recipes
SKIP_LBLREF = False
SKIP_LBLMASK_FP = True
SKIP_LBLCOMPUTE_FP = True
SKIP_LBLCOMPILE_FP = True
SKIP_LBLMASK_SCI = False
SKIP_LBLCOMPUTE_SCI = False
SKIP_LBLCOMPILE_SCI = False

[larnoldgithub]

I had not run LBL before with this setup of APERO, so the SKIP set to True shouldn't have had any impacts.

[njcuk9999]

Do you have a FP directory in the lbl/science directory?

If so could you try just running the following steps (i.e. set them to True and everything else to False)

RUN_LBLMASK_FP = True
RUN_LBLCOMPUTE_FP = True
RUN_LBLCOMPILE_FP = True

I'd like to have the log if possible for these ones to see whats happening.

[larnoldgithub]

I'll do this test asap @njcuk9999

in the mean time, I'm running another LBL with APERO in another flavor of APERO , the quicklook. No lbl ran before withing this profile. I can see already a problem with FP: the Template_s1dv_FP_sc1d_v_file_AB.fits cannot be found

2024-12-17 12:27:46.247|G|LBLCOMPUTE_FP[00048]| Plot directory exists (Path=/data/spirou4/apero-data/quicklook290/lbl/plots)
2024-12-17 12:27:46.247|G|LBLCOMPUTE_FP[00049]| LBL RV directory exists (Path=/data/spirou4/apero-data/quicklook290/lbl/lblrv/FP_FP)
2024-12-17 12:27:46.247|G|LBLCOMPUTE_FP[00048]| Model directory exists (Path=/data/spirou4/apero-data/quicklook290/lbl/models)
2024-12-17 12:27:46.248|G|LBLCOMPUTE_FP[00049]| LBL reftable directory exists (Path=/data/spirou4/apero-data/quicklook290/lbl/lblreftable)
2024-12-17 12:27:46.248|E| File /data/spirou4/apero-data/quicklook290/lbl/templates/Template_s1dv_FP_sc1d_v_file_AB.fits cannot be found
2024-12-17 12:27:46.248|G|LBLCOMPUTE_FP[00049]| LBL rdb directory exists (Path=/data/spirou4/apero-data/quicklook290/lbl/lblrdb)
2024-12-17 12:27:46.248|G|LBLCOMPUTE_FP[00049]| Plot directory exists (Path=/data/spirou4/apero-data/quicklook290/lbl/plots)
2024-12-17 12:27:46.249|G|LBLCOMPUTE_FP[00049]| Model directory exists (Path=/data/spirou4/apero-data/quicklook290/lbl/models)
2024-12-17 12:27:46.249|E| File /data/spirou4/apero-data/quicklook290/lbl/templates/Template_s1dv_FP_sc1d_v_file_AB.fits cannot be found
22:27:46.255-**|LBLCOMPUTE_FP[00048]| QUALITY CONTROL SUCCESSFUL - Well Done -
22:27:46.257-**|LBLCOMPUTE_FP[00049]| QUALITY CONTROL SUCCESSFUL - Well Done -
22:27:46.304-!!|LBLCOMPUTE_FP[00047]| LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
22:27:46.304-!!|LBLCOMPUTE_FP[00047]|   File
22:27:46.304-!!|LBLCOMPUTE_FP[00047]|   /data/spirou4/apero-data/quicklook290/lbl/templates/Template_s1dv_FP_sc1d_v_file_AB.fits
22:27:46.304-!!|LBLCOMPUTE_FP[00047]|   cannot be found
22:27:46.306-!!|LBLCOMPUTE_FP[00046]| LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
22:27:46.306-!!|LBLCOMPUTE_FP[00046]|   File
22:27:46.306-!!|LBLCOMPUTE_FP[00046]|   /data/spirou4/apero-data/quicklook290/lbl/templates/Template_s1dv_FP_sc1d_v_file_AB.fits
22:27:46.306-!!|LBLCOMPUTE_FP[00046]|   cannot be found
22:27:46.319-!!|LBLCOMPUTE_FP[00050]| LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
22:27:46.319-!!|LBLCOMPUTE_FP[00050]|   File
22:27:46.319-!!|LBLCOMPUTE_FP[00050]|   /data/spirou4/apero-data/quicklook290/lbl/templates/Template_s1dv_FP_sc1d_v_file_AB.fits
22:27:46.319-!!|LBLCOMPUTE_FP[00050]|   cannot be found
22:27:46.329-!!|LBLCOMPUTE_FP[00048]| LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
22:27:46.329-!!|LBLCOMPUTE_FP[00048]|   File
22:27:46.329-!!|LBLCOMPUTE_FP[00048]|   /data/spirou4/apero-data/quicklook290/lbl/templates/Template_s1dv_FP_sc1d_v_file_AB.fits
22:27:46.329-!!|LBLCOMPUTE_FP[00048]|   cannot be found
22:27:46.331-!!|LBLCOMPUTE_FP[00049]| LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
22:27:46.331-!!|LBLCOMPUTE_FP[00049]|   File
22:27:46.331-!!|LBLCOMPUTE_FP[00049]|   /data/spirou4/apero-data/quicklook290/lbl/templates/Template_s1dv_FP_sc1d_v_file_AB.fits
22:27:46.331-!!|LBLCOMPUTE_FP[00049]|   cannot be found
22:27:46.583-**|LBLCOMPUTE_FP[00046]| ***************************************************************************
22:27:46.587-**|LBLCOMPUTE_FP[00047]| ***************************************************************************
22:27:46.597-@!|LBLCOMPUTE_FP[00046]| W[40-003-00005]: Recipe apero_lbl_compute_spirou has NOT been successfully
22:27:46.597-@!|LBLCOMPUTE_FP[00046]|   completed (1.851 seconds)
22:27:46.603-@!|LBLCOMPUTE_FP[00047]| W[40-003-00005]: Recipe apero_lbl_compute_spirou has NOT been successfully
22:27:46.603-@!|LBLCOMPUTE_FP[00047]|   completed (1.844 seconds)
22:27:46.613-**|LBLCOMPUTE_FP[00046]| ***************************************************************************
22:27:46.614-**|LBLCOMPUTE_FP[00050]| ***************************************************************************
22:27:46.617-**|LBLCOMPUTE_FP[00048]| ***************************************************************************
22:27:46.631-**|LBLCOMPUTE_FP[00049]| ***************************************************************************
22:27:46.640-**|LBLCOMPUTE_FP[00047]| ***************************************************************************
22:27:46.644-@!|LBLCOMPUTE_FP[00049]| W[40-003-00005]: Recipe apero_lbl_compute_spirou has NOT been successfully
22:27:46.645-@!|LBLCOMPUTE_FP[00048]| W[40-003-00005]: Recipe apero_lbl_compute_spirou has NOT been successfully
22:27:46.644-@!|LBLCOMPUTE_FP[00049]|   completed (1.854 seconds)
22:27:46.645-@!|LBLCOMPUTE_FP[00048]|   completed (1.859 seconds)
22:27:46.645-@!|LBLCOMPUTE_FP[00050]| W[40-003-00005]: Recipe apero_lbl_compute_spirou has NOT been successfully
22:27:46.645-@!|LBLCOMPUTE_FP[00050]|   completed (1.834 seconds)
22:27:46.661-**|LBLCOMPUTE_FP[00049]| ***************************************************************************
22:27:46.663-**|LBLCOMPUTE_FP[00048]| ***************************************************************************
22:27:46.663-**|LBLCOMPUTE_FP[00050]| ***************************************************************************

[larnoldgithub]

Do you have a FP directory in the lbl/science directory?

Yes there is a FP/
I'll run the test you suggest.

[larnoldgithub]

@njcuk9999 should I reset lbl first and then run the FP steps as you suggest ?

[njcuk9999]

I'd like to see whats happening without resetting (to see the current problem).

If there is no template you can just run the RUN_LBLCOMPUTE_FP step and look at the output... you don't even need to use the ini file you can just run apero_lbl_mask_spirou.py --objname=FP that should create the mask for FP, so I'd like the log from that recipe run!

[larnoldgithub]

Here is the log after I launched apero_processing.py LBL_run_LBLCOMPILE_FP_only.ini on Dec19.
As I mentioned this morning at the DRS meeting, it seems to fail becasue of a RAM issue, after 97% of the process was done.

06:14:45.730- |PROC| ***************************************************************************
06:14:45.737- |PROC| *
06:14:45.737- |PROC| * SPIROU @PID-00017346752849674160-F9TO (V0.7.291)
06:14:45.738- |PROC| * py3.9.16 git:v0.7.291-stable-test
06:14:45.738- |PROC| *
06:14:45.744- |PROC| ***************************************************************************
06:14:45.750- |PROC| █████╗ ██████╗ ███████╗██████╗ ██████╗
06:14:45.755- |PROC| ██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔═══██╗
06:14:45.761- |PROC| ███████║██████╔╝█████╗ ██████╔╝██║ ██║
06:14:45.767- |PROC| ██╔══██║██╔═══╝ ██╔══╝ ██╔══██╗██║ ██║
06:14:45.773- |PROC| ██║ ██║██║ ███████╗██║ ██║╚██████╔╝
06:14:45.779- |PROC| ╚═╝ ╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝
06:14:45.785- |PROC| ***************************************************************************
06:14:45.794-|PROC| DRS Setup:
06:14:45.800-|PROC|
06:14:45.800-|PROC| DRS_ROOT: /apero/apero-drs-v0.7.290/apero
06:14:45.801-|PROC| DRS_DATA_RAW: /data/spirou4/arlinks
06:14:45.801-|PROC| DRS_DATA_REDUC: /data/spirou4/apero-data/offline290/red
06:14:45.801-|PROC| DRS_DATA_WORKING: /data/spirou4/apero-data/offline290/tmp
06:14:45.801-|PROC| DRS_CALIB_DB: /data/spirou4/apero-data/offline290/calib
06:14:45.801-|PROC| DRS_TELLU_DB: /data/spirou4/apero-data/offline290/tellu
06:14:45.801-|PROC| DRS_DATA_ASSETS: /data/spirou4/apero-data/offline290/asset
06:14:45.801-|PROC| DRS_DATA_OUT: /data/spirou4/apero-data/offline290/out
06:14:45.802-|PROC| DRS_DATA_MSG: /data/spirou4/apero-data/offline290/msg
06:14:45.802-|PROC| DRS_DATA_RUN: /data/spirou4/apero-data/offline290/run
06:14:45.802-|PROC| DRS_DATA_PLOT: /data/spirou4/apero-data/offline290/plot
06:14:45.802-|PROC| DRS_DATA_OTHER: /data/spirou4/apero-data/offline290/other
06:14:45.802-|PROC| LBL_PATH: /data/spirou4/apero-data/offline290/lbl
06:14:45.802-|PROC| DRS_CONFIG: /apero/config/offline290/user_config.ini
06:14:45.803-|PROC| DRS_CONFIG: /apero/config/offline290/user_constants.ini
06:14:45.803-|PROC| DRS_CONFIG: apero.constants.default.default_config
06:14:45.803-|PROC| DRS_CONFIG: apero.constants.default.default_keywords
06:14:45.803-|PROC| DRS_CONFIG: config.instruments.spirou.default_config.py
06:14:45.803-|PROC| DRS_CONFIG: config.instruments.spirou.default_keywords.py
06:14:45.803-|PROC| DRS_CONFIG: core.instruments.default.default_constants.py
06:14:45.803-|PROC| DRS_CONFIG: core.instruments.spirou.default_constants.py
06:14:45.804-|PROC| DATABASE: MYSQL
06:14:45.804-|PROC| DATABASE-calib: aperooffline290@localhost:calib_offline290_db
06:14:45.804-|PROC| DATABASE-tellu: aperooffline290@localhost:tellu_offline290_db
06:14:45.804-|PROC| DATABASE-findex: aperooffline290@localhost:findex_offline290_db
06:14:45.804-|PROC| DATABASE-log: aperooffline290@localhost:log_offline290_db
06:14:45.804-|PROC| DATABASE-astrom: aperooffline290@localhost:astrom_offline290_db
06:14:45.805-|PROC| DATABASE-lang: aperooffline290@localhost:lang_offline290_db
06:14:45.805-|PROC| DATABASE-reject: aperooffline290@localhost:reject_offline290_db
06:14:45.805-|PROC| PRINT_LEVEL: all
06:14:45.805-|PROC| LOG_LEVEL: all
06:14:45.805-|PROC| DRS_PLOT: 0

06:14:45.811- |PROC| ***************************************************************************
06:14:45.817-|PROC| Arguments used:
06:14:45.823-|PROC|
06:14:45.823-|PROC| --RUNFILE: LBL_run_LBLCOMPILE_FP_only.ini
06:14:45.829- |PROC| ***************************************************************************
06:14:47.709-|PROC| Updating reject database (from google sheets)
06:14:51.636- |PROC| Updating index database with new raw files
06:15:43.047- |PROC| Searching all directories
06:17:17.022- |PROC| Updating index database with new tmp files
06:17:28.146- |PROC| Searching all directories
06:17:31.978- |PROC| Updating index database with new red files
06:21:17.821- |PROC| Searching all directories
06:25:44.275- |PROC| Updating index database with new out files
06:26:10.681- |PROC| Searching all directories
06:26:42.052- |PROC| Updating database with header fixes
06:26:43.803- |PROC| Generating skip table
06:27:10.731- |PROC| Found 723683 previous recipe runs
06:27:10.820-|PROC| Generating run list
06:27:10.991- |PROC| Calculating which nights have no object files
06:27:14.032-@$|PROC| W[10-503-00014]: Remove Engineering Nights: Directory 18AQ07-Apr29,
06:27:14.032-@$|PROC| 18AQ07-Apr30, 19AQ02-Feb12, 19AQ02-Feb14, 19AQ02-Feb18, 19AQ02-Feb19,
06:27:14.032-@$|PROC| 19AQ02-Feb20, 19AQ02-Feb21, 19AQ02-Feb23, 19AQ08-Apr12, 19AQ08-Apr13,
06:27:14.032-@$|PROC| 19AQ08-Apr28, 19AQ08-Apr29, 19BQ05-Sep20, 19BQ05-Sep22, 19BQ07-Oct09,
06:27:14.032-@$|PROC| 19BQ07-Oct10, 19BQ10-Oct29, 19BQ14-Dec05, 19BQ14-Dec16, 19BQ14-Dec17,
06:27:14.032-@$|PROC| 19BQ14-Dec18, 19BQ14-Dec19, 20AQ02-Feb06, 20AQ02-Feb07, 20AQ02-Feb09,
06:27:14.032-@$|PROC| 20AQ02-Feb10, 20AQ02-Feb11, 20AQ02-Feb12, 20AQ02-Feb13, 20AQ06-Mar12,
06:27:14.032-@$|PROC| 20AQ06-Mar15, 20AQ06-Mar16, 20AQ06-Mar17, 20AQ06-Mar18, 20AQ08-May01,
06:27:14.032-@$|PROC| 20AQ08-May03, 20AQ08-May04, 20AQ08-May05, 20AQ15-Jul25, 20BQ01-Aug04,
06:27:14.032-@$|PROC| 20BQ06-Oct01, 20BQ06-Sep23, 20BQ09-Oct26, 20BQ09-Oct27, 20BQ09-Oct28,
06:27:14.032-@$|PROC| 21AQ05-Feb28, 21AQ08-Mar24, 21AQ08-Mar28, 21AQ17-Jul22, 21BQ06-Sep15,
06:27:14.032-@$|PROC| 21BQ14-Nov13, 21BQ14-Nov14, 21BQ17-Dec07, 21BQ17-Dec19, 21BQ17-Dec21,
06:27:14.032-@$|PROC| 21BQ20-Jan06, 22AQ09-Apr16, 22AQ12-May18, 22BQ11-Nov03, 22BQ13-Dec05,
06:27:14.032-@$|PROC| 22BQ13-Dec07, 22BQ13-Dec08, 22BQ13-Dec12, 22BQ13-Nov29, 23AQ01-Feb03,
06:27:14.032-@$|PROC| 23AQ01-Feb06, 23AQ09-Apr02, 23AQ09-Apr03, 23AQ09-Apr04, 23AQ20-Jul18,
06:27:14.032-@$|PROC| 23BQ06-Sep29, 23BQ12-Nov28, 23BQ12-Nov29, 23BQ14-Dec20, 23BQ14-Dec21,
06:27:14.032-@$|PROC| 23BQ16-Jan25, 23BQ16-Jan26, 24AQ03-Mar03, 24AQ03-Mar04, 24AQ05-Mar25,
06:27:14.032-@$|PROC| 24AQ05-Mar27, 24BQ03-Aug18 was removed - no OBJECT type found. Set
06:27:14.032-@$|PROC| "ENGINEERING=True" to include this directory.
06:27:14.568- |PROC| Adding condition for rejected odometer codes
06:27:22.582- |PROC| Loading recipe module files...
06:27:22.906-|PROC| Processing sequence 'LBL_SEQ'
06:27:22.947- |PROC| Adding recipe LBLREF to sequence
06:27:22.990- |PROC| Adding recipe LBLMASK_FP to sequence
06:27:23.088- |PROC| Adding recipe LBLCOMPUTE_FP to sequence
06:27:23.185- |PROC| Adding recipe LBLCOMPILE_FP to sequence
06:27:23.278- |PROC| Adding recipe LBLMASK_SCI to sequence
06:27:24.980- |PROC| Adding recipe LBLCOMPUTE_SCI to sequence
06:27:25.087- |PROC| Adding recipe LBLCOMPILE_SCI to sequence
06:28:05.441-|PROC| processing recipes for 116389 raw data entries
06:28:05.462- |PROC| skipping recipe 'apero_lbl_ref_spirou.py' (RUN_LBLREF=False)
06:28:05.482- |PROC| skipping recipe 'apero_lbl_mask_spirou.py' (RUN_LBLMASK_FP=False)
06:28:05.493- |PROC| skipping recipe 'apero_lbl_compute_spirou.py' (RUN_LBLCOMPUTE_FP=False)
06:28:05.503- |PROC| processing recipe 'apero_lbl_compile_spirou.py' (LBLCOMPILE_FP)
06:30:47.681- |PROC| Added 1 runs
06:30:47.706- |PROC| skipping recipe 'apero_lbl_mask_spirou.py' (RUN_LBLMASK_SCI=False)
06:30:47.726- |PROC| skipping recipe 'apero_lbl_compute_spirou.py' (RUN_LBLCOMPUTE_SCI=False)
06:30:47.737- |PROC| skipping recipe 'apero_lbl_compile_spirou.py' (RUN_LBLCOMPILE_SCI=False)
06:30:48.178-|PROC| Validating all runs (1 run ids found)
06:30:48.199- |PROC| ***************************************************************************
06:30:48.219- |PROC|
06:30:48.219- |PROC| Validating run ID00000 (1 of 1)
06:30:48.236- |PROC| ***************************************************************************
06:30:48.246- |PROC| apero_lbl_compile_spirou.py FP
06:30:48.259- |PROC| run ID00000 validated
06:30:48.270-|PROC| Analyzed 1 runs. Validated 1 runs. Skipped 0 runs.
06:30:48.286-|PROC| Running with 15 cores
06:30:48.357-|PROC|
06:30:48.370-|PROC| ***************************************************************************
06:30:48.381-|PROC| * GROUP 1/1 (LBLCOMPILE_FP)
06:30:48.391-|PROC| ***************************************************************************
06:30:48.401-|PROC|
06:30:48.455-|PROC| ID00000|C01/15| apero_lbl_compile_spirou.py FP --crunfile=LBL_run_LBLCOMPILE_FP_only.ini --program=LBLCOMPILE_FP[00000] --recipe_kind=lbl-compile-fp --shortname=LBLCOMPILE_FP --parallel=True
06:30:48.798- |LBLCOMPILE| ***************************************************************************
06:30:48.804- |LBLCOMPILE| *
06:30:48.805- |LBLCOMPILE| * SPIROU @PID-00017346762484742660-YMSJ (V0.7.291)
06:30:48.805- |LBLCOMPILE| * py3.9.16 git:v0.7.291-stable-test
06:30:48.806- |LBLCOMPILE| *
06:30:48.812- |LBLCOMPILE| * Group: APEROG-PID-00017346752849674160-F9TO_apero_processing_group
06:30:48.817- |LBLCOMPILE| ***************************************************************************
06:30:48.823- |LBLCOMPILE| █████╗ ██████╗ ███████╗██████╗ ██████╗
06:30:48.829- |LBLCOMPILE| ██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔═══██╗
06:30:48.835- |LBLCOMPILE| ███████║██████╔╝█████╗ ██████╔╝██║ ██║
06:30:48.841- |LBLCOMPILE| ██╔══██║██╔═══╝ ██╔══╝ ██╔══██╗██║ ██║
06:30:48.847- |LBLCOMPILE| ██║ ██║██║ ███████╗██║ ██║╚██████╔╝
06:30:48.853- |LBLCOMPILE| ╚═╝ ╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝
06:30:48.859- |LBLCOMPILE| ***************************************************************************
06:30:48.884-|LBLCOMPILE_FP[00000]| DRS Setup:
06:30:48.890-|LBLCOMPILE_FP[00000]|
06:30:48.890-|LBLCOMPILE_FP[00000]| DRS_ROOT: /apero/apero-drs-v0.7.290/apero
06:30:48.890-|LBLCOMPILE_FP[00000]| DRS_DATA_RAW: /data/spirou4/arlinks
06:30:48.891-|LBLCOMPILE_FP[00000]| DRS_DATA_REDUC: /data/spirou4/apero-data/offline290/red
06:30:48.891-|LBLCOMPILE_FP[00000]| DRS_DATA_WORKING: /data/spirou4/apero-data/offline290/tmp
06:30:48.891-|LBLCOMPILE_FP[00000]| DRS_CALIB_DB: /data/spirou4/apero-data/offline290/calib
06:30:48.891-|LBLCOMPILE_FP[00000]| DRS_TELLU_DB: /data/spirou4/apero-data/offline290/tellu
06:30:48.891-|LBLCOMPILE_FP[00000]| DRS_DATA_ASSETS: /data/spirou4/apero-data/offline290/asset
06:30:48.892-|LBLCOMPILE_FP[00000]| DRS_DATA_OUT: /data/spirou4/apero-data/offline290/out
06:30:48.892-|LBLCOMPILE_FP[00000]| DRS_DATA_MSG: /data/spirou4/apero-data/offline290/msg
06:30:48.892-|LBLCOMPILE_FP[00000]| DRS_DATA_RUN: /data/spirou4/apero-data/offline290/run
06:30:48.892-|LBLCOMPILE_FP[00000]| DRS_DATA_PLOT: /data/spirou4/apero-data/offline290/plot
06:30:48.892-|LBLCOMPILE_FP[00000]| DRS_DATA_OTHER: /data/spirou4/apero-data/offline290/other
06:30:48.893-|LBLCOMPILE_FP[00000]| LBL_PATH: /data/spirou4/apero-data/offline290/lbl
06:30:48.893-|LBLCOMPILE_FP[00000]| DRS_CONFIG: /apero/config/offline290/user_config.ini
06:30:48.893-|LBLCOMPILE_FP[00000]| DRS_CONFIG: /apero/config/offline290/user_constants.ini
06:30:48.893-|LBLCOMPILE_FP[00000]| DRS_CONFIG: apero.constants.default.default_config
06:30:48.893-|LBLCOMPILE_FP[00000]| DRS_CONFIG: apero.constants.default.default_keywords
06:30:48.894-|LBLCOMPILE_FP[00000]| DRS_CONFIG: config.instruments.spirou.default_config.py
06:30:48.894-|LBLCOMPILE_FP[00000]| DRS_CONFIG: config.instruments.spirou.default_keywords.py
06:30:48.894-|LBLCOMPILE_FP[00000]| DRS_CONFIG: core.instruments.default.default_constants.py
06:30:48.894-|LBLCOMPILE_FP[00000]| DRS_CONFIG: core.instruments.spirou.default_constants.py
06:30:48.894-|LBLCOMPILE_FP[00000]| DATABASE: MYSQL
06:30:48.895-|LBLCOMPILE_FP[00000]| DATABASE-calib: aperooffline290@localhost:calib_offline290_db
06:30:48.895-|LBLCOMPILE_FP[00000]| DATABASE-tellu: aperooffline290@localhost:tellu_offline290_db
06:30:48.895-|LBLCOMPILE_FP[00000]| DATABASE-findex: aperooffline290@localhost:findex_offline290_db
06:30:48.895-|LBLCOMPILE_FP[00000]| DATABASE-log: aperooffline290@localhost:log_offline290_db
06:30:48.895-|LBLCOMPILE_FP[00000]| DATABASE-astrom: aperooffline290@localhost:astrom_offline290_db
06:30:48.896-|LBLCOMPILE_FP[00000]| DATABASE-lang: aperooffline290@localhost:lang_offline290_db
06:30:48.896-|LBLCOMPILE_FP[00000]| DATABASE-reject: aperooffline290@localhost:reject_offline290_db
06:30:48.896-|LBLCOMPILE_FP[00000]| PRINT_LEVEL: all
06:30:48.896-|LBLCOMPILE_FP[00000]| LOG_LEVEL: all
06:30:48.896-|LBLCOMPILE_FP[00000]| DRS_PLOT: 0
06:30:48.903- |LBLCOMPILE_FP[00000]| ***************************************************************************
06:30:48.909-|LBLCOMPILE_FP[00000]| Arguments used:
06:30:48.915-|LBLCOMPILE_FP[00000]|
06:30:48.915-|LBLCOMPILE_FP[00000]| --OBJNAME: FP
06:30:48.916-|LBLCOMPILE_FP[00000]| --DEBUG: None
06:30:48.916-|LBLCOMPILE_FP[00000]| --PROGRAM: LBLCOMPILE_FP[00000]
06:30:48.916-|LBLCOMPILE_FP[00000]| --RECIPE_KIND: lbl-compile-fp
06:30:48.916-|LBLCOMPILE_FP[00000]| --PARALLEL: True
06:30:48.916-|LBLCOMPILE_FP[00000]| --SHORTNAME: LBLCOMPILE_FP
06:30:48.917-|LBLCOMPILE_FP[00000]| --IDEBUG: None
06:30:48.917-|LBLCOMPILE_FP[00000]| --REF: None
06:30:48.917-|LBLCOMPILE_FP[00000]| --CRUNFILE: LBL_run_LBLCOMPILE_FP_only.ini
06:30:48.917-|LBLCOMPILE_FP[00000]| --NOSAVE: False
06:30:48.917-|LBLCOMPILE_FP[00000]| --FORCE_INDIR: None
06:30:48.918-|LBLCOMPILE_FP[00000]| --FORCE_OUTDIR: None
06:30:48.924- |LBLCOMPILE_FP[00000]| ***************************************************************************
06:30:49.738-|LBLCOMPILE_FP[00000]| 7 DTEMP files found.
06:30:49.758-|LBLCOMPILE_FP[00000]| Running LBL compile for FP_FP
2024-12-19 20:30:49.766|G|LBLCOMPILE_FP[00000]| log directory exists (Path=/data/spirou4/apero-data/offline290/lbl/log)
2024-12-19 20:30:49.802|I|LBLCOMPILE_FP[00000]|
2024-12-19 20:30:49.803|I|LBLCOMPILE_FP[00000]| *******************************************************************************
2024-12-19 20:30:49.803|I|LBLCOMPILE_FP[00000]| LBL Compil
2024-12-19 20:30:49.803|I|LBLCOMPILE_FP[00000]| VERSION: 0.64.0
2024-12-19 20:30:49.804|I|LBLCOMPILE_FP[00000]| INSTRUMENT: SPIROU
2024-12-19 20:30:49.804|I|LBLCOMPILE_FP[00000]| *******************************************************************************
2024-12-19 20:30:49.804|I|LBLCOMPILE_FP[00000]|
2024-12-19 20:30:49.805|I|LBLCOMPILE_FP[00000]| User keyword arguments:
2024-12-19 20:30:49.805|I|LBLCOMPILE_FP[00000]| OBJECT_SCIENCE="FP"
2024-12-19 20:30:49.805|I|LBLCOMPILE_FP[00000]| OBJECT_TEMPLATE="FP"
2024-12-19 20:30:49.805|I|LBLCOMPILE_FP[00000]| DATA_TYPE="FP"
2024-12-19 20:30:49.806|I|LBLCOMPILE_FP[00000]| INSTRUMENT="SPIROU"
2024-12-19 20:30:49.806|I|LBLCOMPILE_FP[00000]| DATA_DIR="/data/spirou4/apero-data/offline290/lbl"
2024-12-19 20:30:49.806|I|LBLCOMPILE_FP[00000]| DATA_SOURCE="APERO"
2024-12-19 20:30:49.806|I|LBLCOMPILE_FP[00000]| SKIP_DONE="False"
2024-12-19 20:30:49.807|I|LBLCOMPILE_FP[00000]| PROGRAM="LBLCOMPILE_FP[00000]"
2024-12-19 20:30:49.807|I|LBLCOMPILE_FP[00000]| RESPROJ_TABLES="{'DTEMP3000': 'temperature_gradient_3000.fits', 'DTEMP3500': 'temperature_gradient_3500.fits', 'DTEMP4000': 'temperature_gradient_4000.fits', 'DTEMP4500': 'temperature_gradient_4500.fits', 'DTEMP5000': 'temperature_gradient_5000.fits', 'DTEMP5500': 'temperature_gradient_5500.fits', 'DTEMP6000': 'temperature_gradient_6000.fits'}"
2024-12-19 20:30:49.814|G|LBLCOMPILE_FP[00000]| Mask directory exists (Path=/data/spirou4/apero-data/offline290/lbl/masks)
2024-12-19 20:30:49.820|G|LBLCOMPILE_FP[00000]| Templates directory exists (Path=/data/spirou4/apero-data/offline290/lbl/templates)
2024-12-19 20:30:49.821|G|LBLCOMPILE_FP[00000]| Calib directory exists (Path=/data/spirou4/apero-data/offline290/lbl/calib)
2024-12-19 20:30:49.831|G|LBLCOMPILE_FP[00000]| Science directory exists (Path=/data/spirou4/apero-data/offline290/lbl/science)
2024-12-19 20:30:49.832|G|LBLCOMPILE_FP[00000]| LBL RV directory exists (Path=/data/spirou4/apero-data/offline290/lbl/lblrv/FP_FP)
2024-12-19 20:30:49.832|G|LBLCOMPILE_FP[00000]| LBL reftable directory exists (Path=/data/spirou4/apero-data/offline290/lbl/lblreftable)
2024-12-19 20:30:49.833|G|LBLCOMPILE_FP[00000]| LBL rdb directory exists (Path=/data/spirou4/apero-data/offline290/lbl/lblrdb)
2024-12-19 20:30:49.833|G|LBLCOMPILE_FP[00000]| Plot directory exists (Path=/data/spirou4/apero-data/offline290/lbl/plots)
2024-12-19 20:30:49.834|G|LBLCOMPILE_FP[00000]| Model directory exists (Path=/data/spirou4/apero-data/offline290/lbl/models)
2024-12-19 20:30:52.228|I|LBLCOMPILE_FP[00000]| Producing LBL RDB 1 table
97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 62010/63713 [6:24:36<23:33, 1.21it/s]
12:55:31.567- |PROC| Updating index database with new raw files
12:56:58.073- |PROC| Searching all directories
12:58:41.856- |PROC| Updating index database with new tmp files
12:58:57.957- |PROC| Searching all directories
12:59:04.438- |PROC| Updating index database with new red files
13:05:29.055- |PROC| Searching all directories
13:12:34.168- |PROC| Updating index database with new out files
13:13:08.647- |PROC| Searching all directories
13:13:33.617- |PROC|
13:13:33.637- |PROC| ***************************************************************************
13:13:33.656- |PROC| Timings:
13:13:33.676- |PROC| ***************************************************************************
13:13:33.696- |PROC|
13:13:33.707- |PROC| ***************************************************************************
13:13:33.717-|PROC| Cumulative time taken = 0.000
13:13:33.727-|PROC| Actual time taken = 24165.330
13:13:33.738-|PROC| Speed up: 0.000 (Number of cores = 15)
13:13:33.749- |PROC| ***************************************************************************
13:13:33.759- |PROC|
13:13:34.000-|PROC| ***************************************************************************
13:13:34.020-|PROC| Recipe apero_processing has been successfully completed (25129.028 seconds)
13:13:34.032-|PROC| ***************************************************************************

[larnoldgithub]

Here is how the RAM was used during the processing.

[larnoldgithub]

@njcuk9999 I got some explanation from Sidik about the plot above: "The pink line is "committed". "committed" means something called malloc() for a huge amount of memory (310Gb) but didn't touch all of it yet. It may never touch all of it - that's called "sparse" use, and Linux luckily allows it -- doesn't fail right away -- because it's a normal programming style these days for some types of software, usually ones that need large arrays which are populated by a lot of data with holes in it... but does it actually make sense that whatever it is doing should require more than 100 gigabytes of RAM? How high did "committed" go in relation to the green for similar past operations which succeeded? This might give us a ratio."

[njcuk9999]

So it make sense that it may go this high - but it shouldn't be the case. I think it is holding open all the FP files (or a good fraction of them) so it isn't just one array it will be many smaller ones... @eartigau and I need to look in to this!

[larnoldgithub]

For the record, the Linux version is Ubuntu 24.04.4 LTS