Problem running HLS PGE with error: "Input directory /home/conda/input_dir does not contain any tif files"

[riverma]

Hi @collinss-jpl et al - I'm receiving the following error trying to execute the opera_pge-dswx_hls-1.0.0-er.1.0 docker image against sample data.

Error:

Running preprocessor for DSWxPreProcessorMixin
Traceback (most recent call last):
  File "/home/conda/opera/scripts/pge_main.py", line 184, in <module>
    pge_main()
  File "/home/conda/opera/scripts/pge_main.py", line 180, in pge_main
    pge_start(run_config_filename)
  File "/home/conda/opera/scripts/pge_main.py", line 154, in pge_start
    pge.run()
  File "/home/conda/opera/pge/base_pge.py", line 373, in run
    self.run_preprocessor(**kwargs)
  File "/home/conda/opera/pge/dswx_pge.py", line 92, in run_preprocessor
    self._validate_inputs()
  File "/home/conda/opera/pge/dswx_pge.py", line 70, in _validate_inputs
    self.logger.critical(self.name, ErrorCode.INPUT_NOT_FOUND, error_msg)
  File "/home/conda/opera/util/logger.py", line 410, in critical
    raise RuntimeError(description)
RuntimeError: Input directory /home/conda/input_dir does not contain any tif files

Here's my setup:

• Docker version: Docker version 20.10.12, build e91ed57 on Mac OSX

• Docker command:
  docker run --rm -u conda:conda -v ${WORKING_DIR}/run_config_dir:/home/conda/runconfig:ro -v ${WORKING_DIR}/test_datasets:/home/conda/input_dir:ro -v ${WORKING_DIR}/output_dir:/home/conda/output_dir -i --tty opera_pge/dswx_hls:1.0.0-er.1.0 --file /home/conda/runconfig/dswx_hls_sample_runconfig.yaml

• ${WORKING_DIR} structure and contents

.
├── opera_pge-dswx_hls-1.0.0-er.1.0.tar.gz
├── output_dir
│   ├── pge_20220203T004935.log
│   └── pge_20220203T005010.log
├── run_config_dir
│   └── dswx_hls_sample_runconfig.yaml
└── test_datasets
    ├── l30_greenland
    │   ├── README.txt
    │   ├── expected_output
    │   │   └── dswx_hls.tif
    │   ├── input_files_hls_v2.0
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B01.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B02.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B03.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B04.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B05.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B06.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B07.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B09.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B10.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.B11.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.Fmask.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.SAA.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.SZA.tif
    │   │   ├── HLS.L30.T22VEQ.2021248T143156.v2.0.VAA.tif
    │   │   └── HLS.L30.T22VEQ.2021248T143156.v2.0.VZA.tif
    │   └── runconfig_dswx_hls.yaml
    └── s30_louisiana
        ├── README.txt
        ├── expected_output
        │   └── dswx_hls.tif
        ├── input_files_hls_v2.0
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B01.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B02.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B03.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B04.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B05.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B06.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B07.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B08.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B09.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B10.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B11.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B12.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.B8A.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.Fmask.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.SAA.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.SZA.tif
        │   ├── HLS.S30.T15SXR.2021250T163901.v2.0.VAA.tif
        │   └── HLS.S30.T15SXR.2021250T163901.v2.0.VZA.tif
        └── runconfig_dswx_hls.yaml

[collinss-jpl]

@riverma This should be a quick fix, this PGE expects the HLS .tif files to be located directly within /home/conda/input_dir, so the corresponding Docker volume argument should be:

-v ${WORKING_DIR}/test_datasets/<dataset_name>/input_files_hls_v2.0:/home/conda/input_dir:ro

where <dataset_name> would be one of s30_louisiana or l30_greenland.

You are, of course, free to reorganize the directory structure of the test datasets on your local machine if you want to simplify the mount argument, but it is critical that, when mounted, the .tif files are available directly within /home/conda/input_dir.

Hope this helps!

[riverma]

This worked well - thanks for that tip @collinss-jpl

Here's my final command and output(s) for reference:

Docker command:

docker run --rm -u conda:conda -v ${WORKING_DIR}/run_config_dir:/home/conda/runconfig:ro -v ${WORKING_DIR}/test_datasets:/home/conda/input_dir/l30_greenland/input_files_hls_v2.0:ro -v ${WORKING_DIR}/output_dir:/home/conda/output_dir -i --tty opera_pge/dswx_hls:1.0.0-er.1.0 --file /home/conda/runconfig/dswx_hls_sample_runconfig.yaml

Standard output:

Running preprocessor for DSWxPreProcessorMixin
Starting SAS execution for DSWxExecutor
Running postprocessor for DSWxPostProcessorMixin

${WORKING_DIR}/output_dir contents:

output_dir/
├── dswx_hls.tif
├── pge_20220203T004935.log
├── pge_20220203T005010.log
├── pge_20220203T182536.log
└── pge_20220203T182624.log