Ctest failure on Jet

[wx20jjung]

In running the ctests on Jet, the rrfs tests fail for both the develop and update. Here is the failure line from one of them:

forrtl: severe (24): end-of-file during read, unit 24, file /mnt/lfs5/HFIP/hfv3gfs/Jim.Jung/ptmp/tmpreg_rrfs_3denvar_rdasens/rrfs_3denvar_rdasens_loproc_u
pdat/coupler.res

[RussTreadon-NOAA]

Tagging @ShunLiu-NOAA and @hu5970 . Have we confirmed that the rrfs ctest works on Jet?

[ShunLiu-NOAA]

Ming will test it on JET once he returns his leave.

[RussTreadon-NOAA]

@wx20jjung , has this issue been resolved or does it still persist on Jet?

[wx20jjung]

I tried the ctest again on jet. Same file missing error forrtl: severe (24): end-of-file during read, unit 24, file /mnt/lfs4/NESDIS/nesdis-rdo1/Jim.Jung/ptmp/tmpreg_rrfs_3denvar_rdasens/rrfs_3denvar_rdasens_loproc_updat/coupler.res

…

On Mon, Aug 5, 2024 at 5:54 AM RussTreadon-NOAA ***@***.***> wrote: @wx20jjung <https://github.com/wx20jjung> , has this issue been resolved or does it still persist on Jet? — Reply to this email directly, view it on GitHub <#768 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMPASA7SMLL2EWF5S4HQODDZP5DUNAVCNFSM6AAAAABKQUZY2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRYGY3DQMJSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RussTreadon-NOAA]

@hu5970 , can we fix the rrfs test on Jet?

[RussTreadon-NOAA]

While working on PR #787 find that the problem Jim reports still exists on Jet

Attempts to run rrfs_3denvar_rdasens fail in job rrfs_3denvar_rdasens_loproc_updat with the error message

forrtl: severe (24): end-of-file during read, unit 24, file /mnt/lfs5/HFIP/hfv3gfs/Russ.Treadon/ptmp/tmpreg_rrfs_3denvar_rdasens/rrfs_3denvar_rdasens_loproc_updat/coupler.res
Image              PC                Routine            Line        Source
gsi.x              0000000005A04A08  Unknown               Unknown  Unknown
gsi.x              0000000005A3BB8A  Unknown               Unknown  Unknown
gsi.x              0000000000A7DEAA  gsi_rfv3io_mod_mp         323  gsi_rfv3io_mod.f90
gsi.x              00000000006CF2A3  convert_fv3_regio          37  fv3_regional_interface.f90
gsi.x              00000000004233E3  gsimod_mp_gsimain        2349  gsimod.F90
gsi.x              0000000000414C35  MAIN__                    620  gsimain.f90

The working copy of coupler.res in the run directory is zero length. Script rrfs_3denvar_rdasens.sh copies this file into the working directory as follows

# update times in coupler.res to current cycle time
cp ${bkpath}/fv3_coupler.res          coupler.res
sed -i "s/yyyy/${YYYY}/" coupler.res
sed -i "s/mm/${MM}/"     coupler.res
sed -i "s/dd/${DD}/"     coupler.res
sed -i "s/hh/${HH}/"     coupler.res

A check of rrfs_3denvar_rdasens_loproc_updat.out shows

bkpath=/lfs5/NESDIS/nesdis-rdo2/David.Huber/save/CASES/regtest/regional/rrfs/2023061012/ges

This directory does not exist

/lfs5/HFIP/hfv3gfs/Russ.Treadon/git/gsi/tmp/develop/build$ ls /lfs5/NESDIS/nesdis-rdo2/David.Huber/save/CASES/regtest/regional/rrfs/2023061012
ls: cannot access '/lfs5/NESDIS/nesdis-rdo2/David.Huber/save/CASES/regtest/regional/rrfs/2023061012': No such file or directory
/lfs5/HFIP/hfv3gfs/Russ.Treadon/git/gsi/tmp/develop/build$ ls -l /lfs5/NESDIS/nesdis-rdo2/David.Huber/save/CASES/regtest/regional/rrfs/
total 4
drwxr-sr-x 5 David.Huber nesdis-rdo2 4096 Jan  5  2023 2021072518

Tagging @DavidHuber-NOAA and @InnocentSouopgui-NOAA for awareness.

[DavidHuber-NOAA]

Thanks for looking into this, @RussTreadon-NOAA. I'll perform an rsync from Hera and get this data updated.

[DavidHuber-NOAA]

The regression test data has been updated on Jet. I will create a nightly rsync check against Hera to prevent this from happening again.

[RussTreadon-NOAA]

rrfs_3denvar_rdasens now runs on Jet as reported in PR #787.

Close this issue.