Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add two band sdl to global_4denvar namelist #758

Merged
merged 6 commits into from
Jun 21, 2024

Conversation

RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Jun 18, 2024

Description
GFS v17 runs gsi.x with scale-dependent localization (SDL). Update the global_4denvar ctest to include SDL.

Fixes #756

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Install feature/sdl on Dogwood and run global_4denvar ctest. Confirm correct namelist parameters are set to activate SDL.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Jun 18, 2024
@RussTreadon-NOAA RussTreadon-NOAA changed the title add two band sdl to global_4denvar namelist (#756) add two band sdl to global_4denvar namelist Jun 18, 2024
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jun 18, 2024

global_4denvar ctests have been run on Dogwood with acceptable results.

@RussTreadon-NOAA
Copy link
Contributor Author

@CatherineThomas-NOAA and @TravisElless-NOAA : This PR activates two-band SDL in the global_4denvar ctest. Do either of you have time to review the changes in this PR?

@CatherineThomas-NOAA
Copy link
Collaborator

@RussTreadon-NOAA: I can review these changes today.

@RussTreadon-NOAA
Copy link
Contributor Author

Thank you @CatherineThomas-NOAA

@TravisElless-NOAA
Copy link

Potential newbie question. Should regression_namelists_db.sh also be updated with these values given this if block?

@CatherineThomas-NOAA
Copy link
Collaborator

My ctests ran on Hera without issue. Looking at the log for the ctests, I see that the parameters are set appropriately:

 NSCLGRP =           2,
 NGVARLOC        =           1,
 NAENSLOC        =           3,
 R_ENSLOCCOV4TIM =   1.00000000000000     ,
 R_ENSLOCCOV4VAR =   1.00000000000000     ,
 R_ENSLOCCOV4SCL =   1.00000000000000     ,
 S_ENS_H =   1000.00000000000     ,   300.000000000000     ,   685.000000000000     , 17*2828.00000000000        ,
 S_ENS_V = 2*-0.500000000000000      ,  0.000000000000000E+000, 17*30.0000000000000        ,
 READIN_LOCALIZATION     = F,
 GLOBAL_SPECTRAL_FILTER_SD       = F,

The only snag is that due to the decreased resolution of the ctest, our chosen second scale is too small:

( pi*rearth*.001/jcap_in <= s_ens_hv <= 5500 )

Our second scale localization value of 300 km is reset automatically:

  s_ens_hv(           1 ) =    300.000000000000       km--too small, min value =
     416.994064886485       km.
  s_ens_hv(           1 ) reset to min value

Once it's reset, the scales are showing as expected:

 in init_sf_xy, jcap,s_ens_hv(    1 -  127), max diff(f0-f)=        48   1000.00    0.4608313731E-11
 in init_sf_xy, jcap,s_ens_hv(    1 -  127), max diff(f0-f)=        48    416.99    0.4824141087E-11
 in init_sf_xy, jcap,s_ens_hv(    1 -  127), max diff(f0-f)=        48    685.00    0.4678923915E-11

@RussTreadon-NOAA, @TravisElless-NOAA: Should we adjust our second scale so these warnings aren't set off? 450 km instead of 300?

@RussTreadon-NOAA
Copy link
Contributor Author

Good suggestion @TravisElless-NOAA: I will add SDL to the global_4denvar case in regression_namelists_db.sh

Good catch @CatherineThomas-NOAA: I will increase the 300 km scale to 450 km.

@RussTreadon-NOAA
Copy link
Contributor Author

Rerun global_4denvar on Dogwood with 450 km scale. No more warning messages

  reading time level            7
 in init_sf_xy, jcap,s_ens_hv(    1 -  127), max diff(f0-f)=        48   1000.00    0.4608313731E-11
 in init_sf_xy, jcap,s_ens_hv(    1 -  127), max diff(f0-f)=        48    450.00    0.4799716180E-11
 in init_sf_xy, jcap,s_ens_hv(    1 -  127), max diff(f0-f)=        48    685.00    0.4678923915E-11
GLBSOI: jiter,jiterstart,jiterlast,jiterend=    1    1    2    1

ctest still passes (it should!)

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/sdl/build
    Start 1: global_4denvar
1/1 Test #1: global_4denvar ...................   Passed  2344.44 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 2344.45 sec

Copy link
Collaborator

@CatherineThomas-NOAA CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Thanks @RussTreadon-NOAA!

@RussTreadon-NOAA
Copy link
Contributor Author

Update modulefiles/gsi_orion.intel.lua in order to compile feature/sdl on Orion and run ctest. Done at aab4a06.

Sorry @CatherineThomas-NOAA - this commit dismissed your approval.

@CatherineThomas-NOAA
Copy link
Collaborator

That's fine @RussTreadon-NOAA. I'll wait until the Orion tests complete before approving again.

@RussTreadon-NOAA
Copy link
Contributor Author

WARNING

Updating to Orion Rocky 9 is not as simple as updating a few files. The Orion and Hercules Rocky 9 crtm-fix/2.4.0.1_emc module do not contain the same coefficient files.

Orion crtm-fix/2.4.0.1_emc has

setenv("CRTM_FIX","/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-qls55kd/fix")

whereas Hercules crtm-fix/2.4.0.1_emc has

setenv("CRTM_FIX","/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-2os2hw2/fix")

A diff -r of these two CRTM_FIX directories finds 447 files differ. This is not correct.

hercules-login-3:~$ diff -r /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-qls55kd/fix /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-2os2hw2/fix |wc
    447    2680  159396

For example,

hercules-login-3:~$ ls -l /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-qls55kd/fix/abi_gr.TauCoeff.bin /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-2os2hw2/fix/abi_gr.TauCoeff.bin
-rw-r--r-- 1 role-epic epic  10972 Dec 20  2023 /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-2os2hw2/fix/abi_gr.TauCoeff.bin
-rw-r--r-- 1 role-epic epic 184588 Jun 14 16:06 /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/intel/2021.9.0/crtm-fix-2.4.0.1_emc-qls55kd/fix/abi_gr.TauCoeff.bin

Given this, revert commits 086e273 and aab4a06 from feature/sdl.

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jun 20, 2024

@CatherineThomas-NOAA : I abandoned the Orion ctest attempt. While updating the module is easy, the CRTM coefficients when loading the Orion crtm-fix/2.4.0.1_emc differ from those one gets when loading the Hercules crtm-fix/2.4.0.1_emc. This is not correct. Leave this to be sorted out in GSI issue #754

I am done making changes to this PR.

Copy link

@TravisElless-NOAA TravisElless-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me. Thanks @RussTreadon-NOAA!

@RussTreadon-NOAA RussTreadon-NOAA merged commit f82dc34 into NOAA-EMC:develop Jun 21, 2024
4 checks passed
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/sdl branch June 24, 2024 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Turn on SDL in global_4denvar ctest
3 participants