global_nest_v1 suite: infinite cumulative surface upwelling long-wave flux

[SamuelTrahanNOAA]

Description

I have a failing test case of C768 with the "FV3_global_nest_v1" suite with infinite values in the accumulated surface upwelling long-wave flux. This comes from an execution of the global workflow. Test case location on Hera is detailed below.

The variable's name depends on the context:

• GFS_typedefs.meta: cumulative_surface_upwelling_longwave_flux_multiplied_by_timestep
• GFS_typedefs.F90: ulwsfc
• GFS_diagnostics.F90: ULWRF
• GFS diag_table: ulwrf_ave

Infinities happen from forecast hour 3 onward. Upwelling and shortwave fluxes don't have this problem, nor do the instantaneous values at the output times.

From skimming the code, the only possibility I can immediately see is this: the instantaneous values are infinite for at least one timestep, but are finite at output times. I can't see why the instantaneous values would go infinite.

To Reproduce:

Test case is here on HERA

NOAA Hera: /scratch2/BMC/wrfruc/Samuel.Trahan/westwater/sijie-inf/cleaned
To run:

1. Copy that directory.
2. Clone your favorite version of the ufs-weather-model
3. Compile with FV3_global_nest_v1 suite enabled. I used ./compile.sh hera "-DAPP=ATM -D32BIT=ON -DCCPP_SUITES=FV3_global_nest_v1" optim intel YES NO
4. Edit job.sh. Point it to your source code, executable, and CPU account.
5. Submit job.sh.
6. Output files should have infinities in ulwrf_ave, as described above.

Additional context

This failing case comes from the the Water in the West experiments. We're encountering many problems while running the ENKF at C768 resolution.
It requires the maximum possible job size on Hera (210 nodes), mostly due to the ever-expanding memory usage of the write component.

[NickSzapiro-NOAA]

Do you know where in the domain this happens? If near sea ice margins, there are some questions on what to do when sea ice forms or vanishes

[SamuelTrahanNOAA]

Some happen in lakes, but most are in oceans. This is February 19, so the North Hemisphere lakes might be frozen.

In the plot below, red dots are infinite and blue are not. Coastlines are in faint white.

[NickSzapiro-NOAA]

Thanks! I misunderstood that this was coupled.

May I ask if it is an option to test updated surface fix files?

There is some possibly related issues posted in #1954. I don't know if anyone has followed up on the reproducibility issues for gnv1_nested on WCOSS2 Brian mentioned in #2005

[SamuelTrahanNOAA]

May I ask if it is an option to test updated surface fix files?

Certainly, I could test new fix files. Can you tell me which files I should try replacing, and with what?

[SamuelTrahanNOAA]

If you're referring to TISFC and TIICE, as noted in #1954 (comment), it'll have to be regenerated. We're using a seven-tile configuration with custom orography data. Hence, I need to know the fix or workaround @GeorgeGayno-NOAA used so we can regenerate the orography with the fix.

I could test the fix by using a workaround in FV3. Overriding the troubled values in FV3/io/fv3atm_restart_io.F90 would mimic correcting the orography data.

[SamuelTrahanNOAA]

Clarification to prior comment. When I say "custom orography data," I mean the output is seven tiles, rather than six. Input is the same as generating similar-resolution six-tile orography data.

If I could figure out Ning Wang's github username, I'd ping him here. He wrote the script and generated this data.

[GeorgeGayno-NOAA]

Clarification to prior comment. When I say "custom orography data," I mean the output is seven tiles, rather than six. Input is the same as generating similar-resolution six-tile orography data.

If I could figure out Ning Wang's github username, I'd ping him here. He wrote the script and generated this data.

Tagging @NingWang325

[SamuelTrahanNOAA]

We have two orographies, and I'm not sure which was used for this test:

1. One I generated using an older version of UFS_UTILS which probably doesn't have the fix in Odd behavior in the ice field #1954
2. Ning's orography, generated by a similar method, but using the latest UFS_UTILS as of a few months ago. It probably does have that fix.

[SamuelTrahanNOAA]

I tried Ning's orography:

/scratch1/BMC/gsd-fv3-dev/NCEPDEV/stmp3/Ning.Wang/nest3/C384

In my test case here:

/scratch2/BMC/wrfruc/Samuel.Trahan/westwater/sijie-inf/ningfix

But I get the same problem.

[SamuelTrahanNOAA]

I've also tried:

1. Turning of fractional ice (frac_ice=.false.)
2. Adding the missing &surf_map namelist

Neither of those fixed the problem.

EDIT: The fractional grid is already turned off. Fractional ice is a separate feature which is specific to certain physics schemes. I don't think it's in use in the global_nest_v1 suite.