Having issue with updated GFS DDTs data structures: PR 2183

[uturuncoglu]

I had a code that brings CDEPS inline capability to FV3 to ingest Great Lake surface properties (SST, sea ice etc.) and update those information through the CCPP. This is a done for a JTTI project. This was working fine but once I sync the development branch with the head of develop, I have some issues. I crackdown the issue to a specific PR - In fv3atm: convert GFS DDTs from blocked data structures to contiguous arrays (#2183) - hash: fcf0022

I was using following code to ingest data coming from CDEPS inline to CCPP,

       case ('So_omask')
!$omp parallel do default(shared) private(i,j,nb,ix)
          do j = jsc, jec
             do i = isc, iec
                nb = atm_block%blkno(i,j)
                ix = atm_block%ixp(i,j)
                GFS_data(nb)%Coupling%mask_dat(ix) = datar82d(i-isc+1,j-jsc+1)
             end do
          end do
       case ('So_t')
!$omp parallel do default(shared) private(i,j,nb,ix)
          do j = jsc, jec
             do i = isc, iec
                nb = atm_block%blkno(i,j)
                ix = atm_block%ixp(i,j)
                GFS_data(nb)%Coupling%tsfco_dat(ix) = datar82d(i-isc+1,j-jsc+1)
             end do
          end do

Then, after PR I had to change it to following,

case ('So_omask')
!$omp parallel do default(shared) private(i,j,nb,ix,im)
          do j = jsc, jec
             do i = isc, iec
                nb = atm_block%blkno(i,j)
                ix = atm_block%ixp(i,j)
                im = GFS_control%chunk_begin(nb)+ix-1
                GFS_Coupling%mask_dat(im) = data2d(i-isc+1,j-jsc+1)
             end do
          end do
       case ('So_t')
!$omp parallel do default(shared) private(i,j,nb,ix,im)
          do j = jsc, jec
             do i = isc, iec
                nb = atm_block%blkno(i,j)
                ix = atm_block%ixp(i,j)
                im = GFS_control%chunk_begin(nb)+ix-1
                GFS_Coupling%tsfco_dat(im) = data2d(i-isc+1,j-jsc+1)
             end do
          end do

but now the result field is corrupted (I am checking by activating aux debug variables in CCPP side and write those to FV3 history file). For example for the working configuration, I could see the mask variable correctly on the output like following,

but the updated code produces following,

So, I tried different things but could not find any solution. Any suggestion will help at this point.

[uturuncoglu]

JFYI, @climbfuji

[climbfuji]

A couple of quick questions:

1. Are you using the same blocks for the CDEPS coupling with CCPP than fv3atm does?
2. Why are you using datar82d in the old blocked data structures call, and data2d (no r8) in the new ones?
3. Did you also make the metadata updates and any necessary ccpp-framework updates (if the framework is a submodule under cdeps, I don't remember)?

[uturuncoglu]

@climbfuji Thanks for your help. Here are my comments,

1. The implementation works like following. CDEPS inline API call accepts host grid (in this case forecast grid) to do the interpolation from actual data grid to destination one. In this process, I am converting FV3 Cubed-sphere grid to ESMF Mesh to pass it to CDEPS. Then I am redistributing the result field (on mesh) to FV3 Cubed-sphere grid (again this is forecast grid which is already available through the FV3 cap and I did not do anything). So, I think that part is working fine since the old version of code works without any issue and since I am redistributing to FV3 forecast grid, it would be fine. It is very similar that we have current fields found in import and export states.
2. I think it is just a cosmetic change but I could try to revery back. The CDEPS always returns in R8, I maybe I change it to be compatible in terms of types. It is hard to remember.
3. I did not do anything in terms of adding CCPP to CDEPS. CDEPS just provides the data and the data connected to CCPP using the same way with other coupling fields. I have new scheme file FV3/ccpp/physics/physics/SFC_Models/Data/sfc_data.F to get the data (of course I am also adding it to the suite file). That is all. I am using same CCPP used by other coupling fields and but updating extra fields.

[climbfuji]

Hmm. When we merged the update to the contiguous arrays, all regression tests were b4b identical. Just saying this because it means that anything that was exercised in rt.sh worked as expected. Maybe this can narrow down the search for the culprit.

[uturuncoglu]

Yes, it is good to know. It seems that The new code is selecting the blocks in a correct way but somehow it blends the data (maybe using lat as Lon etc. not sure) and not map correctly inside of that block. It is very strange. Probably, tiny detail that I could not catch.

[uturuncoglu]

I wonder if where those blocks are defined? CCPP or FV3 cap? I am not sure I need something special when I creating field from the forecast grid but I could not find anything like that in the NUOPC cap. Maybe it is something related with CCPP.

[uturuncoglu]

BTW, this is HRRR configuration. I have only one tile.

[gspetro-NOAA]

@uturuncoglu I know this discussion continued offline and there was a PR in progress. Has it been resolved, or should this discussion remain open?

[uturuncoglu]

@uturuncoglu This is is not resolved yet but I rollback the code that I developed for University of Michigan as a part of the JTTI project by using the particular version of UFS just before the changes in GFS DDTs. It is working now. I am not sure what was changed in GFS DDT to affect out implementation at this point. Since I am not founded by that project and just helped them, I did not go further and spent time to solve it. So, if you want, you could close this issue and if we decide to tackle it and need help I could open this again. Anyway, let me know what you think.