ELM Example Case III. Point Mode with Coupler Bypass option (CPL_BYPASS) 'gswp3'

ELM Examples of Setting-up, Configuring, Building, and Running III

NOTE:

• In these examples, there are 4 user-defined ENVs variables of directories: $E3SM_INPUTDATA, $E3SM_ROOT, $PROJECT_E3SM, for inputdata, e3sm model source codes, and root directory for cases and runs, respectively. USERS must have created these, and cloned both data and e3sm codes.

• $E3SM_INPUTDATA is cloned repository of a few site-level full datasets to run offline ELM, from pt-e3sm-inputdata

• Upon user's machine, $MACH_NAME is needed for E3SM. In these examples, 'mymac' for MacOS/linux, 'wsl' for Windows Subsystem for Linux, OR, 'cades' for a cluster in ORNL, may be directly used.

HERE we manually customize E3SM Land Model to run with option of 'CPL_BYPASS' (coupler_bypass), developed by Dr. Daniel Ricciuto at ORNL. This configuration will run offline ELM about 3 times faster and vey-slightly different results, although not bit-for-bit

++++++++++++++++++++++++++++++++++++++++++++

There are 4-5 forms of this configuration, upon what meteorological dataset to drive the land model. We will demonstrate 1 of them as following: (2) 'GSWP3' - point metdata extracted from half-degree global datasets , which is CPL_BYPASS version of ELM Example Case II. Point Mode with CPL_BYPASS, but with different forcing.

++++++++++++++++++++++++++++++++++++++++++++

CASE III: single-point user-defined satm-elm (CPL_BYPASS metdata_type of 'GSWP3).

• STEP 1. Create a PTCLM case, with --compset ICBxxxx

  1) cd $E3SM_ROOT/cime/scripts

  2) ./create_newcase --case $PROJECT_E3SM/cases/Test_elm-ICB1850CNPRDCTCBC --res ELM_USRDAT --mach $MACH_NAME --compiler gnu --compset ICB1850CNPRDCTCBC --walltime 06:00:00

• STEP 2. Configure the case created in STEP 1 (extracted 1 grid dataset from half-degree global one, namely '1x1pt_US-Brw' from 1901-2014)

  3) cd $PROJECT_E3SM/cases/Test_elm-ICB1850CNPRDCTCBC

  4) ./xmlchange, (OR, vi env_run.xml)
             --id ELM_USRDAT_NAME --val 1x1pt_beo-GRID

 ADDITIONALLY, 
 if 'accelerated spinup', you need to edit 'env_run.xml' as following:
             --id CLM_FORCE_COLDSTART --val on
             --id CLM_ACCELERATED_SPINUP --val on
             --id CLM_BLDNML_OPTS --val '-bgc bgc -nutrient cn -nutrient_comp_pathway rd  -soil_decomp ctc -methane -nitrif_denitrif' 
     (i.e. remove P limitation for ad_spinup)

By default, the model run on machine is upon some pre-configured PE setting. For an example, the case setup by now will run model on 1 node with 8 MPI Task. So that's not good for 1 PT model, so vi env_mach_pes.xml as following so that model runs on 1 node with 1 MPI Tasks

  6) vi env_mach_pes.xml

    <entry id="NTASKS">
      <type>integer</type>
      <values>
        <value compclass="ATM">1</value>
        <value compclass="CPL">1</value>
        <value compclass="OCN">1</value>
        <value compclass="WAV">1</value>
        <value compclass="GLC">1</value>
        <value compclass="ICE">1</value>
        <value compclass="ROF">1</value>
        <value compclass="LND">1</value>
        <value compclass="ESP">1</value>
      </values>
      <desc>number of tasks for each component</desc>
    </entry> 

    <entry id="NTASKS_PER_INST">
      <type>integer</type>
      <values>
        <value compclass="ATM">1</value>
        <value compclass="CPL">1</value>
        <value compclass="OCN">1</value>
        <value compclass="WAV">1</value>
        <value compclass="GLC">1</value>
        <value compclass="ICE">1</value>
        <value compclass="ROF">1</value>
        <value compclass="LND">1</value>
        <value compclass="ESP">1</value>
      </values>
      <desc>number of tasks for each component</desc>
    </entry> 

OR, 
./xmlchange NTASKS=1
./xmlchange NTASKS_PER_INST=1

IF PNETCDF is not available,
./xmlchange PIO_TYPENAME=netcdf

THEN, issue the following command:

  7) ./case.setup 
(./case.setup --clean FOR cleaning-up)

The above command may or may not be successful (mostly with errors relevant to missing input files). AND it's also missing of metdata and setting for CPL_BYPASS. So, modifying user_nl_elm as following:

 metdata_type = 'gswp3'
 metdata_bypass = '$E3SM_INPUTDATA/atm/datm7/1x1pt_US-Brw/cpl_bypass_GSWP3'
 fsurdat = '$E3SM_INPUTDATA/lnd/clm2/surfdata_map/surfdata_1x1pt_beo-GRID_simyr1850_c360x720_c171002.nc'
 flanduse_timeseries = '$E3SM_INPUTDATA/lnd/clm2/surfdata_map/landuse.timeseries_1x1pt_beo-GRID_simyr1850-2015_c180423.nc'
 aero_file = '$E3SM_INPUTDATA//atm/cam/chem/trop_mozart_aero/aero/aerosoldep_monthly_1850_mean_1.9x2.5_c090803.nc'
 CO2_file = '$E3SM_INPUTDATA//atm/datm7/CO2/fco2_datm_1765-2007_c100614.nc'
 nyears_ad_carbon_only = 25
 spinup_mortality_factor = 10

NOTE: (1)the last 2 options are optional, but which can accelerate spinup much better.

(2) Those options are to instruct model read datasets from indicated directory or files.

In this case, the 'metdata' is located in directory metdata_bypass with 7 files GSWP3_*_1901-2014_z14.nc, and an ascii file zone_mappings.txt. So, make sure those files are existed in those inputdata directories.

TWO (2) surface data, 'fsurdat=...', 'flanduse_timeseries=...', are used for spinups and transient runs. Make sure they are in the specified directories and file names are correct. IF NOT, edit as NEEDED.

THEN, re-issue command ./case.setup

• STEP 3. Build the case from STEP 1-2

FOR CPL_BYPASS, ONE more editing IS required before code compiling. Edit Macro.make and Macro.cmake under case directory as following (appending -DCPL_BYPASS into CPPDEFS)

Macro.make:

  CPPDEFS := $(CPPDEFS)  -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DCPL_BYPASS

Macro.cmake:

  set(CPPDEFS "${CPPDEFS}  -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DCPL_BYPASS")

OR, (for newer version ELM since 2021-Oct), add one line into cmake_macros/universal_cmake:

string(APPEND CPPDEFS " -DCPL_BYPASS")

THEN, build the model:

  8) ./case.build
(./case.build --clean      #cleaning-up ALL E3SM model components
 ./case.build --clean lnd  #cleaning-up LND component
 ./case.build --clean-all  #cleaning-up ALL model components and External components)

• STEP 4. Run the case

  ./case.submit

NOTES: *(a) the run direcory is:$PROJECT_E3SM/cases/Test_elm_ICB1850CNPRDCTCBC/run

*(b) The run length is 5 days, with monthly output, i.e. NO real data write out EXCEPT for restart files at the end.

*(c) If you'd like what-ever time-length of the simulation, editing env_run.xml, for an example, like the following:

  --id STOP_OPTION nyears
  --id STOP_N 200
 (These editing will let model run 200 years, with monthly outputs of default variables, AND will produce 2 restart files)

*(d) If simulation may not be finished in WALLCLOCK time, it will crash. Then have to editing env_run.xml to resume run:

   --id CONTINUE_RUN TRUE

NOTE: The unfinished run MUST have produced 2 restart files, .elm.r..nc, .cpl.r..nc. IF NOT, have to redo run from beginning, with smaller interval for producing restart files, like (retart file writing every 10 years):

  --id REST_N 10
  --id REST_OPTION nyears