Using EFSOI with GDAS
last edited June 9, 2022
To use EFSOI, you need to clone the forked global-workflow repository, then checkout the EFSOI branch, then run the usual sequence of scripts. This branch of global-workflow will clone a fork of GSI and checkout the EFSOI development branch. The forked global-workflow repository is merged with the March 10, 2022 hash of the global-workflow develop branch; the forked GSI repository is merged with the May 9, 2022 GSI develop branch. Note that this post-dates the GSI hash a62dec6 of Dec 14, 2021 that the official global-workflow uses. At the time of this writing EFSOI works on Orion and WCOSS.
To set up the current global-workflow repository with the latest EFSOI development:
git clone --recursive https://github.com/AndrewEichmann-NOAA/global-workflow.git
cd global-workflow/
git checkout feature/EFSOI
and then checkout, build, and link global-workflow as usual.
Run ush/rocoto/setup_expt.py as usual for a cycling experiment. For testing 20-member ensembles suffices, and 80-member are used for experiments. GFS need not be run run separately.
In config.base in your expdir, set
export DO_EFSOI="YES"
Then run ush/rocoto/setup_workflow.py from the EFSOI build of global-workflow. This will set up the workflow with the extra EFSOI tasks. The experiment can be started as usual.
During the first complete cycle, the gdaseupdfsoi task - the ensemble update with settings specific to EFSOI - will run with the same priority as gdaseupd, leading to a parallel set of EFSOI-specific gdas tasks ending with post-processing. In the first complete cycle, the first 30-hour ensemble forecast (metatask gdasefmnfsoi) is generated, and the gdasefsoi task will never run. During the second complete cycle, the forecast will be made again for 30 hours, and post-processed to generate 24-hour and 30-hour ensemble means. The gdasefsoi task in this cycle will sit idle until the cycle 24 hours subsequent is active and creates the verifying analysis. The gdasefsoi task from the second complete cycle then runs, using the 24-hour forecast from that cycle and the 30-hour forecast from the previous cycle, creating the final observation sensitivity - osense - file. This is placed in the osense directory in COMROT. The process is repeated for the following cycles for the length of the experiment.
One result of this process is that the EFSOI-specific data, stored in efsoigdas directories with a structure similar to that of enkfgdas directories, has to be kept on disk for a longer time than the other data files, and can eat up space. Likewise the osense files are several hundred MB for each cycle. These osense files can be analyzed with scripts in sorc/gsi.fd/util/EFSOI_Utilities/scripts.
Ensemble Forecast Sensitivity to Observation Impacts is based on a method developed in Langland and Baker (2004) that uses a model adjoint and the Kalman gain to determine the positive or negative impact of individual assimilated observations on the error of a forecast relative to a verifying analysis. The state vector plot below illustrates the concept.
Plot by Rahul Mahajan
The forecast background Xb and analysis Xa of a given cycle are both used to initialize forecasts, Xaf and Xbf. These forecasts are then compared to a verifying analysis Xt to obtain the respective errors of the two forecasts. The difference in the errors at each observation point are traced back to their respective assimilated observations using the following equation:
Kalnay et al. (2012) developed a method to use ensemble forecasts and observation error covariance in lieu of an adjoint and Kalman gain:
Kalnay, E., Ota., Y., Miyoshi, T. and Liu, J. 2012. A simpler formulation of forecast sensitivity to observations: application to ensemble Kalman filters. Tellus, 64A, 18462
Ota., Y., Derber., J., Kalnay., E. and Miyoshi., T., 2013, Ensemble-Based Observation Impact Estimates Using the NCEP GFS. Tellus, 65A, 20038
In more concrete terms within GDAS and global-workflow, the variables in the EFSOI equation are represented as follows:
where the green terms are stored in the initial "osense file" generated during the EFSOI-specific ensemble update task (gdaseupdfsoi) for a given cycle t0, and the values used for the forecast perturbation (the red term) are in 24-hour ensemble member forecasts initialized with the analysis at t0, also generated by gdaseupdfsoi. The forecast errors (the blue terms) are calculated using the 24-hour forecast ensemble mean, the 30-hour forecast ensemble mean from the cycle t0-6hr (which is functionally the same as a 24-hour forecast initialized with the background at t0), and the verifying analysis from t0+24hr. Both the 24-hour and 30-hour ensemble forecasts are run at the same time for t0 with the metatask gdasefmnfsoi, the 24-hour forecast to for the EFSOI calculation for cycle t and the 30-hour forecast for cycle t+6hr, and the ensemble means generated with gdasepmnfsoi. Note that the 24-hour forecasts used are specific to the global model; regional models may use shorter forecasts for the same purpose.
