updated notebooks, sample bmi configs and README

.gitignore

@@ -82,4 +82,6 @@ dist/
 
 # Pickle stuff #
 ################
-model1_state.pkl
+model1_state.pkl
+
+build/

README.md

@@ -15,7 +15,9 @@ This Long Short-Term Memory (LSTM) network was developed for use in the [Next Ge
 This module is dependent on a trained deep learning model. The forward pass of this LSTM model [`nextgen_cuda_lstm.py`](./lstm/nextgen_cuda_lstm.py) is heavily based on NeuralHydrology's [`CudaLSTM`](https://neuralhydrology.readthedocs.io/en/latest/usage/models.html#cudalstm). Other model classes can be applied but [`bmi_lstm.py`](./lstm/bmi_lstm.py) would need to load it in. More information about the python package NeuralHydrology can be found [here](https://neuralhydrology.readthedocs.io/en/latest/).  
 
 ## Sample Data
-All data required for a test run of this model is available in the [`data/`](./data) directory. This includes:
+
+### NLDAS sample data
+Sample data required for a test run of this model is available in the [`data/`](./data) directory. This includes:
 * Forcing data: `usgs-streamflow-nldas_hourly.nc`
 * Observation values: also included in `usgs-streamflow-nldas_hourly.nc`
 * Static attributes: see an example configuration file for a list of these attributes in [`./bmi_config_files`](./bmi_config_files/01022500_hourly_all_attributes_forcings.yml) 
@@ -28,6 +30,9 @@ for four USGS gauges:
 
 Note that the data found in this repository are simply examples. The LSTM model can be run on any watershed, provided the necessary static attributes and dynamic forcings. The full list of attributes differs depending on the trained LSTM model chosen. Example files (`*.yml`) with the required attributes are located in the [`./bmi_config_files`](./bmi_config_files)directory. The attributes required for these configuration files can be found in the [`camels_attributes_v2.0/`](./data/camels_attributes_v2.0) data directory for catchments in the CAMELS dataset or estimated from [Addor, N., A.J. Newman, N. Mizukami, and M.P. Clark. 2017. The CAMELS data set: catchment attributes and meteorology for large-sample studies. Hydrol. Earth Syst. Sci. 21: 5293-5313. https://doi.org/10.5194/hess-21-5293-2017](https://doi.org/10.5194/hess-21-5293-2017).  
 
+### AORC Sample Data
+To run a sample with AORC, you can clone this repository that has data from several camples basins: [https://github.com/NWC-CUAHSI-Summer-Institute/CAMELS_data_sample](https://github.com/NWC-CUAHSI-Summer-Institute/CAMELS_data_sample). You'll need to change the paths in the sample AORC notebook.
+
 ## Configurations
 The LSTM model requires a configuration file for specification of forcings, weights, scalers, run options (like warmup period), runtime period, static basin parameters and model time step. This configuration file needs to be generated for any specific application of the LSTM model.
 

bmi_config_files/01013500_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 2252.7
+basin_id: 1013500
+basin_name: Fish River near Fort Kent, Maine
+elev_mean: 250.31
+initial_state: zero
+lat: 47.23739
+lon: -68.58264
+slope_mean: 21.64152
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/01022500_nh_AORC_hourly_25yr_1210_112435.yml → bmi_config_files/01022500_nh_NLDAS_hourly.yml

@@ -5,7 +5,7 @@ basin_id: '01022500'
 area_sqkm: 620.38
 lat: 44.60797
 lon: -67.93524
-train_cfg_file: ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435/config.yml
+train_cfg_file: ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
 verbose: 0
 elev_mean: 92.68
 slope_mean: 17.79072

bmi_config_files/01333000_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 112.16
+basin_id: 1333000
+basin_name: GREEN RIVER AT WILLIAMSTOWN, MA
+elev_mean: 485.91
+initial_state: zero
+lat: 42.70897
+lon: -73.19677
+slope_mean: 74.78725
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/01547700_nh_NLDAS_hourly.yml

@@ -0,0 +1,11 @@
+area_sqkm: 142.18
+basin_id: 12010000
+basin_name: NASELLE RIVER NEAR NASELLE, WA
+elev_mean: 145.18
+initial_state: zero
+lat: 46.37399
+lon: -123.74348
+slope_mean: 47.2961
+time_step: 1 hour
+train_cfg_file: ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/02046000_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 288.52
+basin_id: 2046000
+basin_name: STONY CREEK NEAR DINWIDDIE, VA
+elev_mean: 86.64
+initial_state: zero
+lat: 37.06709
+lon: -77.60249
+slope_mean: 6.04481
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/02064000_nh_NLDAS_hourly.yml

@@ -0,0 +1,11 @@
+area_sqkm: 142.18
+basin_id: 12010000
+basin_name: NASELLE RIVER NEAR NASELLE, WA
+elev_mean: 145.18
+initial_state: zero
+lat: 46.37399
+lon: -123.74348
+slope_mean: 47.2961
+time_step: 1 hour
+train_cfg_file: ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/03010655_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 254.44
+basin_id: 3010655
+basin_name: Oswayo Creek at Shinglehouse, PA
+elev_mean: 635.06
+initial_state: zero
+lat: 41.96173
+lon: -78.19807
+slope_mean: 32.95227
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/03015500_nh_NLDAS_hourly.yml

@@ -0,0 +1,11 @@
+area_sqkm: 142.18
+basin_id: 12010000
+basin_name: NASELLE RIVER NEAR NASELLE, WA
+elev_mean: 145.18
+initial_state: zero
+lat: 46.37399
+lon: -123.74348
+slope_mean: 47.2961
+time_step: 1 hour
+train_cfg_file: ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/03439000_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 178.67
+basin_id: 3439000
+basin_name: FRENCH BROAD RIVER AT ROSMAN, NC
+elev_mean: 863.18
+initial_state: zero
+lat: 35.14333
+lon: -82.82472
+slope_mean: 63.23118
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/04015330_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 224.35
+basin_id: 4015330
+basin_name: KNIFE RIVER NEAR TWO HARBORS, MN
+elev_mean: 337.69
+initial_state: zero
+lat: 46.94688
+lon: -91.7924
+slope_mean: 19.16862
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/05057200_nh_AORC_hourly_ensemble.yml

@@ -1,17 +1,17 @@
-time_step: '1 hour'
-initial_state: 'zero'
-basin_name: 'BALDHILL CREEK NR DAZEY, ND'
-basin_id: '05057200'
-area_sqkm: 1900
+area_sqkm: 1897.33
+basin_id: 5057200
+basin_name: BALDHILL CREEK NR DAZEY, ND
+elev_mean: 447.5
+initial_state: zero
 lat: 47.22916
 lon: -98.12482
-train_cfg_file: 
+slope_mean: 2.33897
+time_step: 1 hour
+train_cfg_file:
 - ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
 - ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
 - ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
 - ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
 - ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
 - ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
 verbose: 0
-elev_mean: 447.5
-slope_mean: 2.33897

bmi_config_files/05291000_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 1046.78
+basin_id: 5291000
+basin_name: WHETSTONE RIVER NEAR BIG STONE CITY, SD
+elev_mean: 432.13
+initial_state: zero
+lat: 45.29163
+lon: -96.48756
+slope_mean: 8.90129
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/06221400_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 227.88
+basin_id: 6221400
+basin_name: DINWOODY CREEK ABOVE LAKES, NEAR BURRIS, WYO.
+elev_mean: 3336.8
+initial_state: zero
+lat: 43.34551
+lon: -109.41014
+slope_mean: 127.16435
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/07057500_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 1456.44
+basin_id: 7057500
+basin_name: North Fork River near Tecumseh, MO
+elev_mean: 324.68
+initial_state: zero
+lat: 36.62303
+lon: -92.24813
+slope_mean: 11.24276
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/07291000_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 479.3
+basin_id: 7291000
+basin_name: HOMOCHITTO RIVER AT EDDICETON, MS
+elev_mean: 129.13
+initial_state: zero
+lat: 31.50306
+lon: -90.7775
+slope_mean: 7.60783
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/08023080_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 187.61
+basin_id: 8023080
+basin_name: Bayou Grand Cane near Stanley, LA
+elev_mean: 86.85
+initial_state: zero
+lat: 31.97933
+lon: -93.93408
+slope_mean: 4.78985
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/08267500_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 96.26
+basin_id: 8267500
+basin_name: RIO HONDO NEAR VALDEZ, NM
+elev_mean: 3006.6
+initial_state: zero
+lat: 36.54169
+lon: -105.5564
+slope_mean: 149.08403
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/09035900_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 72.84
+basin_id: 9035900
+basin_name: SOUTH FORK OF WILLIAMS FORK NEAR LEAL, CO.
+elev_mean: 3240.75
+initial_state: zero
+lat: 39.79582
+lon: -106.03057
+slope_mean: 123.91195
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0

bmi_config_files/09386900_nh_AORC_hourly_ensemble.yml

@@ -0,0 +1,17 @@
+area_sqkm: 184.94
+basin_id: 9386900
+basin_name: RIO NUTRIA NEAR RAMAH, NM
+elev_mean: 2341.65
+initial_state: zero
+lat: 35.28253
+lon: -108.55341
+slope_mean: 37.39246
+time_step: 1 hour
+train_cfg_file:
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_7/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_8/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_1210_112435_9/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed101_0701_143442/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_25yr_seq999_seed103_2701_171540/config.yml
+- ../trained_neuralhydrology_models/nh_AORC_hourly_slope_elev_precip_temp_seq999_seed101_2801_191806/config.yml
+verbose: 0