Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scalar values from trained neuralhydrology model can be updated to avoid pickle. #51

Open
jmframe opened this issue Jan 23, 2025 · 0 comments

Comments

@jmframe
Copy link
Contributor

jmframe commented Jan 23, 2025

Neural Hydrology has updated the way they save the scalars necessary to process input and output data. Training any new LSTM models with neural hydrology would require reading in scalar values from yaml format, rather than pickle files. This is good, because it means that the BMI LSTM is no longer dependent on pickle, at all. Screenshot below shows the changes needed to read in scalars from yaml format. Also pasted below is updated code to read in scalars from yaml. Finally, this change can also be found in this PR from my branch:

jmframe#11

` scaler_file = os.path.join(self.cfg_train['run_dir'], 'train_data', 'train_data_scaler.yml')

    with open(scaler_file, 'r') as f:
        scaler_data = yaml.safe_load(f)

    self.train_data_scaler = scaler_data

    self.attribute_means = scaler_data.get('attribute_means', {})
    self.attribute_stds = scaler_data.get('attribute_stds', {})
    self.feature_scale = {k: v['data'] for k, v in scaler_data['xarray_feature_scale']['data_vars'].items()}
    self.feature_center = {k: v['data'] for k, v in scaler_data['xarray_feature_center']['data_vars'].items()}
    if self.verbose > 1:
        print(self.feature_center)
        print(self.feature_scale)
        print(self.attribute_means)
        print(self.attribute_stds)

#------------------------------------------------------------`

"""Mean and standard deviation for the inputs and LSTM outputs""" self.out_mean = self.train_data_scaler['xarray_feature_center']['data_vars'][self.cfg_train['target_variables'][0]]['data'] self.out_std = self.train_data_scaler['xarray_feature_scale']['data_vars'][self.cfg_train['target_variables'][0]]['data'] self.input_mean.extend([self.train_data_scaler['xarray_feature_center']['data_vars'][x]['data'] for x in self.cfg_train['dynamic_inputs']]) self.input_std.extend([self.train_data_scaler['xarray_feature_scale']['data_vars'][x]['data'] for x in self.cfg_train['dynamic_inputs']]) self.input_std.extend([self.train_data_scaler['attribute_stds'][x] for x in self.cfg_train['static_attributes']]) self.input_std = np.array(self.input_std)

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant