Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize arrays for speed savings #258

Merged
merged 1 commit into from
Aug 26, 2024
Merged

Initialize arrays for speed savings #258

merged 1 commit into from
Aug 26, 2024

Conversation

dnowacki-usgs
Copy link
Member

This approach improves run speed by 50% for a ~10 GB Vector file.

CC @rbales-usgs

This approach improves run speed by 50% for a ~10 GB Vector file.
@dnowacki-usgs
Copy link
Member Author

On my machine, the run speed went from MIA23TR202d-raw.cdf 527.98s user 273.30s system 94% cpu 14:07.49 total to MIA23TR202d-raw.cdf 319.50s user 38.97s system 88% cpu 6:44.44 total. As far as I can tell, the outputs are identical.

@dnowacki-usgs dnowacki-usgs merged commit a9f2d94 into master Aug 26, 2024
13 checks passed
@dnowacki-usgs dnowacki-usgs deleted the reshape branch August 26, 2024 19:59
@rbales-usgs
Copy link
Contributor

That's a great addition! Thanks for working on that.

PR #257 I submitted last week was primarily submitted to get the SNR and COR thresholds in. Along with using trim_maxabs_diff on burst shaped data. The removal of the Z dims and changing float64 variables to float32 was mostly triggered by the time it takes to write out a continuous vector .nc file. It doesn't take as long to write out the Burst vector nc files. But, if you rather those changes not happen I can edit that branch so that it only includes the addition of the QAQC features.

I will say, dropping the Z dim from the variables makes the dataset a lot more user friendly (faster calculations and don't need to use squeeze when plotting). But if you think it is important to keep them associated with the variables then we can. One thing Steve suggested was make the Z dim a variable attribute instead of a dimension for a variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants