Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce RAM update parquet by beam. #33

Open
h-a-graham opened this issue Sep 23, 2024 · 0 comments
Open

reduce RAM update parquet by beam. #33

h-a-graham opened this issue Sep 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@h-a-graham
Copy link
Contributor

At the moment, the memory usage is quite high when we have to load the entire granule (i.e every beam in the granule) into memory. Instead, we could consider saving each beam to the parquet file with update - that way memory usage should be 1/8th what it is now. Not essential immediately but I can see this being useful for working with the direct access. Imagine being able to use very cheap compute with minimal memory to transfer the hdf files to parquet in another s3 bucket - could be a very low cost way to enhance GEDI access...

@h-a-graham h-a-graham added the enhancement New feature or request label Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant