Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reading npz files takes soooooo long #108

Closed
ExpHP opened this issue Jan 28, 2020 · 1 comment
Closed

reading npz files takes soooooo long #108

ExpHP opened this issue Jan 28, 2020 · 1 comment

Comments

@ExpHP
Copy link
Owner

ExpHP commented Jan 28, 2020

Reading an .npz file is done through a child Python process, and the interprocess communication for this appears to be incredibly time-consuming.

Writing files is fine; nearly instant. Not sure why it's so asymmetric.

I've long wanted to substitute this out for rust code that reads npz files, but unfortunately my PRs to npy-rs have fallen into limbo. (and those are just the beginning of the work necessary to read and write npz files) Some day when I have time I'm gonna finish and publish my fork of that crate.

@ExpHP
Copy link
Owner Author

ExpHP commented Jul 6, 2021

I just published https://github.com/ExpHP/npyz version 0.6 which ought to have everything we need to read the files natively in rust

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant