Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with loading very large number of images (e.g., 55000) #33

Open
zhuowenzhao opened this issue Aug 16, 2022 · 0 comments
Open

Issues with loading very large number of images (e.g., 55000) #33

zhuowenzhao opened this issue Aug 16, 2022 · 0 comments

Comments

@zhuowenzhao
Copy link
Member

Labelmaker has latency in displaying the images and data clinic says run out of memory due to loading npz files all at once.
The latency could be due to (1) the sizes of the images are a little higher (1679, 1475) - we may need to resize them to (~512x512) or something like that, (2) I wonder if the image_order vector + maybe the labels in cache could be causing some latency too - since the list is really large.

This might be resolved by storing metadata in splashML or hdf5 files instead of in the memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant