Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Support disk directive with local executor #5636

Open
schorlton-bugseq opened this issue Jan 3, 2025 · 2 comments · May be fixed by #5652
Open

Feature Request: Support disk directive with local executor #5636

schorlton-bugseq opened this issue Jan 3, 2025 · 2 comments · May be fixed by #5652

Comments

@schorlton-bugseq
Copy link

New feature

Thanks for all of your work on nextflow! As best I can tell from the docs, the disk directive is not implemented with the local executor. Is this accurate? I very naively thought local would have been the first executor for it to be implemented only to realize later that it wasn't having any impact on my workflow (as far as I can tell). This directive would likely be very useful to the community for processing big data.

Without a full appreciation for how the directive is implemented on other platforms, I would think we could call something similar to https://unix.stackexchange.com/a/6014 in .command.run and error if there is not sufficient disk space? Thanks as always for your consideration!

@bentsherman
Copy link
Member

I think we could use df to get the available disk space at the start of a run. The local executor already does a similar thing with memory. But local doesn't actually limit the memory usage of each task, it just keeps track of how much memory it has allocated against the total and blocks pending tasks accordingly. This seems like a reasonable heuristic to use for disk space

@bentsherman
Copy link
Member

Even better, File::getUsableSpace() can give us a disk space estimate in a Java-native way. Some basic tests suggest it is very close to df.

@bentsherman bentsherman linked a pull request Jan 7, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants