Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: inappropriate arguments for map_batches in ray mode #533

Open
HYLcool opened this issue Jan 8, 2025 · 0 comments
Open

[BUG]: inappropriate arguments for map_batches in ray mode #533

HYLcool opened this issue Jan 8, 2025 · 0 comments
Labels
bug Something isn't working dj:dist issues/PRs about distributed data processing

Comments

@HYLcool
Copy link
Collaborator

HYLcool commented Jan 8, 2025

For now, running Data-Juicer on multiple nodes in "ray" mode, which uses map_batches to process datasets, might cause some implicit problems.

The map_batches method has two arguments, num_gpus and concurrency, which are actually cluster-level arguments. However, they are calculated automatically according to the hardware information of a single machine. So, there might be some resource utilization problems when running on multiple nodes for OPs with _accelerator is "cuda".

@HYLcool HYLcool added bug Something isn't working dj:dist issues/PRs about distributed data processing labels Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dj:dist issues/PRs about distributed data processing
Projects
None yet
Development

No branches or pull requests

1 participant