-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add dask_awkward wrapper to Correction and CompoundCorrection #219
Conversation
also make min awkward/dask_awkward versions more easily configurable
@nsmith- please review when you have time, thanks! |
and just to be sure you're fine with the Mostly just an issue of thread safety, but I don't imagine people using correctionlib in python threads (as opposed to processes) that much. |
I'm much more scared of attempting to persist the |
Yeah what I've implemented here was more or less what Martin suggested so far as dask usage patterns are concerned. No need to persist if you wrap it in the delayed object. It'll be handled by any scheduler that conforms to the spec. This is also what's being done over in coffea for corrections and ml models after his suggestion. |
Also add awkward wrapper to CompoundCorrection.
This PR lets us pass dask_awkward.Array into correctionlib corrections.
It does the wrapping of the correction into a delayed object and map_partitions call internally now.
Is significantly cleaner than the
map_partitions
version.