Fixes #14: Refactoring ray portable runner #18
Conversation
|
r: @iasoon ? : ) |
|
I realize that CI is broken. Should we fix it as part of this PR? or as a follow-up PR? |
pabloem
force-pushed
the
refactor-rayportablerunner
branch
3 times, most recently
from
21f68b3 to
a5dff00
Compare
pabloem
force-pushed
the
refactor-rayportablerunner
branch
from
a5dff00 to
c6a36ca
Compare
|
I made all the portability runner tests run : ) |
pabloem
changed the title
| return self.transform_to_buffer_coder, self.data_output, self.stage_timers | ||
|
|
||
| def setup(self): |
There was a problem hiding this comment.
Is there a reason why this method is only called after initialization of the ContextManager?
| (transform_id, id) | ||
| ] = timer_family_spec.timer_family_coder_id | ||
|
|
||
| def __reduce__(self): |
There was a problem hiding this comment.
Nit / general remark: It's a bit unfortunate that we keep running into the serialization issue, and sometimes solve it by using a custom __reduce__, sometimes by registering a custom serializer (ray.util.register_serializer), and sometimes manually (SerializeToString / FromString).
It would be good if we could enable serialization for all the protobuf components in one central place - I'm not sure how that could be done though, as ray.util.register_serializer would have to be called on every ray worker that transmits protobuf objects. Maybe something to discuss?
|
TODO: Rename immutable context managers to |
|
Sounds good to me :) |
Making the code more readable and more efficient.
Also adding support for SDF-initiated checkpointing