Avoid limited memory adaptor issue in balanced KMeans #2570
Conversation
…to fix-sparse-utilities
|
@csadorf we will also want to make sure we port this over to cuVS, since the kmeans in raft will be getting ported shortly after GTC. |
| rmm::mr::managed_memory_resource managed_memory; | ||
| rmm::device_async_resource_ref device_memory = resource::get_workspace_resource(handle); | ||
| rmm::device_async_resource_ref current_device_resource = rmm::mr::get_current_device_resource(); | ||
| rmm::device_async_resource_ref workspace_resource = |
There was a problem hiding this comment.
This is one of those rare cases where we do indeed need to explicitly allocate rmm::mr::managed_memory_resource (the removed TODO comment is actually incorrect).
The need to use managed memory here has nothing to do with the memory limit and the user choice, but rather is a part of the algorithm. We use the managed_memory variable across this file for not-so-big allocations that are accessed by both device and host (see, for example, the build_fine_clusters function above). Hence, using the device-only memory simply breaks the algorithm here.
There was a problem hiding this comment.
Adressed.
However, I'm wondering whether we should be generally using make_managed_vector instead then. @achirkin Was there a specific motivation for the use of rmm::uvectors instead?
There was a problem hiding this comment.
No, only historic reasons: the balanced kmeans code arrived earlier than these managed helpers.
There was a problem hiding this comment.
Thanks for clarifying.
@cjnolet Considering that this code has moved to cuVS anyways I assume there is no point in refactoring this, is there?
There was a problem hiding this comment.
No reason to refactor, just need to fix any issues that is blocking cuML UMAP ATM
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
csadorf
changed the title
|
Keeping this in draft mode until #2541 is merged. |
Prepared in rapidsai/cuvs#659 . |
Thanks so much @csadorf! |
get_large_workspace_resourceinstead ofget_workspace_resourceBased on and merge after #2541 (diff)