Skip to content

Commit

Permalink
pylibcugraph: declare cupy and numpy hard dependencies (#4854)
Browse files Browse the repository at this point in the history
While testing stuff for #4804, I found that `pylibcugraph` has a hard runtime dependency on `cupy` and `numpy`, but isn't declaring them
 
```shell
docker run \
    --rm \
    --gpus 0 \
    -it rapidsai/ci-wheel:latest \
    bash

python -m pip install 'pylibcugraph-cu12==25.2.*,>=0.0.0a0'
python -c "import pylibcugraph"
```

```text
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/pyenv/versions/3.12.7/lib/python3.12/site-packages/pylibcugraph/__init__.py", line 18, in <module>
    from pylibcugraph.graphs import SGGraph, MGGraph
  File "graphs.pyx", line 1, in init pylibcugraph.graphs
  File "utils.pyx", line 20, in init pylibcugraph.utils
ModuleNotFoundError: No module named 'cupy'
```

This declares those dependencies.

It also promotes `cugraph-service-server`'s `numpy` dependency to a hard runtime dependency.

## Notes for Reviewers

### Evidence that `pylibcugraph` *already* has a hard dependency on these libraries

They're used unconditionally here:

https://github.com/rapidsai/cugraph/blob/cddd69ea3f62cabdb3aa2b7b6676e0b74ab4eefc/python/pylibcugraph/pylibcugraph/utils.pyx#L19-L20

But have import guards in other places:

https://github.com/rapidsai/cugraph/blob/cddd69ea3f62cabdb3aa2b7b6676e0b74ab4eefc/python/pylibcugraph/pylibcugraph/sssp.pyx#L127-L139

So this PR doesn't introduce new hard dependencies... it just makes them explicit, to make it easier to install and run `pylibcugraph`.

### How was this not caught in CI?

Import tests aren't run here for conda packages, because conda builds happen on CPU-only nodes.

https://github.com/rapidsai/cugraph/blob/cddd69ea3f62cabdb3aa2b7b6676e0b74ab4eefc/ci/build_python.sh#L27-L30

And `numpy` and `cupy` are probably getting pulled in by some of the wheels' test dependencies, like `cudf`, here:

https://github.com/rapidsai/cugraph/blob/cddd69ea3f62cabdb3aa2b7b6676e0b74ab4eefc/ci/test_wheel.sh#L17

### Should we just make the other unconditional cases conditional with try-catching?

No. Talked with @rlratzel, @ChuckHastings, and @eriknw offline, and we agreed to declare these as hard runtime dependencies (and remove the try-catching in places that had it).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Rick Ratzel (https://github.com/rlratzel)
  - Bradley Dice (https://github.com/bdice)

URL: #4854
  • Loading branch information
jameslamb authored Jan 13, 2025
1 parent ed954dc commit dd228f9
Show file tree
Hide file tree
Showing 9 changed files with 36 additions and 66 deletions.
4 changes: 3 additions & 1 deletion conda/recipes/pylibcugraph/meta.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2023-2024, NVIDIA CORPORATION.
# Copyright (c) 2023-2025, NVIDIA CORPORATION.

{% set version = environ['RAPIDS_PACKAGE_VERSION'].lstrip('v') + environ.get('VERSION_SUFFIX', '') %}
{% set minor_version = version.split('.')[0] + '.' + version.split('.')[1] %}
Expand Down Expand Up @@ -74,7 +74,9 @@ requirements:
{% else %}
- cuda-cudart
{% endif %}
- cupy >=12.0.0
- libcugraph ={{ version }}
- numpy>=1.23,<3.0a0
- pylibraft ={{ minor_version }}
- python
- rmm ={{ minor_version }}
Expand Down
14 changes: 13 additions & 1 deletion dependencies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,10 @@ files:
table: project
includes:
- cuda_wheels
- depends_on_cupy
- depends_on_pylibraft
- depends_on_rmm
- python_run_pylibcugraph
py_test_pylibcugraph:
output: pyproject
pyproject_dir: python/pylibcugraph
Expand Down Expand Up @@ -199,7 +201,7 @@ files:
key: test
includes:
- test_python_common
- test_python_cugraph
- test_python_cugraph_service_server
channels:
- rapidsai
- rapidsai-nightly
Expand Down Expand Up @@ -421,6 +423,11 @@ dependencies:
- matrix: # All CUDA 11 versions
packages:
- cuda-python>=11.8.5,<12.0a0
python_run_pylibcugraph:
common:
- output_types: [conda, pyproject, requirements]
packages:
- *numpy
python_run_cugraph_service_client:
common:
- output_types: [conda, pyproject]
Expand Down Expand Up @@ -468,6 +475,11 @@ dependencies:
- output_types: [conda]
packages:
- *thrift
test_python_cugraph_service_server:
common:
- output_types: [conda, pyproject]
packages:
- *numpy
test_python_pylibcugraph:
common:
- output_types: [conda, pyproject]
Expand Down
4 changes: 0 additions & 4 deletions python/cugraph-service/server/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,12 @@ cugraph-service-server = "cugraph_service_server.__main__:main"

[project.optional-dependencies]
test = [
"certifi",
"networkx>=2.5.1",
"numpy>=1.23,<3.0a0",
"pandas",
"pytest",
"pytest-benchmark",
"pytest-cov",
"pytest-xdist",
"python-louvain",
"scikit-learn>=0.23.1",
"scipy",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../../dependencies.yaml and run `rapids-dependency-file-generator`.

Expand Down
9 changes: 3 additions & 6 deletions python/pylibcugraph/pylibcugraph/bfs.pyx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2022-2024, NVIDIA CORPORATION.
# Copyright (c) 2022-2025, NVIDIA CORPORATION.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
Expand All @@ -14,6 +14,8 @@
# Have cython use python 3 syntax
# cython: language_level = 3

import cupy

from libc.stdint cimport uintptr_t
from libc.stdint cimport int32_t
from libc.limits cimport INT_MAX
Expand Down Expand Up @@ -141,11 +143,6 @@ def bfs(ResourceHandle handle, _GPUGraph graph,
>>> })
"""

try:
import cupy
except ModuleNotFoundError:
raise RuntimeError("bfs requires the cupy package, which could not "
"be imported")
assert_CAI_type(sources, "sources")

if depth_limit <= 0:
Expand Down
12 changes: 3 additions & 9 deletions python/pylibcugraph/pylibcugraph/node2vec.pyx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2022-2024, NVIDIA CORPORATION.
# Copyright (c) 2022-2025, NVIDIA CORPORATION.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
Expand All @@ -14,6 +14,8 @@
# Have cython use python 3 syntax
# cython: language_level = 3

import cupy

from libc.stdint cimport uintptr_t

from pylibcugraph._cugraph_c.types cimport (
Expand Down Expand Up @@ -124,14 +126,6 @@ def node2vec(ResourceHandle resource_handle,
"""

# FIXME: import these modules here for now until a better pattern can be
# used for optional imports (perhaps 'import_optional()' from cugraph), or
# these are made hard dependencies.
try:
import cupy
except ModuleNotFoundError:
raise RuntimeError("node2vec requires the cupy package, which could not "
"be imported")
assert_CAI_type(seed_array, "seed_array")

cdef cugraph_resource_handle_t* c_resource_handle_ptr = \
Expand Down
19 changes: 4 additions & 15 deletions python/pylibcugraph/pylibcugraph/pagerank.pyx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2022-2024, NVIDIA CORPORATION.
# Copyright (c) 2022-2025, NVIDIA CORPORATION.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
Expand All @@ -14,6 +14,9 @@
# Have cython use python 3 syntax
# cython: language_level = 3

import cupy
import numpy

from pylibcugraph._cugraph_c.types cimport (
bool_t,
)
Expand Down Expand Up @@ -167,20 +170,6 @@ def pagerank(ResourceHandle resource_handle,
array([0.11615585, 0.21488841, 0.2988108 , 0.3701449 ], dtype=float32)
"""

# FIXME: import these modules here for now until a better pattern can be
# used for optional imports (perhaps 'import_optional()' from cugraph), or
# these are made hard dependencies.
try:
import cupy
except ModuleNotFoundError:
raise RuntimeError("pagerank requires the cupy package, which could "
"not be imported")
try:
import numpy
except ModuleNotFoundError:
raise RuntimeError("pagerank requires the numpy package, which could "
"not be imported")

cdef cugraph_type_erased_device_array_view_t* \
initial_guess_vertices_view_ptr = \
create_cugraph_type_erased_device_array_view_from_py_obj(
Expand Down
19 changes: 4 additions & 15 deletions python/pylibcugraph/pylibcugraph/personalized_pagerank.pyx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2022-2024, NVIDIA CORPORATION.
# Copyright (c) 2022-2025, NVIDIA CORPORATION.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
Expand All @@ -14,6 +14,9 @@
# Have cython use python 3 syntax
# cython: language_level = 3

import cupy
import numpy

from pylibcugraph._cugraph_c.types cimport (
bool_t,
)
Expand Down Expand Up @@ -177,20 +180,6 @@ def personalized_pagerank(ResourceHandle resource_handle,
array([0.00446455, 0.00379487, 0.53607565, 0.45566472 ], dtype=float32)
"""

# FIXME: import these modules here for now until a better pattern can be
# used for optional imports (perhaps 'import_optional()' from cugraph), or
# these are made hard dependencies.
try:
import cupy
except ModuleNotFoundError:
raise RuntimeError("pagerank requires the cupy package, which could "
"not be imported")
try:
import numpy
except ModuleNotFoundError:
raise RuntimeError("pagerank requires the numpy package, which could "
"not be imported")

cdef cugraph_type_erased_device_array_view_t* \
initial_guess_vertices_view_ptr = \
create_cugraph_type_erased_device_array_view_from_py_obj(
Expand Down
19 changes: 4 additions & 15 deletions python/pylibcugraph/pylibcugraph/sssp.pyx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2022-2024, NVIDIA CORPORATION.
# Copyright (c) 2022-2025, NVIDIA CORPORATION.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
Expand All @@ -14,6 +14,9 @@
# Have cython use python 3 syntax
# cython: language_level = 3

import cupy
import numpy

from pylibcugraph._cugraph_c.types cimport (
bool_t,
)
Expand Down Expand Up @@ -124,20 +127,6 @@ def sssp(ResourceHandle resource_handle,
array([-1, -1, 1, 2], dtype=int32)
"""

# FIXME: import these modules here for now until a better pattern can be
# used for optional imports (perhaps 'import_optional()' from cugraph), or
# these are made hard dependencies.
try:
import cupy
except ModuleNotFoundError:
raise RuntimeError("sssp requires the cupy package, which could not "
"be imported")
try:
import numpy
except ModuleNotFoundError:
raise RuntimeError("sssp requires the numpy package, which could not "
"be imported")

if compute_predecessors is False:
raise ValueError("compute_predecessors must be True for the current "
"release.")
Expand Down
2 changes: 2 additions & 0 deletions python/pylibcugraph/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ authors = [
license = { text = "Apache 2.0" }
requires-python = ">=3.10"
dependencies = [
"cupy-cuda11x>=12.0.0",
"numpy>=1.23,<3.0a0",
"nvidia-cublas",
"nvidia-curand",
"nvidia-cusolver",
Expand Down

0 comments on commit dd228f9

Please sign in to comment.