Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removing Python from build to get CI passing #31

Merged
merged 9 commits into from
Feb 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,40 @@
# <div align="left"><img src="https://rapids.ai/assets/images/rapids_logo.png" width="90px"/>&nbsp;cuVS: Vector Search on the GPU</div>
# <div align="left"><img src="https://rapids.ai/assets/images/rapids_logo.png" width="90px"/>&nbsp;cuVS: Vector Search and Clustering on the GPU</div>

### NOTE: cuVS is currently being

## Contents
<hr>

1. [Useful Resources](#useful-resources)
2. [What is cuVS?](#what-is-cuvs)
3. [Getting Started](#getting-started)
4. [Installing cuVS](#installing)
5. [Contributing](#contributing)
6. [References](#references)

<hr>

## Useful Resources

- [cuVS Reference Documentation](https://docs.rapids.ai/api/cuvs/stable/): API Documentation.
- [cuVS Getting Started](./docs/source/quick_start.md): Getting started with RAFT.
- [Build and Install cuVS](./docs/source/build.md): Instructions for installing and building cuVS.
- [Example Notebooks](./notebooks): Example jupyer notebooks
- [RAPIDS Community](https://rapids.ai/community.html): Get help, contribute, and collaborate.
- [GitHub repository](https://github.com/rapidsai/cuvs): Download the cuVS source code.
- [Issue tracker](https://github.com/rapidsai/cuvs/issues): Report issues or request features.

## What is cuVS?

cuVS contains many algorithms for running approximate nearest neighbors and clustering on the GPU.

## Getting Started



## Installing cuVS

## Contributing

## References

10 changes: 5 additions & 5 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ ARGS=$*
# scripts, and that this script resides in the repo dir!
REPODIR=$(cd $(dirname $0); pwd)

VALIDARGS="clean libcuvs python docs tests template clean --uninstall -v -g -n --compile-static-lib --allgpuarch --no-nvtx --show_depr_warn --incl-cache-stats --time -h"
VALIDARGS="clean libcuvs python docs tests examples clean --uninstall -v -g -n --compile-static-lib --allgpuarch --no-nvtx --show_depr_warn --incl-cache-stats --time -h"
HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<tool>] [--limit-tests=<targets>] [--build-metrics=<filename>]
where <target> is:
clean - remove all existing build artifacts and configuration (start over)
Expand All @@ -27,7 +27,7 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to
python - build the cuvs Python package
docs - build the documentation
tests - build the tests
template - build the example CUVS application template
examples - build the examples

and <flag> is:
-v - verbose build mode
Expand Down Expand Up @@ -433,10 +433,10 @@ if hasArg docs; then
fi

################################################################################
# Initiate build for example CUVS application template (if needed)
# Initiate build for c++ examples (if needed)

if hasArg template; then
pushd ${REPODIR}/cpp/template
if hasArg examples; then
pushd ${REPODIR}/cpp/examples
./build.sh
popd
fi
8 changes: 2 additions & 6 deletions ci/build_docs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,11 @@ rapids-print-env

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
#PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
libcuvs \
libcuvs-headers \
cuvs \
raft-dask
libcuvs

export RAPIDS_VERSION_NUMBER="24.02"
export RAPIDS_DOCS_DIR="$(mktemp -d)"
Expand Down
49 changes: 14 additions & 35 deletions ci/build_python.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,43 +15,22 @@ rapids-print-env

rapids-logger "Begin py build"

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
#CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

version=$(rapids-generate-version)
git_commit=$(git rev-parse HEAD)
export RAPIDS_PACKAGE_VERSION=${version}
echo "${version}" > VERSION
#version=$(rapids-generate-version)
#git_commit=$(git rev-parse HEAD)
#export RAPIDS_PACKAGE_VERSION=${version}
#echo "${version}" > VERSION

package_dir="python"
for package_name in cuvs raft-dask; do
underscore_package_name=$(echo "${package_name}" | tr "-" "_")
sed -i "/^__git_commit__/ s/= .*/= \"${git_commit}\"/g" "${package_dir}/${package_name}/${underscore_package_name}/_version.py"
done
#package_dir="python"
#for package_name in cuvs; do
# underscore_package_name=$(echo "${package_name}" | tr "-" "_")
# sed -i "/^__git_commit__/ s/= .*/= \"${git_commit}\"/g" "${package_dir}/${package_name}/${underscore_package_name}/_version.py"
#done

# TODO: Remove `--no-test` flags once importing on a CPU
# node works correctly
rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
conda/recipes/cuvs


# Build ann-bench for each cuda and python version
rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
--channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \
conda/recipes/cuda-ann-bench

# Build ann-bench-cpu only in CUDA 11 jobs since it only depends on python
# version
RAPIDS_CUDA_MAJOR="${RAPIDS_CUDA_VERSION%%.*}"
if [[ ${RAPIDS_CUDA_MAJOR} == "11" ]]; then
rapids-conda-retry mambabuild \
--no-test \
--channel "${CPP_CHANNEL}" \
--channel "${RAPIDS_CONDA_BLD_OUTPUT_DIR}" \
conda/recipes/cuda-ann-bench-cpu
fi

rapids-upload-conda-to-s3 python
#rapids-conda-retry mambabuild \
# --no-test \
# --channel "${CPP_CHANNEL}" \
# conda/recipes/cuvs
4 changes: 2 additions & 2 deletions ci/build_wheel_cuvs.sh
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/bin/bash
# Copyright (c) 2023, NVIDIA CORPORATION.
# Copyright (c) 2023-2024, NVIDIA CORPORATION.

set -euo pipefail

# Set up skbuild options. Enable sccache in skbuild config options
export SKBUILD_CONFIGURE_OPTIONS="-DRAFT_BUILD_WHEELS=ON -DDETECT_CONDA_ENV=OFF -DFIND_RAFT_CPP=OFF"

ci/build_wheel.sh cuvs python/cuvs
#ci/build_wheel.sh cuvs python/cuvs
34 changes: 17 additions & 17 deletions ci/test_python.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2022-2023, NVIDIA CORPORATION.
# Copyright (c) 2022-2024, NVIDIA CORPORATION.

set -euo pipefail

Expand All @@ -20,18 +20,18 @@ set -u

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
#PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}
RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${PWD}/coverage-results"}
mkdir -p "${RAPIDS_TESTS_DIR}" "${RAPIDS_COVERAGE_DIR}"

rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
libcuvs cuvs
#rapids-mamba-retry install \
# --channel "${CPP_CHANNEL}" \
## --channel "${PYTHON_CHANNEL}" \
# libcuvs #cuvs

rapids-logger "Check GPU usage"
nvidia-smi
Expand All @@ -40,17 +40,17 @@ EXITCODE=0
trap "EXITCODE=1" ERR
set +e

rapids-logger "pytest cuvs"
pushd python/cuvs/cuvs
pytest \
--cache-clear \
--junitxml="${RAPIDS_TESTS_DIR}/junit-cuvs.xml" \
--cov-config=../.coveragerc \
--cov=cuvs \
--cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cuvs-coverage.xml" \
--cov-report=term \
test
popd
#rapids-logger "pytest cuvs"
#pushd python/cuvs/cuvs
#pytest \
# --cache-clear \
# --junitxml="${RAPIDS_TESTS_DIR}/junit-cuvs.xml" \
# --cov-config=../.coveragerc \
# --cov=cuvs \
# --cov-report=xml:"${RAPIDS_COVERAGE_DIR}/cuvs-coverage.xml" \
# --cov-report=term \
# test
#popd

rapids-logger "Test script exiting with value: $EXITCODE"
exit ${EXITCODE}
24 changes: 12 additions & 12 deletions ci/test_wheel_cuvs.sh
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
#!/bin/bash
# Copyright (c) 2023, NVIDIA CORPORATION.
# Copyright (c) 2023-2024, NVIDIA CORPORATION.

set -euo pipefail

mkdir -p ./dist
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
RAPIDS_PY_WHEEL_NAME="cuvs_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 ./dist
#RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
#RAPIDS_PY_WHEEL_NAME="cuvs_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 ./dist

# echo to expand wildcard before adding `[extra]` requires for pip
python -m pip install $(echo ./dist/cuvs*.whl)[test]

# Run smoke tests for aarch64 pull requests
if [[ "$(arch)" == "aarch64" && "${RAPIDS_BUILD_TYPE}" == "pull-request" ]]; then
python ./ci/wheel_smoke_test_cuvs.py
else
python -m pytest ./python/cuvs/cuvs/test
fi
## echo to expand wildcard before adding `[extra]` requires for pip
#python -m pip install $(echo ./dist/cuvs*.whl)[test]
#
## Run smoke tests for aarch64 pull requests
#if [[ "$(arch)" == "aarch64" && "${RAPIDS_BUILD_TYPE}" == "pull-request" ]]; then
# python ./ci/wheel_smoke_test_cuvs.py
#else
# python -m pytest ./python/cuvs/cuvs/test
#fi
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Copyright (c) 2022-2023, NVIDIA CORPORATION.
# Copyright (c) 2022-2024, NVIDIA CORPORATION.

# Just building template so we verify it uses libraft.so and fail if it doesn't build
./build.sh template
./build.sh examples
6 changes: 3 additions & 3 deletions conda/recipes/libcuvs/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -195,9 +195,9 @@ outputs:
home: https://rapids.ai/
license: Apache-2.0
summary: libcuvs tests
- name: libcuvs-template
- name: libcuvs-examples
version: {{ version }}
script: build_libcuvs_template.sh
script: build_libcuvs_examples.sh
build:
script_env: *script_env
number: {{ GIT_DESCRIBE_NUMBER }}
Expand Down Expand Up @@ -241,4 +241,4 @@ outputs:
about:
home: https://rapids.ai/
license: Apache-2.0
summary: libcuvs template
summary: libcuvs examples
File renamed without changes.
8 changes: 4 additions & 4 deletions cpp/template/README.md → cpp/examples/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Example CUVS Project Template
# cuVS C++ Examples

This template project provides a drop-in sample to either start building a new application with, or using CUVS in an existing CMake project.

First, please refer to our [installation docs](https://docs.rapids.ai/api/cuvs/stable/build.html#cuda-gpu-requirements) for the minimum requirements to use CUVS.
First, please refer to our [installation docs](https://docs.rapids.ai/api/cuvs/stable/build.html#cuda-gpu-requirements) for the minimum requirements to use cuVS.

Once the minimum requirements are satisfied, this example template application can be built with the provided `build.sh` script. This is a bash script that calls the appropriate CMake commands, so you can look into it to see the typical CMake based build workflow.

This directory (`CUVS_SOURCE/cpp/template`) can be copied directly in order to build a new application with CUVS.
This directory (`CUVS_SOURCE/cpp/examples`) can be copied directly in order to build a new application with CUVS.

CUVS can be integrated into an existing CMake project by copying the contents in the `configure rapids-cmake` and `configure cuvs` sections of the provided `CMakeLists.txt` into your project, along with `cmake/thirdparty/get_cuvs.cmake`.
cuVS can be integrated into an existing CMake project by copying the contents in the `configure rapids-cmake` and `configure cuvs` sections of the provided `CMakeLists.txt` into your project, along with `cmake/thirdparty/get_cuvs.cmake`.

Make sure to link against the appropriate Cmake targets. Use `cuvs::cuvs` to utilize the shared library.

Expand Down
2 changes: 1 addition & 1 deletion cpp/template/build.sh → cpp/examples/build.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash

# Copyright (c) 2023, NVIDIA CORPORATION.
# Copyright (c) 2023-2024, NVIDIA CORPORATION.

# cuvs empty project template build script

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# =============================================================================
# Copyright (c) 2023, NVIDIA CORPORATION.
# Copyright (c) 2023-2024, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2023, NVIDIA CORPORATION.
* Copyright (c) 2023-2024, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
Loading
Loading