Build failure in rocky 9 x86_64 using CUDA flags

[keixthb]

Steps to reproduce:

On a clean install of Rocky 9.5 I fetch the Nvidia driver with the following:

   $ sudo dnf update -y
   $  sudo dnf install -y tar bzip2 pciutils elfutils-libelf-devel libglvnd-opengl libglvnd-glx libglvnd-devel acpid pkgconfig emacs cmake emacs
   $  sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
   $  sudo dnf config-manager --set-enabled crb
   $  sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
   $  sudo dnf clean all
   $  sudo dnf -y module install nvidia-driver:latest
   $  sudo reboot

I verify the driver works by calling the following:

   $  nvidia-smi

I fetch cuda with:

   $  sudo dnf install -y cuda-toolkit.x86_64  

Add cuda to the path with:

   $  emacs ~/.bashrc

and:

export PATH=/usr/local/cuda/bin:$PATH

Test the nvcc compiler with:

   $  which nvcc

Build kokkos with:

   $  git clone --depth=10 https://github.com/kokkos/kokkos.git ~/kokkos
   $  cd ~/kokkos/
   $  mkdir build
   $  cd build
   $  cmake .. -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_CUDA=ON -DKokkos_ENABLE_CUDA_LAMBDA=ON 
   $  make CXXFLAGS="-O3" CFLAGS="-O3" -j$(nproc)
   $  sudo make install CXXFLAGS="-O3" CFLAGS="-O3" -j$(nproc)

But when I try building kokkos-kernels with the following:

   $  git clone --depth=10 https://github.com/kokkos/kokkos-kernels ~/kokkos-kernels
   $  cd ~/kokkos-kernels
   $  mkdir build 
   $  cd build
   $  cmake .. -DCMAKE_CXX_STANDARD=17 -DKokkos_ENABLE_CUDA=ON -DKokkos_ENABLE_CUDA_LAMBDA=ON 
   $  make CXXFLAGS="-O3" CFLAGS="-O3" -j$(nproc)

Error message:

Error message

[  0%] Building CXX object CMakeFiles/kokkoskernels.dir/blas/eti/generated_specializations_cpp/rot/Blas1_rot_eti_DOUBLE_LAYOUTLEFT_EXECSPACE_CUDA_MEMSPACE_CUDASPACE.cpp.o
/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(323): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(324): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(325): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(326): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(327): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(328): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(329): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(330): error: argument of type "cuDoubleComplex *" (aka "double2 *") is incompatible with parameter of type "const double *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                                  ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(332): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(333): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(334): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(335): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(336): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(337): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(338): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

/home/admin/kokkos-kernels/blas/tpls/KokkosBlas1_rot_tpl_spec_decl.hpp(339): error: argument of type "cuComplex *" (aka "float2 *") is incompatible with parameter of type "const float *"
  (singleton.handle, X.extent_int(0), reinterpret_cast(X.data()), 1, reinterpret_cast(Y.data()), 1, c.data(), reinterpret_cast(s.data())); KokkosBlas::Impl::cublas_internal_safe_call(
                                                                                                                                      ^

16 errors detected in the compilation of "/home/admin/kokkos-kernels/build/blas/eti/generated_specializations_cpp/rot/Blas1_rot_eti_DOUBLE_LAYOUTLEFT_EXECSPACE_CUDA_MEMSPACE_CUDASPACE.cpp".
make[2]: *** [CMakeFiles/kokkoskernels.dir/build.make:958: CMakeFiles/kokkoskernels.dir/blas/eti/generated_specializations_cpp/rot/Blas1_rot_eti_DOUBLE_LAYOUTLEFT_EXECSPACE_CUDA_MEMSPACE_CUDASPACE.cpp.o] Error 2
make[1]: *** [CMakeFiles/Makefile2:866: CMakeFiles/kokkoskernels.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

Information on system:

This was done using a AMD Opteron 4284 with a NVidia GTX 1060.

cmake --version:

cmake version 3.26.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).

gcc --version:

gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Any ideas? Thanks in advance.

[lucbv]

Thanks for reporting this issue, it should be fix one PR #2473 merges

[keixthb]

Can confirm that fixed the error on my machine. Marking as closed.