Nightly test failure, Trilinos builds with intel (icpc): sparse_ioutils "Problem reading HB file test_sparse_ioutils_asym.hb..."

[ndellingwood]

Nightly builds of Trilinos with sems-intel/2021.3 and intel/2023.2.0, using icpc, are failing in the sparse_ioutils test:

18:49:29 [ RUN      ] serial.sparse_ioutils
18:49:29 unknown file: Failure
18:49:29 C++ exception with description "Problem reading HB file test_sparse_ioutils_asym.hb, Line 2 did not have valid values" thrown in the test body.
18:49:29 [  FAILED  ] serial.sparse_ioutils (0 ms)

Following merge of #2155 , @jgfouca can you help take a look? If nothing else, we can disable the subtest for these builds, there are also issues with Trilinos and matrix market readers in similar configurations (e.g. see trilinos/Trilinos#11968)

Reproducer (kokkos-dev-2):

# Repos
git clone -b develop https://github.com/trilinos/Trilinos.git
TRILINOS_DIR=$PWD/Trilinos
git clone -b develop https://github.com/kokkos/kokkos.git
KOKKOS_DIR=$PWD/kokkos
cd $TRILINOS_DIR
ln -s ${PWD}/../kokkos kokkos
cd ..
git clone -b develop https://github.com/kokkos/kokkos-kernels.git
KOKKOSKERNELS_DIR=$PWD/kokkos-kernels
cd $TRILINOS_DIR
ln -s ${PWD}/../kokkos-kernels kokkos-kernels
cd ..

# Setup environment
source /projects/sems/modulefiles/utils/sems-v2-modules-init.sh
module purge
module load sems-cmake sems-intel/2021.3 sems-intel-mkl/2020.4.304 sems-openmpi/4.1.4 sems-ninja
module list

# MKL
export BLAS_LIBRARIES="-L$MKLROOT/mkl/lib;-lblas;-lgfortran;-lgomp"
export LAPACK_LIBRARIES="-L$MKLROOT/mkl/lib;-llapack;-lgfortran;-lgomp"

# Configure Trilinos
cmake \
  -GNinja \
  -D CMAKE_INSTALL_PREFIX="${TRILINOS_INSTALL_DIR}" \
  -D CMAKE_BUILD_TYPE:STRING=RELEASE \
  -D BUILD_SHARED_LIBS:BOOL=OFF \
  -D CMAKE_CXX_COMPILER="`which mpic++`" \
  -D CMAKE_C_COMPILER="`which mpicc`" \
  -D CMAKE_Fortran_COMPILER="`which mpif90`" \
  -D CMAKE_CXX_FLAGS="-no-ip" \
  -D CMAKE_C_FLAGS="-no-ip" \
  -DTPL_ENABLE_MPI=ON \
  -DTPL_ENABLE_BLAS:BOOL=ON \
  -DTPL_BLAS_LIBRARIES:PATH="${BLAS_LIBRARIES}" \
  -DTPL_ENABLE_LAPACK:BOOL=ON \
  -DTPL_LAPACK_LIBRARIES:PATH="${LAPACK_LIBRARIES}" \
  -DTrilinos_ENABLE_ALL_PACKAGES=OFF \
  -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
  -DTrilinos_ENABLE_TESTS=ON \
  -DTrilinos_MUST_FIND_ALL_TPL_LIBS=TRUE \
  -DTrilinos_ENABLE_OpenMP=OFF \
  -DTrilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_SERIAL=ON \
   -D Kokkos_ENABLE_TESTS=ON \
  -D Kokkos_ARCH_SKX=ON \
  -DTrilinos_ENABLE_KokkosKernels=ON \
   -D KokkosKernels_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Tpetra=ON \
   -D Tpetra_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Ifpack2=ON \
   -D Ifpack2_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Sacado=ON \
   -D Sacado_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Stokhos=ON \
   -D Stokhos_ENABLE_TESTS=ON \
\
  -DTPL_ENABLE_Matio=OFF \
\
 -DKokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
 -DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \
$TRILINOS_DIR

Edit: Added missing sym links to kokkos* repos in the Trilinos src dir

[jgfouca]

Thanks @ndellingwood for the reproducer. I will look at this. I had to make a couple minor adjustments to the reproducer:

 -DKokkos_SOURCE_DIR_OVERRIDE:STRING=$KOKKOS_DIR \
 -DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=$KOKKOSKERNELS_DIR \

Also, both kokkos and kokkos-kernels dirs have to be under Trilinos or else you get this error:

  Error: The package 'Kokkos' was given an absolute directory
  '/ascldap/users/jgfouca/kokkos/kokkos' which is *not* under the project's
  source directory '/ascldap/users/jgfouca/Trilinos/'!

[ndellingwood]

@jgfouca oop, thanks, I forgot to add the lines to add the symlinks to the kokkos* repos in the Trilinos source dir, sorry about that

[ndellingwood]

@jgfouca the reproducer should be fixed now

[ndellingwood]

intel builds are happier after merge of #2162 , thanks @jgfouca !