Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hera gnu build broken #831

Closed
RussTreadon-NOAA opened this issue Jan 30, 2025 · 8 comments · Fixed by #833
Closed

Hera gnu build broken #831

RussTreadon-NOAA opened this issue Jan 30, 2025 · 8 comments · Fixed by #833

Comments

@RussTreadon-NOAA
Copy link
Contributor

The gsi_hera.gnu.lua from develop at 27c03e8 is missing -rocky8 on the the prepend_path line. Even after fixing this in a working copy of develop

-prepend_path("MODULEPATH", "/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/modulefiles/Core")

the gnu build fails with the message

[  0%] Building Fortran object src/mgbf/CMakeFiles/mgbf.dir/jp_pkind.f90.o
cd /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/build/src/mgbf && /apps/gnu/gcc-9.2.0/bin/gfortran -DNDEBUG -I/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/build/src/mgbf/include/mgbf -g -fbacktrace -fconvert=big-endian -O3 -DNDEBUG -O3 -Jinclude/mgbf -c /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/mgbf/jp_pkind.f90 -o CMakeFiles/mgbf.dir/jp_pkind.f90.o
/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/mgbf/jp_pkind.f90:22:4:

   22 | use mpi
      |    1
Fatal Error: Cannot open module file 'mpi.mod' for reading at (1): No such file or directory
compilation terminated.
make[2]: *** [src/mgbf/CMakeFiles/mgbf.dir/build.make:88: src/mgbf/CMakeFiles/mgbf.dir/jp_pkind.f90.o] Error 1

This issue is opened to document the fact that the Hera gnu build is broken. This issue will remain open until the Hera gnu build is fixed or the decision is made to not support a GSI gnu build on Hera.

@DavidHuber-NOAA
Copy link
Collaborator

Opened issue JCSDA/spack-stack#1483 to request a new GNU installation of spack-stack on Hera.

@RussTreadon-NOAA
Copy link
Contributor Author

Based on comments from @RatkoVasic-NOAA in GSI-utils #66, I made the following changes to a working copy of develop modulefiles/gsi_hera.gnu.lua

-prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/scratch4/NCEPDEV/stmp/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/modulef
iles/Core")
 --Needed for openmpi build
-prepend_path("MODULEPATH", "/scratch1/NCEPDEV/jcsda/jedipara/spack-stack/modulefiles")
+--prepend_path("MODULEPATH", "/scratch1/NCEPDEV/jcsda/jedipara/spack-stack/modulefiles")

 local python_ver=os.getenv("python_ver") or "3.11.6"
 local stack_gnu_ver=os.getenv("stack_gnu_ver") or "9.2.0"
-local stack_openmpi_ver=os.getenv("stack_openmpi_ver") or "4.1.5"
+local stack_openmpi_ver=os.getenv("stack_openmpi_ver") or "4.1.6"

The first GSI build attempt failed with

[  0%] Building Fortran object src/mgbf/CMakeFiles/mgbf.dir/jp_pkind2.f90.o
/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/mgbf/jp_pkind2.f90:22:4:

   22 | use mpi
      |    1
Fatal Error: Cannot open module file 'mpi.mod' for reading at (1): No such file or directory
/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/mgbf/jp_pkind.f90:22:4:

   22 | use mpi
      |    1
Fatal Error: Cannot open module file 'mpi.mod' for reading at (1): No such file or directory
compilation terminated.

As a test I turned off the MGBF build and tried again. The build got further but failed again

 79%] Building Fortran object src/gsi/CMakeFiles/gsi_fortran_obj.dir/cplr_regional_io.f90.o
[ 80%] Building Fortran object src/gsi/CMakeFiles/gsi_fortran_obj.dir/hybrid_ensemble_isotropic.F90.o
/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/gsi/hybrid_ensemble_isotropic.F90:107:6:

  107 |   use mg_intstate
      |      1
Fatal Error: Cannot open module file 'mg_intstate.mod' for reading at (1): No such file or directory
compilation terminated.
make[2]: *** [src/gsi/CMakeFiles/gsi_fortran_obj.dir/build.make:2610: src/gsi/CMakeFiles/gsi_fortran_obj.dir/hybrid_ensemble_isotropic.F90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:203: src/gsi/CMakeFiles/gsi_fortran_obj.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

Line 107 of src/gsi/hybrid_ensemble_isotropic.F90 is in the block

! For MGBF
  use mg_intstate
  use mg_timers

We do not use MGBF in any of our regional or global ctests. We do not use MGBF in the operational GDAS. Nonetheless, it seems we must compile MGBF in order to build gsi.x.

My guess is that the gnu build will work once the gnu MGBF issues are resolved.

Tagging @TingLei-NOAA and @ShunLiu-NOAA to confirm that we must compile MGBF in order to build gsi.x and enkf.x. If true we need to

  1. fix MGBF so that it compiles with gnu, or
  2. modify the GSI build so that MGBF is an optional component

@TingLei-NOAA
Copy link
Contributor

TingLei-NOAA commented Jan 31, 2025 via email

@RussTreadon-NOAA
Copy link
Contributor Author

Thank you for reminding me of GSI issue #765. This issue remains open.

@TingLei-NOAA
Copy link
Contributor

@RussTreadon-NOAA It seems the mpi dependency is not specified in the mgbf cmake file.
But, now, when I ran build.sh to build gsi, I got problems like :

Lmod has detected the following error: The following module(s) are unknown: "stack-gcc/9.2.0"
"stack-openmpi/4.1.5"

Seems stack_gnu_ver and so on are not specified correctly.
Would you please let you how those parameters be setup ?

@RussTreadon-NOAA
Copy link
Contributor Author

@TingLei-NOAA , did you make the following changes in your working copy of modulefiles/gsi_hera.gnu.lua

--- a/modulefiles/gsi_hera.gnu.lua
+++ b/modulefiles/gsi_hera.gnu.lua
@@ -1,13 +1,13 @@
 help([[
 ]])

-prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/scratch4/NCEPDEV/stmp/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/modulefiles/Core")
 --Needed for openmpi build
-prepend_path("MODULEPATH", "/scratch1/NCEPDEV/jcsda/jedipara/spack-stack/modulefiles")
+--prepend_path("MODULEPATH", "/scratch1/NCEPDEV/jcsda/jedipara/spack-stack/modulefiles")

 local python_ver=os.getenv("python_ver") or "3.11.6"
 local stack_gnu_ver=os.getenv("stack_gnu_ver") or "9.2.0"
-local stack_openmpi_ver=os.getenv("stack_openmpi_ver") or "4.1.5"
+local stack_openmpi_ver=os.getenv("stack_openmpi_ver") or "4.1.6"
 local cmake_ver=os.getenv("cmake_ver") or "3.23.1"
 local prod_util_ver=os.getenv("prod_util_ver") or "2.1.1"
 local openblas_ver=os.getenv("openblas_ver") or "0.3.24"

I made the above changes to a working copy of GSI develop. Look in Hera /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop. After this I executed

Hera(hfe07):/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop$ cd ush
Hera(hfe07):/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/ush$ export COMPILER=gnu
Hera(hfe07):/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/ush$ ./build.sh > build.log 2>&1

tail -25 build.log returns

Dependee "/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/build/src/mgbf/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/build/src/mgbf/CMakeFiles/mgbf.dir/depend.internal".
Scanning dependencies of target mgbf
[  0%] Building C object src/gsi/CMakeFiles/gsi_c_obj.dir/blockIO.c.o
[  0%] Building Fortran object src/mgbf/CMakeFiles/mgbf.dir/jp_pkind2.f90.o
[  0%] Building Fortran object src/mgbf/CMakeFiles/mgbf.dir/jp_pkind.f90.o
[  0%] Building Fortran object src/mgbf/CMakeFiles/mgbf.dir/kinds.f90.o
/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/mgbf/jp_pkind.f90:22:4:

   22 | use mpi
      |    1
Fatal Error: Cannot open module file 'mpi.mod' for reading at (1): No such file or directory
/scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop/src/mgbf/jp_pkind2.f90:22:4:

   22 | use mpi
      |    1
Fatal Error: Cannot open module file 'mpi.mod' for reading at (1): No such file or directory
compilation terminated.
compilation terminated.
make[2]: *** [src/mgbf/CMakeFiles/mgbf.dir/build.make:88: src/mgbf/CMakeFiles/mgbf.dir/jp_pkind.f90.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [src/mgbf/CMakeFiles/mgbf.dir/build.make:101: src/mgbf/CMakeFiles/mgbf.dir/jp_pkind2.f90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:195: src/mgbf/CMakeFiles/mgbf.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  0%] Built target gsi_c_obj
make: *** [Makefile:146: all] Error 2

@TingLei-NOAA
Copy link
Contributor

@RussTreadon-NOAA Thanks. Yes, I directly copied your gsi_hera.gnu.lua and resolved my problem.
The mgbf CMakeLists.txt is upgraded with correct specification on MPI dependency as done in feature/mgbf4gnu at https://github.com/TingLei-daprediction/GSI.git. After this change, the building using gnu succeeded.
Do you want me to open an PR to add this change or @DavidHuber-NOAA could directly incorporate these two lines of changes in his work?

@RussTreadon-NOAA
Copy link
Contributor Author

@TingLei-NOAA , thank you for fixing the mgbf gnu build. Please open a PR to get the changes from feature/mgbf4gnu into develop. I cloned your branch and confirm that the Hera gnu build works for gsi.x and enkf.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants