Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix python wheel packages (312, 313) #6430

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

fix python wheel packages (312, 313) #6430

wants to merge 2 commits into from

Conversation

hgy59
Copy link
Contributor

@hgy59 hgy59 commented Jan 27, 2025

Description

  • python312-wheels and python313-wheels did not include gcc version dependent wheels
  • TC_GCC version checks must be used after include of spksrc.python.mk (or spksrc.spk.mk) to work

Fixes #

Checklist

  • Build rule all-supported completed successfully
  • New installation of package completed successfully
  • Package upgrade completed successfully (Manually install the package again)
  • Package functionality was tested
  • Any needed documentation is updated/created

Type of change

  • Bug fix
  • New Package
  • Package update
  • Includes small framework changes
  • This change requires a documentation update (e.g. Wiki)

@hgy59
fix python wheel packages (312, 313)
2b03884 
- python312-wheels and python313-wheels did not include gcc version dependent wheels
- TC_GCC version checks must be used after include of spksrc.python.mk (or spksrc.spk.mk) to work
@hgy59
Copy link
Contributor Author

hgy59 commented Jan 27, 2025

@th0ma7 I am unsure wheter this fix really works, due to the comment you added at the bottom of the Makefile(s)

##
## Addition of requirements files beyond spksrc.python.mk
## will not be taken into consideration for processing
## unless WHEELS has been previously populated. This is due to
## the makefile analysis phase which discard ifeq instructions.
##

@hgy59
Copy link
Contributor Author

hgy59 commented Jan 27, 2025

I mentioned my issue with numpy but finally found, that python312-wheels did not build numpy at all.
And numpy in python312-wheels has the same error:

===>  cross-pip: [/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/bin/cross-pip]
===>  _PYTHON_HOST_PLATFORM="x86_64-pc-linux-gnu" PATH=:/spksrc/native/python312/work-native/install/usr/local/bin:/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/bin:/spksrc/distrib/cargo/bin:/scripts:~/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games CMAKE_TOOLCHAIN_FILE= MESON_CROSS_FILE= cross-pip wheel --disable-pip-version-check --no-binary :all: --find-links /spksrc/distrib/pip --cache-dir /spksrc/spk/python312-wheels/work-x64-7.1/pip --no-deps --wheel-dir /spksrc/spk/python312-wheels/work-x64-7.1/wheelhouse --no-index --no-build-isolation numpy==2.2.2
Looking in links: /spksrc/distrib/pip
Processing /spksrc/distrib/pip/numpy-2.2.2.tar.gz
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      + /spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/cross/bin/python /tmp/pip-wheel-c9ipqape/numpy_a9af0df1de0b4badb052cff187c75dbe/vendored-meson/meson/meson.py setup /tmp/pip-wheel-c9ipqape/numpy_a9af0df1de0b4badb052cff187c75dbe /tmp/pip-wheel-c9ipqape/numpy_a9af0df1de0b4badb052cff187c75dbe/.mesonpy-emin9j8v -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-wheel-c9ipqape/numpy_a9af0df1de0b4badb052cff187c75dbe/.mesonpy-emin9j8v/meson-python-native-file.ini
      The Meson build system
      Version: 1.5.2
      Source dir: /tmp/pip-wheel-c9ipqape/numpy_a9af0df1de0b4badb052cff187c75dbe
      Build dir: /tmp/pip-wheel-c9ipqape/numpy_a9af0df1de0b4badb052cff187c75dbe/.mesonpy-emin9j8v
      Build type: native build
      Project name: NumPy
      Project version: 2.2.2
      C compiler for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc (gcc 8.5.0 "x86_64-pc-linux-gnu-gcc (GCC) 8.5.0")
      C linker for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc ld.bfd 2.30
      C++ compiler for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ (gcc 8.5.0 "x86_64-pc-linux-gnu-g++ (GCC) 8.5.0")
      C++ linker for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ ld.bfd 2.30
      Cython compiler for the host machine: cython3 (cython 0.29.32)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64

      ../meson.build:37:2: ERROR: Problem encountered: NumPy requires Cython >= 3.0.6

      A full log can be found at /tmp/pip-wheel-c9ipqape/numpy_a9af0df1de0b4badb052cff187c75dbe/.mesonpy-emin9j8v/meson-logs/meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
make[4]: *** [../../mk/spksrc.wheel-compile.mk:105: cross-compile-wheel-numpy-2.2.2] Error 1
make[3]: *** [../../mk/spksrc.wheel-compile.mk:58: wheel_compile_target] Error 2
make[2]: *** [../../mk/spksrc.wheel.mk:80: build_wheel_target] Error 1
make[2]: Leaving directory '/spksrc/spk/python312-wheels'
make[1]: *** [../../mk/spksrc.supported.mk:74: build-arch-x64-7.1] Error 1
make[1]: Leaving directory '/spksrc/spk/python312-wheels'

@hgy59 hgy59 mentioned this pull request Jan 27, 2025
Open
4 tasks
@hgy59
Copy link
Contributor Author

hgy59 commented Jan 27, 2025

@th0ma7 I am unsure wheter this fix really works, due to the comment you added at the bottom of the Makefile(s)

##
## Addition of requirements files beyond spksrc.python.mk
## will not be taken into consideration for processing
## unless WHEELS has been previously populated. This is due to
## the makefile analysis phase which discard ifeq instructions.
##

@th0ma7 if we must define all requirements files in WHEELS variable before the include of spksrc.python.mk, we can't have TC_GCC version dependent requirements files.
This could only be solved by using TCVERSION and ARCH variables (i.e. DSM 7 without comcerto2k = GCC > 5.0)

What is the reason for the comment above and why not in python311-wheels?

@hgy59
Copy link
Contributor Author

hgy59 commented Jan 27, 2025

as expected, all packages with numpy failed

My local builds have the error

../meson.build:37:2: ERROR: Problem encountered: NumPy requires Cython >= 3.0.6

Details for make WHEELS="numpy==2.2.2" wheel-x64-7.1

===>  cross-pip: [/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/bin/cross-pip]
===>  _PYTHON_HOST_PLATFORM="x86_64-pc-linux-gnu" PATH=:/spksrc/native/python312/work-native/install/usr/local/bin:/spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/bin:/spksrc/distrib/cargo/bin:/scripts:~/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games CMAKE_TOOLCHAIN_FILE= MESON_CROSS_FILE= cross-pip wheel --disable-pip-version-check --no-binary :all: --find-links /spksrc/distrib/pip --cache-dir /spksrc/spk/python312-wheels/work-x64-7.1/pip --no-deps --wheel-dir /spksrc/spk/python312-wheels/work-x64-7.1/wheelhouse --no-index --no-build-isolation numpy==2.2.2
Looking in links: /spksrc/distrib/pip
Processing /spksrc/distrib/pip/numpy-2.2.2.tar.gz
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      + /spksrc/spk/python312-wheels/work-x64-7.1/crossenv-default/cross/bin/python /tmp/pip-wheel-ibwawh9h/numpy_d732a200d3404d309922d364bb8c6575/vendored-meson/meson/meson.py setup /tmp/pip-wheel-ibwawh9h/numpy_d732a200d3404d309922d364bb8c6575 /tmp/pip-wheel-ibwawh9h/numpy_d732a200d3404d309922d364bb8c6575/.mesonpy-c_2ui5cx -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-wheel-ibwawh9h/numpy_d732a200d3404d309922d364bb8c6575/.mesonpy-c_2ui5cx/meson-python-native-file.ini
      The Meson build system
      Version: 1.5.2
      Source dir: /tmp/pip-wheel-ibwawh9h/numpy_d732a200d3404d309922d364bb8c6575
      Build dir: /tmp/pip-wheel-ibwawh9h/numpy_d732a200d3404d309922d364bb8c6575/.mesonpy-c_2ui5cx
      Build type: native build
      Project name: NumPy
      Project version: 2.2.2
      C compiler for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc (gcc 8.5.0 "x86_64-pc-linux-gnu-gcc (GCC) 8.5.0")
      C linker for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc ld.bfd 2.30
      C++ compiler for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ (gcc 8.5.0 "x86_64-pc-linux-gnu-g++ (GCC) 8.5.0")
      C++ linker for the host machine: /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ ld.bfd 2.30
      Cython compiler for the host machine: cython3 (cython 0.29.32)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64

      ../meson.build:37:2: ERROR: Problem encountered: NumPy requires Cython >= 3.0.6

      A full log can be found at /tmp/pip-wheel-ibwawh9h/numpy_d732a200d3404d309922d364bb8c6575/.mesonpy-c_2ui5cx/meson-logs/meson-log.txt
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
make[3]: *** [../../mk/spksrc.wheel-compile.mk:105: cross-compile-wheel-numpy-2.2.2] Error 1
make[2]: *** [../../mk/spksrc.wheel-compile.mk:58: wheel_compile_target] Error 2
make[1]: *** [../../mk/spksrc.wheel.mk:115: build_wheel_target] Error 2

but the github build has different error (and has additionally log since CI build is detected)

2025-01-27T15:50:10.4756863Z ===>  Processing requirement [/github/workspace/spk/python312-wheels/src/requirements-crossenv-numpy.txt:numpy==2.2.2]
2025-01-27T15:50:10.4832119Z ===>  make ARCH=aarch64 TCVERSION=7.1 REQUIREMENT="numpy==2.2.2" WHEEL_NAME="numpy" WHEEL_VERSION="2.2.2" WHEEL_TYPE="crossenv" wheel
2025-01-27T15:50:11.2852490Z ===>  Downloading wheel [numpy], version [2.2.2]
2025-01-27T15:50:11.5805456Z wget --secure-protocol=TLSv1_2 -nv -O /github/workspace/distrib/pip/numpy-2.2.2.tar.gz.part -nc https://files.pythonhosted.org/packages/ec/d0/c12ddfd3a02274be06ffc71f3efc6d0e457b0409c4481596881e748cb264/numpy-2.2.2.tar.gz
2025-01-27T15:50:11.8436929Z 2025-01-27 15:50:11 URL:https://files.pythonhosted.org/packages/ec/d0/c12ddfd3a02274be06ffc71f3efc6d0e457b0409c4481596881e748cb264/numpy-2.2.2.tar.gz [20233295/20233295] -> "/github/workspace/distrib/pip/numpy-2.2.2.tar.gz.part" [1]
2025-01-27T15:50:11.8567833Z ===>  Compiling wheel [numpy], version [2.2.2], type [crossenv]
2025-01-27T15:50:11.8582473Z ===>  make WHEEL_NAME="numpy" WHEEL_VERSION="2.2.2" crossenv-aarch64-7.1
2025-01-27T15:50:12.6543381Z ===>  make ARCH="aarch64" TCVERSION="7.1" WHEEL_NAME="numpy" WHEEL_VERSION="2.2.2" crossenv
2025-01-27T15:50:13.4466739Z make[5]: 'crossenv' is up to date.
2025-01-27T15:50:13.4700217Z ===>  pip build [numpy], version: [2.2.2]     
2025-01-27T15:50:14.2883534Z ===>  crossenv: [/github/workspace/spk/python312-wheels/work-aarch64-7.1/crossenv-default]
2025-01-27T15:50:14.2893924Z ===>  pip: [/github/workspace/spk/python312-wheels/work-aarch64-7.1/crossenv-default/bin/cross-pip]
2025-01-27T15:50:14.2905835Z ===>  maturin: [/github/workspace/native/python312/work-native/install/usr/local/bin/maturin]
2025-01-27T15:50:14.2950324Z ===>  _PYTHON_HOST_PLATFORM="aarch64-unknown-linux-gnu" PATH=:/github/workspace/native/python312/work-native/install/usr/local/bin:/github/workspace/spk/python312-wheels/work-aarch64-7.1/crossenv-default/bin:/github/workspace/distrib/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CMAKE_TOOLCHAIN_FILE= MESON_CROSS_FILE= cross-pip wheel --disable-pip-version-check --no-binary :all: --find-links /github/workspace/distrib/pip --cache-dir /github/workspace/spk/python312-wheels/work-aarch64-7.1/pip --no-deps --wheel-dir /github/workspace/spk/python312-wheels/work-aarch64-7.1/wheelhouse --no-index --no-build-isolation numpy==2.2.2
2025-01-27T15:50:14.7190104Z Looking in links: /github/workspace/distrib/pip
2025-01-27T15:50:14.7267766Z Processing /github/workspace/distrib/pip/numpy-2.2.2.tar.gz
2025-01-27T15:50:16.7089064Z   Preparing metadata (pyproject.toml): started
2025-01-27T15:50:18.0282017Z   Preparing metadata (pyproject.toml): finished with status 'error'
2025-01-27T15:50:18.0352368Z   error: subprocess-exited-with-error
2025-01-27T15:50:18.0353108Z   
2025-01-27T15:50:18.0354015Z   × Preparing metadata (pyproject.toml) did not run successfully.
2025-01-27T15:50:18.0354628Z   │ exit code: 1
2025-01-27T15:50:18.0354914Z   ╰─> [69 lines of output]
2025-01-27T15:50:18.0357455Z       + /github/workspace/spk/python312-wheels/work-aarch64-7.1/crossenv-default/cross/bin/python /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/vendored-meson/meson/meson.py setup /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1 /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9/meson-python-native-file.ini
2025-01-27T15:50:18.0359882Z       The Meson build system
2025-01-27T15:50:18.0360136Z       Version: 1.5.2
2025-01-27T15:50:18.0360696Z       Source dir: /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1
2025-01-27T15:50:18.0361564Z       Build dir: /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9
2025-01-27T15:50:18.0362079Z       Build type: native build
2025-01-27T15:50:18.0362344Z       Project name: NumPy
2025-01-27T15:50:18.0362595Z       Project version: 2.2.2
2025-01-27T15:50:18.0362838Z       
2025-01-27T15:50:18.0363708Z       ../meson.build:1:0: ERROR: Could not invoke sanity test executable: [Errno 8] Exec format error: '/tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9/meson-private/sanitycheckc.exe'.
2025-01-27T15:50:18.0364816Z       
2025-01-27T15:50:18.0365385Z       A full log can be found at /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9/meson-logs/meson-log.txt
2025-01-27T15:50:18.0367321Z ##[group]==== CI platform detected, click here for meson-log.txt contents. ====
2025-01-27T15:50:18.0368055Z       Build started at 2025-01-27T15:50:17.769668
2025-01-27T15:50:18.0368968Z       Main binary: /github/workspace/spk/python312-wheels/work-aarch64-7.1/crossenv-default/cross/bin/python
2025-01-27T15:50:18.0370252Z       Build Options: -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9/meson-python-native-file.ini
2025-01-27T15:50:18.0371021Z       Python system: Linux
2025-01-27T15:50:18.0371447Z       The Meson build system
2025-01-27T15:50:18.0371683Z       Version: 1.5.2
2025-01-27T15:50:18.0372065Z       Source dir: /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1
2025-01-27T15:50:18.0372595Z       Build dir: /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9
2025-01-27T15:50:18.0373165Z       Build type: native build
2025-01-27T15:50:18.0373990Z       Running command: /github/workspace/spk/python312-wheels/work-aarch64-7.1/crossenv-default/cross/bin/python /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/numpy/_build_utils/gitversion.py
2025-01-27T15:50:18.0374827Z       --- stdout ---
2025-01-27T15:50:18.0375019Z       2.2.2
2025-01-27T15:50:18.0375176Z       
2025-01-27T15:50:18.0375336Z       --- stderr ---
2025-01-27T15:50:18.0375522Z       
2025-01-27T15:50:18.0375668Z       
2025-01-27T15:50:18.0375830Z       Project name: NumPy
2025-01-27T15:50:18.0376042Z       Project version: 2.2.2
2025-01-27T15:50:18.0376246Z       -----------
2025-01-27T15:50:18.0376828Z       Detecting compiler via: `/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc --version` -> 0
2025-01-27T15:50:18.0377446Z       stdout:
2025-01-27T15:50:18.0377649Z       aarch64-unknown-linux-gnu-gcc (GCC) 8.5.0
2025-01-27T15:50:18.0377948Z       Copyright (C) 2018 Free Software Foundation, Inc.
2025-01-27T15:50:18.0378330Z       This is free software; see the source for copying conditions.  There is NO
2025-01-27T15:50:18.0378786Z       warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
2025-01-27T15:50:18.0379135Z       -----------
2025-01-27T15:50:18.0379340Z       Running command: -cpp -x c -E -dM -
2025-01-27T15:50:18.0379582Z       -----
2025-01-27T15:50:18.0379745Z       -----------
2025-01-27T15:50:18.0386297Z       Detecting linker via: `/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc -Wl,--version -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/lib -L/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/lib -Wl,--rpath-link,/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/lib -Wl,--rpath,/var/packages/python312-wheels/target/lib -L/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/lib -Wl,--rpath-link,/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/lib -Wl,--rpath,/var/packages/python312/target/lib -I/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/usr/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/ -I/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/include -I/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/usr/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/ -I/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/include` -> 0
2025-01-27T15:50:18.0392899Z       stdout:
2025-01-27T15:50:18.0393085Z       GNU ld (GNU Binutils) 2.30
2025-01-27T15:50:18.0393352Z       Copyright (C) 2018 Free Software Foundation, Inc.
2025-01-27T15:50:18.0393738Z       This program is free software; you may redistribute it under the terms of
2025-01-27T15:50:18.0394190Z       the GNU General Public License version 3 or (at your option) a later version.
2025-01-27T15:50:18.0394564Z       This program has absolutely no warranty.
2025-01-27T15:50:18.0394809Z       -----------
2025-01-27T15:50:18.0394983Z       stderr:
2025-01-27T15:50:18.0395157Z       collect2 version 8.5.0
2025-01-27T15:50:18.0407000Z       /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../lib/gcc/aarch64-unknown-linux-gnu/8.5.0/../../../../aarch64-unknown-linux-gnu/bin/ld -plugin /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../libexec/gcc/aarch64-unknown-linux-gnu/8.5.0/liblto_plugin.so -plugin-opt=/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../libexec/gcc/aarch64-unknown-linux-gnu/8.5.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccz9wV7o.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --sysroot=/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot --eh-frame-hdr -dynamic-linker /lib/ld-linux-aarch64.so.1 -X -EL -maarch64linux -pie /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot/usr/lib/../lib64/Scrt1.o /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crti.o /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../lib/gcc/aarch64-unknown-linux-gnu/8.5.0/crtbeginS.o -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/lib -L/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/lib -L/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/lib -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../lib/gcc/aarch64-unknown-linux-gnu/8.5.0 -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../lib/gcc -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../lib/gcc/aarch64-unknown-linux-gnu/8.5.0/../../../../aarch64-unknown-linux-gnu/lib/../lib64 -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot/lib/../lib64 -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot/usr/lib/../lib64 -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../lib/gcc/aarch64-unknown-linux-gnu/8.5.0/../../../../aarch64-unknown-linux-gnu/lib -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot/lib -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot/usr/lib --version --rpath-link /github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/lib --rpath /var/packages/python312-wheels/target/lib --rpath-link /github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/lib --rpath /var/packages/python312/target/lib -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../lib/gcc/aarch64-unknown-linux-gnu/8.5.0/crtendS.o /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/../aarch64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crtn.o
2025-01-27T15:50:18.0418808Z       -----------
2025-01-27T15:50:18.0419335Z       Sanity testing C compiler: /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc
2025-01-27T15:50:18.0419912Z       Is cross compiler: False.
2025-01-27T15:50:18.0426490Z       Sanity check compiler command line: /github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc sanitycheckc.c -o sanitycheckc.exe -I/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/usr/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/ -I/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/include -I/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/usr/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/include -I/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/ -I/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/include -D_FILE_OFFSET_BITS=64 -L/github/workspace/toolchain/syno-aarch64-7.1/work/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/lib -L/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/lib -Wl,--rpath-link,/github/workspace/spk/python312-wheels/work-aarch64-7.1/install/var/packages/python312-wheels/target/lib -Wl,--rpath,/var/packages/python312-wheels/target/lib -L/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/lib -Wl,--rpath-link,/github/workspace/spk/python312/work-aarch64-7.1/install/var/packages/python312/target/lib -Wl,--rpath,/var/packages/python312/target/lib
2025-01-27T15:50:18.0433198Z       Sanity check compile stdout:
2025-01-27T15:50:18.0433421Z       
2025-01-27T15:50:18.0433574Z       -----
2025-01-27T15:50:18.0433752Z       Sanity check compile stderr:
2025-01-27T15:50:18.0433962Z       
2025-01-27T15:50:18.0434106Z       -----
2025-01-27T15:50:18.0434630Z       Running test binary command:  /tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9/meson-private/sanitycheckc.exe
2025-01-27T15:50:18.0435207Z       
2025-01-27T15:50:18.0435891Z       ../meson.build:1:0: ERROR: Could not invoke sanity test executable: [Errno 8] Exec format error: '/tmp/pip-wheel-9vwrho8p/numpy_f23eb40e6aed42b1adb88c1ceea45fe1/.mesonpy-o39ig5h9/meson-private/sanitycheckc.exe'.
2025-01-27T15:50:18.0436651Z       
2025-01-27T15:50:18.0436988Z ##[endgroup]
2025-01-27T15:50:18.0437144Z       
2025-01-27T15:50:18.0437298Z       [end of output]
2025-01-27T15:50:18.0437483Z   
2025-01-27T15:50:18.0437782Z   note: This error originates from a subprocess, and is likely not a problem with pip.
2025-01-27T15:50:18.0438169Z error: metadata-generation-failed
2025-01-27T15:50:18.0438422Z 
2025-01-27T15:50:18.0438672Z × Encountered error while generating package metadata.
2025-01-27T15:50:18.0439001Z ╰─> See above for output.
2025-01-27T15:50:18.0439130Z 
2025-01-27T15:50:18.0439284Z note: This is an issue with the package mentioned above, not pip.
2025-01-27T15:50:18.0439600Z hint: See above for details.
2025-01-27T15:50:18.4325156Z make[4]: *** [../../mk/spksrc.wheel-compile.mk:105: cross-compile-wheel-numpy-2.2.2] Error 1
2025-01-27T15:50:18.4326856Z make[3]: *** [../../mk/spksrc.wheel-compile.mk:58: wheel_compile_target] Error 2
2025-01-27T15:50:18.4334565Z make[2]: *** [../../mk/spksrc.wheel.mk:80: build_wheel_target] Error 1

@hgy59
Copy link
Contributor Author

hgy59 commented Jan 27, 2025

finally github build shows the same error for numpy==1.26.4 as my local build:
it does not use Cython of the crossenv...

2025-01-27T17:48:41.5422723Z ===>  pip build [numpy], version: [1.26.4]     
2025-01-27T17:48:42.2951323Z ===>  crossenv: [/github/workspace/spk/python312-wheels/work-x64-7.1/crossenv-default]
2025-01-27T17:48:42.2962026Z ===>  pip: [/github/workspace/spk/python312-wheels/work-x64-7.1/crossenv-default/bin/cross-pip]
2025-01-27T17:48:42.2973392Z ===>  maturin: [/github/workspace/native/python312/work-native/install/usr/local/bin/maturin]
2025-01-27T17:48:42.3017455Z ===>  _PYTHON_HOST_PLATFORM="x86_64-pc-linux-gnu" PATH=:/github/workspace/native/python312/work-native/install/usr/local/bin:/github/workspace/spk/python312-wheels/work-x64-7.1/crossenv-default/bin:/github/workspace/distrib/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CMAKE_TOOLCHAIN_FILE= MESON_CROSS_FILE= cross-pip wheel --disable-pip-version-check --no-binary :all: --find-links /github/workspace/distrib/pip --cache-dir /github/workspace/spk/python312-wheels/work-x64-7.1/pip --no-deps --wheel-dir /github/workspace/spk/python312-wheels/work-x64-7.1/wheelhouse --no-index --no-build-isolation numpy==1.26.4
2025-01-27T17:48:42.7155951Z Looking in links: /github/workspace/distrib/pip
2025-01-27T17:48:42.7232864Z Processing /github/workspace/distrib/pip/numpy-1.26.4.tar.gz
2025-01-27T17:48:44.4843397Z   Preparing metadata (pyproject.toml): started
2025-01-27T17:48:46.2578547Z   Preparing metadata (pyproject.toml): finished with status 'error'
2025-01-27T17:48:46.2626588Z   error: subprocess-exited-with-error
2025-01-27T17:48:46.2627031Z   
2025-01-27T17:48:46.2627749Z   × Preparing metadata (pyproject.toml) did not run successfully.
2025-01-27T17:48:46.2628391Z   │ exit code: 1
2025-01-27T17:48:46.2628831Z   ╰─> [19 lines of output]
2025-01-27T17:48:46.2632544Z       + /github/workspace/spk/python312-wheels/work-x64-7.1/crossenv-default/cross/bin/python /tmp/pip-wheel-gygvfwya/numpy_ad90cdcd68dc419da738b74c324aca0c/vendored-meson/meson/meson.py setup /tmp/pip-wheel-gygvfwya/numpy_ad90cdcd68dc419da738b74c324aca0c /tmp/pip-wheel-gygvfwya/numpy_ad90cdcd68dc419da738b74c324aca0c/.mesonpy-z41ci2qh -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-wheel-gygvfwya/numpy_ad90cdcd68dc419da738b74c324aca0c/.mesonpy-z41ci2qh/meson-python-native-file.ini
2025-01-27T17:48:46.2634856Z       The Meson build system
2025-01-27T17:48:46.2635095Z       Version: 1.2.99
2025-01-27T17:48:46.2635446Z       Source dir: /tmp/pip-wheel-gygvfwya/numpy_ad90cdcd68dc419da738b74c324aca0c
2025-01-27T17:48:46.2636036Z       Build dir: /tmp/pip-wheel-gygvfwya/numpy_ad90cdcd68dc419da738b74c324aca0c/.mesonpy-z41ci2qh
2025-01-27T17:48:46.2636500Z       Build type: native build
2025-01-27T17:48:46.2636752Z       Project name: NumPy
2025-01-27T17:48:46.2637002Z       Project version: 1.26.4
2025-01-27T17:48:46.2637766Z       C compiler for the host machine: /github/workspace/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc (gcc 8.5.0 "x86_64-pc-linux-gnu-gcc (GCC) 8.5.0")
2025-01-27T17:48:46.2638817Z       C linker for the host machine: /github/workspace/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc ld.bfd 2.30
2025-01-27T17:48:46.2640109Z       C++ compiler for the host machine: /github/workspace/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ (gcc 8.5.0 "x86_64-pc-linux-gnu-g++ (GCC) 8.5.0")
2025-01-27T17:48:46.2641087Z       C++ linker for the host machine: /github/workspace/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-g++ ld.bfd 2.30
2025-01-27T17:48:46.2641715Z       Cython compiler for the host machine: cython3 (cython 0.29.32)
2025-01-27T17:48:46.2642052Z       Host machine cpu family: x86_64
2025-01-27T17:48:46.2642308Z       Host machine cpu: x86_64
2025-01-27T17:48:46.2642527Z       
2025-01-27T17:48:46.2642826Z       ../meson.build:37:2: ERROR: Problem encountered: NumPy requires Cython >= 0.29.34
2025-01-27T17:48:46.2643182Z       
2025-01-27T17:48:46.2643812Z       A full log can be found at /tmp/pip-wheel-gygvfwya/numpy_ad90cdcd68dc419da738b74c324aca0c/.mesonpy-z41ci2qh/meson-logs/meson-log.txt
2025-01-27T17:48:46.2644861Z       [end of output]
2025-01-27T17:48:46.2645385Z   
2025-01-27T17:48:46.2646054Z   note: This error originates from a subprocess, and is likely not a problem with pip.
2025-01-27T17:48:46.2646891Z error: metadata-generation-failed
2025-01-27T17:48:46.2647073Z 
2025-01-27T17:48:46.2647312Z × Encountered error while generating package metadata.
2025-01-27T17:48:46.2647667Z ╰─> See above for output.
2025-01-27T17:48:46.2647803Z 
2025-01-27T17:48:46.2647965Z note: This is an issue with the package mentioned above, not pip.
2025-01-27T17:48:46.2648286Z hint: See above for details.
2025-01-27T17:48:46.6411000Z make[4]: *** [../../mk/spksrc.wheel-compile.mk:105: cross-compile-wheel-numpy-1.26.4] Error 1
2025-01-27T17:48:46.6413005Z make[3]: *** [../../mk/spksrc.wheel-compile.mk:58: wheel_compile_target] Error 2
2025-01-27T17:48:46.6420060Z make[2]: *** [../../mk/spksrc.wheel.mk:80: build_wheel_target] Error 1
2025-01-27T17:48:46.6420641Z make[2]: Leaving directory '/github/workspace/spk/python312-wheels'
2025-01-27T17:48:46.6442513Z make[1]: *** [../../mk/spksrc.supported.mk:74: build-arch-x64-7.1] Error 1
2025-01-27T17:48:46.6443593Z make[1]: Leaving directory '/github/workspace/spk/python312-wheels'
2025-01-27T17:48:46.6446826Z make: Leaving directory '/github/workspace/spk/python312-wheels'

@th0ma7
Copy link
Contributor

th0ma7 commented Jan 28, 2025

@th0ma7 I am unsure wheter this fix really works, due to the comment you added at the bottom of the Makefile(s)

##
## Addition of requirements files beyond spksrc.python.mk
## will not be taken into consideration for processing
## unless WHEELS has been previously populated. This is due to
## the makefile analysis phase which discard ifeq instructions.
##

This means that if WHEELS is still undefined by that point then it won't honor the following += entries. Feel free to rephrase if you have a better wording in mind.

@th0ma7
Copy link
Contributor

th0ma7 commented Jan 28, 2025 •
edited
Loading

Looking quickly at the container image it does have an older cython version matching your output:

$ dpkg -l | grep -i cython
ii  cython3                           0.29.32-2+b1                    amd64        C-Extensions for Python 3

Further investigating... the spksrc generated crossenv/build/python-cc.mk has empty CMAKE_TOOLCHAIN_FILE and
MESON_CROSS_FILE variables.

@th0ma7
Copy link
Contributor

th0ma7 commented Jan 28, 2025 •
edited
Loading

TL;DR (short version)

@hgy59 we need to distinguish build from cross when generating our dynamic crossenv and adjust PATH accordingly.

long version

Been doing some more reading and I may have gotten this wrong all along... been learning as i go on this whole crossenv thing and something stood out tonight.

prerequisites:
build: where the compiler runs (gcc, make, maturin)
host: where the compiled binaries will run (i.e. $(ARCH))

When creating our crossenv, dating from prior to me afaik, we've always been installing all "environmental" wheels in both:

  • crossenv/build: i.e. build
  • crossenv/cross: i.e. host

This is where we got it wrong: maturin is a build tool and has no reason to be in crossenv/cross. Why? because when sourcing .activate script from the crossenv it will then find a maturin version that cannot run (ex: trying to run an ARMv7 maturin binary on x64).

I recall when I added maturin (as mandatory requirement for many wheels), I ended up adding it everywhere (i.e. build and cross, like usual)... Then also in native (as it wasn't working) which hapened to be at the begining of the PATH variable, thus magically finding a proper maturin version. But if you really think about it, it's not needed at all in native as the crossenv/build directory is the escape plan.

To recap:

  1. source .activate or better, use cross-pip or cross-python3 (already doing that, so one thing solved)
  2. then add upfront crossenv/build to our PATH so our matching python build python compiling tools can be made available from PATH.

This leaves native/python3* which then only need probably pip and crossenv wheels to generate the initial crossenv using a python version exactly matching our cross-compiled version, optionnally possibly wheel and setuptools (if mandatory for the initial crossenv generation).

Then, only mandatory python "dependency" required by the code we're building needs to be part of our crossenv/cross environment to allow cross-compiling. This would obviously incude cffi, cryptography, but also numpy and panda for instance.

With this in mind, our dynamic crossenv requirement definition files should probably look more like this (I did not look into details nor tested any of that):

# build
build:pip==24.3.1
build:pip-tools==7.4.1
build:setuptools==75.8.0
build:build==1.2.2
build:wheel==0.45.1
build:Cython==3.0.11
build:flit==3.10.1
build:maturin==1.8.1
build:meson-python==0.17.1
build:poetry==1.8.5
build:scikit-build==0.18.1
build:scikit-build-core==0.10.7
build:setuptools-rust==1.10.2
build:setuptools-scm==8.1.0
#
# cross
cross:pip==24.3.1
cross:setuptools==75.8.0
cross:wheel==0.45.1
#
# keep older version for gcc <= 4.9.x
cross:msgpack==1.0.5
#
cross:cffi==1.17.1
cross:cryptography==44.0.0
cross:typing_extensions==4.12.2

Lastly, why is this working currently? because we cover most usecases with native maturin and common dependencies available in cross. But any "complex" wheel has good chances to fail, such as numpy.

Good news are:

  • we already have the dynamic crossenv functionality... it's now just a matter a enhancing that a bit.
  • then cleaning-up the PATH management which has always been a bit of voodoo-black-magic in the context of cross-compiling wheels... which now finally looks much more saner with this new understanding.
  • everything else stays pretty much the same as the logic does not change

@th0ma7
Copy link
Contributor

th0ma7 commented Jan 28, 2025

Been thinking this a bit more, a temporary quick fix could be to add the removal of maturin and unneeded wheels from crossenv/cross and readjust PATH.

This could save time until a more complete fix is built under spksrc.crossenv.mk ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hgy59 @th0ma7