Installing HDF5.jl on ARM M1 #870

tamasgal · 2021-10-19T15:00:23Z

Using the latest 1.7.0-rc1, HDF5.jl cannot be installed due to a missing binary dependency:

(@v1.7) pkg> add HDF5
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`

(@v1.7) pkg> precompile
Precompiling project...
  ✗ HDF5
  0 dependencies successfully precompiled in 1 seconds (21 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]

Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /Users/tamasgal/.julia/compiled/v1.7/HDF5/jl_wRxaZM.
ERROR: LoadError: UndefVarError: libhdf5 not defined

I am not so familiar with the underlying BinaryBuilder.jl infrastructure, so I wanted to ask around what to do: how can I contribute to https://github.com/JuliaBinaryWrappers/HDF5_jll.jl ?

The text was updated successfully, but these errors were encountered:

musm · 2021-10-19T15:02:43Z

Thanks for the report, the build script is actually https://github.com/JuliaPackaging/Yggdrasil/blob/master/H/HDF5/build_tarballs.jl let me know if you can't find the issue. Otherwise I can look into this as well.

tamasgal · 2021-10-19T15:03:49Z

Yes I just found it out a few seconds ago thanks!

As far as I can see, I only need to create a PR and update the list of platforms. I'll try!

zahachtah · 2021-10-26T09:34:18Z

Is this in the pipeline yet? Only package I use that does not wanna install on M1. If I knew anything about PR's or build scripts I'd love to help...

tamasgal · 2021-10-26T12:18:58Z

I am still struggling to get the compilation script working, help is appreciated ;)

simonbyrne · 2021-10-27T04:43:58Z

HDF5 doesn't support cross-compilation (see JuliaPackaging/Yggdrasil#567), so at the moment the build script just pulls in prebuilt binaries from elsewhere.

musm · 2021-10-27T05:04:27Z

The quickest solution would be for someone on M1 platform to compile the HDF5 library and upload the binaries somewhere so we can add them to https://github.com/JuliaPackaging/Yggdrasil/blob/master/H/HDF5/build_tarballs.jl, until we can sort out the more complicated cross-compilation process.

tamasgal · 2021-10-27T14:13:31Z

I built it on M1 with a standard configure:

http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz
http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz.md5

The MD5 checksum is 5b5085aee04af16c0d906b4b0129dfa7

Here is the configuration:

	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.12.1
                  Configured on: Wed Oct 27 16:00:30 CEST 2021
                  Configured by: tamasgal@silentbox
                    Host system: arm-apple-darwin20.6.0
              Uname information: Darwin silentbox 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm64
                       Byte sex: little-endian
             Installation point: /Users/tamasgal/tmp/hdf5/hdf5-1.12.1/hdf5

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /opt/homebrew/opt/llvm@11/bin/clang
                       CPPFLAGS:
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS:
                        C Flags:
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wno-c++-compat -Wno-format-nonliteral -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var    -Wno-missing-noreturn  -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: no

                           Java: no


Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                Build HDF5 Tests: yes
                Build HDF5 Tools: yes
                    Threadsafety: no (recursive RW locks: no)
             Default API mapping: v112
  With deprecated public symbols: yes
          I/O filters (external): deflate(zlib)
                             MPE: no
                   Map (H5M) API: no
                      Direct VFD: no
                      Mirror VFD: no
              (Read-Only) S3 VFD: no
            (Read-Only) HDFS VFD: no
                         dmalloc: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
                Use file locking: best-effort
       Strict file format checks: no
    Optimization instrumentation: no

musm · 2021-10-27T14:56:10Z

Great thanks @tamasgal, do we know if there's a way to test M1 on CI or at least an M1 VM I or someone else could use to test the precompiled binaries?

tamasgal · 2021-10-27T15:54:45Z

I remember that Stefan Karpinski mentioned in a talk a few months ago that they are using set of Mac Minis for Julia for testing, which is kind of a shame for Apple, but at least a solution.

I cannot provide such a farm but at least I can manually check the test suite until we have a solution for an M1-CI 😕

cortner · 2021-11-25T00:16:33Z

Is there a workaround for this?

EDIT: I can install HDF5 via home-brew, then point HDF5.jl to that installation. But when using HDF5 I get

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: InitError: UndefVarError: libblosc not defined
Stacktrace:
  [1] __init__()
    @ Blosc ~/.julia/packages/Blosc/vjmKP/src/Blosc.jl:6
...
 [15] top-level scope
    @ none:1
during initialization of module Blosc
in expression starting at /Users/ortner/.julia/packages/HDF5/pIJra/src/blosc_filter.jl:1
in expression starting at /Users/ortner/.julia/packages/HDF5/pIJra/src/HDF5.jl:1

Unfortunately I did not figure out how to fix Blosc in a similar way.

abhayap · 2021-12-06T18:47:19Z

I built it on M1 with a standard configure:

http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz.md5

The MD5 checksum is 5b5085aee04af16c0d906b4b0129dfa7

Here is the configuration:

	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.12.1
                  Configured on: Wed Oct 27 16:00:30 CEST 2021
                  Configured by: tamasgal@silentbox
                    Host system: arm-apple-darwin20.6.0
              Uname information: Darwin silentbox 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm64
                       Byte sex: little-endian
             Installation point: /Users/tamasgal/tmp/hdf5/hdf5-1.12.1/hdf5

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /opt/homebrew/opt/llvm@11/bin/clang
                       CPPFLAGS:
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS:
                        C Flags:
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wno-c++-compat -Wno-format-nonliteral -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var    -Wno-missing-noreturn  -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: no

                           Java: no


Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                Build HDF5 Tests: yes
                Build HDF5 Tools: yes
                    Threadsafety: no (recursive RW locks: no)
             Default API mapping: v112
  With deprecated public symbols: yes
          I/O filters (external): deflate(zlib)
                             MPE: no
                   Map (H5M) API: no
                      Direct VFD: no
                      Mirror VFD: no
              (Read-Only) S3 VFD: no
            (Read-Only) HDFS VFD: no
                         dmalloc: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
                Use file locking: best-effort
       Strict file format checks: no
    Optimization instrumentation: no

@tamasgal I have downloaded the binary you built. Thank you! How do I use it so that I can get it working with Julia? Where should I copy it?

This updates c-blosc 1 to 1.21.1: https://github.com/Blosc/c-blosc/releases/tag/v1.21.1 Builds for experimental platforms to resolve macOS on M1. Fix JuliaPackaging#4051 Fix JuliaIO/Blosc.jl#85 Fix JuliaIO/HDF5.jl#870

mkitti · 2021-12-10T03:57:01Z

I'm trying to address the Blosc dependency issues in JuliaPackaging/Yggdrasil#4052

#875 does not load Blosc by default. #880 drops Blosc as HDF5.jl dependency, deferring to H5Zblosc.jl (a subdirectory package).

tamasgal · 2021-12-18T11:30:42Z

This issue is closed but it still does not work. libhdf5 is not found, although HDF5_jll is downloaded...

(@v1.7) pkg> generate foo
  Generating  project foo:
    foo/Project.toml
    foo/src/foo.jl

(@v1.7) pkg> add HDF5
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
   Installed Blosc ───── v0.7.2
   Installed Blosc_jll ─ v1.21.1+0
   Installed Compat ──── v3.41.0
  Downloaded artifact: Blosc
    Updating `~/.julia/environments/v1.7/Project.toml`
  [f67ccb44] + HDF5 v0.15.7
    Updating `~/.julia/environments/v1.7/Manifest.toml`
  [a74b3585] + Blosc v0.7.2
  [34da2185] + Compat v3.41.0
  [f67ccb44] + HDF5 v0.15.7
  [692b3bcd] + JLLWrappers v1.3.0
  [21216c6a] + Preferences v1.2.2
  [0b7ba130] + Blosc_jll v1.21.1+0
  [0234f1f7] + HDF5_jll v1.12.0+1
  [5ced341a] + Lz4_jll v1.9.3+0
  [458c3c95] + OpenSSL_jll v1.1.10+0
  [3161d3a3] + Zstd_jll v1.5.0+0
  [8bb1440f] + DelimitedFiles
  [37e2e46d] + LinearAlgebra
  [a63ad114] + Mmap
  [1a1011a3] + SharedArrays
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [e66e0078] + CompilerSupportLibraries_jll
  [4536629a] + OpenBLAS_jll
  [8e850b90] + libblastrampoline_jll
Precompiling project...
  ✗ HDF5
  3 dependencies successfully precompiled in 2 seconds (18 already precompiled)
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package

(@v1.7) pkg> precompile
Precompiling project...
  ✗ HDF5
  0 dependencies successfully precompiled in 1 seconds (21 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]

Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /Users/tamasgal/.julia/compiled/v1.7/HDF5/jl_quHb64.
ERROR: LoadError: UndefVarError: libhdf5 not defined
Stacktrace:
  [1] top-level scope
    @ ~/.julia/packages/HDF5/pIJra/src/api_types.jl:119
  [2] include(mod::Module, _path::String)
    @ Base ./Base.jl:418
  [3] include(x::String)
    @ HDF5 ~/.julia/packages/HDF5/pIJra/src/HDF5.jl:1
  [4] top-level scope
    @ ~/.julia/packages/HDF5/pIJra/src/HDF5.jl:51
  [5] include
    @ ./Base.jl:418 [inlined]
  [6] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1318
  [7] top-level scope
    @ none:1
  [8] eval
    @ ./boot.jl:373 [inlined]
  [9] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [10] top-level scope
    @ none:1
in expression starting at /Users/tamasgal/.julia/packages/HDF5/pIJra/src/api_types.jl:119
in expression starting at /Users/tamasgal/.julia/packages/HDF5/pIJra/src/HDF5.jl:1

mkitti · 2021-12-19T18:58:36Z

Let's open an issue at https://github.com/JuliaPackaging/Yggdrasil

That is the best place to track this issue.

giordano · 2022-01-10T23:07:10Z

After JuliaPackaging/Yggdrasil#4218

     Testing Running tests...
[ Info: libhdf5 v1.12.1
Test Summary: | Pass  Broken  Total   Time
HDF5.jl       |  752       1    753  21.5s
     Testing HDF5 tests passed

julia> versioninfo()
Julia Version 1.8.0-DEV.1264
Commit 835d3d37d4 (2022-01-09 21:25 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin20.6.0)
  CPU: Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.0 (ORCJIT, cyclone)

abhayap · 2022-01-10T23:16:42Z

Thanks @giordano!

tamasgal · 2022-01-11T10:44:38Z

It works like a charm now, thanks @giordano :)

giordano · 2022-01-11T10:52:57Z

I just realised I accidentally closed this issue by pushing to my fork of Yggdrasil. Thanks GitHub 🙄

musm · 2022-01-11T15:47:20Z

Thanks @giordano

…aPackaging#4052) This updates c-blosc 1 to 1.21.1: https://github.com/Blosc/c-blosc/releases/tag/v1.21.1 Builds for experimental platforms to resolve macOS on M1. Fix JuliaPackaging#4051 Fix JuliaIO/Blosc.jl#85 Fix JuliaIO/HDF5.jl#870

ryan-a-anderson · 2023-12-29T06:25:23Z

Hi – I am still running into the same issue as above using Julia v1.8. "using HDF5" sends the error "UndefVarError: libhdf5 not defined". How can I get around this? On Mac M1

mkitti · 2023-12-29T07:33:08Z

@ryan-a-anderson Please create a new issue. Provide the output of ] status and ] status -m.

sairus7 mentioned this issue Oct 25, 2021

Installing HDF5.jl on armv7l 32 bit #871

Open

tamasgal mentioned this issue Nov 25, 2021

25% faster decompression with cloudflare's zlib fork JuliaHEP/UnROOT.jl#136

Closed

johnnychen94 mentioned this issue Nov 27, 2021

Please add support for Julia 1.7 running on Apple Silicon (ARM64) on MacOS JuliaIO/MAT.jl#168

Closed

mkitti mentioned this issue Dec 10, 2021

Update Blosc1 to version 1.21.1; support experimental platforms JuliaPackaging/Yggdrasil#4052

Merged

giordano closed this as completed in JuliaPackaging/Yggdrasil#4052 Dec 10, 2021

mkitti reopened this Dec 10, 2021

giordano closed this as completed in giordano/Yggdrasil@79392e9 Dec 13, 2021

tamasgal mentioned this issue Dec 23, 2021

libhdf5 not defined (Apple M1) when installing HDF5.jl JuliaPackaging/Yggdrasil#4105

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installing HDF5.jl on ARM M1 #870

Installing HDF5.jl on ARM M1 #870

tamasgal commented Oct 19, 2021

musm commented Oct 19, 2021

tamasgal commented Oct 19, 2021

zahachtah commented Oct 26, 2021

tamasgal commented Oct 26, 2021

simonbyrne commented Oct 27, 2021

musm commented Oct 27, 2021

tamasgal commented Oct 27, 2021

musm commented Oct 27, 2021

tamasgal commented Oct 27, 2021

cortner commented Nov 25, 2021 •

edited

Loading

abhayap commented Dec 6, 2021

mkitti commented Dec 10, 2021

tamasgal commented Dec 18, 2021 •

edited

Loading

mkitti commented Dec 19, 2021

giordano commented Jan 10, 2022

abhayap commented Jan 10, 2022

tamasgal commented Jan 11, 2022

giordano commented Jan 11, 2022

musm commented Jan 11, 2022

ryan-a-anderson commented Dec 29, 2023

mkitti commented Dec 29, 2023

Installing HDF5.jl on ARM M1 #870

Installing HDF5.jl on ARM M1 #870

Comments

tamasgal commented Oct 19, 2021

musm commented Oct 19, 2021

tamasgal commented Oct 19, 2021

zahachtah commented Oct 26, 2021

tamasgal commented Oct 26, 2021

simonbyrne commented Oct 27, 2021

musm commented Oct 27, 2021

tamasgal commented Oct 27, 2021

musm commented Oct 27, 2021

tamasgal commented Oct 27, 2021

cortner commented Nov 25, 2021 • edited Loading

abhayap commented Dec 6, 2021

mkitti commented Dec 10, 2021

tamasgal commented Dec 18, 2021 • edited Loading

mkitti commented Dec 19, 2021

giordano commented Jan 10, 2022

abhayap commented Jan 10, 2022

tamasgal commented Jan 11, 2022

giordano commented Jan 11, 2022

musm commented Jan 11, 2022

ryan-a-anderson commented Dec 29, 2023

mkitti commented Dec 29, 2023

cortner commented Nov 25, 2021 •

edited

Loading

tamasgal commented Dec 18, 2021 •

edited

Loading