Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't build on an M1/ARM Mac #80

Open
2 tasks done
cpaika opened this issue Jan 19, 2024 · 6 comments
Open
2 tasks done

Can't build on an M1/ARM Mac #80

cpaika opened this issue Jan 19, 2024 · 6 comments

Comments

@cpaika
Copy link

cpaika commented Jan 19, 2024

What happens?

I'm attempting to clone this repo to make changes. I cloned the repo, and initialized the duckdb submodule. When I run make it fails 82% of the way in building the parquet duckdb extension:

[ 82%] Built target tpch_loadable_extension
[ 82%] Linking CXX shared library parquet.duckdb_extension
Undefined symbols for architecture arm64:
  "_parquet_init", referenced from:
     -exported_symbol[s_list] command line option
  "_parquet_version", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [extension/parquet/parquet.duckdb_extension] Error 1
make[2]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [release] Error 2

The same error occurs when I run GEN=ninja make as well.

To Reproduce

Clone this repository, initialize the submodule, and run make

OS:

Mac OS X

SQLite Version:

Default

DuckDB Version:

main

DuckDB Client:

N/A

Full Name:

Christopher Paika

Affiliation:

Tesla

Have you tried this on the latest main branch?

  • I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

  • I agree
@cpaika cpaika changed the title Can't build on an M1/Arm Mac Can't build on an M1/ARM Mac Jan 19, 2024
@Mytherin
Copy link
Contributor

Thanks for the report! I'm running on an M1 mac as well and not running into this problem - are you perhaps on an older MacOS/compiler version or using a custom/different compiler?

$> clang++ --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Alternatively do you perhaps have any compilation flags defined in your env that could change symbol visibility/linking behavior?

@cpaika
Copy link
Author

cpaika commented Jan 20, 2024

Upgraded to Sonoma to match your clang++ version, still getting the exception unfortunately:

Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

My InstalledDir is different, unsure if that matters if the versions are the same.

It seems like the problem is building the duckdb submodule, is there anything special I have to do to that other than check it out?

@Mytherin
Copy link
Contributor

Hm, odd. I still suspect there's something different about the configuration/environment setup that causes the change in behavior. As a work-around you could try uncommenting this line which is likely where the problem originates from.

@breckcs
Copy link

breckcs commented Jan 21, 2024

I work for the same organization as @cpaika, so I have a similar M1 Mac setup. I can build duckdb (build/release/duckdb) and the sqlite_scanner extension (build/release/extension/sqlite_scanner/sqlite_scanner.duckdb_extension). My build fails later linking the parquet.duckdb_extension:

[ 91%] Linking CXX shared library parquet.duckdb_extension
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
ld: Undefined symbols:
  _parquet_init, referenced from:
      <initial-undefines>
  _parquet_version, referenced from:
      <initial-undefines>
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/build.make:721: extension/parquet/parquet.duckdb_extension] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:7770: extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
gmake[1]: *** [Makefile:136: all] Error 2
gmake[1]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
make: *** [release] Error 2

The steps I used:

  1. git clone https://github.com/duckdb/sqlite_scanner.git
  2. cd sqlite_scanner
  3. git submodule init
  4. git submodule update
  5. make

My clang++ version:

% clang++ --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

@cpaika, the difference in InstalledDir may be because you don't have Xcode installed.

@carlopi
Copy link
Contributor

carlopi commented Jan 25, 2024

I think this might come from the fact that somehow the linker links against a [lib]parquet_extension.so or some similar problem connected to dynamic libraries vs static libraries.

Can you try check something what the command ls build/relese/**/*.so returns? Or add some more logging to the build?

@breckcs
Copy link

breckcs commented Jan 25, 2024

If I comment out this line, it builds.

That line was originally introduced by @jkub here, then modified by @Mytherin here then here.

With CMAKE_VERBOSE_MAKEFILE set to ON, the output when the build works:

[ 91%] Linking CXX shared library parquet.duckdb_extension
cd /Users/cbreck/github/sqlite_scanner/build/release/extension/parquet && /opt/homebrew/Cellar/cmake/3.28.1/bin/cmake -E cmake_link_script CMakeFiles/parquet_loadable_extension.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -O3 -DNDEBUG   -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -dynamiclib -Wl,-headerpad_max_install_names -o parquet.duckdb_extension -install_name @rpath/parquet.duckdb_extension CMakeFiles/parquet_loadable_extension.dir/column_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/column_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_crypto.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_extension.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_metadata.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_statistics.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_timestamp.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/serialize_parquet.cpp.o CMakeFiles/parquet_loadable_extension.dir/zstd_file_system.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_constants.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_types.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/protocol/TProtocol.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TTransportException.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TBufferTransports.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy.cc.o "CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy-sinksource.cc.o" CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_ddict.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/huf_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress_block.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/entropy_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/fse_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/zstd_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/error_private.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/xxhash.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/fse_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/hist.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/huf_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_literals.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_sequences.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_superblock.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_double_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_lazy.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_ldm.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_opt.cpp.o  ../../src/libduckdb_static.a -Wl,-dead_strip ../tpch/libtpch_extension.a libparquet_extension.a ../../src/libduckdb_static.a ../tpch/libtpch_extension.a libparquet_extension.a ../../third_party/fsst/libduckdb_fsst.a ../../third_party/fmt/libduckdb_fmt.a ../../third_party/libpg_query/libduckdb_pg_query.a ../../third_party/re2/libduckdb_re2.a ../../third_party/miniz/libduckdb_miniz.a ../../third_party/utf8proc/libduckdb_utf8proc.a ../../third_party/hyperloglog/libduckdb_hyperloglog.a ../../third_party/fastpforlib/libduckdb_fastpforlib.a ../../third_party/skiplist/libduckdb_skiplistlib.a ../../third_party/mbedtls/libduckdb_mbedtls.a
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
gmake[3]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'

And when it fails:

[ 91%] Linking CXX shared library parquet.duckdb_extension
cd /Users/cbreck/github/sqlite_scanner/build/release/extension/parquet && /opt/homebrew/Cellar/cmake/3.28.1/bin/cmake -E cmake_link_script CMakeFiles/parquet_loadable_extension.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -O3 -DNDEBUG   -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -dynamiclib -Wl,-headerpad_max_install_names -o parquet.duckdb_extension -install_name @rpath/parquet.duckdb_extension CMakeFiles/parquet_loadable_extension.dir/column_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/column_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_crypto.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_extension.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_metadata.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_reader.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_statistics.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_timestamp.cpp.o CMakeFiles/parquet_loadable_extension.dir/parquet_writer.cpp.o CMakeFiles/parquet_loadable_extension.dir/serialize_parquet.cpp.o CMakeFiles/parquet_loadable_extension.dir/zstd_file_system.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_constants.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/parquet/parquet_types.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/protocol/TProtocol.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TTransportException.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/thrift/thrift/transport/TBufferTransports.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy.cc.o "CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/snappy/snappy-sinksource.cc.o" CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_ddict.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/huf_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/decompress/zstd_decompress_block.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/entropy_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/fse_decompress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/zstd_common.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/error_private.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/common/xxhash.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/fse_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/hist.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/huf_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_literals.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_sequences.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_compress_superblock.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_double_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_fast.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_lazy.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_ldm.cpp.o CMakeFiles/parquet_loadable_extension.dir/__/__/third_party/zstd/compress/zstd_opt.cpp.o  ../../src/libduckdb_static.a -Wl,-dead_strip -Wl,-exported_symbol,_parquet_init -Wl,-exported_symbol,_parquet_version -Wl,-exported_symbol,_parquet_storage_ini* ../tpch/libtpch_extension.a libparquet_extension.a ../../src/libduckdb_static.a ../tpch/libtpch_extension.a libparquet_extension.a ../../third_party/fsst/libduckdb_fsst.a ../../third_party/fmt/libduckdb_fmt.a ../../third_party/libpg_query/libduckdb_pg_query.a ../../third_party/re2/libduckdb_re2.a ../../third_party/miniz/libduckdb_miniz.a ../../third_party/utf8proc/libduckdb_utf8proc.a ../../third_party/hyperloglog/libduckdb_hyperloglog.a ../../third_party/fastpforlib/libduckdb_fastpforlib.a ../../third_party/skiplist/libduckdb_skiplistlib.a ../../third_party/mbedtls/libduckdb_mbedtls.a
ld: warning: ignoring duplicate libraries: '../../src/libduckdb_static.a', '../tpch/libtpch_extension.a', 'libparquet_extension.a'
ld: Undefined symbols:
  _parquet_init, referenced from:
      <initial-undefines>
  _parquet_version, referenced from:
      <initial-undefines>
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[3]: *** [extension/parquet/CMakeFiles/parquet_loadable_extension.dir/build.make:724: extension/parquet/parquet.duckdb_extension] Error 1
gmake[3]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
gmake[2]: *** [CMakeFiles/Makefile2:7773: extension/parquet/CMakeFiles/parquet_loadable_extension.dir/all] Error 2
gmake[2]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
gmake[1]: *** [Makefile:139: all] Error 2
gmake[1]: Leaving directory '/Users/cbreck/github/sqlite_scanner/build/release'
make: *** [release] Error 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants