Error on importing bonsai on Mac M2/arm

[chefkoch-de42]

Hi,
I run in import errors when trying to use bonsai lib on MacBookPro M2 (arm9). Tested with 3.9,3.10 and 3.11 Trackback is pasted below.
If I run the same import in Rosetta2 (x86) emulation environment it works perfectly.

Thanks for taking care

python 3.9

$ python
Python 3.9.16 (main, Dec  7 2022, 10:06:04)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bonsai
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/admin/tmp/venv/lib/python3.9/site-packages/bonsai/__init__.py", line 3, in <module>
    from .ldapconnection import LDAPConnection
  File "/Users/admin/tmp/venv/lib/python3.9/site-packages/bonsai/ldapconnection.py", line 5, in <module>
    from bonsai._bonsai import ldapconnection, ldapsearchiter
ImportError: dlopen(/Users/admin/tmp/venv/lib/python3.9/site-packages/bonsai/_bonsai.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_ldap_create_passwordpolicy_control'

$ pip3 freeze | grep bonsai
bonsai==1.5.1

python 3.10

$ python
Python 3.10.10 (main, Feb 16 2023, 02:49:39) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bonsai
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/admin/tmp/venv/lib/python3.10/site-packages/bonsai/__init__.py", line 3, in <module>
    from .ldapconnection import LDAPConnection
  File "/Users/admin/tmp/venv/lib/python3.10/site-packages/bonsai/ldapconnection.py", line 5, in <module>
    from bonsai._bonsai import ldapconnection, ldapsearchiter
ImportError: dlopen(/Users/admin/tmp/venv/lib/python3.10/site-packages/bonsai/_bonsai.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_ldap_create_passwordpolicy_control'

$ pip3 freeze | grep bonsai
bonsai==1.5.1

python 3.11

$ python
Python 3.11.2 (main, Feb 16 2023, 02:55:59) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bonsai
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/admin/tmp/venv/lib/python3.11/site-packages/bonsai/__init__.py", line 3, in <module>
    from .ldapconnection import LDAPConnection
  File "/Users/admin/tmp/venv/lib/python3.11/site-packages/bonsai/ldapconnection.py", line 5, in <module>
    from bonsai._bonsai import ldapconnection, ldapsearchiter
ImportError: dlopen(/Users/admin/tmp/venv/lib/python3.11/site-packages/bonsai/_bonsai.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ber_pvt_opt_on'

$ pip3 freeze | grep bonsai
bonsai==1.5.1

[noirello]

Hi, in the case of 3.9 and 3.10: there aren't any wheels for macOS ARM64. You have to build the module from source. Make sure you follow the instructions, and that your shared lib (_bonsai.*.dylib) is linked to the correct (brew installed) openldap libraries. The simple pip install probably built the module from source as well, but linked the library to the system-vendored old libldap libraries.

In the case of 3.11 is more interesting. Apparently, the uploaded wheel is marked as universal2. Something has changed in that Python version, but I've got my doubts that without any other configurations, it could create a valid wheel. I recommend to build from source in this case also, juts like previously for 3.9 and 3.10.

[chefkoch-de42]

HI, thx for the quick answer.
Building the lib for 3.10 from worked for me and I was able to run my program.
With 3.11 I was able to import bonsai lib in python console via "import bonsai" without errors. Testing my prog does not work, because of other dependency problems.

Regards

[cwlls]

I am also attempting to install bonsai from source on my Mac M1. I am pretty new to this process, but even following the instructions I end up with the same error message:

bonsai-1.5.1-py3.10-macosx-13.4-arm64.egg/bonsai/_bonsai.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_ldap_create_passwordpolicy_control'

My setup.cfg file contains the following (my homebrew prefix is /opt/homebrew):

[build_ext]
library_dirs = /opt/homebrew/opt/openldap/lib
include_dirs = /opt/homebrew/opt/openldap/include

Running otool -L lists the following, which makes me think It's still linking to the system LDAP library:

iroh@shaw bonsai-1.5.1 % otool -L build/lib.macosx-13.4-arm64-cpython-310/bonsai/_bonsai.cpython-310-darwin.so 
build/lib.macosx-13.4-arm64-cpython-310/bonsai/_bonsai.cpython-310-darwin.so:
	/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

Any help would be greatly appreciated, thanks!
Chris

[cwlls]

After some messing around with this for a while I discovered that I was using a pyenv Python installation instead of the one installed with homebrew. After switching back to the Homebrew installed Python I was able to recompile/reinstall and things look proper and work correctly.

Thanks for all your work!
Chris Wells

[Niccolum]

Hello!

Is it possible to create wheels for macos arm64 and upload there to pypi? Or, because of old openldap it is not possible?

[noirello]

Hi,

The main issue that I don't have a M1 Mac at my disposal, and neither GH Actions nor Azure Pipelines supports M1 VM-s right now, AFAIK. This makes building an M1 compatible wheel difficult, especially testing it.

[cwlls]

I have an M1 Mac and a vested interest in getting an M1 compatible wheel going. Let me know if you want some help building/testing.

Hi,

The main issue that I don't have a M1 Mac at my disposal, and neither GH Actions nor Azure Pipelines supports M1 VM-s right now, AFAIK. This makes building an M1 compatible wheel difficult, especially testing it.

[noirello]

I tried to create universal2 wheel with cibuildwheel, but the required OpenSSL and OpenLDAP libraries installed with brew doesn't support universal builds.

[JacobCoffee]

Hi, GH actions now supports Arm based runners

• https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/
• https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/
• https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

is it possible to have this pushed through?

[noirello]

Yes, of course.

It's already done on the dev branch, actually. I just haven't got the time to create a new release. I'll try to make time for it this weekend.

[noirello]

The new release has been made. 1.5.3 is uploaded to PyPI.

[Tikrong]

Faced the same issue (symbol not found in flat namespace '_ldap_create_passwordpolicy_control') today with python 3.9 on macos with m1 processor. Tried installing bonsai 1.5.3 with pip and from source, installing openldap and changing .cfg file as written in the docs. Still, when I client I get error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tikrong/Desktop/projects/ad_api/utilities/ad.py", line 2, in <module>
    from bonsai import LDAPClient
  File "/Users/tikrong/Desktop/projects/ad_api/.venv/lib/python3.9/site-packages/bonsai/__init__.py", line 3, in <module>
    from .ldapconnection import LDAPConnection
  File "/Users/tikrong/Desktop/projects/ad_api/.venv/lib/python3.9/site-packages/bonsai/ldapconnection.py", line 5, in <module>
    from bonsai._bonsai import ldapconnection, ldapsearchiter
ImportError: dlopen(/Users/tikrong/Desktop/projects/ad_api/.venv/lib/python3.9/site-packages/bonsai/_bonsai.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_ldap_create_passwordpolicy_control'

Would appreciate any help

[noirello]

Hi, there's no wheel for Python 3.9 on M1. The Python 3.9 on the new m1 github runners are unavailable.
You need to build it on your own. Here are some instructions how to do that.

[Tikrong]

I tried installing it via pip on python 3.10, also tried building from source using instructions, but it doesn't work, get the same error:
ImportError: dlopen(/Users/tikrong/Desktop/projects/study/.venv/lib/python3.10/site-packages/bonsai/_bonsai.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_ldap_create_passwordpolicy_control'

Forgot to mention last time, that I use pyenv to manage python versions. Tried to install in virtual environment and and outside of it, had no luck. Ready to try new approaches and help fix this issue if I can.

[noirello]

I don't know what goes wrong there. I looks like to me, that even the 3.10 tries to locally compile the module from source.
Tried to check the PyPI uploaded wheels for 1.5.3, but I couldn't install docker on the macos gh runners. So I just installed and imported the module without running the test cases, and that went without errors.

You should check the _bonsai.cpython-310-darwin.so with otool, the output should be something similar to this:

otool -L /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/bonsai/_bonsai.cpython-310-darwin.so
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/bonsai/_bonsai.cpython-310-darwin.so:
	@loader_path/.dylibs/libldap.2.dylib (compatibility version 3.0.0, current version 3.200.0)
	@loader_path/.dylibs/liblber.2.dylib (compatibility version 3.0.0, current version 3.200.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)

[PRESIDENT810]

Faced the same issue (symbol not found in flat namespace '_ldap_create_passwordpolicy_control') today with python 3.9 on macos with m1 processor. Tried installing bonsai 1.5.3 with pip and from source, installing openldap and changing .cfg file as written in the docs. Still, when I client I get error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tikrong/Desktop/projects/ad_api/utilities/ad.py", line 2, in <module>
    from bonsai import LDAPClient
  File "/Users/tikrong/Desktop/projects/ad_api/.venv/lib/python3.9/site-packages/bonsai/__init__.py", line 3, in <module>
    from .ldapconnection import LDAPConnection
  File "/Users/tikrong/Desktop/projects/ad_api/.venv/lib/python3.9/site-packages/bonsai/ldapconnection.py", line 5, in <module>
    from bonsai._bonsai import ldapconnection, ldapsearchiter
ImportError: dlopen(/Users/tikrong/Desktop/projects/ad_api/.venv/lib/python3.9/site-packages/bonsai/_bonsai.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_ldap_create_passwordpolicy_control'

Would appreciate any help

I think the trick for fixing _ldap_create_passwordpolicy_control is to link to a LDAP library with this symbol. I just checked my M3 Mac's library:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ldap.h this file has the declaration of function _ldap_create_passwordpolicy_control

but if you check /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP.tbd you'll find it doesn't actually have this symbol. (.tbd is the text-based stub file for the actually dylib which is probably not visible to users).

So I assume if you are linking to MacOS's LDAP library, you'll never fix this issue since this symbol never exists even if header file includes it, which makes it a undefined symbol. You probably need to link to other LDAP libraries with this symbol present