faiss fails to import on python==3.12 because of deprecated numpy.distutils
#3936
Comments
13 tasks
|
acked, Nix is not the official supported way of installing faiss. If you need to be unblocked quickly, can you try installing faiss from conda-forge? I will check if https://anaconda.org/anaconda/py-cpuinfo can be used to replace the deprecated module |
I am not a There is no particular urgency don't worry. |
|
I am also facing this exact same issue using Debian. |
|
@cachitas is this on x86 or aarch64? |
|
aarch64 |
|
This doesn't appear to be aarch64 specific: based on the numpy documentation it applies to all numpy uses in Python 3.12 and newer. |
asadoughi
changed the title
faiss fails to import on python==3.12 & aarch64-linux because of deprecated numpy.distutilsfaiss fails to import on python==3.12 because of deprecated numpy.distutils
|
i have this issue: Traceback (most recent call last):
File "/home/vboxuser/projects/chatbot-ia/index.py", line 3, in <module>
app = create_app()
^^^^^^^^^^^^
File "/home/vboxuser/projects/chatbot-ia/src/__init__.py", line 9, in create_app
from .routes import rag,db
File "/home/vboxuser/projects/chatbot-ia/src/routes/db.py", line 3, in <module>
import faiss
File "/home/vboxuser/projects/chatbot-ia/env/lib/python3.12/site-packages/faiss/__init__.py", line 16, in <module>
from .loader import *
File "/home/vboxuser/projects/chatbot-ia/env/lib/python3.12/site-packages/faiss/loader.py", line 88, in <module>
instruction_sets = supported_instruction_sets()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vboxuser/projects/chatbot-ia/env/lib/python3.12/site-packages/faiss/loader.py", line 53, in supported_instruction_sets
if is_sve_supported():
^^^^^^^^^^^^^^^^^^
File "/home/vboxuser/projects/chatbot-ia/env/lib/python3.12/site-packages/faiss/loader.py", line 43, in is_sve_supported
import numpy.distutils.cpuinfo
ModuleNotFoundError: No module named 'numpy.distutils' with arm64 |
|
I have the same issue when trying to use faiss on an arm linux platform and numpy==1.26.0. I think the problematic import |
8 tasks
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 14, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 15, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 15, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 15, 2025
…#4126) Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - Python upgraded to 3.12 because otherwise we fail with the below error, and I thought we might want to keep telemetry? Let me know otherwise ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-anaconda-telemetry-0.1.1-py312h06a4308_0 requires python >=3.12,<3.13.0a0, but none of the providers can be installed Could not solve for environment specs The following packages are incompatible ├─ conda-anaconda-telemetry is installable and it requires │ └─ python >=3.12,<3.13.0a0 , which can be installed; └─ python 3.11** is not installable because it conflicts with any installable versions previously reported. ``` - Solver changes: conda/conda-libmamba-solver#283 (comment) We need to manually install libmamba from conda-forge instead of defaults, then use it. If it and libarchive come from different channels, build fails with: ``` Error while loading conda entry point: conda-libmamba-solver (libarchive.so.20: cannot open shared object file: No such file or directory) CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic ``` or the very cryptic: (it refers to glibc version...) ``` Your installed version is: 2.35 ``` - Why not use the classic solver, why do we need the above change to keep using libmamba? classic is not able to resolve dependencies for us. Others suggest just to use libmamba: conda/conda-build#5516 (comment). Classic fails with ``` DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'__glibc', '__archspec', '__unix', '__conda', '__linux'} ``` - cpuinfo.py copied in because Python 3.12 removes distutils, and setuptools does not have feature parity. People have complained that they can't use FAISS with Python > 3.11 before and this is the dependency that I see. facebookresearch#3936 - cuda-toolkit upgraded to 12.5.1 because of this error on 12.4.1: ``` nvlink fatal : Input file '/home/runner/miniconda3/bin/../lib/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (126 vs 124) ``` - Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 19, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 20, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 20, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 20, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 20, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 20, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 20, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this issue
Jan 21, 2025
Summary: Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: this build kept trying to reference 11.2 when 14.2 was installed. Current builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - conda_build_config.yaml: I removed Python 3.12 because the build fails with the below error. Python 3.12 also doesn't really work (facebookresearch#3936) I think? So I removed it. ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` - Note: test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Differential Revision: D68043874
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
To know whether the system supports SVE, faiss uses deprecated
numpy.distutils.cpuinfo.This has been removed and crashes on Python 3.12 (on
aarch64-linuxsystems) with:Platform
OS: NixOS
Faiss version: 1.9.0
Installed from: Nix
Faiss compilation options:
Running on:
Interface:
Reproduction instructions
import faissThe text was updated successfully, but these errors were encountered: