Rust Issue on Unix in 0.21.0 Version

[insculptor]

Hi,

While installing the tokenizers library after 0.21.0 version release, the installation is failing in Unix. (Python 3.9.16)

It is looking for a Rust installation.

12:27:55 Looking in indexes: https://pypi.org/simple
12:27:55 Collecting tokenizers (from -r requirements.txt (line 1))
12:27:56 Downloading https:///pypi/pypi-/tokenizers-0.21.0.tar.gz (343 kB)
12:27:56 Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
Checking for Rust toolchain....
Cargo, the Rust package manager, is not installed or is not on PATH.
This package requires Rust and Cargo to compile extensions. Install it through
the system's package manager or via https://rustup.rs/

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

When I revert to 0.20.3 , it works fine.

[Narsil]

We need a lot more information to understand why that is.

Python will use the source code(and therefore a rust compiler) whenever it fails to resolve the binary.
We support quite a lot of various manylinux versions... https://pypi.org/project/tokenizers/0.21.0/#files

So if yours is not supported/resolved we need as much information as possible on OS, glibc version, python version etc..And also which tool is being used to install the dependency (with it's version).
Anything to your environment that might mess up things, like conda, docker or whatever similar tools.