[Bug] Error occurs when loading in bitsandbytes format.

[upskyy]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 172, in __init__
self.quant_method = quant_config.get_quant_method(self,
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/bitsandbytes.py", line 114, in get_quant_method
return BitsAndBytesLinearMethod(self)
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/bitsandbytes.py", line 141, in __init__
raise ImportError("Please install bitsandbytes>=0.44.0 via "
ImportError: Please install bitsandbytes>=0.44.0 via `pip install bitsandbytes>=0.44.0` to use bitsandbytes quantizer.

Reproduction

python3 -m sglang.launch_server --model-path /models --tokenizer-path /models --host 0.0.0.0 --port 30000 --tokenizer-mode auto --dtype bfloat16 --mem-fraction-static 0.5 --random-seed 0 --enable-torch-compile --disable-cuda-graph --schedule-conservativeness 1.3 --kv-cache-dtype fp8_e5m2 --quantization bitsandbytes --load-format bitsandbytes

Environment

root@33e74a81f115:/sglang/python# python3 -m sglang.check_env                                                                                                                                                         

Python: 3.10.16 (main, Dec  4 2024, 08:53:37) [GCC 9.4.0]
CUDA available: True
GPU 0: NVIDIA A100-SXM4-80GB
GPU 0 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 550.127.05
PyTorch: 2.5.1+cu124
sglang: 0.4.0.post2
flashinfer: 0.1.6+cu124torch2.4
triton: 3.1.0
transformers: 4.47.0
torchao: 0.6.1
numpy: 1.26.4
aiohttp: 3.11.10
fastapi: 0.115.6
hf_transfer: 0.1.8
huggingface_hub: 0.26.3
interegular: 0.3.3
modelscope: 1.21.0
orjson: 3.10.12
packaging: 24.2
psutil: 6.1.0
pydantic: 2.10.3
multipart: 0.0.19
zmq: 26.2.0
uvicorn: 0.32.1
uvloop: 0.21.0
vllm: 0.6.4.post1
openai: 1.57.0
anthropic: 0.40.0
decord: 0.6.0

[upskyy]

I think you just need to update bitsandbytes to 0.45.0.