Llama3.1-8b with LoRa: This model does not support adapter loading.

[ilyalasy]

System Info

Docker image: ghcr.io/huggingface/text-generation-inference:2.2.0
GPU: T4
OS: Ubuntu 20.04.6 LTS

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

My env.list:

HF_MODEL_ID=meta-llama/Meta-Llama-3.1-8B
HF_TOKEN=<TOKEN>
QUANTIZE=bitsandbytes
LORA_ADAPTERS=<path/to/custom/adapter>
USE_FLASH_ATTENTION=False

Command:

docker run --gpus all --shm-size 1g -p 8080:80 --env-file ./env.list -v $PWD/data:/data \
    ghcr.io/huggingface/text-generation-inference:2.2.0 --model-id meta-llama/Meta-Llama-3.1-8B

Error:

Traceback (most recent call last):

  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())

  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 118, in serve
    server.serve(

  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 297, in serve
    asyncio.run(

  File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)

  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()

  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 256, in serve_inner
    model.load_adapter(

  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/model.py", line 198, in load_adapter
    raise ValueError("This model does not support adapter loading.")

ValueError: This model does not support adapter loading.

Expected behavior

Expected to successfully run the TGI server.