Deepseek-v2

[ehartford]

Model description

deepseek-v2.5 doesnt work

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

https://huggingface.co/deepseek-ai/DeepSeek-V2.5-1210

[drbh]

Hi @ehartford thanks for opening this issue, would you mind sharing the TGI version you're using?

It seems that deepseek 2.5 uses the deepseek_v2 architecture as shown in the config.json: https://huggingface.co/deepseek-ai/DeepSeek-V2.5-1210/blob/6f134cbe88cb9284a8ce696e8ac8eefd0bc24ede/config.json#L23

Since deepseek_v2 support was added in version v2.2.0 as of this PR #2224 the newer model should load and avoid the error shown above.

**If using a Docker container I'd make sure to avoid using :latest and pin the specific version like ghcr.io/huggingface/text-generation-inference:3.0.1 only because latest is opaque and often the cause of versioning issues