[Question] what to do when model doesn't have tokenizer.model?

[steveepreston]

while tokenizer.model is required in yaml config, but there are many models that doesn't have tokenizer.model (example: unsloth/Llama-3.2-1B)

In these cases, how can we use tokenizer.json or tokenizer_config.json that are included in almost all model instead of tokenizer.model?

[RdoubleA]

In your case specifically, you can use the original Llama 3.2 1B tokenizer.model from https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct (if the unsloth version is based off the instruct model, use the base one otherwise). If unsloth modified any of the special tokens, then you will need a new tokenizer.model.

I don't believe you can load in the tokenizer without the tokenizer.model file, because it contains the BPE encoding itself.

[steveepreston]

@RdoubleA Thanks for explain, got the case.
I list some other random models that doesn't have a tokenizer.model:

deepseek-ai/DeepSeek-V3
Qwen/QVQ
nvidia/Llama-3.1-Nemotron
openai/gpt2
mistralai/Mistral-Nemo
CohereForAI/c4ai
facebook/opt-125m

I don't have any idea to what should be done here.