tokenizer.decode() and tokenizer.convert_ids_to_tokens() return different results

[thangld201]

System Info

In [2]: tokenizers.__version__, transformers.__version__
Out[2]: ('0.21.0', '4.47.0')

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer
token_id=124
in_tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
print([in_tokenizer.convert_ids_to_tokens([token_id])[0], in_tokenizer.decode(token_id)])


token_id=124
in_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
print([in_tokenizer.convert_ids_to_tokens(token_id), in_tokenizer.decode(token_id)])

Expected behavior

# Output
['À', '�']
['À', '�']

# Expected Output
['À', 'À']
['À', 'À']

[sambhavnoobcoder]

this is an issue from the Qwen tokeniser and the discrepancy in the way it treats special characters . I have made a fix and pushed a PR in #35643 . @thangld201 you can give it a look and maintainers @Rocketknight1 @ArthurZucker also please look at it as reviews are appreciated .

[tdard]

As of right now, you will observe the discrepancy only on other characters than utf-8 (here, À is in latin1).

The culprit is the following method of the Qwen2Tokenizer that's being called in the decode method:

    def convert_tokens_to_string(self, tokens):
        """Converts a sequence of tokens (string) in a single string."""
        text = "".join(tokens)
        text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors)
        return text

I'd say this could be fixed by following a similar approach to other tokenizers such as Llama's, but I'm not so sure about the approach. I'd be willing to help if someone knowledgeable could pinpoint whether that's correct or not !

[thangld201]

@sambhavnoobcoder @tdard This issue also happens with Llama 3.1, I updated the example code. Probably also present in other models (but I haven't confirmed yet).

[ArthurZucker]

Hey, this is not an issue, it's expected

[ArthurZucker]

These tokenizers use ByteLevel encoding, meaning the tokens are byte representation of non utf8 characters. SO, when you ask to convert_ids_to_tokens you get the string representation, which is not possible to decode to utf8. THat's why when you ask to decode it fails 😉

[thangld201]

@ArthurZucker Isn't the output misleading ? Those token ids map to valid characters but when decoded they all turned into weird replacement characters.

[ArthurZucker]

They are not valid characters, they are valid UTF-8 representation of characters. But the token is not actually À. I am not sure how missleading it is, we can add documentation about it if you want, but the matter of the fact is that those characters are the same as what b'\x80\x81\x82'.decode(errors="replace") would give: '���'

[thangld201]

@ArthurZucker Thank you for clarifying! But why don't we convert them to utf-8 in .decode() ? Why display '�' ?

[ArthurZucker]

Again, because decode outputs strings not string representation of tokens. If you want the display the representation then you use convert_tokens_to_ids