[BUG: is the documentation example code outdated?

[JINO-ROHIT]

Python -VV

3.11

Pip Freeze

-

Reproduction Steps

running the example snippet from tokenization guide throws an error - https://docs.mistral.ai/guides/tokenization/

running this line -

tokenizer = MistralTokenizer.v3(is_tekken=True)

throws an error

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 32549: character maps to <undefined>

Expected Behavior

no error

Additional Context

No response

Suggested Solutions

No response

[jeanfmac]

Hi,
Did you receive any kind of feedback or found a solution? I am facing the same issue.
Thanks.

[JINO-ROHIT]

not really, its been a while since i checked back again sorry

[jeanfmac]

Hi,
The issue is related to UTF8.
If you are on windows 10 like me, you can follow this link.
That worked for me.
Best.

[JINO-ROHIT]

whup, thanks jean!