Using GPT2Tokenizer convert Arabic characters to symbols

[AbeerAbuZayed]

I am trying to use (aragpt2-base) for Arabic text classification task as follows:

tokenizer = GPT2Tokenizer.from_pretrained(model_path)
model = GPT2ForSequenceClassification.from_pretrained(model_path, num_labels)

When I use the tokenizer it converts the Arabic characters to symbols like this ĠÙĥتÙĬر. Any advice would be appreciated.

[WissamAntoun]

This is normal don't worry about it, GPT2 tokenizer works on a byte-level so it will treat each character as a 4 bytes, so printing it as it is won't work.

[AbeerAbuZayed]

Doesn't this impact model performance?

[WissamAntoun]

No, this is how the model was pretrained

[AbeerAbuZayed]

Great! Another question, please. Since it does classification on the last token and the padding is to the left, does it require to add any special token to the input sequence?

[WissamAntoun]

Hmm i don't think so. But you can try it maybe it makes a difference.

[AbeerAbuZayed]

I have tried both, with and without adding bos_token and eos_token to the input sequence. In both cases, the results are very low! I think there is something wrong happening, I am trying to figure it out.

Many thanks for all of your answers.