[Fix metaspace prepending scheme] ⛓️💥⛓️💥 #1568
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
changed the title
ArthurZucker
changed the title
| normalized.prepend(&self.str_rep); | ||
| } | ||
| let result = if self.prepend_scheme == PrependScheme::First { | ||
| let first = &mut pretokenized.splits[0]; |
There was a problem hiding this comment.
This will panic if splits is empty, isn't
pretokenized.split(|_, mut normalized| {
->
pretokenized.split(|i, mut normalized| {
...
if i == 0Enough for a change ?
There was a problem hiding this comment.
Also for good measure, the reason it checks offsets.original == 0 instead of i==0 is because users can send pre pretokenized strings, meanings you can have split n still being "the first token". (Since it's transparent for us at this stage if it was pretiokenized strings from us, or from the users, only the offsets retain that information)
| @@ -493,6 +493,7 @@ def test_splitting(self): | |||
| tokenizer.pre_tokenizer.split = False | |||
| tokenizer.add_tokens([AddedToken("<REPR_END>", rstrip=True, lstrip=True)]) | |||
| assert tokenizer.encode("<REPR_END>inform<s>. Hey. .", add_special_tokens=False).tokens == [ | |||
| "▁", | |||
There was a problem hiding this comment.
This seems like a bug no ?
<REPR_END> is a token on it's own, we shouldn't prepend with _ no ?
Fixes Metaspace again