Skip to content

Commit

Permalink
fix last truncated token
Browse files Browse the repository at this point in the history
  • Loading branch information
janbar committed Dec 19, 2023
1 parent 002e4fa commit d2b2c2c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion noson/src/private/tokenizer.h
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ inline void __tokenize(const std::string& str, const char *delimiters, std::vect
}
while (trimnull && str.find_first_of(delimiters, pb) == pb);
}
if (!trimnull || pa < (str.size() - 1))
if (!trimnull || pa < str.size())
tokens.push_back(str.substr(pa));
}

Expand Down

0 comments on commit d2b2c2c

Please sign in to comment.