Code

Bpe clones#1707

Code

Open

sftse wants to merge 6 commits intohuggingface:mainhuggingface/tokenizers:mainfrom sftse:bpe-clonessftse/tokenizers:bpe-clonesCopy head branch name to clipboard

+49-115

Commits

Commits on Jan 3, 2025

• Dont iterate over chars to determine length

  sftse

  committed
  5947e34

  Copy full SHA for 5947e34

• Lose some unnecessary allocations

  sftse

  committed
  2fabb1f

  Copy full SHA for 2fabb1f

• fmt

  sftse

  committed
  783fe7a

  Copy full SHA for 783fe7a

• Small refactors

  sftse

  committed
  9659474

  Copy full SHA for 9659474

• Remove needless unwrap

  sftse

  committed
  54b1b78

  Copy full SHA for 54b1b78

• Subtyping will autocoerce lifetimes by itself

  sftse

  committed
  7ba6512

  Copy full SHA for 7ba6512