Code

Draft backtrack#1712

Code

Draft

ArthurZucker wants to merge 11 commits intomainhuggingface/tokenizers:mainfrom backtrackhuggingface/tokenizers:backtrackCopy head branch name to clipboard

+2,282-17

Commits

Commits on Jan 3, 2025

• initial-commit

  ArthurZucker

  committed
  ac46243

  Copy full SHA for ac46243

• update test

  ArthurZucker

  committed
  3e884ac

  Copy full SHA for 3e884ac

• update benches as well

  ArthurZucker

  committed
  0907a4d

  Copy full SHA for 0907a4d

• nits

  ArthurZucker

  committed
  dca4568

  Copy full SHA for dca4568

• add no pretokenizer bench

  ArthurZucker

  committed
  d334fb4

  Copy full SHA for d334fb4

• push latest changes

  ArthurZucker

  committed
  a3ed0c3

  Copy full SHA for a3ed0c3

Commits on Jan 4, 2025

• updates

  ArthurZucker

  committed
  7c9e534

  Copy full SHA for 7c9e534

• nits

  ArthurZucker

  committed
  ee18ba9

  Copy full SHA for ee18ba9

Commits on Jan 6, 2025

• update serialization to support initializing from BPE

  ArthurZucker

  committed
  4b63a7a

  Copy full SHA for 4b63a7a

• nits

  ArthurZucker

  committed
  a7baf1b

  Copy full SHA for a7baf1b

Commits on Jan 21, 2025

• current state

  ArthurZucker

  committed
  224f432

  Copy full SHA for 224f432