Code

Refactor pre tokenization tool#219

Code

Open

eliebak wants to merge 7 commits intohuggingface:mainhuggingface/nanotron:mainfrom eliebak:refactor-pre-tokenization-tooleliebak/nanotron:refactor-pre-tokenization-toolCopy head branch name to clipboard

+115-81

Commits

Commits on Aug 18, 2024

• add Shuffle and ParquetReader support

  eliebak

  committed
  126a90b

  Copy full SHA for 126a90b

Commits on Aug 21, 2024

• add slurm support and refactor the code

  eliebak

  committed
  5f5f2f1

  Copy full SHA for 5f5f2f1

Commits on Aug 22, 2024

• no email in local pipeline

  eliebak

  committed
  5a038cb

  Copy full SHA for 5a038cb

Commits on Aug 23, 2024

• fix: cpus-per-task typo

  eliebak

  committed
  03f2c5d

  Copy full SHA for 03f2c5d

Commits on Aug 29, 2024

• add mem_per_cpu_gb flag

  eliebak

  committed
  bb61d8d

  Copy full SHA for bb61d8d

• fix typo readers => reader

  eliebak

  committed
  5dc5861

  Copy full SHA for 5dc5861

• fix if => elif

  eliebak

  committed
  7502bae

  Copy full SHA for 7502bae