Code

fp8#266

Code

Open

xrsrke wants to merge 46 commits intomainhuggingface/nanotron:mainfrom xrsrke/fp8_for_nanotronhuggingface/nanotron:xrsrke/fp8_for_nanotronCopy head branch name to clipboard

+6,617-887

Commits

Commits on Oct 9, 2024

• move the basics of fp8 to this branch

  xrsrke

  committed
  44e6574

  Copy full SHA for 44e6574

Commits on Oct 28, 2024

• add fp8 tensor

  xrsrke

  committed
  6c9a4d0

  Copy full SHA for 6c9a4d0

• add fp8 linear

  xrsrke

  committed
  0f8f672

  Copy full SHA for 0f8f672

Commits on Oct 29, 2024

• add fp8 tensor parallel

  xrsrke

  committed
  7dfe3ac

  Copy full SHA for 7dfe3ac

Commits on Oct 30, 2024

• add fp8 tp profiler

  xrsrke

  committed
  dda00a4

  Copy full SHA for dda00a4

Commits on Nov 1, 2024

• update profiling script

  xrsrke

  committed
  b4156dc

  Copy full SHA for b4156dc

• remove uncessary .contiguous() in fp8 backward

  xrsrke

  committed
  39a4960

  Copy full SHA for 39a4960

• remove unnecessary .transpose in fp8 linear backward

  xrsrke

  committed
  1ddc44c

  Copy full SHA for 1ddc44c

• remove uncessary transpose input in the fwd pass, and contiguous weight initialization

  xrsrke

  committed
  c827594

  Copy full SHA for c827594

• add bencmark speed with 5% speed up

  xrsrke

  committed
  c937375

  Copy full SHA for c937375

• add speed benchmark

  xrsrke

  committed
  f3e3495

  Copy full SHA for f3e3495

Commits on Nov 3, 2024

• 65% speed up in fwd+bwd pass with m=n=k=32768

  xrsrke

  committed
  4b26cf1

  Copy full SHA for 4b26cf1

• add dumb transpose in fp8_matmul_kernel

  xrsrke

  committed
  edb1e87

  Copy full SHA for edb1e87

Commits on Nov 4, 2024

• remove transpose in kernel

  xrsrke

  committed
  e93cf55

  Copy full SHA for e93cf55

Commits on Nov 18, 2024

• backup before doing monkey dispatch fp8 tp

  xrsrke

  committed
  9510f57

  Copy full SHA for 9510f57

Commits on Nov 20, 2024

• remove fp8 tp from llama's modeling code, fix no grad in param, remove rms norm due to illegal memory

  xrsrke

  committed
  478984a

  Copy full SHA for 478984a

• refactor NanotronParameter to support fp8

  xrsrke

  committed
  c7d9e8a

  Copy full SHA for c7d9e8a

Commits on Nov 21, 2024

• keep FP8 NanotronParameter's dtype in 8 bit, move converting model to fp8 after mark sharded, tied, and parametrization

  xrsrke

  committed
  23d66cb

  Copy full SHA for 23d66cb

• add tests for create_param_that_share_metadata, and generating hash

  xrsrke

  committed
  2864391

  Copy full SHA for 2864391

Commits on Nov 22, 2024

• add fp8 optim init

  xrsrke

  committed
  1800efe

  Copy full SHA for 1800efe

• refactor fp8 linear, tp, parameter tests

  xrsrke

  committed
  c5bcbe7

  Copy full SHA for c5bcbe7

Commits on Nov 25, 2024

• move master weights to gradient accumulator

  xrsrke

  committed
  a4d6f15

  Copy full SHA for a4d6f15

Commits on Nov 28, 2024

• refactor + add test for fp8 initialization

  xrsrke

  committed
  fbbbf4d

  Copy full SHA for fbbbf4d

• new changes

  xrsrke

  committed
  b764b97

  Copy full SHA for b764b97

Commits on Nov 29, 2024

• fix hanging due to NanotronParameter.__repr__ (param.data == NanotronParameter)

  xrsrke

  committed
  afdfbf1

  Copy full SHA for afdfbf1

Commits on Nov 30, 2024

• by default, do not quantize the first and last layer

  xrsrke

  committed
  79341ea

  Copy full SHA for 79341ea

Commits on Dec 18, 2024

• fix nan in fwd pass

  xrsrke

  committed
  b440408

  Copy full SHA for b440408

Commits on Dec 19, 2024

• fix grad_clipping for fp8

  xrsrke

  committed
  4723335

  Copy full SHA for 4723335

Commits on Jan 9, 2025

• fix didn't update fp8 parameters in optim.step() due to grad_accum

  xrsrke

  committed
  dd3259b

  Copy full SHA for dd3259b

• remove ablated fp8 config, and uncessary files/code

  xrsrke

  committed
  a3a13ce

  Copy full SHA for a3a13ce

Commits on Jan 10, 2025

• clean up

  xrsrke

  committed
  e8b114b

  Copy full SHA for e8b114b

Commits on Jan 11, 2025

• add

  xrsrke

  committed
  ebea115

  Copy full SHA for ebea115

• Merge branch 'main' into xrsrke/fp8_for_nanotron

  xrsrke

  authored
  9a99ab6

  Copy full SHA for 9a99ab6

Commits on Jan 13, 2025

• resolve merge conflicts

  xrsrke

  committed
  c0cb423

  Copy full SHA for c0cb423

• add tp_recompute_allgather to column linear

  xrsrke

  committed
  21b2408

  Copy full SHA for 21b2408

Commits on Jan 14, 2025

• fix bias[None, :] in tp's functional

  xrsrke

  committed
  f8c40ad

  Copy full SHA for f8c40ad

• fix test_base_model

  xrsrke

  committed
  3dde0af

  Copy full SHA for 3dde0af

Commits on Jan 15, 2025

• fix tensor slicing in NanotronParameter

  xrsrke

  committed
  31af4f7

  Copy full SHA for 31af4f7

• disable nanoset

  xrsrke

  committed
  678343e

  Copy full SHA for 678343e

Commits on Jan 17, 2025

• fix import te in cicd tests

  xrsrke

  committed
  18ec446

  Copy full SHA for 18ec446

• fix failed fp8 tests

  xrsrke

  committed
  b857a9a

  Copy full SHA for b857a9a

• remove constants.CONFIG and remove manual attn_output dtype conversion to resid dtype

  xrsrke

  committed
  222bd00

  Copy full SHA for 222bd00

• remove fp8 parameters

  xrsrke

  committed
  f70230e

  Copy full SHA for f70230e

• remove importing fp8 modules in fp8's __init__ because it causes cicd to fail

  xrsrke

  committed
  7c7949c

  Copy full SHA for 7c7949c

• untyping tex.DType

  xrsrke

  committed
  d113dda

  Copy full SHA for d113dda

• add back parallel tests

  xrsrke

  committed
  bcf70c4

  Copy full SHA for bcf70c4