Code

training with pytorch lightning#230

Code

Open

ekg wants to merge 71 commits intostate-spaces:mainstate-spaces/mamba:mainfrom ekg:trainingekg/mamba:trainingCopy head branch name to clipboard

+267-5

Commits

Commits on Dec 28, 2023

• adding requirements

  ekg

  committed
  135675d

  Copy full SHA for 135675d

• FSDP training draft

  ekg

  committed
  c6463f2

  Copy full SHA for c6463f2

• debugging

  ekg

  committed
  186bf87

  Copy full SHA for 186bf87

• Revert "debugging"

  Show description for 52828a5
  ekg

  committed
  52828a5

  Copy full SHA for 52828a5

Commits on Dec 29, 2023

• rewrite to use fairscale fsdp

  ekg

  committed
  6900450

  Copy full SHA for 6900450

Commits on Dec 31, 2023

• tweak import

  ekg

  committed
  94490e9

  Copy full SHA for 94490e9

• this doesnt work so we will try with lightning

  ekg

  committed
  07c7661

  Copy full SHA for 07c7661

• rewrite to use lightning

  ekg

  committed
  2162a41

  Copy full SHA for 2162a41

• install pytorch-lightning

  ekg

  committed
  b8d47b0

  Copy full SHA for b8d47b0

• use newer format for call to the trainer

  ekg

  committed
  f6ca729

  Copy full SHA for f6ca729

• require tensorboard

  ekg

  committed
  d85522e

  Copy full SHA for d85522e

• provide loss based on cross entropy loss

  ekg

  committed
  3788cf5

  Copy full SHA for 3788cf5

• use automatic mixed precision

  ekg

  committed
  c63b8b7

  Copy full SHA for c63b8b7

• adjust use of AMP

  ekg

  committed
  8c34d55

  Copy full SHA for 8c34d55

• training appears to work

  ekg

  committed
  64d00ca

  Copy full SHA for 64d00ca

• using all possible contexts

  ekg

  committed
  700d810

  Copy full SHA for 700d810

• add stride parameter

  ekg

  committed
  338b1c4

  Copy full SHA for 338b1c4

Commits on Jan 1, 2024

• simple generation script

  ekg

  committed
  9cd2b7f

  Copy full SHA for 9cd2b7f

• load from checkpoint

  ekg

  committed
  df54375

  Copy full SHA for df54375

• fix import

  ekg

  committed
  bf25e3d

  Copy full SHA for bf25e3d

• not good strategy

  ekg

  committed
  076ea81

  Copy full SHA for 076ea81

• add perplexity logging

  ekg

  committed
  fe888e6

  Copy full SHA for fe888e6

• control output directory and use save_pretrained

  ekg

  committed
  990c564

  Copy full SHA for 990c564

• forward to mixer seq mamba save and load

  ekg

  committed
  8bb6ab8

  Copy full SHA for 8bb6ab8

• hmm

  ekg

  committed
  1ec72c0

  Copy full SHA for 1ec72c0

• add proper model loader

  ekg

  committed
  b3a5120

  Copy full SHA for b3a5120

• use save_pretrained in training

  ekg

  committed
  a48373f

  Copy full SHA for a48373f

• make the model loadable from file

  ekg

  committed
  8781a64

  Copy full SHA for 8781a64

• is it working?

  ekg

  committed
  2a54d2d

  Copy full SHA for 2a54d2d

Commits on Jan 2, 2024

• use real variable

  ekg

  committed
  c1a7bc3

  Copy full SHA for c1a7bc3

• try to get real text

  ekg

  committed
  0b78105

  Copy full SHA for 0b78105

• tweak setup

  ekg

  committed
  14dfaaf

  Copy full SHA for 14dfaaf

• work towards memory mapped input

  ekg

  committed
  d974130

  Copy full SHA for d974130

• try to improve inference with better dynamic range

  ekg

  committed
  70aec11

  Copy full SHA for 70aec11

• extreme version of dynamic range in inference

  ekg

  committed
  bbdcd2e

  Copy full SHA for bbdcd2e

• there was no obvious difference

  ekg

  committed
  cf012f9

  Copy full SHA for cf012f9

• FSDP strategy to shard model

  ekg

  committed
  5097260

  Copy full SHA for 5097260

• correct typo

  ekg

  committed
  87079a9

  Copy full SHA for 87079a9

• oops

  ekg

  committed
  20d5422

  Copy full SHA for 20d5422

• try tokenizer parallelism

  ekg

  committed
  e93ed26

  Copy full SHA for e93ed26

Commits on Jan 3, 2024

• save full checkpoints

  ekg

  committed
  4816409

  Copy full SHA for 4816409

• add saving method for working with fsdp trainer

  ekg

  committed
  71a0fd7

  Copy full SHA for 71a0fd7

Commits on Jan 4, 2024

• use the fsdp saver

  ekg

  committed
  e551b80

  Copy full SHA for e551b80

• fix save pattern to use the right function

  ekg

  committed
  55c5e99

  Copy full SHA for 55c5e99

• be explicit

  ekg

  committed
  314ca5a

  Copy full SHA for 314ca5a

• try to save a chkpt differently

  ekg

  committed
  b1ddc30

  Copy full SHA for b1ddc30

• hmm

  ekg

  committed
  24acea9

  Copy full SHA for 24acea9

• well that was useless

  ekg

  committed
  4307181

  Copy full SHA for 4307181

• well...

  ekg

  committed
  bfd7625

  Copy full SHA for bfd7625

• opps

  ekg

  committed
  07673b7

  Copy full SHA for 07673b7

• what happens

  ekg

  committed
  7f953c6

  Copy full SHA for 7f953c6

• manual mode save

  ekg

  committed
  c9814c4

  Copy full SHA for c9814c4

• ok

  ekg

  committed
  28e0bc1

  Copy full SHA for 28e0bc1

Commits on Jan 5, 2024

• un-oops

  ekg

  committed
  67d5fa2

  Copy full SHA for 67d5fa2

• why not

  ekg

  committed
  744c181

  Copy full SHA for 744c181

• why not

  ekg

  committed
  ae30ae3

  Copy full SHA for ae30ae3

• magari

  ekg

  committed
  4caf9b3

  Copy full SHA for 4caf9b3

• rewrite the state dict because of stuff

  ekg

  committed
  d11121c

  Copy full SHA for d11121c

• import ordereddict

  ekg

  committed
  1b9250c

  Copy full SHA for 1b9250c

• try to build a 2.8b model with fsdp

  ekg

  committed
  d6e79c5

  Copy full SHA for d6e79c5

• does this decrease memory

  ekg

  committed
  99778ca

  Copy full SHA for 99778ca

• not really

  ekg

  committed
  4d69658

  Copy full SHA for 4d69658

• print stuff out

  ekg

  committed
  ec8abec

  Copy full SHA for ec8abec

• cleanup print statements

  ekg

  committed
  7359088

  Copy full SHA for 7359088

• count tokens/s

  ekg

  committed
  61f7fb3

  Copy full SHA for 61f7fb3

• in time

  ekg

  committed
  28b06f1

  Copy full SHA for 28b06f1

Commits on Jan 26, 2024

• rewrite to be byte level

  ekg

  committed
  f302984

  Copy full SHA for f302984

• little cleanup

  ekg

  committed
  cfdcb9b

  Copy full SHA for cfdcb9b

• use the word "context" rather than block

  ekg

  committed
  4151fda

  Copy full SHA for 4151fda

Commits on Feb 28, 2024

• Merge branch 'main' of https://github.com/state-spaces/mamba into training

  ekg

  committed
  869c957

  Copy full SHA for 869c957

Commits on Mar 8, 2024

• Merge branch 'main' of https://github.com/state-spaces/mamba into training

  ekg

  committed
  e93dfc7

  Copy full SHA for e93dfc7