Code

32B with Hybrid Sharding#134

Code

Draft

dirkgr wants to merge 74 commits into32Ballenai/OLMo-core:32Bfrom dirkg/32BHybridallenai/OLMo-core:dirkg/32BHybridCopy head branch name to clipboard

+304-76

Commits

Commits on Dec 23, 2024

• Experiments with hybrid sharding

  dirkgr

  committed
  1a6adde

  Copy full SHA for 1a6adde

• Updated notebook

  dirkgr

  committed
  638d8b2

  Copy full SHA for 638d8b2

• One replica per node

  dirkgr

  committed
  6cea547

  Copy full SHA for 6cea547

Commits on Dec 24, 2024

• Two nodes

  dirkgr

  committed
  bbe0d98

  Copy full SHA for bbe0d98

• Updated dashboard

  dirkgr

  committed
  4a567d6

  Copy full SHA for 4a567d6

• Load optimizer and model in the other order

  dirkgr

  committed
  8c666a5

  Copy full SHA for 8c666a5

• Update the notebook

  dirkgr

  committed
  31555fb

  Copy full SHA for 31555fb

Commits on Dec 27, 2024

• Updated notebook

  dirkgr

  committed
  1c4ae16

  Copy full SHA for 1c4ae16

Commits on Dec 30, 2024

• Merge remote-tracking branch 'origin/32B' into dirkg/32BHybrid

  dirkgr

  committed
  d315524

  Copy full SHA for d315524

• Problem doesn't repro with four nodes

  dirkgr

  committed
  4f85f4d

  Copy full SHA for 4f85f4d

• SkipAdam optimizer has problems during startup?

  dirkgr

  committed
  2ce06ae

  Copy full SHA for 2ce06ae

• Go back to FSDP to see if we can survive more than 9 steps

  dirkgr

  committed
  1558306

  Copy full SHA for 1558306

Commits on Jan 3, 2025

• Merge remote-tracking branch 'origin/32B' into dirkg/32BHybrid

  dirkgr

  committed
  6e92ed2

  Copy full SHA for 6e92ed2

• Turn hybrid back on

  dirkgr

  committed
  fe2bb8b

  Copy full SHA for fe2bb8b

• Get faster answers

  dirkgr

  committed
  068b96e

  Copy full SHA for 068b96e

• More informative error message

  dirkgr

  committed
  56be57c

  Copy full SHA for 56be57c

• Forgot about GPUs

  dirkgr

  committed
  ad773fe

  Copy full SHA for ad773fe

• Try it with more nodes

  dirkgr

  committed
  d8c2d2a

  Copy full SHA for d8c2d2a

Commits on Jan 4, 2025

• Put SkipStep back

  dirkgr

  committed
  f751c77

  Copy full SHA for f751c77

Commits on Jan 5, 2025

• Don't have that many nodes available now

  dirkgr

  committed
  388f30c

  Copy full SHA for 388f30c

Commits on Jan 6, 2025

• Merge remote-tracking branch 'origin/32B' into dirkg/32BHybrid

  dirkgr

  committed
  79d3370

  Copy full SHA for 79d3370

• Adds a way to load unsharded checkpoints

  dirkgr

  committed
  4cae22b

  Copy full SHA for 4cae22b

• Turn off compiling the optimizer

  dirkgr

  committed
  a2b9bdd

  Copy full SHA for a2b9bdd

• Checkpoint directories work different now

  dirkgr

  committed
  b5aa0f8

  Copy full SHA for b5aa0f8

Commits on Jan 7, 2025

• 30 minute timeout

  dirkgr

  committed
  9f65f58

  Copy full SHA for 9f65f58

• Support key_mapping when loading unsharded checkpoints

  dirkgr

  committed
  ed5a94c

  Copy full SHA for ed5a94c

• Don't broadcast from 0, because apparently that hangs

  dirkgr

  committed
  94a08bc

  Copy full SHA for 94a08bc

• More logging

  dirkgr

  committed
  fff3276

  Copy full SHA for fff3276

• I think this is always None, but I want to be sure.

  dirkgr

  committed
  549693b

  Copy full SHA for 549693b

• Better log message

  dirkgr

  committed
  18a96f4

  Copy full SHA for 18a96f4

• Even better messaging

  dirkgr

  committed
  98a5a2d

  Copy full SHA for 98a5a2d

• Log all threads before downloading

  dirkgr

  committed
  0b957c6

  Copy full SHA for 0b957c6

• Let's see download progress

  dirkgr

  committed
  df62dd5

  Copy full SHA for df62dd5

• Make printing stack traces more readable

  dirkgr

  committed
  9afadd2

  Copy full SHA for 9afadd2

• Don't log all threads

  dirkgr

  committed
  0b8f047

  Copy full SHA for 0b8f047

• 15 minute timeout

  dirkgr

  committed
  7166a09

  Copy full SHA for 7166a09

• Revert "Let's see download progress"

  Show description for f4f2a58
  dirkgr

  committed
  f4f2a58

  Copy full SHA for f4f2a58

• Use the rank we mean

  dirkgr

  committed
  67547b5

  Copy full SHA for 67547b5

• Finalize config after applying overrides

  dirkgr

  committed
  56c53b2

  Copy full SHA for 56c53b2

• Download checkpoint before doing anything else

  dirkgr

  committed
  16d99b4

  Copy full SHA for 16d99b4

• We don't have gsutil in the path.

  dirkgr

  committed
  8e32582

  Copy full SHA for 8e32582

• Install google cloud CLI in the Docker container

  dirkgr

  committed
  2e976a7

  Copy full SHA for 2e976a7

• Installing this way doesn't work

  dirkgr

  committed
  9457dcc

  Copy full SHA for 9457dcc

Commits on Jan 8, 2025

• Less output from gsutil

  dirkgr

  committed
  a93d01d

  Copy full SHA for a93d01d

• Disable evals so we can experiment faster

  dirkgr

  committed
  899686a

  Copy full SHA for 899686a

• Revert "Disable evals so we can experiment faster"

  Show description for 50f60c3
  dirkgr

  committed
  50f60c3

  Copy full SHA for 50f60c3

• Adds an unsharding script

  dirkgr

  committed
  0caa78a

  Copy full SHA for 0caa78a

• Adds gcloud utilities to the image

  dirkgr

  committed
  05979c2

  Copy full SHA for 05979c2

• Merge branch 'dirkg/32BHybrid' of https://github.com/allenai/OLMo-core into dirkg/32BHybrid

  dirkgr

  committed
  034dea9

  Copy full SHA for 034dea9

• Bring back selective activation checkpointing

  dirkgr

  committed
  24db150

  Copy full SHA for 24db150

• Merge branch 'dirkg/32BHybrid' of https://github.com/allenai/OLMo-core into dirkg/32BHybrid

  dirkgr

  committed
  82dd576

  Copy full SHA for 82dd576

• Checkpoint more stuff

  dirkgr

  committed
  7d0b268

  Copy full SHA for 7d0b268

• Fix directory naming

  dirkgr

  committed
  e55b84b

  Copy full SHA for e55b84b

• Initialize modules that are wrapped

  dirkgr

  committed
  51ae3a4

  Copy full SHA for 51ae3a4

• More informative error

  dirkgr

  committed
  a2f010e

  Copy full SHA for a2f010e

• Build parameter groups even with fancy activation checkpointing schemes

  dirkgr

  committed
  8d69ce1

  Copy full SHA for 8d69ce1

• Don't preserve randomness

  dirkgr

  committed
  c13e4cf

  Copy full SHA for c13e4cf

• We can use globbing to checkpoint

  dirkgr

  committed
  e39a11c

  Copy full SHA for e39a11c

• Switch back to full sharding

  dirkgr

  committed
  aace4af

  Copy full SHA for aace4af

• More precise checkpointing

  dirkgr

  committed
  c79584e

  Copy full SHA for c79584e

• Make a baseline with 16 nodes

  dirkgr

  committed
  e1f5956

  Copy full SHA for e1f5956

Commits on Jan 9, 2025

• Checkpoint attention

  dirkgr

  committed
  af57282

  Copy full SHA for af57282

• Checkpoint some more

  dirkgr

  committed
  1759a53

  Copy full SHA for 1759a53

• Revert "Checkpoint some more"

  Show description for 5395984
  dirkgr

  committed
  5395984

  Copy full SHA for 5395984

• Revert "Checkpoint attention"

  Show description for 76db353
  dirkgr

  committed
  76db353

  Copy full SHA for 76db353

• Reorder ranks in GCP

  dirkgr

  committed
  50d93cf

  Copy full SHA for 50d93cf

• Rank 0 needs to remain rank 0

  dirkgr

  committed
  3a6ebff

  Copy full SHA for 3a6ebff

Commits on Jan 14, 2025

• Merge remote-tracking branch 'origin/32B' into dirkg/32BHybrid

  dirkgr

  committed
  c7b616b

  Copy full SHA for c7b616b

• Back to hybrid

  dirkgr

  committed
  5cf48f5

  Copy full SHA for 5cf48f5

• Settle in for a lot of experiments

  dirkgr

  committed
  0723199

  Copy full SHA for 0723199

• Let's make it fast first

  dirkgr

  committed
  f3a7466

  Copy full SHA for f3a7466

• Start from nothing

  dirkgr

  committed
  d8aa5da

  Copy full SHA for d8aa5da

• Let's load from somewhere then

  dirkgr

  committed
  f4aec4c

  Copy full SHA for f4aec4c

• Unused import

  dirkgr

  committed
  ea14f94

  Copy full SHA for ea14f94