Code

32 b#121

Code

Draft

dirkgr wants to merge 118 commits intomainallenai/OLMo-core:mainfrom 32Ballenai/OLMo-core:32BCopy head branch name to clipboard

+2,325-139

Commits

Commits on Dec 8, 2024

• Save more often

  dirkgr

  committed
  b94e702

  Copy full SHA for b94e702

• Don't check for cancelation all the time

  dirkgr

  committed
  368abb8

  Copy full SHA for 368abb8

• Make sure we use the same CE loss that we used for the 13B

  dirkgr

  committed
  c277d54

  Copy full SHA for c277d54

• We're going to 5T!

  dirkgr

  committed
  7c74d8b

  Copy full SHA for 7c74d8b

• We can live with a bigger eval batch size.

  dirkgr

  committed
  53d61fe

  Copy full SHA for 53d61fe

Commits on Dec 9, 2024

• Add MMLU downstream eval

  dirkgr

  committed
  514abb8

  Copy full SHA for 514abb8

• Module isn't callable

  dirkgr

  committed
  011113e

  Copy full SHA for 011113e

• Qwen-ish

  dirkgr

  committed
  2577397

  Copy full SHA for 2577397

• Make model bigger

  dirkgr

  committed
  93637a1

  Copy full SHA for 93637a1

Commits on Dec 10, 2024

• It's now a 32B.

  dirkgr

  committed
  784377d

  Copy full SHA for 784377d

• 6T tokens

  dirkgr

  committed
  eec7e10

  Copy full SHA for eec7e10

• Official save folder

  dirkgr

  committed
  bd5edee

  Copy full SHA for bd5edee

• 6.5T tokens

  dirkgr

  committed
  f516f09

  Copy full SHA for f516f09

• Merge remote-tracking branch 'origin/main' into 32B

  dirkgr

  committed
  49264f5

  Copy full SHA for 49264f5

• Merged

  dirkgr

  committed
  4bb5d5c

  Copy full SHA for 4bb5d5c

• Change project name and location

  dirkgr

  committed
  1ff1371

  Copy full SHA for 1ff1371

• Revert "Merged"

  Show description for 4375612
  dirkgr

  committed
  4375612

  Copy full SHA for 4375612

• Revert "Module isn't callable"

  Show description for 20b9b08
  dirkgr

  committed
  20b9b08

  Copy full SHA for 20b9b08

• Revert "Make sure we use the same CE loss that we used for the 13B"

  Show description for 7736198
  dirkgr

  committed
  7736198

  Copy full SHA for 7736198

• We still want it fused!

  dirkgr

  committed
  8e0613f

  Copy full SHA for 8e0613f

• One-in-two activation checkpointing

  dirkgr

  committed
  5652953

  Copy full SHA for 5652953

• Merge remote-tracking branch 'origin/main' into 32B

  dirkgr

  committed
  323c786

  Copy full SHA for 323c786

• Smaller microbatch

  dirkgr

  committed
  4f676e2

  Copy full SHA for 4f676e2

• Wrap 3 in 4 blocks

  dirkgr

  committed
  d4e63fa

  Copy full SHA for d4e63fa

• Don't compile the loss.

  dirkgr

  committed
  7c22386

  Copy full SHA for 7c22386

Commits on Dec 11, 2024

• Turn off broken eval

  dirkgr

  committed
  f38bff4

  Copy full SHA for f38bff4

• Go back to mbsz of 4

  dirkgr

  committed
  3bf2440

  Copy full SHA for 3bf2440

• Set drop_last for DownstreamEvaluator to False

  2015aroras

  committed
  ab5afcf

  Copy full SHA for ab5afcf

• Bring back Copa now that we have Shane's fix

  dirkgr

  committed
  47f9545

  Copy full SHA for 47f9545

• Merge remote-tracking branch 'origin/32B' into 32B

  dirkgr

  committed
  ee6aa90

  Copy full SHA for ee6aa90

• Check if beaker loading issues are due to beaker changes by updating beaker-py version

  2015aroras

  committed
  c656a41

  Copy full SHA for c656a41

• Try hsdp with 2 nodes per replica

  2015aroras

  committed
  7852e1e

  Copy full SHA for 7852e1e

• Revert "Try hsdp with 2 nodes per replica"

  Show description for b19e76d
  2015aroras

  committed
  b19e76d

  Copy full SHA for b19e76d

Commits on Dec 12, 2024

• Try activation checkpointing 3 in 4

  2015aroras

  committed
  a02dd95

  Copy full SHA for a02dd95

• Try activation checkpointing 3 in 4 + all feedforwards checkpointed

  2015aroras

  committed
  6eaa5a3

  Copy full SHA for 6eaa5a3

Commits on Dec 13, 2024

• Decrease microbatch size

  2015aroras

  committed
  b2a07de

  Copy full SHA for b2a07de

• Try activation checkpointing on just feed forwards

  2015aroras

  committed
  9985d31

  Copy full SHA for 9985d31

Commits on Dec 16, 2024

• Fix name

  dirkgr

  committed
  4cc6a62

  Copy full SHA for 4cc6a62

• Try to run with hybrid sharding.

  dirkgr

  committed
  1060499

  Copy full SHA for 1060499

• More batch

  dirkgr

  committed
  fb2a274

  Copy full SHA for fb2a274

• Revert "More batch"

  Show description for 1073613
  dirkgr

  committed
  1073613

  Copy full SHA for 1073613

• There is something wrong with how the common object is set up.

  dirkgr

  committed
  c553b98

  Copy full SHA for c553b98

• We need a less sharded checkpoint and I guess this is the only way we can do it.

  dirkgr

  committed
  e49d4b7

  Copy full SHA for e49d4b7

• Revert "We need a less sharded checkpoint and I guess this is the only way we can do it."

  Show description for 9608482
  dirkgr

  committed
  9608482

  Copy full SHA for 9608482

• Async checkpointer may have problems with large checkpoints?

  dirkgr

  committed
  4804004

  Copy full SHA for 4804004

• For loading checkpoints, it seems we need a longer timeout

  dirkgr

  committed
  fd4edb8

  Copy full SHA for fd4edb8

• Revert "Async checkpointer may have problems with large checkpoints?"

  Show description for 1f79446
  dirkgr

  committed
  1f79446

  Copy full SHA for 1f79446

• Flight to safety

  dirkgr

  committed
  072c616

  Copy full SHA for 072c616

Commits on Dec 17, 2024

• Increase microbatch size up to 2 * 4096

  2015aroras

  committed
  6ba3e23

  Copy full SHA for 6ba3e23

Commits on Dec 18, 2024

• Watching the 32B in a notebook

  dirkgr

  committed
  07cc66c

  Copy full SHA for 07cc66c

• Merge branch '32B' of https://github.com/allenai/OLMo-core into 32B

  dirkgr

  committed
  18e9a32

  Copy full SHA for 18e9a32

Commits on Dec 19, 2024

• Merge branch 'main' into 32B

  2015aroras

  committed
  2150b36

  Copy full SHA for 2150b36

• Enable HSDP with pre-downloading

  2015aroras

  committed
  c8cf403

  Copy full SHA for c8cf403

• Turn off hsdp

  2015aroras

  committed
  d9cb6cf

  Copy full SHA for d9cb6cf

• Revert "Turn off hsdp"

  Show description for 5f2cf19
  2015aroras

  committed
  5f2cf19

  Copy full SHA for 5f2cf19

• Add option to set thread_count

  2015aroras

  committed
  19c8758

  Copy full SHA for 19c8758

• Run formatter

  2015aroras

  committed
  9a12202

  Copy full SHA for 9a12202

• Limit thread count

  2015aroras

  committed
  d5e6e2b

  Copy full SHA for d5e6e2b

• Decrease microbatch size

  2015aroras

  committed
  ea0acce

  Copy full SHA for ea0acce

• Increase microbatch size, increase activation checkpointing

  2015aroras

  committed
  d2a00a7

  Copy full SHA for d2a00a7

Commits on Dec 20, 2024

• Decrease microbatch size

  2015aroras

  committed
  016e426

  Copy full SHA for 016e426

• Decrease thread_count

  2015aroras

  committed
  a28ca37

  Copy full SHA for a28ca37

• Thread count 1

  2015aroras

  committed
  1c33794

  Copy full SHA for 1c33794

• Back to FSDP

  2015aroras

  committed
  484d01c

  Copy full SHA for 484d01c

• Back to HSDP, but with less replicas

  2015aroras

  committed
  275364c

  Copy full SHA for 275364c

• Merge branch 'main' into 32B

  2015aroras

  committed
  54d5623

  Copy full SHA for 54d5623

• Microbatch size back to 1

  2015aroras

  committed
  4644e6e

  Copy full SHA for 4644e6e

• Revert "Microbatch size back to 1"

  Show description for d7ed30e
  2015aroras

  committed
  d7ed30e

  Copy full SHA for d7ed30e

• Back to FSDP

  2015aroras

  committed
  0c47992

  Copy full SHA for 0c47992

• Revert "Back to FSDP"

  Show description for 246eff6
  2015aroras

  committed
  246eff6

  Copy full SHA for 246eff6

• Enable NCCL debug

  2015aroras

  committed
  b956e3f

  Copy full SHA for b956e3f

• More debug info

  2015aroras

  committed
  f877907

  Copy full SHA for f877907

• Merge branch 'main' into 32B

  2015aroras

  committed
  58bef95

  Copy full SHA for 58bef95

• Disable pre_download, set higher thread count

  2015aroras

  committed
  c84708f

  Copy full SHA for c84708f

• FSDP with AC of selected ops

  2015aroras

  committed
  56c4ab3

  Copy full SHA for 56c4ab3

Commits on Dec 21, 2024

• Back to AC of just feedforward layers

  2015aroras

  committed
  b5f3a86

  Copy full SHA for b5f3a86

• Add new inloop evals

  2015aroras

  committed
  3fbdeb0

  Copy full SHA for 3fbdeb0

• Turn off NCCL debug

  2015aroras

  committed
  b335cdf

  Copy full SHA for b335cdf

• Merge branch 'main' into 32B

  2015aroras

  committed
  30f8f59

  Copy full SHA for 30f8f59

Commits on Dec 22, 2024

• Make checkpoint writing respect thread count config

  2015aroras

  committed
  e17e4b8

  Copy full SHA for e17e4b8

• Add skip step optimizer changes

  2015aroras

  committed
  ba49cc4

  Copy full SHA for ba49cc4

• Update 32B config with skip step adamw

  2015aroras

  committed
  25ede33

  Copy full SHA for 25ede33

• Try fix skip step optimizer

  2015aroras

  committed
  ac01e83

  Copy full SHA for ac01e83

• Try manual _std_mean impl

  2015aroras

  committed
  ddd61ac

  Copy full SHA for ddd61ac

• Add skip step fixes

  2015aroras

  committed
  973a26c

  Copy full SHA for 973a26c

• Have separate save and load thread counts

  2015aroras

  committed
  baf5700

  Copy full SHA for baf5700

• Decrease threads used for saving

  2015aroras

  committed
  b6762d8

  Copy full SHA for b6762d8

• Skipped steps and automatic spike analysis

  dirkgr

  committed
  d98f06d

  Copy full SHA for d98f06d

• Use compile=True for optimizer

  2015aroras

  committed
  4a68e9e

  Copy full SHA for 4a68e9e

Commits on Dec 23, 2024

• Make gcs upload pass generation

  2015aroras

  committed
  d81cd12

  Copy full SHA for d81cd12

• Update CHANGELOG

  2015aroras

  committed
  0a04034

  Copy full SHA for 0a04034

• Run formatter

  2015aroras

  committed
  5acc7eb

  Copy full SHA for 5acc7eb

• Make generation 0 when object does not exist

  2015aroras

  committed
  213b03e

  Copy full SHA for 213b03e

• Merge branch 'shanea/fix-upload-retries' into 32B

  2015aroras

  committed
  b4994b0

  Copy full SHA for b4994b0

• Run formatting

  2015aroras

  committed
  3b84351

  Copy full SHA for 3b84351

• Remove unneeded import

  2015aroras

  committed
  178d9ad

  Copy full SHA for 178d9ad

• Add missing reload

  2015aroras

  committed
  0b737aa

  Copy full SHA for 0b737aa

Commits on Dec 27, 2024

• Updated notebook

  dirkgr

  committed
  3e6f9f1

  Copy full SHA for 3e6f9f1

• Updated dashboard

  dirkgr

  committed
  663d63a

  Copy full SHA for 663d63a

• Update the notebook

  dirkgr

  committed
  496919b

  Copy full SHA for 496919b

• Updated notebook

  dirkgr

  committed
  a1854bd

  Copy full SHA for a1854bd

Commits on Dec 28, 2024

• Retry on bad request

  dirkgr

  committed
  f2de5f4

  Copy full SHA for f2de5f4

• Add some more retries

  dirkgr

  committed
  33c0f58

  Copy full SHA for 33c0f58

Commits on Dec 29, 2024

• Updated the notebook

  dirkgr

  committed
  86afc43

  Copy full SHA for 86afc43

Commits on Dec 30, 2024

• Update the dashboard

  dirkgr

  committed
  2e45a79

  Copy full SHA for 2e45a79

Commits on Dec 31, 2024

• Fix the way we use the step in the optimizer

  dirkgr

  committed
  e4e8fbb

  Copy full SHA for e4e8fbb

• Dashboard update

  dirkgr

  committed
  146caaf

  Copy full SHA for 146caaf

Commits on Jan 3, 2025

• Update dashboard

  dirkgr

  committed
  393a462

  Copy full SHA for 393a462

Commits on Jan 6, 2025

• New report

  dirkgr

  committed
  d39c59d

  Copy full SHA for d39c59d

Commits on Jan 7, 2025

• Dashboard update

  dirkgr

  committed
  16983c4

  Copy full SHA for 16983c4

Commits on Jan 8, 2025

• No more ephemeral checkpoints

  dirkgr

  committed
  5e4d04f

  Copy full SHA for 5e4d04f

• Don't eval so much

  dirkgr

  committed
  eba0418

  Copy full SHA for eba0418

• When you wait on someone, you bring them water.

  dirkgr

  committed
  5605001

  Copy full SHA for 5605001

• Updating the dashboard

  dirkgr

  committed
  7ce7efa

  Copy full SHA for 7ce7efa

Commits on Jan 9, 2025

• Reorder ranks in GCP

  dirkgr

  committed
  05aa94f

  Copy full SHA for 05aa94f

• Rank 0 needs to remain rank 0

  dirkgr

  committed
  9c86bf9

  Copy full SHA for 9c86bf9

• Slightly less checkpointing

  dirkgr

  committed
  e27b91d

  Copy full SHA for e27b91d

• Revert "Slightly less checkpointing"

  Show description for 52b9b77
  dirkgr

  committed
  52b9b77

  Copy full SHA for 52b9b77