Code

[wip] Entmax loss#3

Code

Open

bpopeters wants to merge 26 commits intomaindeep-spin/Megatron-DeepSpeed:mainfrom entmax-lossdeep-spin/Megatron-DeepSpeed:entmax-lossCopy head branch name to clipboard

+456-79

Commits

Commits on Jan 9, 2024

• add entmax loss training for gpt-like models

  bpopeters

  committed
  8543c0e

  Copy full SHA for 8543c0e

• add entmax gpt example for debugging

  bpopeters

  committed
  d3be645

  Copy full SHA for d3be645

Commits on Jan 25, 2024

• fix loss indentation problem that breaks entmax training

  bpopeters

  committed
  f946b8f

  Copy full SHA for f946b8f

• remove nonfunctional, unused, and unnecessary shell script

  bpopeters

  committed
  1c76bf8

  Copy full SHA for 1c76bf8

Commits on Mar 21, 2024

• compute support during training (although do not log it yet)

  bpopeters

  committed
  563f968

  Copy full SHA for 563f968

Commits on Mar 27, 2024

• fix model training bug, accumulate support across gpus

  bpopeters

  committed
  7228a8b

  Copy full SHA for 7228a8b

Commits on Apr 1, 2024

• slightly more informative support logging

  bpopeters

  committed
  202caed

  Copy full SHA for 202caed

Commits on May 20, 2024

• update return_support to return_support_size in all cases

  bpopeters

  committed
  8a83cf0

  Copy full SHA for 8a83cf0

Commits on May 22, 2024

• [wip] compute more sparsity stats

  bpopeters

  committed
  48a1781

  Copy full SHA for 48a1781

• add more support logging statistics

  bpopeters

  committed
  19b478f

  Copy full SHA for 19b478f

Commits on Jun 26, 2024

• add multiple loss functions (with some print statements to make sure my assumptions of dimensions are correct)

  bpopeters

  committed
  c4cf60c

  Copy full SHA for c4cf60c

• add force_decoded_accuracy as a zero-shot evaluation metric

  bpopeters

  committed
  a02cf72

  Copy full SHA for a02cf72

• update logging with teacher forced accuracy

  bpopeters

  committed
  a8733c4

  Copy full SHA for a8733c4

• update xavier_uniform init (which will probably be unused

  bpopeters

  committed
  e51656d

  Copy full SHA for e51656d

• avoid unpacking error when labels is None

  bpopeters

  committed
  055f869

  Copy full SHA for 055f869

• remove print statements for eval computation

  bpopeters

  committed
  e24ebc6

  Copy full SHA for e24ebc6

• call item() because json cannot handle tensors

  bpopeters

  committed
  f77cf0c

  Copy full SHA for f77cf0c

Commits on Jun 27, 2024

• add accuracy at k

  bpopeters

  committed
  68a4a1b

  Copy full SHA for 68a4a1b

• fix guardrail for new task

  bpopeters

  committed
  24a5ed3

  Copy full SHA for 24a5ed3

• small change to try to get LAMBADA to run

  bpopeters

  committed
  77fe108

  Copy full SHA for 77fe108

• don't load optimizer state for lambada

  bpopeters

  committed
  c41f8b0

  Copy full SHA for c41f8b0

• write lambada to results file

  bpopeters

  committed
  4cfc133

  Copy full SHA for 4cfc133

Commits on Jun 28, 2024

• fix entmax_bisect_loss return type

  bpopeters

  committed
  0182532

  Copy full SHA for 0182532

Commits on Jul 12, 2024

• refactor zeroshot gpt evaluation for sparsemax score

  bpopeters

  committed
  9e7f53d

  Copy full SHA for 9e7f53d

• add missing parentheses

  bpopeters

  committed
  f3c66c4

  Copy full SHA for f3c66c4

• get item from sparsemax score tensor

  bpopeters

  committed
  4f2c4fc

  Copy full SHA for 4f2c4fc