Code

[Feature] Infini Attention#169

Code

Open

xrsrke wants to merge 43 commits intomainhuggingface/nanotron:mainfrom xrsrke/infini_attention_this_actually_workshuggingface/nanotron:xrsrke/infini_attention_this_actually_worksCopy head branch name to clipboard

+50,854-251

Commits

Commits on Apr 22, 2024

• add retrieving memory

  xrsrke

  committed
  2ab9858

  Copy full SHA for 2ab9858

• add computing the output

  xrsrke

  committed
  7659415

  Copy full SHA for 7659415

Commits on Apr 23, 2024

• add linear delta memory update

  xrsrke

  committed
  936a45b

  Copy full SHA for 936a45b

• fix computing global attention

  xrsrke

  committed
  8f3a61e

  Copy full SHA for 8f3a61e

• fix

  xrsrke

  committed
  d681ee5

  Copy full SHA for d681ee5

Commits on Apr 27, 2024

• support gqa and new theta

  xrsrke

  committed
  74545cc

  Copy full SHA for 74545cc

• support loading hf ckp

  xrsrke

  committed
  30b95a6

  Copy full SHA for 30b95a6

• tested, llama3 wiht infini, get 5 loss

  xrsrke

  committed
  fef75ff

  Copy full SHA for fef75ff

Commits on May 1, 2024

• add gqa optional

  xrsrke

  committed
  234d33d

  Copy full SHA for 234d33d

Commits on May 7, 2024

• fix v minus 1 in memory update

  xrsrke

  committed
  2f50a7b

  Copy full SHA for 2f50a7b

• noy apply pe to memory

  xrsrke

  committed
  597ff44

  Copy full SHA for 597ff44

• use this for exp19

  xrsrke

  committed
  2e74dd6

  Copy full SHA for 2e74dd6

• use this for exp19

  xrsrke

  committed
  e36ed87

  Copy full SHA for e36ed87

Commits on May 14, 2024

• don't change the max positional encodings

  xrsrke

  committed
  440e674

  Copy full SHA for 440e674

• add generating training data

  xrsrke

  committed
  63cdce5

  Copy full SHA for 63cdce5

Commits on May 15, 2024

• generate finetuning data with a target context length

  xrsrke

  committed
  f62e1a5

  Copy full SHA for f62e1a5

Commits on May 16, 2024

• fix splitting sequence

  xrsrke

  committed
  b76c61c

  Copy full SHA for b76c61c

Commits on May 21, 2024

• add logging the internals of neural network

  xrsrke

  committed
  f2c1b91

  Copy full SHA for f2c1b91

Commits on May 23, 2024

• add monitor

  xrsrke

  committed
  d1d54f1

  Copy full SHA for d1d54f1

Commits on May 25, 2024

• fix sequence splitting

  xrsrke

  committed
  c57a3e9

  Copy full SHA for c57a3e9

Commits on May 27, 2024

• fix decode for non-megatron-sp

  xrsrke

  committed
  05c48b3

  Copy full SHA for 05c48b3

• backup with new sp splitting, but wrong reduction dimension in infini attention

  xrsrke

  committed
  f3a39b1

  Copy full SHA for f3a39b1

Commits on May 29, 2024

• backup

  xrsrke

  committed
  b1deec5

  Copy full SHA for b1deec5

• save run evals and generate data

  xrsrke

  committed
  4e18fac

  Copy full SHA for 4e18fac

Commits on May 31, 2024

• save generation

  xrsrke

  committed
  97d3861

  Copy full SHA for 97d3861

• add specifying segment length, ctx length, and turn on/off memory from config

  xrsrke

  committed
  5a15f2e

  Copy full SHA for 5a15f2e

• support turning off memory

  xrsrke

  committed
  6273da1

  Copy full SHA for 6273da1

Commits on Jun 7, 2024

• fix modeling code for bs = 1

  xrsrke

  committed
  18b3ef3

  Copy full SHA for 18b3ef3

Commits on Jun 11, 2024

• fix wrong interdimate activations after splitting qkv with bs > 1

  xrsrke

  committed
  692b052

  Copy full SHA for 692b052

Commits on Jun 12, 2024

• code use for exp 33

  xrsrke

  committed
  328fb71

  Copy full SHA for 328fb71

Commits on Jun 13, 2024

• fix inference

  xrsrke

  committed
  838cce6

  Copy full SHA for 838cce6

Commits on Jun 19, 2024

• refactor passkey retrivial data generation script

  xrsrke

  committed
  25f1f35

  Copy full SHA for 25f1f35

Commits on Jun 20, 2024

• refactor for generating finetuning data and run evals

  xrsrke

  committed
  dcbe86e

  Copy full SHA for dcbe86e

Commits on Jun 25, 2024

• add debugging

  xrsrke

  committed
  a6b8b5e

  Copy full SHA for a6b8b5e

• add l2 nor, kurtosis, abmax, weight update magnitude

  xrsrke

  committed
  15fd9dd

  Copy full SHA for 15fd9dd

Commits on Jun 28, 2024

• each segment in the attention layer uses the same balance factors

  xrsrke

  committed
  9ef1f4c

  Copy full SHA for 9ef1f4c

Commits on Jun 30, 2024

• support custom balance factor's act func, init

  xrsrke

  committed
  a91d1fe

  Copy full SHA for a91d1fe

Commits on Jul 3, 2024

• it works, backup for exp 57

  xrsrke

  committed
  962a8e1

  Copy full SHA for 962a8e1

• backup

  xrsrke

  committed
  03b43ea

  Copy full SHA for 03b43ea

Commits on Jul 29, 2024

• backup before cleaning

  xrsrke

  committed
  8aa0ee5

  Copy full SHA for 8aa0ee5

Commits on Aug 2, 2024

• clean up the core parts

  xrsrke

  committed
  75485da

  Copy full SHA for 75485da

• clean

  xrsrke

  committed
  3f860fd

  Copy full SHA for 3f860fd

• clean

  xrsrke

  committed
  0d99cfb

  Copy full SHA for 0d99cfb