Code

[Feature] Batch Decoding #477

Code

Open

AnaRhisT94 wants to merge 2 commits intostate-spaces:mainstate-spaces/mamba:mainfrom AnaRhisT94:mainAnaRhisT94/mamba:mainCopy head branch name to clipboard

+323-28

Commits

Commits on Jul 18, 2024

• Implemented decoding N tokens, instead of one token after another.

  

  ilanaizelman

  committed
  1fd4882

  Copy full SHA for 1fd4882

• added code which shows that chunked decoding works, and also latency test that it's much faster.

  

  ilanaizelman

  committed
  678ce14

  Copy full SHA for 678ce14