Code

Metal: Improved reduce and softmax#1819

Code

Open

ivarflakstad wants to merge 38 commits intomainhuggingface/candle:mainfrom ivarflakstad/metal-reduce-3huggingface/candle:ivarflakstad/metal-reduce-3Copy head branch name to clipboard

+1,502-334

Commits

Commits on Jan 22, 2024

• Improve reduce perf and add contiguous impl

  ivarflakstad

  committed
  d590284

  Copy full SHA for d590284

• Improve arg reduce and add contiguous impl

  ivarflakstad

  committed
  1f4c544

  Copy full SHA for 1f4c544

• Improve softmax kernel. 33%-39% higher thrpt

  ivarflakstad

  committed
  2056866

  Copy full SHA for 2056866

• Merge branch 'main' into ivarflakstad/metal-reduce-2

  ivarflakstad

  committed
  086b6ef

  Copy full SHA for 086b6ef

• fmt

  ivarflakstad

  committed
  077e781

  Copy full SHA for 077e781

Commits on Jan 30, 2024

• Fixed all bugs. Improved code quality. Added tests.

  ivarflakstad

  committed
  8babfe0

  Copy full SHA for 8babfe0

Commits on Feb 7, 2024

• Stash for debugging

  ivarflakstad

  committed
  cf92c96

  Copy full SHA for cf92c96

• Stash for debugging 2

  ivarflakstad

  committed
  3bc4fcb

  Copy full SHA for 3bc4fcb

• Fixing argmax bug and improve performance

  Show description for 2ee1a0c
  

  ivarflakstad

  and

  FL33TW00D

  committed
  2ee1a0c

  Copy full SHA for 2ee1a0c

Commits on Feb 9, 2024

• Fix test and add is_valid_simgroup_reduce_type trait

  ivarflakstad

  committed
  a3fc6c4

  Copy full SHA for a3fc6c4

Commits on Feb 11, 2024

• Online softmax. Improved threadgroup reduce. Tidying up a bit.

  ivarflakstad

  committed
  db08d66

  Copy full SHA for db08d66

• Remove redundant threadgroup_barrier from arg reduce

  ivarflakstad

  committed
  1f63401

  Copy full SHA for 1f63401

Commits on Feb 13, 2024

• Mostly tidying up. Some improvements

  ivarflakstad

  committed
  dec2db6

  Copy full SHA for dec2db6

• Simplify indexed struct

  ivarflakstad

  committed
  237369c

  Copy full SHA for 237369c

• tidying

  ivarflakstad

  committed
  bcdbcd1

  Copy full SHA for bcdbcd1

• Reuse operation operator instead of passing it in as a parameter

  ivarflakstad

  committed
  286598a

  Copy full SHA for 286598a

• Fix how operators are applied to indexed<vec<T,N>>

  ivarflakstad

  committed
  14652a6

  Copy full SHA for 14652a6

Commits on Feb 20, 2024

• Vectorized load. Scalar block reduce. Hitting max throughput for f32 reduce.

  ivarflakstad

  committed
  c0d80b4

  Copy full SHA for c0d80b4

• Vectorized load for online softmax. Involves a reinterpret_cast of src which may be suboptimal.

  ivarflakstad

  committed
  2c78cab

  Copy full SHA for 2c78cab

Commits on Feb 26, 2024

• Metal as_type casting vec<bfloat, N> -> vec<float, N/2> for simd and fast math

  ivarflakstad

  committed
  847b34f

  Copy full SHA for 847b34f

• Use constant for input instead of const device. Fix strided reduce.

  ivarflakstad

  committed
  70dd35f

  Copy full SHA for 70dd35f

• Use contiguous reduce in tests

  ivarflakstad

  committed
  8ca2c03

  Copy full SHA for 8ca2c03

• Rename finalize -> to_scalar

  ivarflakstad

  committed
  a3a5994

  Copy full SHA for a3a5994

Commits on Feb 28, 2024

• Support integer types max/min (switch with trait-inferred impl later)

  ivarflakstad

  committed
  9b54e41

  Copy full SHA for 9b54e41

Commits on Mar 7, 2024

• Was worried I was skipping work -> shuffling the 1D test cases

  ivarflakstad

  committed
  3633135

  Copy full SHA for 3633135

• Add build.rs to avoid metal kernel jit compile overhead

  ivarflakstad

  committed
  a9e26b5

  Copy full SHA for a9e26b5

Commits on Mar 8, 2024

• Improve build. Extract utils

  ivarflakstad

  committed
  8186cee

  Copy full SHA for 8186cee

Commits on Sep 2, 2024

• Compile metal kernels for both macos and ios

  ivarflakstad

  committed
  568b543

  Copy full SHA for 568b543

• Merge branch 'main' into ivarflakstad/metal-reduce-3

  ivarflakstad

  committed
  afa1ea1

  Copy full SHA for afa1ea1

Commits on Jan 13, 2025

• Fixed over xmas and then forgot about it

  ivarflakstad

  committed
  c3e4da7

  Copy full SHA for c3e4da7

• Merge branch 'main' into ivarflakstad/metal-reduce-3

  ivarflakstad

  committed
  5c59669

  Copy full SHA for 5c59669

• Add calculate_reduce_threads util

  ivarflakstad

  committed
  bcbbad6

  Copy full SHA for bcbbad6

• Remove old reduce.metal

  ivarflakstad

  committed
  e8499c8

  Copy full SHA for e8499c8

Commits on Jan 20, 2025

• Improve f16/bf16 softmax precision by accumulating in f32

  ivarflakstad

  committed
  4c94925

  Copy full SHA for 4c94925

• Remove build.rs (for now)

  ivarflakstad

  committed
  eb6985e

  Copy full SHA for eb6985e

• Move softmax bench to candle-nn

  ivarflakstad

  committed
  b094d09

  Copy full SHA for b094d09

• Remove redundant thread calc util fn

  ivarflakstad

  committed
  14cbd5e

  Copy full SHA for 14cbd5e

Commits on Jan 21, 2025

• Merge remote-tracking branch 'origin/main' into ivarflakstad/metal-reduce-3

  LaurentMazare

  committed
  c174d5b

  Copy full SHA for c174d5b