Code

Compression support#79

Code

Merged

KillingSpark merged 136 commits intomasterKillingSpark/zstd-rs:masterfrom huffman_encodingKillingSpark/zstd-rs:huffman_encodingCopy head branch name to clipboardDec 13, 2024

+2,830-358

Commits

Commits on Nov 28, 2024

• start a huffman encder

  KillingSpark

  committed
  42c046e

  Copy full SHA for 42c046e

• build huffman table according to spec

  KillingSpark

  committed
  a0d2031

  Copy full SHA for a0d2031

• calc max num bits from weights

  KillingSpark

  committed
  246e609

  Copy full SHA for 246e609

• create valid weight distributions

  KillingSpark

  committed
  35833ba

  Copy full SHA for 35833ba

• implement redistributing weights to fit a given maximum weight

  KillingSpark

  committed
  7f5ac83

  Copy full SHA for 7f5ac83

• fix chosen weight to redistribute

  KillingSpark

  committed
  05788ac

  Copy full SHA for 05788ac

• fix huffman code distribution

  KillingSpark

  committed
  48efed9

  Copy full SHA for 48efed9

• build huffman table for counts

  KillingSpark

  committed
  bad020d

  Copy full SHA for bad020d

• allow building from a slice of data

  KillingSpark

  committed
  0e29503

  Copy full SHA for 0e29503

• start actual huffman encoder

  KillingSpark

  committed
  39e612d

  Copy full SHA for 39e612d

• test for prefix free codes, shows it's still broken

  KillingSpark

  committed
  190f08c

  Copy full SHA for 190f08c

• fix broken prefix allocation

  KillingSpark

  committed
  a7cd816

  Copy full SHA for a7cd816

• test more weight distributions

  KillingSpark

  committed
  e799da7

  Copy full SHA for e799da7

• add roundtrip test for huffman coding that's fit to also be fuzzed

  KillingSpark

  committed
  f7d32c7

  Copy full SHA for f7d32c7

• fix roundtrip test

  KillingSpark

  committed
  cd794a8

  Copy full SHA for cd794a8

• huffman streams are encoded backwards

  KillingSpark

  committed
  1980dbf

  Copy full SHA for 1980dbf

• rewrite bit writer to fill bytes starting at the lower bits

  KillingSpark

  committed
  90ede9e

  Copy full SHA for 90ede9e

• start fuzzing huffman encoder

  KillingSpark

  committed
  bfa1c9e

  Copy full SHA for bfa1c9e

• test that max num bits doesnt get too big

  KillingSpark

  committed
  289e664

  Copy full SHA for 289e664

• only execute fuzz targets when present

  KillingSpark

  committed
  df60a7a

  Copy full SHA for df60a7a

• Fix weight redistribution to limit to a specific number of bits

  KillingSpark

  committed
  4c05304

  Copy full SHA for 4c05304

• add two crashes that are already fixed

  KillingSpark

  committed
  6d4920d

  Copy full SHA for 6d4920d

• expand test to cover a weight distribution of all sizes

  KillingSpark

  committed
  2726eaa

  Copy full SHA for 2726eaa

• cargo fmt

  KillingSpark

  committed
  9d07056

  Copy full SHA for 9d07056

• no need to gate encoding behind std

  KillingSpark

  committed
  10ee6e1

  Copy full SHA for 10ee6e1

• mak clippy happy

  KillingSpark

  committed
  f7a5ad2

  Copy full SHA for f7a5ad2

• more std import shenanigans

  KillingSpark

  committed
  ed219cc

  Copy full SHA for ed219cc

• cargo fmt

  KillingSpark

  committed
  0e2435b

  Copy full SHA for 0e2435b

• start fse encoder

  KillingSpark

  committed
  02c4789

  Copy full SHA for 02c4789

• need different table representation for encoding

  KillingSpark

  committed
  d5e2543

  Copy full SHA for d5e2543

• build fse table for encoding

  KillingSpark

  committed
  96925de

  Copy full SHA for 96925de

• start test for fse table creation, fix edgecase for 0 probability symbols in decoder

  KillingSpark

  committed
  22bfdb3

  Copy full SHA for 22bfdb3

• table generation now matches

  KillingSpark

  committed
  9039fd9

  Copy full SHA for 9039fd9

• first happy fse roundtrip

  KillingSpark

  committed
  4ca250d

  Copy full SHA for 4ca250d

• cargo fmt

  KillingSpark

  committed
  b97f962

  Copy full SHA for b97f962

• unnecessary import

  KillingSpark

  committed
  55b2918

  Copy full SHA for 55b2918

• more roundtrips

  KillingSpark

  committed
  fd9c968

  Copy full SHA for fd9c968

• fuzz fse encoder

  KillingSpark

  committed
  3221f28

  Copy full SHA for 3221f28

• minimum acc_log is 5

  KillingSpark

  committed
  2f4e5d0

  Copy full SHA for 2f4e5d0

• implement fse table encoding

  KillingSpark

  committed
  78ce10a

  Copy full SHA for 78ce10a

• misaligned should never report 8

  KillingSpark

  committed
  8ebd0db

  Copy full SHA for 8ebd0db

• encode huffman tables according to spec

  KillingSpark

  committed
  b7b7d0b

  Copy full SHA for b7b7d0b

• start compression mode based solely on compressing literals

  KillingSpark

  committed
  561d7d5

  Copy full SHA for 561d7d5

• add test for a simple frame encode/decode cycle

  KillingSpark

  committed
  91d46cc

  Copy full SHA for 91d46cc

• fix blocksize for compressed blocks, a block may only contain 1<<18 - 1 uncompressed literals

  KillingSpark

  committed
  9c3e610

  Copy full SHA for 9c3e610

• add support for RLE blocks

  KillingSpark

  committed
  fc48aa5

  Copy full SHA for fc48aa5

• start fuzzing compression interop

  KillingSpark

  committed
  e13886f

  Copy full SHA for e13886f

• added support for literals less than 5

  KillingSpark

  committed
  cb8d4b7

  Copy full SHA for cb8d4b7

• fix single stream encoding

  KillingSpark

  committed
  402d180

  Copy full SHA for 402d180

• add compression and decoding to encoding fuzzer

  KillingSpark

  committed
  454ad7b

  Copy full SHA for 454ad7b

• add compression and decoding to encoding fuzzer

  KillingSpark

  committed
  fc91d6e

  Copy full SHA for fc91d6e

• huffman weights may only use a fse compression with 6 bits

  KillingSpark

  committed
  1106501

  Copy full SHA for 1106501

• implement decreasing the fse probabilities to fit max_log

  KillingSpark

  committed
  eeb940f

  Copy full SHA for eeb940f

• remove table modes from empty sequence sections

  KillingSpark

  committed
  61d5fe3

  Copy full SHA for 61d5fe3

• check in decoder if a sequence section with 0 sequences contains nothing else

  KillingSpark

  committed
  0176e0f

  Copy full SHA for 0176e0f

• not using single segment frames and a window size allows inefficient use of the compressed blocks

  KillingSpark

  committed
  3d259ea

  Copy full SHA for 3d259ea

• switch back to huffman encoding literals

  KillingSpark

  committed
  45d5fba

  Copy full SHA for 45d5fba

• improve interop fuzz

  KillingSpark

  committed
  1a19804

  Copy full SHA for 1a19804

• make window size big enough to fit whole 128kb block into it

  KillingSpark

  committed
  facb5d5

  Copy full SHA for facb5d5

• extend encodecorpus to test compression

  KillingSpark

  committed
  932a12a

  Copy full SHA for 932a12a

• if compressing a block makes it larger just encode it as a raw block and move on

  KillingSpark

  committed
  3df96dd

  Copy full SHA for 3df96dd

• raise the first fse probability to avoid 0 num bit states not the last

  KillingSpark

  committed
  79e1c99

  Copy full SHA for 79e1c99

• cargo fmt

  KillingSpark

  committed
  5ca4eaa

  Copy full SHA for 5ca4eaa

• make clippy happy

  KillingSpark

  committed
  443cac8

  Copy full SHA for 443cac8

• bitwriter can now work on a &mut Vec as output

  KillingSpark

  committed
  aaad836

  Copy full SHA for aaad836

• 4x huffman encoding doesnt use 4 vecs anymore

  KillingSpark

  committed
  e95fd06

  Copy full SHA for e95fd06

• add compression support to zstd binary

  KillingSpark

  committed
  a8ab25e

  Copy full SHA for a8ab25e

• optimize bitwriter to collect bits in a u64 before flushing to the output

  KillingSpark

  committed
  79c2491

  Copy full SHA for 79c2491

• make source a generic read impl

  KillingSpark

  committed
  0f96d67

  Copy full SHA for 0f96d67

• make the target a generic write impl

  KillingSpark

  committed
  39aac8e

  Copy full SHA for 39aac8e

• todo

  KillingSpark

  committed
  c83918b

  Copy full SHA for c83918b

• remove unnecessary code from hot function

  KillingSpark

  committed
  eeb4e08

  Copy full SHA for eeb4e08

• remove one += from hot function

  KillingSpark

  committed
  aea6a90

  Copy full SHA for aea6a90

• more efficient check for hot path

  KillingSpark

  committed
  a355e30

  Copy full SHA for a355e30

• dry common code

  KillingSpark

  committed
  00e61d6

  Copy full SHA for 00e61d6

• employ bufreader in zstd binary

  KillingSpark

  committed
  6b26b2d

  Copy full SHA for 6b26b2d

• start match finding algorithm

  KillingSpark

  committed
  8493d58

  Copy full SHA for 8493d58

• start implementing a skip list for use in the match generator

  KillingSpark

  committed
  262c034

  Copy full SHA for 262c034

• use hashmap to do find indexes of matches of a minimum length

  KillingSpark

  committed
  e175cd9

  Copy full SHA for e175cd9

• only return None if the whole last data slice has been processed

  KillingSpark

  committed
  a6024dc

  Copy full SHA for a6024dc

• matcher can now also accept data that doesnt need matches

  KillingSpark

  committed
  219dbe2

  Copy full SHA for 219dbe2

• start working on encoding the sequences

  KillingSpark

  committed
  dc8068a

  Copy full SHA for dc8068a

• implement translation tables for ll/ml/of tables to codes and additional bits for the bitstream

  KillingSpark

  committed
  14bc05d

  Copy full SHA for 14bc05d

• implement FSE encoding sequence codes

  KillingSpark

  committed
  c9f4f03

  Copy full SHA for c9f4f03

• default tables

  KillingSpark

  committed
  8457c25

  Copy full SHA for 8457c25

• offset in last window slice is calculated differently

  KillingSpark

  committed
  44fcb62

  Copy full SHA for 44fcb62

• properly encode number of sequences

  KillingSpark

  committed
  845008d

  Copy full SHA for 845008d

• cargo fmt

  KillingSpark

  committed
  3f34638

  Copy full SHA for 3f34638

• matcher now correctly returns a last literals sequence for the last unmatched bytes in a block

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  5b0a142

  Copy full SHA for 5b0a142

• add debug assertions that check that matches are correctly generated

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  e4fcb81

  Copy full SHA for e4fcb81

• simplify offset calc, doesn't need a special case for the last slice

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  37ec1e8

  Copy full SHA for 37ec1e8

• offsets are encoded as +3 to allow 1,2,3 to represent the last 3 offsets

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  fc2a5cb

  Copy full SHA for fc2a5cb

• fix error in default table impl

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  90e0e62

  Copy full SHA for 90e0e62

• fix edgecases around small data slices or few sequences

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  a5774de

  Copy full SHA for a5774de

• remove warning

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  25315c9

  Copy full SHA for 25315c9

• fix doc test

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  7d32f52

  Copy full SHA for 7d32f52

• fix fuzzing

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  70e1501

  Copy full SHA for 70e1501

• make clippy happy

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  677ddf4

  Copy full SHA for 677ddf4

• if literals get bigger by encoding them with a huffman tree, just write the literals raw

  KillingSpark

  committed
  741a3f9

  Copy full SHA for 741a3f9

• expand nostd_io module to include write_all, read_to_end and impl Write for Vec<u8>

  KillingSpark

  committed
  fd0ae1f

  Copy full SHA for fd0ae1f

• use slices instead of fixed size arrays as key for match generator

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  b3f20f6

  Copy full SHA for b3f20f6

• use more efficient method of windowing slices

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  247d870

  Copy full SHA for 247d870

• cargo fmt

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  0530eb6

  Copy full SHA for 0530eb6

• disable clippy lint for collapsible if. I think it's clearer this way in this instance

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  54939d4

  Copy full SHA for 54939d4

• encoding now actually streams the data instead of reading the source until the end before processing

  KillingSpark

  committed
  f9c2484

  Copy full SHA for f9c2484

• delete residual eprintln

  KillingSpark

  committed
  1690bd9

  Copy full SHA for 1690bd9

• make clippy happy

  KillingSpark

  committed
  48fac5a

  Copy full SHA for 48fac5a

• remove decoding check from zstd binary

  KillingSpark

  committed
  afc8cc7

  Copy full SHA for afc8cc7

• unused import

  KillingSpark

  committed
  cb07e1c

  Copy full SHA for cb07e1c

• zstd binary reports compression progress now

  KillingSpark

  committed
  d81c76b

  Copy full SHA for d81c76b

• this definitely still needs work...

  KillingSpark

  committed
  7cbeeac

  Copy full SHA for 7cbeeac

Commits on Nov 29, 2024

• No need for a hashtable actually

  KillingSpark

  committed
  421d1c1

  Copy full SHA for 421d1c1

• make clippy happy

  KillingSpark

  committed
  8b42cb6

  Copy full SHA for 8b42cb6

• better key function

  

  Moritz Borcherding

  committed
  699c075

  Copy full SHA for 699c075

• cargo fmt

  

  Moritz Borcherding

  committed
  899410d

  Copy full SHA for 899410d

• even better key function

  

  Moritz Borcherding

  committed
  79e45c4

  Copy full SHA for 79e45c4

• fix multiply overflow

  KillingSpark

  committed
  f98a837

  Copy full SHA for f98a837

• reenable test

  KillingSpark

  committed
  7d23961

  Copy full SHA for 7d23961

• better poly in key function

  KillingSpark

  committed
  869a7bf

  Copy full SHA for 869a7bf

Commits on Dec 2, 2024

• improve match length finding

  

  Moritz Borcherding

  committed
  d17af02

  Copy full SHA for d17af02

• inline makes a significant difference

  

  Moritz Borcherding

  committed
  feb6f5f

  Copy full SHA for feb6f5f

• document the matcher trait and make it slightly easier to use

  

  Moritz Borcherding

  committed
  228353b

  Copy full SHA for 228353b

• cargo fmt

  

  Moritz Borcherding

  committed
  9fbe7dd

  Copy full SHA for 9fbe7dd

• fix hash collision in test

  

  Moritz Borcherding

  committed
  8988bc3

  Copy full SHA for 8988bc3

• also reuse suffix store allocations

  

  Moritz Borcherding

  committed
  b0c65fa

  Copy full SHA for b0c65fa

• fix test and enable niche optimization for Option<NonZeroUsize>

  

  Moritz Borcherding

  committed
  e99c64f

  Copy full SHA for e99c64f

• std -> core import

  

  Moritz Borcherding

  committed
  124793c

  Copy full SHA for 124793c

Commits on Dec 4, 2024

• small optimization around fse encoding

  

  Moritz Borcherding

  committed
  7819c14

  Copy full SHA for 7819c14

Commits on Dec 12, 2024

• doc and comments in fse encoder

  

  Moritz Borcherding

  committed
  28c3084

  Copy full SHA for 28c3084

Commits on Dec 13, 2024

• doc and comments in huff0 encoder

  

  Moritz Borcherding

  committed
  61829d7

  Copy full SHA for 61829d7

• more doc comments

  

  Moritz Borcherding

  committed
  b9029f0

  Copy full SHA for b9029f0

• doc and comments in match generator

  

  Moritz Borcherding

  committed
  a56a40b

  Copy full SHA for a56a40b

• more doc and slightly better Matcher api

  

  Moritz Borcherding

  committed
  4499aac

  Copy full SHA for 4499aac

• add convenience functions for compressing data

  

  Moritz Borcherding

  committed
  1237041

  Copy full SHA for 1237041

• update readme and changelog

  

  Moritz Borcherding

  committed
  99dee17

  Copy full SHA for 99dee17

• cargo fmt

  

  Moritz Borcherding

  committed
  6c2bcd7

  Copy full SHA for 6c2bcd7