lib/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-09-18 10:03:27 +03:00
Code Activity
11,127 Commits 32 Branches 77 Tags
v1.5.7-kernel
Commit Graph

11127 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell
5d9fb47850 [linux] Opt out of row based match finder for the kernel 
The row based match finder is slower without SIMD. We used to detect the
presence of SIMD to set the lower bound to 17, but that breaks
determinism. Instead, specifically opt into it for the kernel, because
it is one of the rare cases that doesn't have SIMD support.
v1.5.7-kernel 		2025-03-11 12:54:22 -07:00
Sergey Senozhatsky
7418f234d1 lib: zstd: export API needed for dictionary support 
Patch series "zram: introduce custom comp backends API", v7.

This series introduces support for run-time compression algorithms tuning,
so users, for instance, can adjust compression/acceleration levels and
provide pre-trained compression/decompression dictionaries which certain
algorithms support.

At this point we stop supporting (old/deprecated) comp API.  We may add
new acomp API support in the future, but before that zram needs to undergo
some major rework (we are not ready for async compression).

Some benchmarks for reference (look at column #2)

*** init zstd
/sys/block/zram0/mm_stat
1750659072 504622188 514355200        0 514355200        1        0    34204    34204

*** init zstd dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750650880 465908890 475398144        0 475398144        1        0    34185    34185

*** init zstd level=8 dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750654976 430803319 439873536        0 439873536        1        0    34185    34185

*** init lz4
/sys/block/zram0/mm_stat
1750646784 664266564 677060608        0 677060608        1        0    34288    34288

*** init lz4 dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750650880 619990300 632102912        0 632102912        1        0    34278    34278

*** init lz4hc
/sys/block/zram0/mm_stat
1750630400 609023822 621232128        0 621232128        1        0    34288    34288

*** init lz4hc dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750659072 505133172 515231744        0 515231744        1        0    34278    34278

Recompress
init zram zstd (prio=0), zstd level=5 (prio 1), zstd with dict (prio 2)

*** zstd
/sys/block/zram0/mm_stat
1750982656 504630584 514269184        0 514269184        1        0    34204    34204

*** idle recompress priority=1 (zstd level=5)
/sys/block/zram0/mm_stat
1750982656 488645601 525438976        0 514269184        1        0    34204    34204

*** idle recompress priority=2 (zstd dict)
/sys/block/zram0/mm_stat
1750982656 460869640 517914624        0 514269184        1        0    34185    34204

This patch (of 24):

We need to export a number of API functions that enable advanced zstd
usage - C/D dictionaries, dictionaries sharing between contexts, etc.

Link: https://lkml.kernel.org/r/20240902105656.1383858-1-senozhatsky@chromium.org
Link: https://lkml.kernel.org/r/20240902105656.1383858-2-senozhatsky@chromium.org
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Nick Terrell <terrelln@fb.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
 2025-03-08 12:03:02 -08:00
Yann Collet
f8745da6ff Merge pull request #4298 from facebook/dev 
v1.5.7
v1.5.7 		2025-02-18 16:04:24 -08:00
Yann Collet
ea027ab21c Merge pull request #4299 from facebook/ga_harden 
Improved Github Actions scorecards
 2025-02-18 15:26:29 -08:00
Yann Collet
b14d76d888 pinned dependency hash 2025-02-18 14:53:31 -08:00
Yann Collet
5c465fcabe harden github actions script Android NDK Build 2025-02-18 14:50:03 -08:00
Yann Collet
99a12e6f72 Merge pull request #4297 from facebook/changelog157 
update changelog for v1.5.7
 2025-02-18 14:35:24 -08:00
Yann Collet
c26bde119b update changelog for v1.5.7 2025-02-18 13:59:24 -08:00
Yann Collet
22c39b9891 Merge pull request #4274 from luau-project/intel-compilers 
CI: enable Intel LLVM C compiler (icx) check
 2025-02-13 20:21:35 -08:00
Yann Collet
beccbc6f74 Merge pull request #4293 from facebook/rip-ubuntu20 
Fixed CI
 2025-02-13 20:21:10 -08:00
Yann Collet
2a58b04752 disabled BTI test 
this test seems impossible on Ubuntu-24
 2025-02-11 13:20:15 -08:00
Yann Collet
85c39b78cf faster aarch64 test execution 2025-02-11 13:11:29 -08:00
Yann Collet
0b8119f0ad minor arm64 test script clarification 2025-02-11 12:58:35 -08:00
Yann Collet
b73e06b83e updated FreeBSD VM to 14.2 
14.1 seems no longer supported by gcloud
 2025-02-11 12:52:59 -08:00
Yann Collet
2b7c661ad2 update arm64 test 
for ubuntu-24
 2025-02-11 12:41:38 -08:00
Yann Collet
75bcae1272 removing x32 tests 
this ABI is no longer supported by Ubuntu,
and there is a wider consensus that this ABI is on the way out,
with more and more distributions dropping it,
and lingering questions about support of x32 in the kernel.
 2025-02-11 11:37:18 -08:00
Yann Collet
fc1baf3463 move qemu tests to ubuntu24 2025-02-11 09:57:18 -08:00
Yann Collet
815ca8c678 ubuntu 20 is being deprecated 
change ubuntu20 tests to use more modern ubuntu variants
 2025-02-11 09:19:08 -08:00
Yann Collet
d84d70bd04 Merge pull request #4288 from facebook/stronger_patchfrom 
Improve compression ratio of the --patch-from mode
 2025-02-10 12:09:16 -08:00
Yann Collet
d2c562b803 update hrlog comment 2025-02-10 10:48:56 -08:00
Yann Collet
67fad95f79 derive hashratelog from hashlog when only hashlog is set 2025-02-10 10:46:37 -08:00
Yann Collet
339bca6606 update ldm compression results 2025-02-10 10:46:37 -08:00
Yann Collet
09d7e34ed8 adjust mml 2025-02-10 10:46:37 -08:00
Yann Collet
d5e4698267 fix boundary condition 2025-02-10 10:46:37 -08:00
Yann Collet
72406b71c3 update hrlog rule to favor compression ratio a bit more at low levels 2025-02-10 10:46:37 -08:00
Yann Collet
7c5b6002c9 update results from the --long mode 2025-02-10 10:46:37 -08:00
Yann Collet
bf218c142a updated LDM documentation 2025-02-10 10:46:37 -08:00
Yann Collet
f26cc54f37 dynamic bucket sizes 2025-02-10 10:46:37 -08:00
Yann Collet
4609a40b89 dynamically adjust hratelog and ldmml based on strategy 2025-02-10 10:46:37 -08:00
Yann Collet
5b8575adaa Merge pull request #4289 from facebook/autoultra 
--ultra automatically triggered with --long and --patch-from
 2025-02-10 10:38:28 -08:00
Yann Collet
e3a9351402 Merge pull request #4290 from facebook/lmax 
--max command
 2025-02-10 10:38:00 -08:00
Yann Collet
071a4a0904 Merge pull request #4291 from facebook/dependabot/github_actions/github/codeql-action-3.28.9 
Bump github/codeql-action from 3.28.1 to 3.28.9
 2025-02-09 23:45:47 -08:00
Yann Collet
613901b6d3 modifies command echoing for macos / linux compatibility 2025-02-09 23:41:32 -08:00
Yann Collet
468e1453a5 disable --max in 32-bit mode 2025-02-09 23:16:14 -08:00
Yann Collet
1603cbe83e update test for 32-bit mode 
--max doesn't work in 32-bit mode, due to address space limitation
 2025-02-09 23:02:14 -08:00
dependabot[bot]
7a2fce5a1f Bump github/codeql-action from 3.28.1 to 3.28.9 
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.28.1 to 3.28.9.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](b6a472f63d...9e8d0789d4)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-02-10 05:36:44 +00:00
Yann Collet
f86024ccd2 reinforce man page warning 
32-bit is unsuitable for --max
 2025-02-09 17:25:35 -08:00
Yann Collet
39d1d82fa8 adjusted mml heuristic 2025-02-09 17:24:12 -08:00
Yann Collet
41b7193757 added --max to test suite 2025-02-09 12:26:36 -08:00
Yann Collet
8ae1330708 add man entry for --max 2025-02-09 11:51:25 -08:00
Yann Collet
630b47a158 added a new --max command 
set parameters to maximum compression
(even more than -22)
 2025-02-09 11:38:44 -08:00
Yann Collet
aebffd66ec --ultra automatically triggered with --long and --patch-from 
the purpose of --ultra is to make the user explicitly opt-in
to generate very large window size (> 8 MB).

The agreement to generate very large window size is already implicit
when selecting --long or --patch-from.

Consequently, `--ultra ` is automatically enabled when `--long` or `--patch-from` is set.
 2025-02-08 22:47:05 -08:00
Yann Collet
0396480109 Merge pull request #4284 from facebook/mesonNoexec 
Add noexecstack flag for gcc/clang C and CPP in Meson
 2025-02-06 10:55:44 -08:00
Yann Collet
ef2bf57811 Merge pull request #4276 from facebook/fix_patchfrom_hc_speed 
fix speed of --patch-from mode at high compression levels
 2025-02-05 20:12:56 -08:00
Yann Collet
23e5f80390 Revert "pass dictionary loading method as parameter" 
This reverts commit 821fc567f9.
 2025-02-05 18:47:26 -08:00
Yann Collet
c7cd7dc04b better MT fluidity 
--patch-from no longer blocked on first job dictionary loading
 2025-02-05 18:42:00 -08:00
Yann Collet
f11bd19c7f ensure cdict is properly reset to NULL 2025-02-05 18:42:00 -08:00
Yann Collet
7406d2b6eb skips the need to create a temporary cdict for --patch-from 
thus saving a bit of memory and a little bit of cpu time
 2025-02-05 18:42:00 -08:00
Yann Collet
220abe6da8 reduced memory usage 
by avoiding to duplicate in memory
a dictionary that was passed by reference.
 2025-02-05 18:42:00 -08:00
Yann Collet
85a44b233a always free .cdictLocal 2025-02-05 18:41:59 -08:00