d5986f235f
fix #4332 : setting ZSTD_NBTHREADS=0 via environment variables
2025-03-10 00:12:34 -07:00
b16d193512
[test] Add tests for determinism
...
Run compression & validate the compressed file matches a known checksum.
To update the output run:
```
make -C tests update-cli-tests
```
2025-03-07 10:31:19 -05:00
0de4991942
Add a method for checking if ZSTD was compiled with flags that impact determinism
2025-03-07 10:31:19 -05:00
99cf130cfc
Merge pull request #4329 from Cyan4973/cmd_split
...
New commands --split and --jobsize
2025-03-05 07:46:04 -08:00
190a620974
[zstd] Remove global variables in dictBuilder
...
D50949782 fixed a race condition updating `g_displayLevel` by disabling display.
Instead of disabling display, delete the global variable and always "capture" a local `displayLevel` variable.
This also fixes `DISPLAYUPDATE()` by requiring the user to pass in the last update time as the first parameter.
2025-03-05 10:35:01 -05:00
d5b84f5a27
[zstd] Backport D49756856
2025-03-05 10:35:01 -05:00
5e7d721235
Merge pull request #4327 from Cyan4973/win_artifacts
...
Generate Zstandard binaries for Windows with lz4 support
2025-03-04 16:08:01 -08:00
5ae1cb9fa1
added a cli test for new command --jobsize
2025-03-04 15:24:43 -08:00
f5a0e047cb
fix typo
2025-03-04 15:12:35 -08:00
0b40c513fd
update fileio to employ jobSize
2025-03-04 14:55:25 -08:00
0298df50f9
update cli-tests
2025-03-04 14:43:06 -08:00
9b8b414833
updated playTests.sh to employ the new commands
2025-03-04 13:22:01 -08:00
19ffcf81b8
changed -B# command into --jobsize
...
to reduce confusion with the term "block".
-B# remains supported for existing scripts,
but it's no longer documented, so it's effectively a hidden shortcut.
2025-03-04 13:17:10 -08:00
fcfb3160dc
convert benchmark unit to use the new naming scheme
...
chunks instead of blocks
2025-03-04 13:04:59 -08:00
d5dbdd6ece
changed command --blocksize into --split
...
to reduce confusion with the concept of "blocks" inside a Zstandard frame.
We are now talking about "independent chunks" being produced by a `split` operation.
updated documentation accordingly.
Note: old commands "-B#` and `--blocksize=#` remain supported,
to maintain compatibility with existing scripts.
2025-03-04 12:53:23 -08:00
bceb8f2dbc
attempt to no longer depend on system library presence for autodetection
...
use explicit opt-in and path setting
2025-03-03 17:20:43 -08:00
9ec1a7c0ac
update zstd build recipe to also support lz4 format
2025-03-03 15:36:38 -08:00
7df457a51d
Merge pull request #4324 from facebook/dependabot/github_actions/actions/checkout-4
...
Bump actions/checkout from 3 to 4
2025-03-03 10:39:50 -08:00
cab2a29c36
Update windows-artifacts.yml
...
edit version comment
2025-03-02 23:11:35 -08:00
3c6c472f01
Update scorecards.yml
...
edit version comment
2025-03-02 23:11:08 -08:00
46b2e05a33
Update publish-release-artifacts.yml
...
fix version comment
2025-03-02 23:10:09 -08:00
83db5376d7
Bump actions/checkout from 3 to 4
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases )
- [Commits](https://github.com/actions/checkout/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com >
2025-03-03 05:44:17 +00:00
0727fc5ff9
lib: zstd: export API needed for dictionary support
...
Patch series "zram: introduce custom comp backends API", v7.
This series introduces support for run-time compression algorithms tuning,
so users, for instance, can adjust compression/acceleration levels and
provide pre-trained compression/decompression dictionaries which certain
algorithms support.
At this point we stop supporting (old/deprecated) comp API. We may add
new acomp API support in the future, but before that zram needs to undergo
some major rework (we are not ready for async compression).
Some benchmarks for reference (look at column #2 )
*** init zstd
/sys/block/zram0/mm_stat
1750659072 504622188 514355200 0 514355200 1 0 34204 34204
*** init zstd dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750650880 465908890 475398144 0 475398144 1 0 34185 34185
*** init zstd level=8 dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750654976 430803319 439873536 0 439873536 1 0 34185 34185
*** init lz4
/sys/block/zram0/mm_stat
1750646784 664266564 677060608 0 677060608 1 0 34288 34288
*** init lz4 dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750650880 619990300 632102912 0 632102912 1 0 34278 34278
*** init lz4hc
/sys/block/zram0/mm_stat
1750630400 609023822 621232128 0 621232128 1 0 34288 34288
*** init lz4hc dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750659072 505133172 515231744 0 515231744 1 0 34278 34278
Recompress
init zram zstd (prio=0), zstd level=5 (prio 1), zstd with dict (prio 2)
*** zstd
/sys/block/zram0/mm_stat
1750982656 504630584 514269184 0 514269184 1 0 34204 34204
*** idle recompress priority=1 (zstd level=5)
/sys/block/zram0/mm_stat
1750982656 488645601 525438976 0 514269184 1 0 34204 34204
*** idle recompress priority=2 (zstd dict)
/sys/block/zram0/mm_stat
1750982656 460869640 517914624 0 514269184 1 0 34185 34204
This patch (of 24):
We need to export a number of API functions that enable advanced zstd
usage - C/D dictionaries, dictionaries sharing between contexts, etc.
Link: https://lkml.kernel.org/r/20240902105656.1383858-1-senozhatsky@chromium.org
Link: https://lkml.kernel.org/r/20240902105656.1383858-2-senozhatsky@chromium.org
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org >
Cc: Nick Terrell <terrelln@fb.com >
Cc: Minchan Kim <minchan@kernel.org >
Cc: Sergey Senozhatsky <senozhatsky@chromium.org >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2025-02-28 17:09:46 -05:00
a9337e6739
Merge pull request #4321 from Cyan4973/win_artifacts
...
Automate Windows Artifact generation and publication
2025-02-28 12:35:06 -08:00
e99be473a6
automatically upload windows artifact on Release Tag creation
2025-02-28 12:04:06 -08:00
bbbb2379ed
test win artifact generation on pushing to release
...
this will get a chance to check that win artifact generation works correctly
before creating the release tag.
2025-02-28 09:42:50 -08:00
a6d232dcad
avoid double zipping
2025-02-28 09:33:22 -08:00
88100bc62f
proper usage of standard variables
2025-02-27 16:59:40 -08:00
4e1723a7e4
fixed the script so that it fails when a copy fails
...
and also: fix the list of files, as `zdict.h` was incorrectly set.
2025-02-27 16:18:44 -08:00
7340657c6f
update build_package.bat by using a subrouting
2025-02-27 16:18:44 -08:00
77e58994d2
updated zlib version to more recent v1.3.1
2025-02-27 16:18:44 -08:00
e94e09dd7b
ensure that a copy error results in the task failing clearly
...
error code != 0, red status
checked by intentionally inserting an error in another run
2025-02-27 16:18:44 -08:00
017ada44b5
Merge pull request #4322 from Cyan4973/meson_win
...
fix the meson windows build
2025-02-27 16:17:34 -08:00
bbf2801ddd
try to fix the meson windows build
...
apparently could be just a missing include
2025-02-27 15:45:59 -08:00
a1a5154b69
Merge pull request #4312 from Cyan4973/musl_ci
...
introduce ZSTD_USE_C90_QSORT
2025-02-27 14:27:21 -08:00
22b2fd2517
Merge pull request #4317 from hirohira9119/fix-function-signature
...
Fix function signature mismatch for ZSTD_convertBlockSequences
2025-02-27 13:03:03 -08:00
d6fbaaac99
Merge pull request #4320 from sebres/patch-1
...
build_package.bat: fix path to zstd_errors.h, avoid silently ignore of the errors if build failed
2025-02-26 15:15:03 -08:00
0ac659fb00
Merge pull request #4319 from Cyan4973/Wconversion
...
step towards -Wconversion
2025-02-26 15:14:52 -08:00
dca9791862
fixed minor C++ compat warnings
2025-02-26 14:30:29 -08:00
4d81b0000d
GHA/windows-artifacts.yml: don't ignore the error if build-package batch failed
2025-02-26 20:13:44 +01:00
f0d3173203
build_package.bat: don't swallow the error(s) by copy, exit with error if failed somewhere
2025-02-26 20:02:48 +01:00
97bc43cc68
build_package.bat: fix path to zstd_errors.h (it is in lib not in lib/common)
...
closes gh-4318
2025-02-26 19:27:44 +01:00
db2d205ada
fixed -Wconversion for lib/decompress/zstd_decompress_block.c
2025-02-26 10:01:05 -08:00
2413f17322
fixed -Wconversion for cover.c
2025-02-26 08:33:01 -08:00
8ffa27d93b
fixed -Wconversion for divsufsort.c
2025-02-26 08:12:11 -08:00
e635221f1b
fixed -Wconversion for zdict
2025-02-26 08:07:51 -08:00
30281d889f
fix conversion warning
2025-02-26 07:41:34 -08:00
2840631dc1
Fix function signature mismatch for ZSTD_convertBlockSequences
2025-02-26 08:23:48 +09:00
76779f52c2
Merge pull request #4310 from Cyan4973/cmake_target_compile
...
[cmake] use target_compile_definitions
2025-02-25 10:04:10 -08:00
3ff0d794b8
Merge pull request #4309 from Cyan4973/check_manual
...
Check API manual and man pages before release
2025-02-24 15:17:11 -08:00
