1
0
mirror of https://github.com/facebook/zstd.git synced 2025-04-19 00:44:19 +03:00

530 Commits

Author SHA1 Message Date
Yann Collet
76c2fdc7b7 better naming
and more narrow scope of local variables
2025-03-28 09:18:17 -07:00
Yann Collet
eb168a0afc add --patch-apply command
as an equivalent for `-d --patch-from`.
Requested by @sergeevabc in #2173.
2025-03-25 14:50:39 -07:00
Yann Collet
19541b4d1a fix minor unused variable warning 2025-03-11 00:17:45 -07:00
Yann Collet
c583c2c39e warn when requesting decompression with multiple threads
restore #2918 fix
2025-03-10 22:50:00 -07:00
Yann Collet
2ff87aefac fix FreeBSD
use an alias instead of a function

also: added more traces and updated version nb to v1.5.8
2025-03-10 19:04:41 -07:00
Yann Collet
56e2ebf5c3 removed useless assert() 2025-03-10 09:54:06 -07:00
Yann Collet
d5986f235f fix #4332: setting ZSTD_NBTHREADS=0 via environment variables 2025-03-10 00:12:34 -07:00
Nick Terrell
0de4991942 Add a method for checking if ZSTD was compiled with flags that impact determinism 2025-03-07 10:31:19 -05:00
Yann Collet
f5a0e047cb fix typo 2025-03-04 15:12:35 -08:00
Yann Collet
0b40c513fd update fileio to employ jobSize 2025-03-04 14:55:25 -08:00
Yann Collet
0298df50f9 update cli-tests 2025-03-04 14:43:06 -08:00
Yann Collet
19ffcf81b8 changed -B# command into --jobsize
to reduce confusion with the term "block".

-B# remains supported for existing scripts,
but it's no longer documented, so it's effectively a hidden shortcut.
2025-03-04 13:17:10 -08:00
Yann Collet
fcfb3160dc convert benchmark unit to use the new naming scheme
chunks instead of blocks
2025-03-04 13:04:59 -08:00
Yann Collet
d5dbdd6ece changed command --blocksize into --split
to reduce confusion with the concept of "blocks" inside a Zstandard frame.

We are now talking about "independent chunks" being produced by a `split` operation.

updated documentation accordingly.

Note: old commands "-B#` and `--blocksize=#` remain supported,
to maintain compatibility with existing scripts.
2025-03-04 12:53:23 -08:00
Yann Collet
5b8575adaa
Merge pull request #4289 from facebook/autoultra
--ultra automatically triggered with --long and --patch-from
2025-02-10 10:38:28 -08:00
Yann Collet
468e1453a5 disable --max in 32-bit mode 2025-02-09 23:16:14 -08:00
Yann Collet
39d1d82fa8 adjusted mml heuristic 2025-02-09 17:24:12 -08:00
Yann Collet
630b47a158 added a new --max command
set parameters to maximum compression
(even more than -22)
2025-02-09 11:38:44 -08:00
Yann Collet
aebffd66ec --ultra automatically triggered with --long and --patch-from
the purpose of --ultra is to make the user explicitly opt-in
to generate very large window size (> 8 MB).

The agreement to generate very large window size is already implicit
when selecting --long or --patch-from.

Consequently, `--ultra ` is automatically enabled when `--long` or `--patch-from` is set.
2025-02-08 22:47:05 -08:00
Yann Collet
56500044c4 bench: nb threads specified with -v 2025-01-12 12:01:57 -08:00
Yann Collet
56cfb7816a codemod: ZSTD_paramSwitch_e -> ZSTD_ParamSwitch_e 2024-12-20 10:36:58 -08:00
Yann Collet
125f05282b publish new symbol ZSTD_compressSequencesAndLiterals() 2024-12-20 10:36:04 -08:00
daniellerozenblit
17beeb5d1a
Change CLI to employ multithreading by default (#4211)
* Change CLI to employ multithreading by default

* Document changes to benchmarking, print number of threads for display level >= 4, and add lower bound of 1 for the default number of threads
2024-12-12 13:09:29 -05:00
Dimitrios Apostolou
194062a4e7 Fix printing of filesize if >4GB 2024-11-20 16:11:17 +01:00
Yann Collet
f34bc9cee6 improve man page on benchmark mode
update the man page in troff format,
and the README with latest `--help` content and complementary details about benchmark mode.

also: display level 0 when doing decompression benchmark
2024-10-23 00:16:13 -07:00
Yann Collet
0079d515b1 Modify benchmark to only load sources once
After a regrettable update,
the benchmark module ended up reloading sources for every compression level.

While the delay itself is likely torelable,
the main issue is that the `--quiet` mode now also displays a loading summary between each compression line.
This wasn't the original intention, which is to produce a compact view of all compressions.

This is fixed in this version,
where sources are loaded only once, for all compression levels,
and loading summary is only displayed once.
2024-10-22 02:18:48 -07:00
Yann Collet
039f404faa update documentation to specify that Dictionary can be used for benchmark
fix #4139
2024-09-25 16:56:01 -07:00
Yann Collet
fbd9e628ae added tests 2024-03-11 12:17:34 -07:00
Yann Collet
c610a01d7d fix #3719
only disable `--rm` at end of command line parsing,
so that `-c` only disables `--rm` if it's effectively selected,
and not if it's overriden by a later `-o FILE` command.
2024-03-11 11:38:55 -07:00
Yann Collet
4d2bf7f0f2 removed sprintf usage from zstdcli.c
some static analyzers flag this standard C90 function as unsafe.
2024-02-24 23:03:40 -08:00
Yann Collet
68a232c591 benchmark more can test targetCBlockSize 2024-02-23 13:13:03 -08:00
Yann Collet
d0b7da30e2 add a lorem ipsum generator
this generator replaces the statistical generator
for the general case when no statistic is requested.

Generated data features a compression level speed / ratio curve
which is more in line with expectation.
2024-01-29 15:00:32 -08:00
Ruslan Sayfutdinov
8052cd0131
cli: better errors on arguent parsing 2023-12-18 13:59:33 +00:00
daniellerozenblit
c28031df8f
Add new line + [no-] to mmap-dict help output (#3601) 2023-04-06 13:01:58 -04:00
Yann Collet
e769da1645
Merge pull request #3526 from facebook/bench_zstd_api
Simplify benchmark unit invocation API from CLI
2023-03-09 13:11:11 -08:00
daniellerozenblit
e0fc9fd90b
Merge pull request #3486 from daniellerozenblit/patch-from-low-memory-mode
Mmap large dictionaries in patch-from mode
2023-03-09 15:30:09 -05:00
Danielle Rozenblit
96e55c14f2 ability to disable mmap + struct to manage FIO dictionary 2023-03-08 08:06:10 -08:00
Yann Collet
1e38e07b3d simplified BMK_benchFilesAdvanced() 2023-03-06 12:34:13 -08:00
Yann Collet
9efc14804e minor: fixed zlib wrapper internal benchmark
another possibility could be to link it to programs/benchfn .
Not worth the effort.
2023-03-06 12:20:06 -08:00
Yann Collet
db79219f70 simplify BMK_syntheticTest() 2023-03-06 12:15:22 -08:00
Dimitri Papadopoulos
547794ef40
Fix typos found by codespell 2023-02-18 10:31:48 +01:00
Danielle Rozenblit
2d8afd9ce1 add manual flag to mmap dictionary 2023-02-14 09:42:23 -08:00
Danielle Rozenblit
610c8b9e33 initial commit 2023-02-09 07:37:37 -08:00
Yann Collet
9cabd155fd return error code when benchmark fails
such scenario can happen, for example,
when trying a decompression-only benchmark on invalid data.
Other possibilities include an allocation error in an intermediate step.

So far, the benchmark would return immediately, but still return 0.
On command line, this would be confusing, as the program appears successful (though it does not display any successful message).

Now it returns !0, which can be interpreted as an error by command line.
2023-02-07 00:35:51 -08:00
Yann Collet
af09777b24 ensure that benchmark mode can only be invoked with zstd format
fix #3463
2023-01-31 09:04:29 -08:00
Yann Collet
82ca00811a change logic when stderr is not console : don't update progress status
but keep warnings and final operation statement.

updated tests/cli-tests/ accordingly
2023-01-26 13:00:52 -08:00
Yonatan Komornik
7b3f03bc9d
Merge pull request #3457 from yoniko/fix-rowhash-cli
[Bugfix] CLI row hash flags set the wrong values


`--[no-]row-match-finder` do the opposite of what they are supposed to.
In effect the no option would activate row hash while the other option will disable it.
This commit fixes the issue and changes the code to use the more readable enum values.
2023-01-25 22:40:25 -08:00
Yonatan Komornik
6422d1d7a8 Bugfix: --[no-]row-match-finder do the opposite of what they are supposed to 2023-01-25 17:59:35 -08:00
Yann Collet
8c85b29e32 disable --rm on -o command
make it more similar to -c (aka `stdout`) convention.
2023-01-25 16:09:25 -08:00
Yann Collet
cee6bec9fa refactor : --rm is ignored with stdout
`zstd` CLI has progressively moved to the policy of
ignoring `--rm` command when the output is `stdout`.
The primary drive is to feature a behavior more consistent with `gzip`,
when `--rm` is the default, but is also ignored when output is `stdout`.
Other policies are certainly possible, but would break from this `gzip` convention.

The new policy was inconsistenly enforced, depending on the exact list of commands.
For example, it was possible to circumvent it by using `-c --rm` in this order,
which would re-establish source removal.

- Update the CLI so that it necessarily catch these situations and ensure that `--rm` is always disabled when output is `stdout`.
- Added a warning message in this case (for verbosity 3 `-v`).
- Added an `assert()`, which controls that `--rm` is no longer active with `stdout`
- Added tests, which control the behavior, even when `--rm` is added after `-c`
- Removed some legacy code which where trying to apply a specific policy for the `stdout` + `--rm` case, which is no longer possible
2023-01-20 18:04:55 -08:00