1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-28 00:01:53 +03:00
Commit Graph

2289 Commits

Author SHA1 Message Date
82ca00811a change logic when stderr is not console : don't update progress status
but keep warnings and final operation statement.

updated tests/cli-tests/ accordingly
2023-01-26 13:00:52 -08:00
7b3f03bc9d Merge pull request #3457 from yoniko/fix-rowhash-cli
[Bugfix] CLI row hash flags set the wrong values


`--[no-]row-match-finder` do the opposite of what they are supposed to.
In effect the no option would activate row hash while the other option will disable it.
This commit fixes the issue and changes the code to use the more readable enum values.
2023-01-25 22:40:25 -08:00
6422d1d7a8 Bugfix: --[no-]row-match-finder do the opposite of what they are supposed to 2023-01-25 17:59:35 -08:00
02434e0867 enforce a hard fail when input files are set to be erased
in scenarios where it's supposed to not be possible.

suggested by @terrelln
2023-01-25 16:18:20 -08:00
8c85b29e32 disable --rm on -o command
make it more similar to -c (aka `stdout`) convention.
2023-01-25 16:09:25 -08:00
b6fd91ba84 update man 2023-01-20 18:07:55 -08:00
cee6bec9fa refactor : --rm is ignored with stdout
`zstd` CLI has progressively moved to the policy of
ignoring `--rm` command when the output is `stdout`.
The primary drive is to feature a behavior more consistent with `gzip`,
when `--rm` is the default, but is also ignored when output is `stdout`.
Other policies are certainly possible, but would break from this `gzip` convention.

The new policy was inconsistenly enforced, depending on the exact list of commands.
For example, it was possible to circumvent it by using `-c --rm` in this order,
which would re-establish source removal.

- Update the CLI so that it necessarily catch these situations and ensure that `--rm` is always disabled when output is `stdout`.
- Added a warning message in this case (for verbosity 3 `-v`).
- Added an `assert()`, which controls that `--rm` is no longer active with `stdout`
- Added tests, which control the behavior, even when `--rm` is added after `-c`
- Removed some legacy code which where trying to apply a specific policy for the `stdout` + `--rm` case, which is no longer possible
2023-01-20 18:04:55 -08:00
3d25502c2d Merge pull request #3432 from felixhandte/fix-perms
Fix CLI Handling of Permissions and Ownership (Again)
2023-01-20 19:19:05 -05:00
772229afd5 Merge pull request #3442 from felixhandte/pgo-tests
Test PGO Builds
2023-01-20 19:18:51 -05:00
87e169d05d Add Additional Flags to PGO Build
In GCC, we can add a couple more flags to give us confidence that the profile
data is actually being found and used.

Also, my system for example doesn't have a binary installed under the name
`llvm-profdata`, but it does have, e.g., `llvm-profdata-13`, etc. So this
commit adds a variable that can be overridden.
2023-01-20 17:32:49 -05:00
ea684c335a added c89 build test to CI 2023-01-19 14:59:30 -08:00
bbe65d760c Merge pull request #3423 from facebook/ptime
Refactor timefn, restore support for clock_gettime()
2023-01-18 13:27:42 -08:00
7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
2023-01-18 11:57:54 -08:00
1e3eba65a6 Copy Permissions from Source File 2023-01-18 11:57:35 -08:00
0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
a5ed28f1fb Use Existing Src File Stat in *_dstFile() Funcs
One fewer `stat()` call to make per operation!
2023-01-17 14:08:22 -08:00
5653f96776 Pass stat_t into *_dstFile() Functions 2023-01-17 14:05:15 -08:00
2ad6855ac1 FIO_openSrcFile() Returns File Stat 2023-01-17 14:01:06 -08:00
5d8cfa6b96 Deprecate advanced streaming functions (#3408)
* deprecate advanced streaming functions

* remove internal usage of the deprecated functions

* nit

* suppress warnings in tests/zstreamtest.c

* purge ZSTD_initDStream_usingDict

* nits

* c90 compat

* zstreamtest.c already disables deprecation warnings!

* fix initDStream() return value

* fix typo

* wasn't able to import private symbol properly, this commit works around that

* new strategy for zbuff

* undo zbuff deprecation warning changes

* move ZSTD_DISABLE_DEPRECATE_WARNINGS from .h to .c
2023-01-13 14:51:47 -05:00
a2ef23dec0 restore support of clock_gettime() for POSIX systems
This should notably allow posix systems with timespec_get()
to have access to a high resolution timer,
instead of falling back to C90's clock_t.
2023-01-12 21:08:15 -08:00
bcfb7ad03c refactor timefn
The timer storage type is no longer dependent on OS.
This will make it possible to re-enable posix precise timers
since the timer storage type will no longer be sensible to #include order.
See #3168 for details of pbs of previous interface.

Suggestion by @terrelln
2023-01-12 19:24:31 -08:00
8b130009e3 minor simplification refactoring for timefn
`UTIL_getSpanTimeMicro()` can be factored in a generic way,
reducing OS-dependent code.
2023-01-06 16:12:54 -08:00
90597d78ea Merge pull request #3394 from terrelln/issue-3010
[cli-tests] Test file stat read/write
2022-12-27 16:20:05 -08:00
11aba9b316 make man 2022-12-22 14:13:24 -05:00
382026f096 Man Page Tweaks, Edits, Formatting Fixes
This started as an application of the edits suggested in #3201 and expanded
from there.
2022-12-22 14:13:17 -05:00
40a7188130 Fix make clangbuild & add CI
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
2022-12-21 17:31:04 -08:00
b6e8112261 [util] Add traces enabled by --trace-file-stat
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
2022-12-21 17:19:34 -08:00
7607b96ed9 Merge pull request #3385 from jonpalmisc/improve_help
Improve help/usage (`-h`, `-H`) formatting
2022-12-21 10:56:09 -08:00
9c93dd71cd Clean up welcome message 2022-12-20 20:13:00 -05:00
678335c4f3 Rewrite help output to improve readability 2022-12-20 20:12:59 -05:00
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
2022-12-20 12:44:56 -05:00
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
e2fc93340f Merge branch 'dev' into http-to-https 2022-12-15 10:46:13 -05:00
15f32ad74c [fileio] Separate parameter adaption from display update rate
Split the logic for parameter adaption from the logic to update the display rate.
This decouples the two updates, so changes to display updates don't affect
parameter adaption.

Also add a test case that checks that parameter adaption actually happens.

This fixes Issue #3353, where --adapt is broken when --no-progress is passed.
2022-12-14 17:08:21 -08:00
fbff7827fa Refactor progress bar & summary line logic
* Centralize the logic about whether to print the progress bar or not in
  the `*_PROGRESS()` macros.
* Centralize the logc about whether to print the summary line or not in
  `FIO_shouldDisplayFileSummary()` and
  `FIO_shouldDisplayMultipleFileSummary()`.
* Make `--progress` work for non-zstd (de)compressors.
* Clean up several edge cases in compression and decompression progress
  printing along the way. E.g. wrong log level, or missing summary line.

One thing I don't like about stdout mode, which sets the display level
to 1, is that warnings aren't displayed. After this PR, we could change
stdout mode from lowering the display level, to defaulting to implied
`--no-progress`. But, I think that deserves a separate PR.
2022-12-14 15:44:09 -08:00
e58a39f84e Allow tests to fake stdin/stdout/stderr is a console
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
2022-12-14 15:44:09 -08:00
4dffc35f2e Convert references to https from http 2022-12-14 06:58:35 -08:00
dc39409a03 Print checksum value for single frame files in cli with -v -l options 2022-12-12 10:53:57 -08:00
4d82a4d3f2 minor : reorder --help
more logical context grouping
2022-11-22 07:23:45 -08:00
b1182fe242 Merge pull request #3300 from yiyuaner/dev
Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable
2022-11-02 15:28:02 -04:00
5334339421 Rewrite checking condition bufStart pointer NULL 2022-11-01 02:31:09 +03:00
361d86998a Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable 2022-10-24 20:37:15 +08:00
9abecfbb7e replaced Zstandard by zstd in man page
identified by @paulmenzel in #3286
2022-10-11 23:32:18 -07:00
2bd70eff06 zstd-pgo: also clean zstd binary from objects
Just a precaution, because it works anyway.
2022-10-06 00:38:23 +07:00
2ffcb2d6a8 fixed zstd-pgo target for GCC
Since your Makefile uses obj/$(HASH_DIR) for object files, this code does not work correctly for GCC. Because profiles are saved in one directory, and are expected in another when reading.

`$(RM) zstd *.o` - this line doesn't delete object files.

Clang stores profiles in the current directory, so the problem doesn't appear when compiling with Clang.

Also this code will work if BUILD_DIR is set.
2022-10-05 17:32:16 +07:00
358787764f Fix for zstd CLI accepts bogus values for numeric parameters (#3268)
* add checks to mal-formed numeric values for memory and memlimit parameters

Signed-off-by: Ly Cao <lycao@fb.com>

* changed errorMsg to a literal string instead of static string in main

* moved bogus numeric error to NEXT_UINT32 + add macro NEXT_TSIZE

Signed-off-by: Ly Cao <lycao@fb.com>

Signed-off-by: Ly Cao <lycao@fb.com>
Co-authored-by: Ly Cao <lycao@fb.com>
2022-09-21 13:20:01 -07:00
f7251f88b9 Merge pull request #3247 from haampie/fix/grep
Fix make variable
2022-09-19 10:47:38 -07:00
5129b4ab10 simplify clean target maintenance within programs/ 2022-09-12 12:19:35 -07:00