1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-29 11:21:22 +03:00
Commit Graph

2251 Commits

Author SHA1 Message Date
4373c5ab88 mmap dicitonaries for large dictionaries in patch-from 2023-02-13 07:26:22 -08:00
c78f434aa4 Fix zstd-dll build missing dependencies (#3496)
* Fixes zstd-dll build (https://github.com/facebook/zstd/issues/3492):
- Adds pool.o and threading.o dependency to the zstd-dll target
- Moves custom allocation functions into header to avoid needing to add dependency on common.o
- Adds test target for zstd-dll
- Adds github workflow that buildis zstd-dll
2023-02-12 12:32:31 -08:00
95ffc767f6 updated man pages 2023-02-09 14:40:39 -08:00
cc4e941745 initial commit 2023-02-09 07:40:25 -08:00
610c8b9e33 initial commit 2023-02-09 07:37:37 -08:00
6740f8f0b8 add error message for the (rare) compression error scenario 2023-02-07 10:02:09 -08:00
58e7067c7d added more accurate error messages
for the decompression-only benchmark mode.
2023-02-07 08:47:39 -08:00
9cabd155fd return error code when benchmark fails
such scenario can happen, for example,
when trying a decompression-only benchmark on invalid data.
Other possibilities include an allocation error in an intermediate step.

So far, the benchmark would return immediately, but still return 0.
On command line, this would be confusing, as the program appears successful (though it does not display any successful message).

Now it returns !0, which can be interpreted as an error by command line.
2023-02-07 00:35:51 -08:00
f746c37d00 Use File Descriptor in Setting Stat on Output File
Note that the `fd` is only valid while the file is still open. So we need to
move the setting calls to before we close the file. However! We cannot do so
with the `utime()` call (even though `futimens()` exists) because the follow-
ing `close()` call to the `fd` will reset the atime of the file. So it seems
the `utime()` call has to happen after the file is closed.
2023-02-06 14:02:31 -08:00
a5a2418df4 Introduce Variants of Some Functions that Take Optional File Descriptors
Somewhat surprisingly, calling `fchmod()` is non-trivially faster than calling
`chmod()`, and so on.

This commit introduces alternate variants to some common file util functions
that take an optional fd. If present, they call the `f`-variant of the
underlying function. Otherwise, they fall back to the regular filename-taking
version of the function.
2023-02-06 13:55:34 -08:00
79bdb8cbb6 AsyncIO performance regression for small files fix (#3474)
- Do not use threaded AsyncIO when handling small files.
- Some typo / doc fixes
2023-02-02 15:19:22 -08:00
af09777b24 ensure that benchmark mode can only be invoked with zstd format
fix #3463
2023-01-31 09:04:29 -08:00
82ca00811a change logic when stderr is not console : don't update progress status
but keep warnings and final operation statement.

updated tests/cli-tests/ accordingly
2023-01-26 13:00:52 -08:00
7b3f03bc9d Merge pull request #3457 from yoniko/fix-rowhash-cli
[Bugfix] CLI row hash flags set the wrong values


`--[no-]row-match-finder` do the opposite of what they are supposed to.
In effect the no option would activate row hash while the other option will disable it.
This commit fixes the issue and changes the code to use the more readable enum values.
2023-01-25 22:40:25 -08:00
6422d1d7a8 Bugfix: --[no-]row-match-finder do the opposite of what they are supposed to 2023-01-25 17:59:35 -08:00
02434e0867 enforce a hard fail when input files are set to be erased
in scenarios where it's supposed to not be possible.

suggested by @terrelln
2023-01-25 16:18:20 -08:00
8c85b29e32 disable --rm on -o command
make it more similar to -c (aka `stdout`) convention.
2023-01-25 16:09:25 -08:00
b6fd91ba84 update man 2023-01-20 18:07:55 -08:00
cee6bec9fa refactor : --rm is ignored with stdout
`zstd` CLI has progressively moved to the policy of
ignoring `--rm` command when the output is `stdout`.
The primary drive is to feature a behavior more consistent with `gzip`,
when `--rm` is the default, but is also ignored when output is `stdout`.
Other policies are certainly possible, but would break from this `gzip` convention.

The new policy was inconsistenly enforced, depending on the exact list of commands.
For example, it was possible to circumvent it by using `-c --rm` in this order,
which would re-establish source removal.

- Update the CLI so that it necessarily catch these situations and ensure that `--rm` is always disabled when output is `stdout`.
- Added a warning message in this case (for verbosity 3 `-v`).
- Added an `assert()`, which controls that `--rm` is no longer active with `stdout`
- Added tests, which control the behavior, even when `--rm` is added after `-c`
- Removed some legacy code which where trying to apply a specific policy for the `stdout` + `--rm` case, which is no longer possible
2023-01-20 18:04:55 -08:00
3d25502c2d Merge pull request #3432 from felixhandte/fix-perms
Fix CLI Handling of Permissions and Ownership (Again)
2023-01-20 19:19:05 -05:00
772229afd5 Merge pull request #3442 from felixhandte/pgo-tests
Test PGO Builds
2023-01-20 19:18:51 -05:00
87e169d05d Add Additional Flags to PGO Build
In GCC, we can add a couple more flags to give us confidence that the profile
data is actually being found and used.

Also, my system for example doesn't have a binary installed under the name
`llvm-profdata`, but it does have, e.g., `llvm-profdata-13`, etc. So this
commit adds a variable that can be overridden.
2023-01-20 17:32:49 -05:00
ea684c335a added c89 build test to CI 2023-01-19 14:59:30 -08:00
bbe65d760c Merge pull request #3423 from facebook/ptime
Refactor timefn, restore support for clock_gettime()
2023-01-18 13:27:42 -08:00
7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
2023-01-18 11:57:54 -08:00
1e3eba65a6 Copy Permissions from Source File 2023-01-18 11:57:35 -08:00
0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
a5ed28f1fb Use Existing Src File Stat in *_dstFile() Funcs
One fewer `stat()` call to make per operation!
2023-01-17 14:08:22 -08:00
5653f96776 Pass stat_t into *_dstFile() Functions 2023-01-17 14:05:15 -08:00
2ad6855ac1 FIO_openSrcFile() Returns File Stat 2023-01-17 14:01:06 -08:00
5d8cfa6b96 Deprecate advanced streaming functions (#3408)
* deprecate advanced streaming functions

* remove internal usage of the deprecated functions

* nit

* suppress warnings in tests/zstreamtest.c

* purge ZSTD_initDStream_usingDict

* nits

* c90 compat

* zstreamtest.c already disables deprecation warnings!

* fix initDStream() return value

* fix typo

* wasn't able to import private symbol properly, this commit works around that

* new strategy for zbuff

* undo zbuff deprecation warning changes

* move ZSTD_DISABLE_DEPRECATE_WARNINGS from .h to .c
2023-01-13 14:51:47 -05:00
a2ef23dec0 restore support of clock_gettime() for POSIX systems
This should notably allow posix systems with timespec_get()
to have access to a high resolution timer,
instead of falling back to C90's clock_t.
2023-01-12 21:08:15 -08:00
bcfb7ad03c refactor timefn
The timer storage type is no longer dependent on OS.
This will make it possible to re-enable posix precise timers
since the timer storage type will no longer be sensible to #include order.
See #3168 for details of pbs of previous interface.

Suggestion by @terrelln
2023-01-12 19:24:31 -08:00
8b130009e3 minor simplification refactoring for timefn
`UTIL_getSpanTimeMicro()` can be factored in a generic way,
reducing OS-dependent code.
2023-01-06 16:12:54 -08:00
90597d78ea Merge pull request #3394 from terrelln/issue-3010
[cli-tests] Test file stat read/write
2022-12-27 16:20:05 -08:00
11aba9b316 make man 2022-12-22 14:13:24 -05:00
382026f096 Man Page Tweaks, Edits, Formatting Fixes
This started as an application of the edits suggested in #3201 and expanded
from there.
2022-12-22 14:13:17 -05:00
40a7188130 Fix make clangbuild & add CI
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
2022-12-21 17:31:04 -08:00
b6e8112261 [util] Add traces enabled by --trace-file-stat
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
2022-12-21 17:19:34 -08:00
7607b96ed9 Merge pull request #3385 from jonpalmisc/improve_help
Improve help/usage (`-h`, `-H`) formatting
2022-12-21 10:56:09 -08:00
9c93dd71cd Clean up welcome message 2022-12-20 20:13:00 -05:00
678335c4f3 Rewrite help output to improve readability 2022-12-20 20:12:59 -05:00
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
2022-12-20 12:44:56 -05:00
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
e2fc93340f Merge branch 'dev' into http-to-https 2022-12-15 10:46:13 -05:00
15f32ad74c [fileio] Separate parameter adaption from display update rate
Split the logic for parameter adaption from the logic to update the display rate.
This decouples the two updates, so changes to display updates don't affect
parameter adaption.

Also add a test case that checks that parameter adaption actually happens.

This fixes Issue #3353, where --adapt is broken when --no-progress is passed.
2022-12-14 17:08:21 -08:00
fbff7827fa Refactor progress bar & summary line logic
* Centralize the logic about whether to print the progress bar or not in
  the `*_PROGRESS()` macros.
* Centralize the logc about whether to print the summary line or not in
  `FIO_shouldDisplayFileSummary()` and
  `FIO_shouldDisplayMultipleFileSummary()`.
* Make `--progress` work for non-zstd (de)compressors.
* Clean up several edge cases in compression and decompression progress
  printing along the way. E.g. wrong log level, or missing summary line.

One thing I don't like about stdout mode, which sets the display level
to 1, is that warnings aren't displayed. After this PR, we could change
stdout mode from lowering the display level, to defaulting to implied
`--no-progress`. But, I think that deserves a separate PR.
2022-12-14 15:44:09 -08:00
e58a39f84e Allow tests to fake stdin/stdout/stderr is a console
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
2022-12-14 15:44:09 -08:00