1
0
mirror of https://github.com/facebook/zstd.git synced 2025-08-07 06:23:00 +03:00

179 Commits

Author SHA1 Message Date
Yann Collet
bfeb456fcd fix #3764
This answer a static analysis tool,
which complains that a buffer allocation is not free() just before exit().

In general, this requirement is not necessary, because invoking exit() will make the OS reclaim all buffers from the terminated process.

I could articulate this new requirement in a "not too heavy" way with the use of a new macro, `CONTROL_EXIT()`.
But "not too heavy" is still a form of maintenance burden: whenever the code is modified, by adding, removing or changing some of these buffers, it requires some form of coordination with exit points, which is easy to let go wrong.
Besides, I wouldn't be surprised if there were some more complex scenarios left, typically across multiple levels of functions, where a call to `exit()` is made while some other buffers, inaccessible from the function, are still allocated. Tackling such issues would require a very different approach, typically forbidding the use of `exit()`, which was meant to simplify code maintenance by reducing the nb and complexity of error paths.
I question the need to make the code more complex to read and maintain, just to tackle a largely theoretical problem with no practical impact on target platforms.
2023-10-07 19:34:22 -07:00
W. Felix Handte
f746c37d00 Use File Descriptor in Setting Stat on Output File
Note that the `fd` is only valid while the file is still open. So we need to
move the setting calls to before we close the file. However! We cannot do so
with the `utime()` call (even though `futimens()` exists) because the follow-
ing `close()` call to the `fd` will reset the atime of the file. So it seems
the `utime()` call has to happen after the file is closed.
2023-02-06 14:02:31 -08:00
W. Felix Handte
a5a2418df4 Introduce Variants of Some Functions that Take Optional File Descriptors
Somewhat surprisingly, calling `fchmod()` is non-trivially faster than calling
`chmod()`, and so on.

This commit introduces alternate variants to some common file util functions
that take an optional fd. If present, they call the `f`-variant of the
underlying function. Otherwise, they fall back to the regular filename-taking
version of the function.
2023-02-06 13:55:34 -08:00
Yann Collet
8c85b29e32 disable --rm on -o command
make it more similar to -c (aka `stdout`) convention.
2023-01-25 16:09:25 -08:00
W. Felix Handte
7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
W. Felix Handte
0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
2023-01-18 11:57:54 -08:00
W. Felix Handte
0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
Nick Terrell
b6e8112261 [util] Add traces enabled by --trace-file-stat
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
2022-12-21 17:19:34 -08:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
Nick Terrell
e58a39f84e Allow tests to fake stdin/stdout/stderr is a console
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
2022-12-14 15:44:09 -08:00
Elliot Gorokhovsky
b1182fe242 Merge pull request #3300 from yiyuaner/dev
Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable
2022-11-02 15:28:02 -04:00
appleman4321
5334339421 Rewrite checking condition bufStart pointer NULL 2022-11-01 02:31:09 +03:00
yiyuaner
361d86998a Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable 2022-10-24 20:37:15 +08:00
Elliot Gorokhovsky
28ceb63503 Merge pull request #3220 from embg/issue3200
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
2022-08-01 14:04:57 -04:00
Yonatan Komornik
ae4670466c stdin multiple file fixes (#3222)
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
2022-07-29 16:13:07 -07:00
Qiongsi Wu
b1bbb0eb4c [AIX] Fix Compiler Flags and Bugs on AIX to Pass All Tests (#3219)
* Fixing compiler warnings

* Replace the old -s flag with the -Wl,-s flag

* Fixing compiler warnings

* Fixing the linker strip flag and tests/code not working as expected on AIX
2022-07-29 12:21:59 -07:00
Elliot Gorokhovsky
e1873ad576 Fix buffer underflow for null dir1 2022-07-29 11:10:47 -07:00
Dominique Pelle
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Xi Ruoyao
d109cef201 fix the assertion in readLinesFromFile (#3084)
* fix the assertion in readLinesFromFile

When the file is not terminated by endline, readLineFromFile will append
a '\0' for the last line.  In this case pos + lineLength == dstCapacity.

* test: don't print very long text garbage
2022-03-04 14:56:44 -05:00
binhdvo
d7e1736375 Fix build for cygwin/bsd (#2882) 2021-11-29 14:11:39 -05:00
Dimitris Apostolou
ebbd675998 Fix typos 2021-11-13 10:04:04 +02:00
Yann Collet
eab692211e removed pretty-print of sizes in benchmark
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
2021-09-03 12:51:02 -07:00
W. Felix Handte
a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
Binh Vo
6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
W. Felix Handte
2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB")
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
2021-06-10 12:53:07 -04:00
W. Felix Handte
9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte
93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte
464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte
9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte
bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
2021-06-10 12:53:07 -04:00
Scott Baker
8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott Baker
1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott Baker
376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott Baker
20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott Baker
64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott Baker
1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott Baker
e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott Baker
eefdbcd93a Make the variable types match 2021-06-10 12:53:07 -04:00
Scott Baker
b6b23dfe64 Convert names to CamelCase 2021-06-10 12:53:07 -04:00
Scott Baker
b70175e5ec Put the human_size() function in util.c 2021-06-10 12:53:07 -04:00
Yann Collet
df05b2ba7c fix --filelist compatibility with Windows cr+lf line ending 2021-05-05 18:01:55 -07:00
W. Felix Handte
b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
W. Felix Handte
33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
Nick Terrell
a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
Sen Huang
f27e326456 Restrict dictmode regression tests only to advanced API, fix some compiler warnings 2021-03-25 10:39:08 -07:00
Paul Bone
4d6c78fb89 Only set numPhysicalCores if ratio is valid 2021-03-03 10:59:00 +11:00
Paul Bone
eb1a09df61 If cpuinfo parsing fails fallback to sysconf 2021-03-03 10:58:51 +11:00
W. Felix Handte
61db590ad8 Detect .. in Paths Correctly
This commit addresses #2509.
2021-02-26 12:29:42 -05:00
W. Felix Handte
a774c57973 Use umask() to Constrain Created File Permissions
This commit addresses #2491.

Note that a downside of this solution is that it is global: `umask()` affects
all file creation calls in the process. I believe this is safe since
`fileio.c` functions should only ever be used in the zstd binary, and these
are (almost) the only files ever created by zstd, and AIUI they're only
created in a single thread. So we can get away with messing with global state.

Note that this doesn't change the permissions of files created by `dibio.c`.
I'm not sure what those should be...
2021-02-17 15:27:39 -05:00