1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-04 07:42:31 +03:00
Commit Graph

731 Commits

Author SHA1 Message Date
ae4670466c stdin multiple file fixes (#3222)
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
2022-07-29 16:13:07 -07:00
2b9fde932b Fix small file passthrough (#3215) 2022-07-29 12:22:46 -07:00
b1bbb0eb4c [AIX] Fix Compiler Flags and Bugs on AIX to Pass All Tests (#3219)
* Fixing compiler warnings

* Replace the old -s flag with the -Wl,-s flag

* Fixing compiler warnings

* Fixing the linker strip flag and tests/code not working as expected on AIX
2022-07-29 12:21:59 -07:00
d7eb829af5 Detect multiple dictIDs in one file 2022-07-08 12:20:50 -05:00
cc8c98485a zstd -lv <file> to show dictID 2022-07-05 21:28:33 -05:00
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
470eb8330a Fix required decompression memory usage reported by -vv + --long
The use of --long alters the window size internally in the underlying
library (lib/compress/zstd_compress.c:ZSTD_getCParamsFromCCtxParams),
which changes the memory required for decompression. This means that the
reported requirement from the zstd binary when -vv is specified is
incorrect.

A full fix for this would be to add an API call to be able to retrieve
the required decompression memory from the library, but as a
lighterweight fix we can just take account of the fact we've enabled
long mode and update our verbose output appropriately.

Fixes #2968
2022-02-02 12:39:27 +00:00
cc0657f27d AsyncIO compression part 2 - added async read and asyncio to compression code (#3022)
* Compression asyncio:
- Added asyncio functionality for compression flow
- Added ReadPool for async reads, implemented in both comp and decomp flows
2022-01-31 15:43:41 -08:00
e60eba58bf Print zlib/lz4/lzma library versions in verbose version output
Knowing the version of zlib/lz4/lzma we're linking against is very
useful for debugging issues with those libraries, so print it out in the
verbosity 4 version output.

Also print this information at the top of `playTests.sh`.
2022-01-26 18:25:58 -08:00
70df5de1b2 AsyncIO compression part 1 - refactor of existing asyncio code (#3021)
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h

* Bugfix - rename fileio_asycio to fileio_asyncio

* Added copyrights & license to new files

* CR fixes
2022-01-24 14:43:02 -08:00
feaaf7a6b1 slightly shortened status and summary lines in very verbose mode 2022-01-21 21:38:35 -08:00
24318093cc slightly shortened compression status update line
to fit within 80 columns limit.
2022-01-21 14:08:46 -08:00
8ab95f24da Merge pull request #2985 from yoniko/zstd-output-file-buffer
ZSTD CLI: Use buffered output
2022-01-21 13:57:05 -08:00
1598e6c634 Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
57a86d9ec6 Avoid updating timestamps when the destination is stdout
Fixes: 9cd6c1ff4d
Fixes: https://github.com/facebook/zstd/issues/2997
2022-01-13 16:47:18 -05:00
df5013b463 ZSTD CLI: Use buffered output for improved performance 2022-01-07 17:10:13 -08:00
308a11b8e8 Fix stderr progress logging for decompression
When decompressing with `-q` and an output file, the progress bar was mistakenly printed. This is a minimal fix, with a larger refactor to be stacked on top of it.

Fixes #2967.
2022-01-07 15:09:56 -08:00
9985e10fda Reject Irregular Dictionary Files
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
2021-12-08 16:17:04 -05:00
e7b0ae385e Merge pull request #2890 from 15596858998/dec_1201
fixbug CLI's -D fails when the argument is not a regular file
2021-12-06 13:18:15 -05:00
3133d1e86e Merge pull request #2876 from 15596858998/dev
Solve the bug of extra output newline character
2021-12-01 15:10:08 -08:00
a8adfa7f67 fixbug CLI's -D fails when the argument is not a regular file 2021-12-01 21:47:35 +08:00
e315a047c7 更新 fileio.c
Solve the logic problem of wrong output of newline characters.
2021-11-23 22:08:23 +08:00
7fbd126e08 Suppress spurious unused parameter warning 2021-11-12 10:44:26 -08:00
375e3aad6c Ensure formatting directives for displaying size_t are portable 2021-11-11 13:17:32 -08:00
df9b7755cb Fix const-ness of FIO_displayCompressionParameters 2021-11-10 17:25:27 -08:00
0665d4c1c2 Display command line parameters with concrete values in verbose mode 2021-11-05 12:01:20 -07:00
b3888193d9 Report memory required to decompress while compressing in verbose mode 2021-11-04 16:25:40 -07:00
b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
30fe49af4e Fix patch-from help msg typo 2021-09-07 10:08:35 -04:00
9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
2021-06-10 12:53:07 -04:00
35576e63ce Convert tabs to spaces 2021-06-10 12:53:07 -04:00
77001f00fb Use human_size() on the "multiple files compressed" output also 2021-06-10 12:53:07 -04:00
4e0d9f1cc8 Move the variable declarations to the top 2021-06-10 12:53:07 -04:00
b6b23dfe64 Convert names to CamelCase 2021-06-10 12:53:07 -04:00
b70175e5ec Put the human_size() function in util.c 2021-06-10 12:53:07 -04:00
26fab1d963 Make the CLI output the file sizes in human readable format 2021-06-10 12:53:07 -04:00
d2f31b6627 Fix --progress flag to properly control progress display and default progress display on when using -v 2021-06-08 17:24:38 -04:00
d4548c96cb fileio: clamp value of windowLog in patch-mode (#2637)
With small enough input files, the inferred value of fileWindowLog could
be smaller than ZSTD_WINDOWLOG_MIN.

This can be reproduced like so:
$ echo abc > small
$ echo abcdef > small2
$ zstd --patch-from small small2 -o patch
previously, this would fail with the error "zstd: error 11 : Parameter is out of bound"
2021-05-12 16:11:15 -04:00
sen
6030cdfede Add --progress flag (#2595) 2021-05-06 14:50:28 -04:00
4f9c6fdb7f Attempt to Fix Windows Build Error 2021-05-05 13:13:56 -04:00
da61918c75 Also Pass Mode Bits in on Windows
I think in some unix emulation environments on Windows, (cygwin?) mode bits
are somehow respected. So we might as well pass them in. Can't hurt.
2021-05-05 13:10:34 -04:00
45c4918ccf Fix Build for Windows 2021-05-05 13:10:34 -04:00
1fb10ba831 Don't Block Removing File on Being Able to Read It
`open()`'s mode bits are only applied to files that are created by the call.
If the output file already exists, but is not readable, the `fopen()` would
fail, preventing us from removing it, which would mean that the file would
not end up with the correct permission bits.

It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()`
should be sufficient, AFAICT.
2021-05-05 13:10:34 -04:00
b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00