1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-30 22:23:13 +03:00

1965 Commits

Author SHA1 Message Date
32bb667138 added test to cli-tests 2022-12-13 12:45:41 -08:00
bb3c01c853 Migrate other test usages of boolean LDM flag to paramSwitch enum 2022-11-21 16:20:38 -05:00
3720910d06 Fix fuzzer failure 2022-11-21 16:09:04 -05:00
db74d043d6 Speed optimizations with macro 2022-10-27 10:20:44 -07:00
0d5d571080 Merge pull request #3285 from daniellerozenblit/optimal-huff-depth
Optimal huf depth
2022-10-18 10:31:44 -04:00
1d153c923c Make fuzzing work without ZSTD_MULTITHREAD (#3291) 2022-10-17 14:31:47 -07:00
75cd42afd7 Update regression results and better variable naming for HUF_cardinality 2022-10-14 13:37:19 -07:00
117fddcd81 Update regression results 2022-10-12 13:49:44 -07:00
8888a2ddcc CI failure fixes 2022-10-11 13:12:19 -07:00
21bd8c3b3c Removed unused variable (#3272) 2022-09-22 08:20:46 -07:00
358787764f Fix for zstd CLI accepts bogus values for numeric parameters (#3268)
* add checks to mal-formed numeric values for memory and memlimit parameters

Signed-off-by: Ly Cao <lycao@fb.com>

* changed errorMsg to a literal string instead of static string in main

* moved bogus numeric error to NEXT_UINT32 + add macro NEXT_TSIZE

Signed-off-by: Ly Cao <lycao@fb.com>

Signed-off-by: Ly Cao <lycao@fb.com>
Co-authored-by: Ly Cao <lycao@fb.com>
2022-09-21 13:20:01 -07:00
361b7bd1d0 Merge branch 'dev' into clean 2022-09-12 10:56:00 -07:00
a06e953db9 some additional comments, remove apt-get from clang jobs, better test titles 2022-09-08 18:30:07 -07:00
c0b46738b4 streamline make clean list maintenance
When creating a new `Makefile` target to build,
it's also necessary to update the `clean` target,
which purpose is to remove built targets when they are present.

This process is simple, but it's also easy to forget :
since there is a large distance between the position in the `Makefile` where the new built target is added,
and the place where the list of files to `clean` is defined.
Moreover, the list of files becomes pretty long over time,
hence it's difficult to visually ensure that all built targets are present there,
or that no old target (no longer produced) is no longer in the list

This PR tries to improve this process by adding a CLEAN variable.
Now, when a new built target is added to the `Makefile`,
it should preceded by :
```
CLEAN += newTarget
newTarget:
<TAB> ...recipe...
```

This new requirement is somewhat similar to `.PHONY: newTarget` for non-built targets.

This new method offers the advantage of locality :
there is no separate place in the file to maintain a list of files to clean.
This makes maintenance of `make clean` easier.
2022-09-07 16:36:03 -07:00
bc7492cefa added zstreamtest_ubsan to make file + added ubsan zstreamtest job for CI tests 2022-09-07 11:54:17 -07:00
282a955d33 added test that exposes zero offset to null pointer error when built with clang 2022-09-07 08:58:08 -07:00
4a6783bbbb remove zstream_ubsan from git and add to gitignore 2022-09-07 07:40:43 -07:00
69022ad886 null decompress buffer test and ubsan flag added 2022-09-06 14:34:55 -07:00
3b4e47092e Document pass-through behavior (#3242)
Adds documentation to help and man pages for legacy pass-through behavior
when force is set and destination is stdout. Documents --pass-through in
man pages
2022-08-15 10:29:54 -07:00
03cc84fddb Add explicit --pass-through flag and default to enabled for *cat (#3223)
Fixes #3211.

Adds the `--[no-]pass-through` flag which enables/disables pass-through mode.

* `zstdcat`, `zcat`, and `gzcat` default to `--pass-through`.
  Pass-through mode can be disabled by passing `--no-pass-through`.
* All other binaries default to not setting pass-through mode.
  However, we preserve the legacy behavior of enabling pass-through
  mode when writing to stdout with `-f` set, unless pass-through
  mode is explicitly disabled with `--no-pass-through`.

Adds a new test for this behavior that should codify the behavior we want.
2022-08-04 17:15:59 -07:00
d0dcc9d775 fix issue #3144 (#3226)
* fix issue #3144

* add test case for verbose-wlog

Co-authored-by: zengyijing <yijingzeng@fb.com>
2022-08-04 13:51:14 -07:00
a70ca2bd7d Fix off-by-one error in superblock mode (#3221)
Fixes #3212.

Long literal and match lengths had an off-by-one error in ZSTD_getSequenceLength.
Fix the off-by-one error, and add a golden compression test that catches the bug.
Also run all the golden tests in the cli-tests framework.
2022-08-03 11:28:39 -07:00
28ceb63503 Merge pull request #3220 from embg/issue3200
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
2022-08-01 14:04:57 -04:00
ae4670466c stdin multiple file fixes (#3222)
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
2022-07-29 16:13:07 -07:00
f9f27de91c Disallow empty output directory 2022-07-29 14:48:33 -07:00
d4a5bc4efc Add warning when multi-thread decompression is requested (#3208)
When user pass in argument for both decompression and multi-thread, print a warning message
to indicate that multi-threaded decompression is not supported.

* Add warning when multi-thread decompression is requested
* add test case for multi-threaded decoding warning
   Expectation is for -d -T0 we will not throw any warning,
   and see warning for any other -d -T(>1) inputs
2022-07-29 12:51:58 -07:00
2b9fde932b Fix small file passthrough (#3215) 2022-07-29 12:22:46 -07:00
b1bbb0eb4c [AIX] Fix Compiler Flags and Bugs on AIX to Pass All Tests (#3219)
* Fixing compiler warnings

* Replace the old -s flag with the -Wl,-s flag

* Fixing compiler warnings

* Fixing the linker strip flag and tests/code not working as expected on AIX
2022-07-29 12:21:59 -07:00
43f21a600e Intial commit to address 3090. Added support to decompress empty block. (#3118)
* Intial commit to address 3090. Added support to decompress empty block

* Update zstd_decompress_block.c

Addressed review comments for the case of 'set_basic'

* Update lib/decompress/zstd_decompress_block.c

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>

* Update lib/decompress/zstd_decompress_block.c

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
2022-07-14 11:54:34 -07:00
e9d6fc867a Merge pull request #3177 from embg/dms_prefetch2
Add prefetchCDictTables CCtxParam (+10-20% cold dict compression speed)
2022-06-24 08:24:43 -07:00
bb4a3c71ef Update README.md for fuzzers (#3174)
* Update README.md for fuzzers

* Add ls corpora/*crash command

* nit

* Clarify wording and add Nick's command

* Minor clarification
2022-06-22 21:02:07 -04:00
747e06f4f6 Add tests 2022-06-22 17:05:23 -04:00
91aeade735 Streaming decompression can detect incorrect header ID sooner
Streaming decompression used to wait for a minimum of 5 bytes before attempting decoding.
This meant that, in the case that only a few bytes (<5) were provided,
and assuming these bytes are incorrect,
there would be no error reported.
The streaming API would simply request more data, waiting for at least 5 bytes.

This PR makes it possible to detect incorrect Frame IDs as soon as the first byte is provided.

Fix #3169
2022-06-21 23:09:03 -07:00
1bc8019e10 Update results.csv 2022-05-11 10:27:35 -07:00
3be9a81e46 Update results.csv 2022-05-04 16:05:37 -04:00
460780f804 Merge pull request #3094 from dirkmueller/usage_cleanup
Split help in long and short version, cleanup formatting
2022-04-05 07:09:54 -07:00
678bfff4fe fix minor bug in sequence_compression_api tester
margin was merely slightly too short for extra splitting.
2022-03-29 16:45:09 -07:00
7fbe60d577 Split help in long and short version, cleanup formatting
Adopt the more standard Usage: formatting style
List short and long options alongside where available
Print lists as a table
Use command style description
2022-03-29 12:57:47 +02:00
f229daaf42 Merge pull request #3052 from dirkmueller/gzip_keep
Keep original file if -c or --stdout is given
2022-03-28 10:35:21 -07:00
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
7a3997c21a Handle newer less versions in zstdless testing
Newer less versions appear to have changed how stderr
and stdout are showing error messages. hardcode the
expected behavior to make the tests pass with any less version.

Also set locale to C so that the strings are matching.
2022-03-10 09:47:33 +01:00
3f4f8b04ed Keep original file if -c or --stdout is given
Set removeSrcFile back to false when -c or --stdout is used to improve
compatibility with gzip(1) behavior.

gzip(1) is removing the original file on compression unless --stdout or
/-c is used. zstd is defaulting to keep the file unless --rm is used or
when it is called via a gzip symlink, in which it is removing by
default. Specifying -c/--stdout turns this behavior off.
2022-03-09 23:38:59 +01:00
8814aa5bfa Merge pull request #3059 from dirkmueller/gzip_fast_best
Implement more gzip compatibility (#3037)
2022-03-08 15:32:06 -05:00
cf1894b324 Fix typos 2022-03-05 23:47:25 +02:00
d109cef201 fix the assertion in readLinesFromFile (#3084)
* fix the assertion in readLinesFromFile

When the file is not terminated by endline, readLineFromFile will append
a '\0' for the last line.  In this case pos + lineLength == dstCapacity.

* test: don't print very long text garbage
2022-03-04 14:56:44 -05:00
da737c7ab8 [programs] Fix infinite loop when empty input is passed to trainer
When an empty input file was passed to the dictionary trainer, it would infinite loop.
The added test case exposes the bug, and is fixed with this PR.
2022-03-02 11:05:26 -08:00
71d9dab76f Replace XOR with subtraction for readability 2022-02-16 16:49:42 -05:00
856c7dc51d Fix fuzzer.c nits and replace CLZ fallback 2022-02-16 11:40:05 -05:00
7c674a0919 Add tests for bitwise intrinsics 2022-02-14 18:36:33 -05:00
db2f4a6532 Move bitwise builtins into bits.h 2022-02-14 11:16:03 -05:00