1
0
mirror of https://github.com/facebook/zstd.git synced 2025-08-08 17:22:10 +03:00
Commit Graph

2014 Commits

Author SHA1 Message Date
Elliot Gorokhovsky
bb3c01c853 Migrate other test usages of boolean LDM flag to paramSwitch enum 2022-11-21 16:20:38 -05:00
Elliot Gorokhovsky
3720910d06 Fix fuzzer failure 2022-11-21 16:09:04 -05:00
Danielle Rozenblit
db74d043d6 Speed optimizations with macro 2022-10-27 10:20:44 -07:00
daniellerozenblit
0d5d571080 Merge pull request #3285 from daniellerozenblit/optimal-huff-depth
Optimal huf depth
2022-10-18 10:31:44 -04:00
Daniel Kutenin
1d153c923c Make fuzzing work without ZSTD_MULTITHREAD (#3291) 2022-10-17 14:31:47 -07:00
Danielle Rozenblit
75cd42afd7 Update regression results and better variable naming for HUF_cardinality 2022-10-14 13:37:19 -07:00
Danielle Rozenblit
117fddcd81 Update regression results 2022-10-12 13:49:44 -07:00
Danielle Rozenblit
8888a2ddcc CI failure fixes 2022-10-11 13:12:19 -07:00
Yonatan Komornik
21bd8c3b3c Removed unused variable (#3272) 2022-09-22 08:20:46 -07:00
ctkhanhly
358787764f Fix for zstd CLI accepts bogus values for numeric parameters (#3268)
* add checks to mal-formed numeric values for memory and memlimit parameters

Signed-off-by: Ly Cao <lycao@fb.com>

* changed errorMsg to a literal string instead of static string in main

* moved bogus numeric error to NEXT_UINT32 + add macro NEXT_TSIZE

Signed-off-by: Ly Cao <lycao@fb.com>

Signed-off-by: Ly Cao <lycao@fb.com>
Co-authored-by: Ly Cao <lycao@fb.com>
2022-09-21 13:20:01 -07:00
Yann Collet
361b7bd1d0 Merge branch 'dev' into clean 2022-09-12 10:56:00 -07:00
Danielle Rozenblit
a06e953db9 some additional comments, remove apt-get from clang jobs, better test titles 2022-09-08 18:30:07 -07:00
Yann Collet
c0b46738b4 streamline make clean list maintenance
When creating a new `Makefile` target to build,
it's also necessary to update the `clean` target,
which purpose is to remove built targets when they are present.

This process is simple, but it's also easy to forget :
since there is a large distance between the position in the `Makefile` where the new built target is added,
and the place where the list of files to `clean` is defined.
Moreover, the list of files becomes pretty long over time,
hence it's difficult to visually ensure that all built targets are present there,
or that no old target (no longer produced) is no longer in the list

This PR tries to improve this process by adding a CLEAN variable.
Now, when a new built target is added to the `Makefile`,
it should preceded by :
```
CLEAN += newTarget
newTarget:
<TAB> ...recipe...
```

This new requirement is somewhat similar to `.PHONY: newTarget` for non-built targets.

This new method offers the advantage of locality :
there is no separate place in the file to maintain a list of files to clean.
This makes maintenance of `make clean` easier.
2022-09-07 16:36:03 -07:00
Danielle Rozenblit
bc7492cefa added zstreamtest_ubsan to make file + added ubsan zstreamtest job for CI tests 2022-09-07 11:54:17 -07:00
Danielle Rozenblit
282a955d33 added test that exposes zero offset to null pointer error when built with clang 2022-09-07 08:58:08 -07:00
Danielle Rozenblit
4a6783bbbb remove zstream_ubsan from git and add to gitignore 2022-09-07 07:40:43 -07:00
Danielle Rozenblit
69022ad886 null decompress buffer test and ubsan flag added 2022-09-06 14:34:55 -07:00
Chris Burgess
3b4e47092e Document pass-through behavior (#3242)
Adds documentation to help and man pages for legacy pass-through behavior
when force is set and destination is stdout. Documents --pass-through in
man pages
2022-08-15 10:29:54 -07:00
Nick Terrell
03cc84fddb Add explicit --pass-through flag and default to enabled for *cat (#3223)
Fixes #3211.

Adds the `--[no-]pass-through` flag which enables/disables pass-through mode.

* `zstdcat`, `zcat`, and `gzcat` default to `--pass-through`.
  Pass-through mode can be disabled by passing `--no-pass-through`.
* All other binaries default to not setting pass-through mode.
  However, we preserve the legacy behavior of enabling pass-through
  mode when writing to stdout with `-f` set, unless pass-through
  mode is explicitly disabled with `--no-pass-through`.

Adds a new test for this behavior that should codify the behavior we want.
2022-08-04 17:15:59 -07:00
zengyijing
d0dcc9d775 fix issue #3144 (#3226)
* fix issue #3144

* add test case for verbose-wlog

Co-authored-by: zengyijing <yijingzeng@fb.com>
2022-08-04 13:51:14 -07:00
Nick Terrell
a70ca2bd7d Fix off-by-one error in superblock mode (#3221)
Fixes #3212.

Long literal and match lengths had an off-by-one error in ZSTD_getSequenceLength.
Fix the off-by-one error, and add a golden compression test that catches the bug.
Also run all the golden tests in the cli-tests framework.
2022-08-03 11:28:39 -07:00
Elliot Gorokhovsky
28ceb63503 Merge pull request #3220 from embg/issue3200
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
2022-08-01 14:04:57 -04:00
Yonatan Komornik
ae4670466c stdin multiple file fixes (#3222)
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
2022-07-29 16:13:07 -07:00
Elliot Gorokhovsky
f9f27de91c Disallow empty output directory 2022-07-29 14:48:33 -07:00
Tom Wang
d4a5bc4efc Add warning when multi-thread decompression is requested (#3208)
When user pass in argument for both decompression and multi-thread, print a warning message
to indicate that multi-threaded decompression is not supported.

* Add warning when multi-thread decompression is requested
* add test case for multi-threaded decoding warning
   Expectation is for -d -T0 we will not throw any warning,
   and see warning for any other -d -T(>1) inputs
2022-07-29 12:51:58 -07:00
Chris Burgess
2b9fde932b Fix small file passthrough (#3215) 2022-07-29 12:22:46 -07:00
Qiongsi Wu
b1bbb0eb4c [AIX] Fix Compiler Flags and Bugs on AIX to Pass All Tests (#3219)
* Fixing compiler warnings

* Replace the old -s flag with the -Wl,-s flag

* Fixing compiler warnings

* Fixing the linker strip flag and tests/code not working as expected on AIX
2022-07-29 12:21:59 -07:00
udayanbapat
43f21a600e Intial commit to address 3090. Added support to decompress empty block. (#3118)
* Intial commit to address 3090. Added support to decompress empty block

* Update zstd_decompress_block.c

Addressed review comments for the case of 'set_basic'

* Update lib/decompress/zstd_decompress_block.c

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>

* Update lib/decompress/zstd_decompress_block.c

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
2022-07-14 11:54:34 -07:00
Elliot Gorokhovsky
e9d6fc867a Merge pull request #3177 from embg/dms_prefetch2
Add prefetchCDictTables CCtxParam (+10-20% cold dict compression speed)
2022-06-24 08:24:43 -07:00
Elliot Gorokhovsky
bb4a3c71ef Update README.md for fuzzers (#3174)
* Update README.md for fuzzers

* Add ls corpora/*crash command

* nit

* Clarify wording and add Nick's command

* Minor clarification
2022-06-22 21:02:07 -04:00
Elliot Gorokhovsky
747e06f4f6 Add tests 2022-06-22 17:05:23 -04:00
Yann Collet
91aeade735 Streaming decompression can detect incorrect header ID sooner
Streaming decompression used to wait for a minimum of 5 bytes before attempting decoding.
This meant that, in the case that only a few bytes (<5) were provided,
and assuming these bytes are incorrect,
there would be no error reported.
The streaming API would simply request more data, waiting for at least 5 bytes.

This PR makes it possible to detect incorrect Frame IDs as soon as the first byte is provided.

Fix #3169
2022-06-21 23:09:03 -07:00
W. Felix Handte
1bc8019e10 Update results.csv 2022-05-11 10:27:35 -07:00
Elliot Gorokhovsky
3be9a81e46 Update results.csv 2022-05-04 16:05:37 -04:00
Yann Collet
460780f804 Merge pull request #3094 from dirkmueller/usage_cleanup
Split help in long and short version, cleanup formatting
2022-04-05 07:09:54 -07:00
Yann Collet
678bfff4fe fix minor bug in sequence_compression_api tester
margin was merely slightly too short for extra splitting.
2022-03-29 16:45:09 -07:00
Dirk Müller
7fbe60d577 Split help in long and short version, cleanup formatting
Adopt the more standard Usage: formatting style
List short and long options alongside where available
Print lists as a table
Use command style description
2022-03-29 12:57:47 +02:00
Nick Terrell
f229daaf42 Merge pull request #3052 from dirkmueller/gzip_keep
Keep original file if -c or --stdout is given
2022-03-28 10:35:21 -07:00
Dominique Pelle
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Dirk Müller
7a3997c21a Handle newer less versions in zstdless testing
Newer less versions appear to have changed how stderr
and stdout are showing error messages. hardcode the
expected behavior to make the tests pass with any less version.

Also set locale to C so that the strings are matching.
2022-03-10 09:47:33 +01:00
Dirk Müller
3f4f8b04ed Keep original file if -c or --stdout is given
Set removeSrcFile back to false when -c or --stdout is used to improve
compatibility with gzip(1) behavior.

gzip(1) is removing the original file on compression unless --stdout or
/-c is used. zstd is defaulting to keep the file unless --rm is used or
when it is called via a gzip symlink, in which it is removing by
default. Specifying -c/--stdout turns this behavior off.
2022-03-09 23:38:59 +01:00
Felix Handte
8814aa5bfa Merge pull request #3059 from dirkmueller/gzip_fast_best
Implement more gzip compatibility (#3037)
2022-03-08 15:32:06 -05:00
Dimitris Apostolou
cf1894b324 Fix typos 2022-03-05 23:47:25 +02:00
Xi Ruoyao
d109cef201 fix the assertion in readLinesFromFile (#3084)
* fix the assertion in readLinesFromFile

When the file is not terminated by endline, readLineFromFile will append
a '\0' for the last line.  In this case pos + lineLength == dstCapacity.

* test: don't print very long text garbage
2022-03-04 14:56:44 -05:00
Nick Terrell
da737c7ab8 [programs] Fix infinite loop when empty input is passed to trainer
When an empty input file was passed to the dictionary trainer, it would infinite loop.
The added test case exposes the bug, and is fixed with this PR.
2022-03-02 11:05:26 -08:00
Elliot Gorokhovsky
71d9dab76f Replace XOR with subtraction for readability 2022-02-16 16:49:42 -05:00
Elliot Gorokhovsky
856c7dc51d Fix fuzzer.c nits and replace CLZ fallback 2022-02-16 11:40:05 -05:00
Elliot Gorokhovsky
7c674a0919 Add tests for bitwise intrinsics 2022-02-14 18:36:33 -05:00
Elliot Gorokhovsky
db2f4a6532 Move bitwise builtins into bits.h 2022-02-14 11:16:03 -05:00
Dirk Müller
e653e97f77 Implement more gzip compatibility (#3037)
-n --no-name is the current behavior already, so we can implement
this as a noop.

--best is an alias for -9 in gzip

add basic cli tests.
2022-02-09 23:09:18 +01:00