1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-30 22:23:13 +03:00

2060 Commits

Author SHA1 Message Date
7c7b9244d6 update regression results 2021-12-16 16:07:54 -08:00
9a32492730 updated regression results.csv 2021-12-16 14:39:30 -08:00
c5f1e826ca Merge pull request #2925 from embg/dict_training_sample_limit_size
Allow user to specify memory limit for dictionary training
2021-12-15 15:58:17 -05:00
71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00
5e2fede604 Merge pull request #2921 from felixhandte/neg-lvl-stagger-step
Stagger Stepping in Negative Levels
2021-12-14 14:13:57 -05:00
450fca9704 Update Regression Tests w/ New Sizes 2021-12-13 17:29:32 -05:00
3e2a70b6fb Merge pull request #2905 from 15596858998/dev_1205
add test case
2021-12-13 13:45:23 -08:00
c69d13eb99 Update playTests.sh 2021-12-13 08:58:42 +08:00
0c26d98c0d Merge pull request #2910 from felixhandte/reject-irregular-dicts
Reject Irregular Dictionary Files
2021-12-09 11:44:37 -05:00
9985e10fda Reject Irregular Dictionary Files
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
2021-12-08 16:17:04 -05:00
fb3522a3fe fixed very minor cast warning under cygwin 2021-12-08 09:48:56 -08:00
38dfc4699e Imply -q when stderr is not a tty (#2884)
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
ba7e8f1051 add test case 2021-12-05 19:12:52 +08:00
01ecd6ffc0 Merge pull request #2892 from terrelln/issue-2785
[CircleCI] Fix short-tests-0
2021-12-02 16:20:56 -05:00
1e514feec6 Makefile: sort all wildcard file list expansions
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
91f5891dd0 [CircleCI] Fix short-tests-0
short-tests-0 were silently failing. I think because of the && make clean construction. Switch to ; instead.

Also fix all the test failures that were exposed.

`make all` is failing on CircleCI because it is missing Docker. Move that test
to GitHub actions, and switch the pedantic CircleCI test to `make allmost`.
2021-12-01 17:43:46 -08:00
9b97fdf74f Merge pull request #2887 from terrelln/issue-2815
[zdict] Remove ZDICT_CONTENTSIZE_MIN restriction for ZDICT_finalizeDictionary
2021-12-01 18:15:53 -05:00
49d578763d reduce storage requirement
51 MB seems excessive for CI storate and considering the nature of the test.

(note : maybe we should consider using `/tmp` for files generated during tests,
as tmpfs is typically using RAM, thus preserving storage.)
2021-12-01 15:10:55 -08:00
3133d1e86e Merge pull request #2876 from 15596858998/dev
Solve the bug of extra output newline character
2021-12-01 15:10:08 -08:00
8031dc7a48 Merge pull request #2885 from yoniko/limit-level-32bit-systems
Limit `ZSTD_maxCLevel` to 21 for 32-bit binaries.
2021-12-01 14:19:16 -08:00
360c2630e4 [test] Test that the exec-stack bit isn't set on libzstd.so
Tests that libzstd.so doesn't have the exec-stack bit set using
readelf. If the stack is marked executable systemd will refuse
to link against zstd. We now test that it isn't set on every PR.

Adds a test for PR #2857
Fixes Issue #2865
2021-11-30 18:13:00 -08:00
0356e05f07 [zdict] Remove ZDICT_CONTENTSIZE_MIN restriction for ZDICT_finalizeDictionary
Allow the `dictContentSize` to be any size. The finalized dictionary
content size must be at least as large as the maximum repcode (8). So we
add zero bytes to the dictionary to ensure that we meet that
requirement.

I've removed this restriction because its been causing us headaches when
people complain that dictionary training failed. It fails because there
isn't enough useful content to put in the dictionary. Either because
every sample is exactly the same and less than ZDICT_CONTENTSIZE_MIN bytes,
or there isn't enough content. Instead, we should succeed in creating
the dictionary, and it is up to the user to decide if it is worthwhile.
It is possible that the tables alone provide enough value.

NOTE: This allows us to produce dictionaries with finalized
`dictContentSize < ZDICT_CONTENTSIZE_MIN`. But, they are still valid
zstd dictionaries. We could remove the `ZDICT_CONTENTSIZE_MIN` macro,
but I've decided to leave that for now, so we don't break users.
2021-11-30 18:02:26 -08:00
ef2cba609d ZSTD_maxCLevel now limited to 21 for 32-bit binaries.
CI tests for constrained memory runs with max level on 32-bit binaries.
2021-11-30 10:31:52 -08:00
7ce6f71c6f 更新 playTests.sh 2021-11-21 12:35:58 +08:00
ebbd675998 Fix typos 2021-11-13 10:04:04 +02:00
ddae153947 Merge pull request #2847 from Svetlitski-FB/improve-verbose-output-2
Display command line parameters with concrete values in verbose mode
2021-11-12 18:51:18 -08:00
9b28c26cbf Integrate verbose mode tests into playTests.sh 2021-11-12 14:10:21 -08:00
f6ffd39230 Add test case for detailed compression parameter verbose output 2021-11-11 11:59:54 -08:00
931778ed9b Fix fullbench CI failure (#2851) 2021-11-09 15:15:35 -05:00
6a7ede3dfc Reduce size of dctx by reutilizing dst buffer (#2751)
* Reduce size of dctx by reutilizing dst buffer

Co-authored-by: Binh Vo <binhvo@fb.com>
2021-10-25 10:38:01 -04:00
23c1a2d260 Merge pull request #2774 from felixhandte/zstd-dfast-pipelined-single
Pipelined Implementation of ZSTD_dfast
2021-10-13 16:38:43 -04:00
c6c482fe07 [binary-tree] Fix underflow of nbCompares
Fix underflow of `nbCompares` by switching to an `int` and comparing
`nbCompares > 0`. This is a minimal fix, because I don't want to change
the logic. These loops seem to be doing `nbCompares + 1` comparisons.

The bug was reported by Dan Carpenter and found by Smatch static
checker.

https://lore.kernel.org/all/20211008063704.GA5370@kili/
2021-10-08 13:22:55 -07:00
168d0a3c89 Fix Flaky Test
This test depended on `_extDict` and `_noDict` compressing identically, which
is not a guarantee we make, AFAIK.
2021-10-05 16:18:09 -04:00
c2c32839dc Update results.csv 2021-10-05 16:18:00 -04:00
7868f38019 Merge pull request #2747 from Helflym/dev
Add AIX support in Makefile
2021-10-01 08:13:39 -07:00
9360367371 Update regression test 2021-09-28 08:29:11 -07:00
b8fd6bf30c Skip most long matches in lazy hash table update 2021-09-28 08:19:39 -07:00
0d45540695 decompress: conditionally remove bmi2 from context
Use an helper function, which will just return 0 in case
the feature is disabled.
Allows constant propagation and removal of dead code.
2021-09-26 14:41:37 +02:00
sen
044c8b4722 Merge pull request #2779 from senhuang42/fse_fix
Fix NCountWriteBound
2021-09-22 13:51:21 -04:00
sen
1e99d36361 Merge pull request #2788 from senhuang42/param_switch
Use new paramSwitch enum for row matchfinder and block splitter
2021-09-22 13:27:55 -04:00
99b5e7b8c2 Add test case for FSE over-write 2021-09-22 12:03:46 -04:00
b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
a5f2c45528 Huffman ASM 2021-09-20 14:46:43 -07:00
d7542aacd9 [fuzzer] Add huf_decompress fuzzer
Add a fuzzer for Huffman decompression. Fix several bugs in Huffman
decompression, mostly related to `op == NULL` and pointer underflow.
2021-09-17 15:00:49 -07:00
8bf699aa59 [build] Add support for ASM files in Make + CMake
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
2021-09-17 14:13:53 -07:00
fd94b9d1c9 Merge branch 'dev' into opt_investigation 2021-09-14 01:15:51 -07:00
d45d0ad9d8 Update regression test 2021-09-13 12:41:02 -04:00
4a498fb9c3 Add a dictionary training large corpus test 2021-09-13 12:29:17 -04:00
b6b2855b80 updated regression tests 2021-09-12 10:22:35 -07:00
f58e63bee7 Merge branch 'dev' into opt_investigation 2021-09-12 01:42:49 -07:00