Elliot Gorokhovsky
581ecdc021
fix 32-bit gcc mingw test
2023-01-11 10:11:55 -08:00
Elliot Gorokhovsky
73fbe32bd5
remove cmake 2019 test
2023-01-11 10:11:55 -08:00
Elliot Gorokhovsky
96b2d0c7c5
upgrade setup-msbuild from v1.1.3 to v1.3
2023-01-11 10:11:55 -08:00
Elliot Gorokhovsky
a931310a4a
Overhaul windows CI
2023-01-11 10:11:47 -08:00
Yann Collet
98ca8b4456
Merge pull request #3414 from facebook/dependabot/github_actions/actions/checkout-3.3.0
...
Bump actions/checkout from 3.2.0 to 3.3.0
2023-01-09 11:33:29 -08:00
dependabot[bot]
6f17a5d8df
Bump actions/checkout from 3.2.0 to 3.3.0
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](755da8c3cf...ac59398561support@github.com >
2023-01-09 05:10:46 +00:00
Yann Collet
71dbe8f9d4
minor: fix conversion warnings
2023-01-04 20:00:04 -08:00
Yann Collet
834fd07a99
Merge pull request #3391 from facebook/fix3228
...
improve compression ratio of small alphabets
2023-01-03 16:01:52 -08:00
Yann Collet
c79fb4d78d
update levels.sh test
...
comparing level 19 to level 22 and expecting a stricter better result from level 22
is not that guaranteed,
because level 19 and 22 are very close to each other,
especially for small files,
so any noise in the final compression result
result in failing this test.
Level 22 could be compared to something much lower, like level 15,
But level 19 is required anyway, because there is a clamping test which depends on it.
Removed level 22, kept level 19
2023-01-03 14:04:41 -08:00
Yann Collet
ebba9ff425
update regression results
2023-01-03 14:04:23 -08:00
Yann Collet
5434de01e2
improve compression ratio of small alphabets
...
fix #3328
In situations where the alphabet size is very small,
the evaluation of literal costs from the Optimal Parser is initially incorrect.
It takes some time to converge, during which compression is less efficient.
This is especially important for small files,
because there will not be enough data to converge,
so most of the parsing is selected based on incorrect metrics.
After this patch, the scenario ##3328 gets fixed,
delivering the expected 29 bytes compressed size (smallest known compressed size).
2023-01-03 12:22:37 -08:00
daniellerozenblit
1c818e3a0a
Merge pull request #3302 from daniellerozenblit/optimal-huff-depth-speed
...
Optimal huff depth speed improvements
2023-01-03 12:51:51 -05:00
Danielle Rozenblit
87becc567d
update regression results.csv
2023-01-03 08:41:40 -08:00
Danielle Rozenblit
df714ddb0f
implement suggestions
2023-01-03 07:20:21 -08:00
Yann Collet
3248910432
Merge pull request #3248 from facebook/opt_comments1
...
[easy] add a few comments to the optimal parser code base for improved clarity
2022-12-28 18:03:57 -08:00
Yann Collet
d07e72bb13
fixed incorrect assert
...
commented Fweight instead
2022-12-28 17:23:40 -08:00
Yann Collet
4a1a79a512
just add some comments to zstd_opt for improved clarity
2022-12-28 16:24:12 -08:00
Yann Collet
9fbbd74871
Merge pull request #3400 from danlark1/dev
...
Move deprecated annotation before static to allow C++ compilation for clang
2022-12-28 15:50:26 -08:00
Yann Collet
bcbd395c1c
Merge pull request #3395 from terrelln/2022-12-21-deprecated-test
...
[tests] Remove deprecated function from longmatch.c test
2022-12-28 15:49:50 -08:00
Yann Collet
00c85b28e7
update ZSTD_CCts_setCParams() inline documentation
...
specify behavior when changing compression parameters during MT compression,
reported by @embg
2022-12-28 15:08:18 -08:00
Yann Collet
481a2e1010
Merge pull request #3403 from facebook/setCParams
...
ZSTD_CCtx_setCParams
2022-12-28 14:07:13 -08:00
Elliot Gorokhovsky
2a402626dd
External matchfinder API ( #3333 )
...
* First building commit with sample matchfinder
* Set up ZSTD_externalMatchCtx struct
* move seqBuffer to ZSTD_Sequence*
* support non-contiguous dictionary
* clean up parens
* add clearExternalMatchfinder, handle allocation errors
* Add useExternalMatchfinder cParam
* validate useExternalMatchfinder cParam
* Disable LDM + external matchfinder
* Check for static CCtx
* Validate mState and mStateDestructor
* Improve LDM check to cover both branches
* Error API with optional fallback
* handle RLE properly for external matchfinder
* nit
* Move to a CDict-like model for resource ownership
* Add hidden useExternalMatchfinder bool to CCtx_params_s
* Eliminate malloc, move to cwksp allocation
* Handle CCtx reset properly
* Ensure seqStore has enough space for external sequences
* fix capitalization
* Add DEBUGLOG statements
* Add compressionLevel param to matchfinder API
* fix c99 issues and add a param combination error code
* nits
* Test external matchfinder API
* C90 compat for simpleExternalMatchFinder
* Fix some @nocommits and an ASAN bug
* nit
* nit
* nits
* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h
* nit
* nit
* nits
* Update copyright headers
* Fix CMake zstreamtest build
* Fix copyright headers (again)
* typo
* Add externalMatchfinder demo program to make contrib
* Reduce memory consumption for small blockSize
* ZSTD_postProcessExternalMatchFinderResult nits
* test sum(matchlen) + sum(litlen) == srcSize in debug builds
* refExternalMatchFinder -> registerExternalMatchFinder
* C90 nit
* zstreamtest nits
* contrib nits
* contrib nits
* allow block splitter + external matchfinder, refactor
* add windowSize param
* add contrib/externalMatchfinder/README.md
* docs
* go back to old RLE heuristic because of the first block issue
* fix initializer element is not a constant expression
* ref contrib from zstd.h
* extremely pedantic compiler warning fix, meson fix, typo fix
* Additional docs on API limitations
* minor nits
* Refactor maxNbSeq calculation into a helper function
* Fix copyright
2022-12-28 16:45:14 -05:00
Yann Collet
b17743e41b
Signal parameter change during MT compression
2022-12-28 13:14:58 -08:00
Yann Collet
89342d1e07
New xp library symbol : ZSTD_CCtx_setCParams()
...
Inspired by #3395 ,
offer a new capability to set all parameters defined in a ZSTD_compressionParameters structure
with a single symbol invocation
to improve user code brevity.
2022-12-27 23:49:22 -08:00
Yann Collet
90597d78ea
Merge pull request #3394 from terrelln/issue-3010
...
[cli-tests] Test file stat read/write
2022-12-27 16:20:05 -08:00
Yann Collet
6640377783
cmake build: fix nit
...
reported by @jaimeMF in https://github.com/facebook/zstd/pull/3392#discussion_r1056643794
2022-12-23 14:18:11 -08:00
Daniel Kutenin
48f4aa7307
Move deprecated annotation before static to allow C++ compilation for clang
...
This fixes last 2 instances of https://github.com/facebook/zstd/issues/3250
2022-12-23 12:07:31 +00:00
Yann Collet
089b2797e3
Merge pull request #3398 from facebook/fix3316
...
spec update : require minimum nb of literals for 4-streams mode
2022-12-22 16:57:05 -08:00
Yann Collet
6a9c525903
spec update : require minimum nb of literals for 4-streams mode
...
Reported by @shulib :
the specification for 4-streams mode
doesn't work when the amount of literals to compress is 5 bytes.
Extending it, it also doesn't work for sizes 1 or 2.
This patch updates the specification and the implementation
to require a minimum of 6 literals to trigger or accept the 4-streams mode.
The impact is expected to be a no-op :
the 4-streams mode is never triggered for such small quantity of literals anyway,
since it would be wasteful (it costs ~7.3 bytes more than single-stream mode).
An informal lower limit is set at ~256 bytes,
so the technical minimum is very far from this limit.
This is just meant for completeness of the specification.
2022-12-22 16:14:34 -08:00
Yann Collet
8209bfc4f3
Merge pull request #3399 from facebook/fix2577
...
Support decompression of compressed blocks of size ZSTD_BLOCKSIZE_MAX
2022-12-22 14:05:36 -08:00
Yann Collet
ea2895cef4
Support decompression of compressed blocks of size ZSTD_BLOCKSIZE_MAX exactly
2022-12-22 12:40:27 -08:00
Felix Handte
f5ea3a196f
Merge pull request #3397 from felixhandte/man-page-tweaks
...
Man Page Tweaks, Edits, Formatting Fixes
2022-12-22 14:49:59 -05:00
W. Felix Handte
11aba9b316
make man
2022-12-22 14:13:24 -05:00
W. Felix Handte
382026f096
Man Page Tweaks, Edits, Formatting Fixes
...
This started as an application of the edits suggested in #3201 and expanded
from there.
2022-12-22 14:13:17 -05:00
Nick Terrell
7fe7a166c2
[cli-tests] Add tests that use --trace-file-stat
...
Basic tests for (de)compressing in the following modes:
* file to file
* file to stdout
* stdin to file
* stdin to stdout
These are basic tests, and aren't testing more advanced scenarios, but
it adds the groundwork for more complex tests as needed.
Fixes #3010 .
2022-12-21 18:32:12 -08:00
Nick Terrell
4b40e405d3
[tests] Remove deprecated function from longmatch.c test
...
Thanks to @eli-schwartz for pointing it out!
We should maybe consider adding a helper function for applying
`ZSTD_parameters` and `ZSTD_compressionParameters` to a context.
That would aid the transition to the new API in situations like this.
2022-12-21 17:52:10 -08:00
Nick Terrell
40a7188130
Fix make clangbuild & add CI
...
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
2022-12-21 17:31:04 -08:00
Nick Terrell
31a703ec13
[cmake] Add noexecstack to compiler/linker flags
2022-12-21 17:30:14 -08:00
Nick Terrell
7df6e25b85
[cli-tests] Add --set-exact-output to update the expected output
...
`./run.py --set-exact-output` will update `stdout.expect` and
`stderr.expect` to match the expected output. This doesn't apply to
outputs which use `.glob` or `.ignore`.
2022-12-21 17:21:09 -08:00
Nick Terrell
b6e8112261
[util] Add traces enabled by --trace-file-stat
...
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
2022-12-21 17:19:34 -08:00
Yann Collet
7607b96ed9
Merge pull request #3385 from jonpalmisc/improve_help
...
Improve help/usage (`-h`, `-H`) formatting
2022-12-21 10:56:09 -08:00
Felix Handte
0694f1435d
Merge pull request #3386 from felixhandte/pin-other-action-commit-hashes
...
Pin Remaining Action Dependencies (Except OSS-Fuzz)
2022-12-21 11:24:23 -05:00
Jon Palmisciano
125bcde17c
Update tests to expect new CLI help output
2022-12-20 20:13:00 -05:00
Jon Palmisciano
9c93dd71cd
Clean up welcome message
2022-12-20 20:13:00 -05:00
Jon Palmisciano
678335c4f3
Rewrite help output to improve readability
2022-12-20 20:12:59 -05:00
W. Felix Handte
150aa23ef4
Pin Remaining Action Dependencies (Except OSS-Fuzz)
...
The one that isn't pinned is the OSS-Fuzz builder and runner. They don't
offer tagged releases. I could pin to the current master commit, but I'm not
sure how desirable that is.
2022-12-20 17:10:01 -05:00
Felix Handte
0790e86c56
Merge pull request #3384 from felixhandte/pin-checkout-action-commit-hash
...
Pin actions/checkout Dependency to Specific Commit Hash
2022-12-20 16:08:03 -05:00
Nick Terrell
651a381060
[build][cmake] Fix cmake with custom assembler
...
Tell CMake to explicitly compile our assembly as C code, because we
require it is compiled by a C compiler, and it is only enabled for
clang/gcc.
Fixes #3193 .
2022-12-20 13:02:21 -08:00
Danielle Rozenblit
c26f348dc8
fix CI errors
2022-12-20 12:43:46 -08:00
Danielle Rozenblit
2def93bf10
Merge branch 'optimal-huff-depth-speed' of github.com:daniellerozenblit/zstd into optimal-huff-depth-speed
2022-12-20 12:28:07 -08:00
