1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-30 22:23:13 +03:00
Commit Graph

19 Commits

Author SHA1 Message Date
649a9c85c3 seekable_format: Add unit test for multiple decompress calls
This does the following:
1. Compress test data into multiple frames
2. Perform a series of small decompressions and seeks forward, checking
   that compressed data wasn't reread unnecessarily.
3. Perform some seeks forward and backward to ensure correctness.
2023-03-29 21:35:52 -07:00
1df9f36c6c Improved seekable format ingestion speed for small frame size
As reported by @P-E-Meunier in https://github.com/facebook/zstd/issues/2662#issuecomment-1443836186,
seekable format ingestion speed can be particularly slow
when selected `FRAME_SIZE` is very small,
especially in combination with the recent row_hash compression mode.
The specific scenario mentioned was `pijul`,
using frame sizes of 256 bytes and level 10.

This is improved in this PR,
by providing approximate parameter adaptation to the compression process.

Tested locally on a M1 laptop,
ingestion of `enwik8` using `pijul` parameters
went from 35sec. (before this PR) to 2.5sec (with this PR).
For the specific corner case of a file full of zeroes,
this is even more pronounced, going from 45sec. to 0.5sec.

These benefits are unrelated to (and come on top of) other improvement efforts currently being made by @yoniko for the row_hash compression method specifically.

The `seekable_compress` test program has been updated to allows setting compression level,
in order to produce these performance results.
2023-03-09 18:00:30 -08:00
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
2022-12-20 12:44:56 -05:00
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
aece0f258a free memory in test case 2022-12-13 08:15:16 -08:00
f17652931c seekable_format no header when compressing empty string to stream 2022-02-08 14:06:00 +01:00
sen
d6be7659b0 Add seekable roundtrip fuzzer (#2617) 2021-05-06 10:08:21 -04:00
3c6f5d5eca Fix seekable test to provide valid descriptor 2021-03-13 00:00:08 +02:00
6e390ced1f Merge branch 'seekTable' of github.com:facebook/zstd into seekTable 2021-03-03 22:44:38 -08:00
16ec1cf355 added test case for seekTable API
and simple roundtrip test
2021-03-03 18:56:23 -08:00
a1d7b9d654 fixed gcc conversion warnings 2021-03-03 15:17:12 -08:00
24d59a655d Merge branch 'dev' into seekTable 2021-03-03 15:08:40 -08:00
ac95a30455 various minor style fixes 2021-03-02 16:03:18 -08:00
029f974ddc strengthen compilation flags 2021-03-02 15:43:52 -08:00
527a20c3cd Fix seekable decompress hanging 2021-03-02 14:30:03 -08:00
3cbdbb888b ZSTD_seekable_decompress() example that hangs. 2021-03-02 14:25:17 -08:00
26f89d47aa Clean up makefile for seekable tests 2020-12-03 09:25:04 -05:00
152b55879c Add unit tests to seekable 2020-12-02 15:33:12 -05:00