1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-30 22:23:13 +03:00
Commit Graph

788 Commits

Author SHA1 Message Date
1d65917323 Move Single-File Build Script from contrib/ to build/ 2021-05-05 16:07:51 -04:00
53a60e98de seekable decompression fixes (#2594)
* seekable_format: fix from-file reading (not in-memory)

It tries to check the buffer boundary, but there is no buffer for
from-file reading.

* seekable_decompression: break when ZSTD_seekable_decompress() returns zero

* seekable_decompression_mem: break when ZSTD_seekable_decompress() returns zero

* seekable_format: cap the offset+len up to the last dOffset

This will allow to read the whole file w/o gotting corruption error if
the offset is more then the data left in file, i.e.:

    $ ./seekable_compression seekable_compression.c 8192 | head
    $ zstd -cdq seekable_compression.c.zst | wc -c
    4737

Before this patch:

    $ ./seekable_decompression seekable_compression.c.zst 0 10000000 | wc -c
    ZSTD_seekable_decompress() error : Corrupted block detected
    0

After:

    $ ./seekable_decompression seekable_compression.c.zst 0 10000000 | wc -c
    4737
2021-05-05 10:05:41 -04:00
09149beaf8 [1.5.0] Move zstd_errors.h and zdict.h to lib/ root
`zstd_errors.h` and `zdict.h` are public headers, so they deserve to be
in the root `lib/` directory with `zstd.h`, not mixed in with our private
headers.
2021-04-30 15:13:54 -07:00
fbb9006e18 [linux-kernel] Replace kernel-style comments
Replace kernel-style comments with regular comments.

E.g.

```
/** Before */

/* After */

/**
 * Before
 */

/*
 * After
 */

/***********************************
 * Before
 ***********************************/

/* *********************************
 * After
 ***********************************/
```
2021-04-29 15:50:23 -07:00
a423305e7b Remove ZBUFF tests 2021-04-19 17:27:05 -04:00
d334ad2ff4 [contrib][linux-kernel] Add zstd_min_clevel() and zstd_max_clevel() 2021-03-30 10:30:59 -07:00
a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
f8ac0ea7ef Merge pull request #2539 from terrelln/linux-kernel-fixes
Fixes for the next linux kernel patch version
2021-03-24 10:34:29 -07:00
cd1551d261 [lib][tracing] Add ZSTD_NO_TRACE macro
When defined, it disables tracing, and avoids including the header.
2021-03-16 11:47:27 -07:00
7222614a19 [contrib][freestanding] Remove tracing support
Remove tracing support from `freestanding.py` to keep things simple.
2021-03-16 11:47:27 -07:00
e4b914e663 [contrib][linux] Expose zstd headers to avoid duplication
Expose the zstd headers in `include/linux` to avoid struct duplication.
This makes the member names not follow Kernel style guidelines, and
exposes the zstd symbols. But, the LMKL reviewers are okay with that.
2021-03-16 11:47:22 -07:00
49a9e070f5 [contrib][linux-kernel] Update test include stubs
Update the test include stubs so they are able to run the current zstd
version in the kernel, so I can compare stack usage.
2021-03-16 11:40:24 -07:00
d2dd35ae65 [contrib][linux-kernel] Fix unaligned.h
Fix the `unaligned.h` shim in the tests that was causing corruption in
the tests. Note that this is a problem with the test shim, not the
kernel code.
2021-03-16 11:40:24 -07:00
3c6f5d5eca Fix seekable test to provide valid descriptor 2021-03-13 00:00:08 +02:00
21697b9c9e Fix seek table descriptor check when loading 2021-03-12 23:07:15 +02:00
2fa4c8c405 added code comments for new API ZSTD_seekTable 2021-03-03 22:54:04 -08:00
6e390ced1f Merge branch 'seekTable' of github.com:facebook/zstd into seekTable 2021-03-03 22:44:38 -08:00
16ec1cf355 added test case for seekTable API
and simple roundtrip test
2021-03-03 18:56:23 -08:00
713d4953f7 fixed gcc-7 conversion warning 2021-03-03 18:00:41 -08:00
6c0bfc468c fixed wrong assert condition 2021-03-03 15:30:55 -08:00
a1d7b9d654 fixed gcc conversion warnings 2021-03-03 15:17:12 -08:00
24d59a655d Merge branch 'dev' into seekTable 2021-03-03 15:08:40 -08:00
ac95a30455 various minor style fixes 2021-03-02 16:03:18 -08:00
029f974ddc strengthen compilation flags 2021-03-02 15:43:52 -08:00
c7e42e147b fixed const guarantees
read-only objects are properly const-ified in parameters
2021-03-02 15:24:30 -08:00
a80b10f5e6 fix potential leak on exit 2021-03-02 15:03:37 -08:00
527a20c3cd Fix seekable decompress hanging 2021-03-02 14:30:03 -08:00
3cbdbb888b ZSTD_seekable_decompress() example that hangs. 2021-03-02 14:25:17 -08:00
ce6d1b9376 Merge pull request #2113 from mdittmer/expose-seek-table
[contrib] Support seek table-only API
2021-03-02 10:50:47 -08:00
adb54293ab Stop using deprecated reset?Stream functions
These are replaced by the corresponding context resets. When
converting resetCStream, CCtx_setPledgedSrcSize isn't called if the
source size is "unknown".

This helps reduce the reliance on "static only" symbols, as well as
reducing the use of deprecated functions.

Signed-off-by: Stephen Kitt <steve@sk2.org>
2021-02-23 21:56:01 +01:00
54a4998a80 Add basic tracing functionality 2021-02-05 16:28:52 -08:00
b45d22c851 [contrib][recovery] Add recovery_directory program
This program takes a file with concatenated zstd frames and splits the
file up by frame. E.g. if `dir.zst` has 4 frames:

```
> ./recover_directory dir.zst recovery/file
Recovering 4 files...
Recovered recovery/file0
Recovered recovery/file1
Recovered recovery/file2
Recovered recovery/file3
Complete
```
2021-01-15 08:45:22 -08:00
4f7584e7a3 Allow freestanding lib script regex to detect XXH64( 2021-01-07 12:29:12 -05:00
66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
0b39531d75 moving all references to release branch
was previously `master`
2020-12-16 23:00:35 -08:00
sen
f34d2f4192 Merge pull request #2408 from senhuang42/seekable_hang_fix
Remove possibility of hanging when using seekable decompression
2020-12-07 08:46:27 -05:00
26f89d47aa Clean up makefile for seekable tests 2020-12-03 09:25:04 -05:00
152b55879c Add unit tests to seekable 2020-12-02 15:33:12 -05:00
10f87d2f43 [contrib][linux] Add SPDX-License-Identifier 2020-12-02 12:00:44 -08:00
d0cc2e3868 [contrib][linux] Fix make import-upstream 2020-12-02 11:43:28 -08:00
3256c00c6e [contrib][linux] Follow style guide more closely 2020-12-02 11:07:38 -08:00
2c570e7f87 [contrib][linux] Add a make target to import upstream 2020-12-02 10:42:33 -08:00
9db49a3989 Add a forward progress requirement bound to seekable streaming decompression 2020-12-02 12:24:16 -05:00
87b38156be [contrib][linux-kernel] Fix test warnings 2020-12-02 01:12:42 -08:00
f58e9a9830 [contrib][linux] Add wrapper API
Add the kernel wrapper API. This keeps the same API and semantics as the
existing kernel API with name changes to be more kernel style and avoid
symbol collisions with zstd.
2020-12-02 01:08:08 -08:00
6a1e526ea7 [lib] Add ZSTD_COMPRESS_HEAPMODE tuning parameter 2020-09-24 19:42:04 -07:00
b841387218 [freestanding] Improve macro resolution to handle #if X 2020-09-24 19:42:04 -07:00
caecd8c211 Allow user to override ASAN/MSAN detection
Rename ADDRESS_SANITIZER -> ZSTD_ADDRESS_SANITIZER and same for
MEMORY_SANITIZER. Also set it to 0/1 instead of checking for defined.
This allows the user to override ASAN/MSAN detection for platforms that
don't support it.
2020-09-24 19:42:04 -07:00
683150e59f [linux-kernel] Avoid including <string.h> in the tests 2020-09-24 19:42:04 -07:00
9ae0483858 Reorganize zstd_deps.h and mem.h + replace mem.h for the kernel 2020-09-24 19:41:59 -07:00