From a29b6ed2510795bb7bca93bb6fadb09cf4c0345d Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 17 Apr 2023 15:43:27 -0700 Subject: [PATCH 1/2] added decoder errata paragraph for compressed blocks of size exactly 128 KB which used to be disallowed by the spec but have become allowed in more recent version of the spec. While this limitation is fixed in decoders v1.5.4+, implementers should refrain from generating such block with their custom encoder as they could be misclassified as corrupted by older decoder versions. --- doc/decompressor_errata.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/doc/decompressor_errata.md b/doc/decompressor_errata.md index b162e7fd6..6c99cb01f 100644 --- a/doc/decompressor_errata.md +++ b/doc/decompressor_errata.md @@ -6,12 +6,29 @@ Each entry will contain: 1. The last affected decompressor versions. 2. The decompressor components affected. 2. Whether the compressed frame could ever be produced by the reference compressor. -3. An example frame. +3. An example frame when it can be short enough to be displayed as hexadecimal 4. A description of the bug. The document is in reverse chronological order, with the bugs that affect the most recent zstd decompressor versions listed first. +Compressed block with a size of exactly 128 KB +------------------------------------------------ + +**Last affected version**: v1.5.2 + +**Affected decompressor component(s)**: Library & CLI + +**Produced by the reference compressor**: No + +The zstd decoder incorrectly rejected blocks of type `Compressed_Block` when their size was exactly 128 KB. + +This type of block was never generated by the reference compressor. + +These blocks used to be disallowed by the spec up until spec version 0.3.2 when the restriction was lifted by [PR#1689](https://github.com/facebook/zstd/pull/1689). + +> A Compressed_Block has the extra restriction that Block_Size is always strictly less than the decompressed size. If this condition cannot be respected, the block must be sent uncompressed instead (Raw_Block). + Compressed block with 0 literals and 0 sequences ------------------------------------------------ From 0d6954b4cc309b430dd010dbeb20b112e7092644 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 19 Apr 2023 00:24:35 -0700 Subject: [PATCH 2/2] added golden file for the new decompressor erratum --- doc/decompressor_errata.md | 5 ++++- doc/educational_decoder/zstd_decompress.c | 2 +- tests/golden-decompression/block-128k.zst | Bin 0 -> 131081 bytes 3 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 tests/golden-decompression/block-128k.zst diff --git a/doc/decompressor_errata.md b/doc/decompressor_errata.md index 6c99cb01f..e170a62c1 100644 --- a/doc/decompressor_errata.md +++ b/doc/decompressor_errata.md @@ -6,7 +6,7 @@ Each entry will contain: 1. The last affected decompressor versions. 2. The decompressor components affected. 2. Whether the compressed frame could ever be produced by the reference compressor. -3. An example frame when it can be short enough to be displayed as hexadecimal +3. An example frame (hexadecimal string when it can be short enough, link to golden file otherwise) 4. A description of the bug. The document is in reverse chronological order, with the bugs that affect the most recent zstd decompressor versions listed first. @@ -21,7 +21,10 @@ Compressed block with a size of exactly 128 KB **Produced by the reference compressor**: No +**Example Frame**: see zstd/tests/golden-decompression/block-128k.zst + The zstd decoder incorrectly rejected blocks of type `Compressed_Block` when their size was exactly 128 KB. +Note that `128 KB - 1` was accepted, and `128 KB + 1` is forbidden by the spec. This type of block was never generated by the reference compressor. diff --git a/doc/educational_decoder/zstd_decompress.c b/doc/educational_decoder/zstd_decompress.c index 9ade76502..3196b78dc 100644 --- a/doc/educational_decoder/zstd_decompress.c +++ b/doc/educational_decoder/zstd_decompress.c @@ -1399,7 +1399,7 @@ size_t ZSTD_get_decompressed_size(const void *src, const size_t src_len) { /******* END OUTPUT SIZE COUNTING *********************************************/ /******* DICTIONARY PARSING ***************************************************/ -dictionary_t* create_dictionary() { +dictionary_t* create_dictionary(void) { dictionary_t* const dict = calloc(1, sizeof(dictionary_t)); if (!dict) { BAD_ALLOC(); diff --git a/tests/golden-decompression/block-128k.zst b/tests/golden-decompression/block-128k.zst new file mode 100644 index 0000000000000000000000000000000000000000..cdaeae39d106fa58bd60e8b167bd4d7704bd493c GIT binary patch literal 131081 zcmeIuu@L|e00gja^fu7yv=+C8l^kb}WQ6oJPG(8nw*>YwAz<>b*1`HT5V8DO@ z0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VK zfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5 zV8DO@0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM z7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b* z1`HT5V8DO@0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd z0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VKfB^#r3>YwA zz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEj zFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VKfB^#r z3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@ z0|pEjFkrxd0RsjM7%*VKfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM7%*VK mfB^#r3>YwAz<>b*1`HT5V8DO@0|pEjFkrxd0RsjM{DB9JumnK> literal 0 HcmV?d00001