A ring buffer of LZ4_DECODER_RING_BUFFER_SIZE(maxBlockSize) is allocated statically,
so its size is compiled into applications.
16d695b1 enlarged its margin from 14 to 64 bytes to fit the fast loop's wide copies,
leaving applications compiled with the former macro exposed to corrupted output.
The decoder now also supports the former margin, without advertising it.
When decoding restarts at the beginning of the ring, the oldest referenced history
lies in the external dictionary, ahead of the write position, at a constant distance.
When this distance is shorter than FASTLOOP_SAFE_DISTANCE, the block is decoded by the safe loop,
which writes at most 16 bytes beyond a copy, like the v1.8.x decoder the 14-byte margin was sized for.
This only concerns rings with less than 64 bytes of margin, hence never the current macro,
for the blocks following such a restart, until 64 KB have been decoded.
Other decoders are unaffected : the test is resolved at compile time outside of usingExtDict.
Applications compiled with older headers decode correctly with the new library, without recompiling.
The static assert now checks what correctness requires, i.e. the safe loop's margin,
which the former macro satisfies : on this point, lz4.c compiles again with an older lz4.h (#1863).
The fuzzer's worst-case ring test now also runs with the former ring size :
it fails without this change, and passes with it.
The other ring condition (decoder ring larger than the encoder's by maxBlockSize + 64 bytes, 2f449bb3)
is unchanged : the decoder can't know the encoder's ring size.
Firefox stores its .jsonlz4 / .mozlz4 files (session, bookmark backups,
search engines) as "mozLz40\0", the decompressed size (LE32),
then a single LZ4 block extending to the end of the file
(see MozLZ4 in Firefox's xpcom/ioutils/IOUtils.cpp).
`lz4 -d`, `lz4 -t` and `lz4cat` now recognize this format,
and decode it like Firefox's reader :
- the decompressed size is only used as decoding capacity
- empty content has no block (Firefox doesn't write one)
Allocations are bounded : sizes > LZ4_MAX_INPUT_SIZE are rejected,
as they can't be compressed into a single block,
and input is read up to LZ4_COMPRESSBOUND(size), the largest valid block.
Data starting with the first half of the magic only ("mozL") keeps
the previous behavior : `lz4cat` passes it through unchanged,
including the bytes read to check the second half.
Decoding only : lz4 doesn't produce this non-standard format.
Based on #1259 by @UnitedMarsupials, rewritten for current dev
(no mmap, bounded allocations, full magic check, tests).
Follow-up to the previous commit: instead of returning early,
clamp a negative dictSize to 0, like LZ4_saveDictHC().
Returning early left the stream referencing the previous dictionary,
which the caller is about to overwrite, since it asked to save it:
the next block could then reference stale data.
With dictSize = 0, the stream continues without dictionary.
fuzzer: the next block copies the end of the dictionary,
and must decode without it (fails with the early return),
and saveBuffer is small enough for ASan to detect
the 64 KB copy (fails on dev).
The legacy compressor used acceleration = -cLevel,
while lz4frame and bench use -cLevel + 1.
Legacy levels were shifted by one: `lz4 -l --fast=1` produced the same
output as `lz4 -l -1`, and `-l --fast=2` matched the frame `--fast=1`.
Use the same mapping, capped at LZ4_ACCELERATION_MAX
(avoids a signed overflow with --fast=2147483647).
Levels >= 0 are unchanged.
test-lz4-basic.sh: `-l --fast=1` must compress less than `-l -1`
(fails on dev), and `-l --fast=2147483647` runs.
Follow-up to "lz4frame: avoid signed overflow", per the #1824 review:
saturate at LZ4_ACCELERATION_MAX rather than INT_MAX,
since higher values make no difference.
LZ4_ACCELERATION_DEFAULT and LZ4_ACCELERATION_MAX move from lz4.c
to the LZ4_STATIC_LINKING_ONLY section of lz4.h, which lz4.c, lz4frame.c,
bench.c and frametest.c already include, so their values derive from it.
The LZ4_compress_fast() documentation already referred to both constants.
bench.c computed the acceleration the same way (-cLevel + 1),
so `lz4 -b --fast=2147483647` hit the same signed overflow:
apply the same cap there.
test-lz4-basic.sh: --fast=2147483647 must produce the same frame
as --fast=65536 (fails on dev, where acceleration wraps to the default),
and its benchmark runs (caught by the UBSan CI job before this fix).
lz4.1: --fast values above 65536 make no difference.
LZ4F_compressBlock() and LZ4F_compressBlock_continue() compute the
acceleration as -level + 1. For compressionLevel == INT_MIN + 1 this
overflows (INT_MAX + 1), and for INT_MIN the negation itself overflows.
Both are undefined behavior (UBSan reports them, e.g. with
`lz4 --fast=2147483647`). In practice the result wraps to a negative
acceleration, which LZ4_compress_fast*() replaces with the default (1),
so the fastest requested levels silently compress at default speed.
Compute the acceleration in a helper that saturates instead of
overflowing. LZ4_compress_fast*() already caps acceleration at
LZ4_ACCELERATION_MAX, so the result is unchanged for every other level.
Add a frametest check that levels -65537, INT_MIN + 1 and INT_MIN
produce the same frame as level -65536.
readU32FromChar() silently wrapped around on values larger than 32 bits,
and compression levels were then cast to int without a range check.
As a result, out-of-range arguments did something unrelated instead of
failing, for example:
lz4 --fast=3000000000 compressed at maximum (HC) level
lz4 -3000000000 used the fastest level (output larger than input)
lz4 -4294967305 used level 9
lz4 -B4294967300 used 64 KB blocks
lz4 -B4194305K used 1 KB blocks
readU32FromChar() now exits with an error when the value (including
K/M suffixes) overflows 32 bits, and compression level arguments
(-#, -e#, --fast=#) are rejected when larger than INT_MAX.
LZ4_CLEVEL and LZ4_NBWORKERS values that overflow are ignored with the
existing warning, like other invalid values.
Also fix the --fast=0 and --fast=-1 tests in test-lz4-basic.sh: they
used a misspelled file name (dg20K instead of dg20k), so lz4 failed
because the file was missing rather than because of the option.
LZ4_decompress_safe_usingDict(), LZ4_decompress_safe_partial_usingDict(),
LZ4_decompress_fast_usingDict() and LZ4_setStreamDecode() only assert()
that dictSize is not negative, then cast it to size_t. In release builds
the dictionary becomes ~SIZE_MAX bytes long, so every match offset is
accepted and decoding reads before the dictionary buffer.
LZ4_loadDictHC() has the same assert(): on lz4mid levels the fill target
of LZ4MID_fillHTable() wraps and it reads up to 32 KB before the
dictionary.
The decoders now return -1, LZ4_setStreamDecode() returns 0 (its
documented error), and LZ4_loadDictHC() loads no dictionary, as
LZ4_loadDict() already does for a negative size.
Near the end of a partial decoding, a match with offset 0 was copied onto
itself, leaving previous dst content in decoded data. doc/lz4_Block_format.md
warns against it (information disclosure), and other decoding paths
already produce 0 bytes. Clear the match segment instead.
Removing the 4-byte content checksum from a valid frame must make
`lz4 --list` fail. Before the previous commit, the checksum was skipped
with fseek(), which succeeds past the end of file, so --list reported
the truncated frame as valid (exit 0) while `lz4 -t` rejected it.
The CLI is not covered by the fuzzers; this test runs in milliseconds.
Since v1.9.3 (c5d6f8a8), partial decoding clamps literals to the end of input.
With a negative srcSize, iend points before src: (size_t)(iend-ip) became huge,
and the literals memmove() ran far past both buffers.
Reject it in the existing srcSize==0 test of LZ4_decompress_generic(),
which becomes srcSize<=0: the generated code only changes je into jle.
The NEWS line also covers the LZ4_decompress_fast*() fix of #1875 (since v1.9.4).
main() allocated `buffer` and `dict` with malloc() and never freed them,
neither on success nor on any of the five error paths.
LeakSanitizer reports 133118 bytes leaked in 2 allocations.
Route all exits through a single `_end` label that frees both buffers.
Checked with gcc -fsanitize=address,undefined: no leak reported, on the
success path and on a forced error path.
A negative originalSize made oend point before the output buffer:
(size_t)(oend-op) then became huge, disabling every output bound check,
so decoding wrote and read past the buffers.
Reject it once, in LZ4_decompress_unsafe_generic(), which backs all
LZ4_decompress_fast*() entry points, as LZ4_decompress_generic()
already does for the safe variants.
When input ended, LZ4F_read() returned 0 (end of file)
even if the decoder still expected more data :
a file missing its last block, end mark or checksum
decoded like a complete one.
It now returns ERROR_io_read when input ends within a frame.
`hint` alone tells whether a frame is in progress : it is now 0 between frames,
and inputLimit() provides the minimal frame header size there.
There is no new state to keep in sync.
Resume points recorded for LZ4F_seek() are unchanged.
testTruncatedFile() checks complete and truncated files,
with 1 or 2 frames, all checksum combinations, and various read sizes.
testFileSeek() now records a resume point at end of file,
and checks that restarting from it reaches end of file cleanly.
Reported in #1814 and #1857.
A failed open sets the handle to NULL.
LZ4F_writeClose() returns LZ4F_OK_NoError on success,
and reports an error from a previous LZ4F_write() (Linux only : uses /dev/full).
Values <= 0 produced corrupted output, and large negative values
read out of bounds. The clamp is now a single helper, LZ4_clampAcceleration(),
used by all 4 entry points.
The skipChecksums decompression option is documented to disable
checksum calculation and verification, but block checksums of compressed blocks
were still calculated and verified. Only uncompressed blocks honored it.
Decompression with skipChecksums of frames using block checksums is now up to +23% faster
(lz4 -bd --no-crc).
Tests : testSkipChecksums(), corrupting each kind of checksum.
Random access within compressed files (#1304), like fseek() / ftell(),
supporting SEEK_SET and SEEK_CUR.
Seeking forward decodes and discards data.
Seeking backward restarts from the closest resume point, recorded while reading :
frame starts, and block starts within frames using independent blocks
(resumed with LZ4F_resumeFrame()), spaced by at least 1 MB of decoded data.
Within frames using linked blocks, decoding restarts from the frame start.
Tests : testFileSeek(), random seeks and reads within a file
mixing linked / independent frames, checksums, content size, skippable and empty frames.
Prepares a decompression context to decode a frame from one of its blocks,
for random access within frames (#1304).
It receives frame parameters (LZ4F_frameInfo_t), the block's position
within decoded content, and the data the block may refer to.
Unlike the former recipe (LZ4F_getFrameInfo() + LZ4F_decompress_usingDict() + skipChecksums),
it supports frames declaring their content size, which remains verified,
and only skips the content checksum : block checksums remain verified.
To this end, the internal skipChecksum flag is split into block and content flags.
Also documented that hints extend up to the end of next block header,
which is how block starts can be located.
Tests : testResumeFrame(), for all combinations of block mode, checksums,
content size, and dictionary.
On short-period data (ABAB...) with linked blocks,
all candidates in the 64 KB history match up to the end of the block.
The match finder compares each of them in full (up to 16384 x 64 KB per block),
making level 12 ~40x slower than on regular data
(OSS-Fuzz 573265410 : timeout in compress_frame_fuzzer).
frametest compares the CPU time of both cases (1 MB each),
so that the test doesn't depend on platform speed (valgrind, qemu, sanitizers).
Fails on dev : periodic data 2.09 s, reference 0.05 s.
Following #1304, Wireshark implements random access within LZ4 frames :
a first pass, directed by hints, records the position of each block start,
both within the frame and within decoded data.
Decoding later resumes from any block start, using a reset state :
the saved frame header is decoded with LZ4F_getFrameInfo(),
the last 64 KB of decoded data are provided with LZ4F_decompress_usingDict()
(linked blocks only), and checksums are skipped for the rest of the frame.
testRandomAccess() checks this recipe, for all combinations of
block mode, block checksum, and content checksum, including an uncompressed block.
It also checks that hints always point to the end of next block header
(or to the end of frame once endMark is consumed),
whatever the amount of input and output provided,
and that LZ4F_getFrameInfo() invoked mid-frame provides the same hint.
Note : frames declaring their content size can't be resumed this way.
LZ4F_decompress() hints were wrong in 2 cases, for frames with block checksums:
- while collecting a compressed block (dstage_storeCBlock),
the block checksum was counted twice, since tmpInTarget already includes it :
the hint exceeded the end of next block header by 4 bytes.
This also affected LZ4F_getFrameInfo() invoked mid-frame.
- when input ended in the middle of the checksum of an uncompressed block,
the hint was not set at all, and remained 1.
The single-thread lz4 CLI reads exactly the hinted amount,
and discards leftover input at the end of a frame.
With block checksums, no content checksum, and a last block compressed > 64 KB,
it read and discarded the beginning of the next frame,
silently truncating the output of concatenated frames.
Added this scenario to test-lz4-frame-concatenation.sh.
LZ4_loadDict() ignores dictionaries smaller than HASH_UNIT.
A large first block must then give the same output as a new stream,
directly or through LZ4_attach_dictionary().
Fails on dev : after LZ4_loadDict(), the output differs.
The fuzzer can detect it too, as a difference between
LZ4_compress_fast_continue() using extDict and using extDictCtx (seed 9016, cycle 8086).
freestanding.c only provides _start for x86_64 Linux targets,
while tests/Makefile decided to add -ffreestanding -nostdlib based on uname.
Whenever host and target disagree (CC="gcc -m32", cross-compilers),
the binary is linked without any entry point and crashes.
Query the compiler's preprocessor instead, using the same condition
as the guard in freestanding.c, so that both decisions are always consistent.
fix#1464
Worst case, with dstCapacity = maxBlockSize : decoding restarts at the beginning of the ring,
a block ends with a wide copy writing beyond its end, a 1-byte block follows,
then a block references the oldest byte allowed by the encoder's ring.
With the formerly documented margin (maxBlockSize), the output is corrupted since v1.9.0.
The new block overwrites the previous one (its dictionary) at the beginning, at the same place,
entirely, or at the end. Output decoded from a separate buffer must be correct.
Fails on dev : LZ4_compress_fast_continue() only handles an overwritten beginning.
Worst case : decoding restarts at the beginning of the ring with maxBlockSize-1 bytes left at its end,
then a wide copy (long literals or long match) is followed by a match at maximum distance.
The copy writes beyond its end, over the oldest history, still referenced by the match.
Fails on dev : on x86, the margin must be 29 bytes (LZ4_wildCopy32) instead of 14.
Attaching a dictionary to a new state, then compressing with one-shot fast reset,
must give the same output as a new state.
It currently doesn't : the block finds no match, and debug builds hit an assert.
testReproducibleFrames() now also covers level 10 (optimal parser).
It currently fails : after an HC frame, a level 10 frame with linked blocks
can differ from a new context's (66929 vs 66919 bytes).
Basic tests take ~1 s longer.
frametest : at level 10, for linked and independent blocks,
a frame started with LZ4F_compressBegin_usingDict()
or LZ4F_compressBegin_usingCDict() differs depending on favorDecSpeed,
and decodes correctly.
The same check without dictionary ensures the test input
is sensitive to favorDecSpeed.
This test fails on current dev :
- LZ4_loadDictHC() fully re-initializes the HC state,
which also resets favorDecSpeed.
- with a CDict, the first block copies the dictionary state
over the working state, which overwrites favorDecSpeed.
- unit test : a 64 KB block, compressed on a new state,
then as a new stream on the same (now used) state, must be identical,
including after dropping history with LZ4_saveDict(stream, buffer, 0).
The input starts with a 4-byte sequence which reappears right after,
followed by a different 5th byte.
- fuzzer : a state keeps a history across cycles;
each block starts a new stream on it with LZ4_resetStream_fast(),
and blocks >= 64 KB must match the output of a new stream.
64 KB is beyond the table clearing threshold, whatever LZ4_MEMORY_USAGE.
These tests fail on current dev :
LZ4_resetStream_fast() keeps the table, and a used table rejects its stale entries,
while a cleared table tests position 0 for unused hash slots.
The small-dst tests compared frames byte by byte,
so each compression started from a new context,
since output could depend on the context's history.
Frames no longer depend on the context's history,
so these tests now reuse a single context,
including after failed operations.
This extends their coverage to context reuse.
Compare its output with LZ4_compress_fast_extState(), which fully initializes the state.
Whether the fast reset clears the table or reuses it,
the output must be the same :
below 64 KB, the table uses a 4-byte hash,
so reusing a table can't miss a match that a cleared table would find.
Beyond, the table is always cleared.
This guards a change of the threshold deciding to clear the table.
Compress the same input with a new context, then with a reused one
after various histories (fast linked frame, HC frame, independent blocks,
or simply the previous configuration), and require identical frames.
Covers fast, mid and HC levels, LZ4F_compressFrame_usingCDict()
and streaming by chunks, linked and independent blocks, autoFlush,
with and without CDict.
The input starts with a 4-byte sequence which reappears right after,
followed by a different 5th byte.
This test fails on current dev :
at fast levels with linked blocks, a frame compressed with a reused context
can differ from the one compressed with a new context (both are valid).
A failed compression marks the HC state as dirty,
and LZ4_resetStreamHC_fast() then fully re-initializes it.
- fuzzer : after a failed compression, the next stream
still favors decompression speed, as it would without the failure.
- frametest : with independent blocks at level 10 and favorDecSpeed,
a block following an incompressible block is identical
to the same block compressed alone.
LZ4F compresses each block into srcSize-1 bytes,
so any incompressible block is a failed compression.
These tests fail on current dev :
the re-initialization also resets favorDecSpeed,
so LZ4F ignores it after the first incompressible block of a frame.
The context size test only covered a single frame.
Now, a sequence of frames, varying block size, block mode and compression level,
is processed by the same contexts,
checking after each frame that reported sizes match allocated memory.
This test fails on current dev :
after a frame with smaller blocks, LZ4F_dctx_size() under-reports.
Bounds should guarantee success, without being requirements :
compression must succeed if and only if its output fits,
and then produce the same output as with a bound-sized buffer.
frametest unit tests :
- LZ4F_compressBegin*() into exactly the header size (7-19 bytes),
and a failed begin doesn't disturb a frame in progress
- LZ4F_compressFrame() and LZ4F_compressFrame_usingCDict() :
exact frame size succeeds with identical output, any smaller capacity fails
(every capacity for small frames, element boundaries otherwise)
- streaming : random scripts of updates, uncompressed updates and flushes;
each operation into exactly its output size, failures just before element boundaries,
frame invalidated after a failure, context reusable for a new frame
- LZ4F_flush() and LZ4F_compressEnd() require a frame in progress
- mixed updates : rewrite the insufficient capacity test
frametest fuzzer : random undersized capacities for compressFrame, updates, flushes and compressEnd
ossfuzz :
- compress_frame_fuzzer : success if and only if the frame fits, identical output
- new compress_frame_stream_fuzzer : streaming with random capacities,
in lockstep with a reference context receiving bound-sized buffers
- round_trip_frame_uncompressed_fuzzer : pass remaining capacity
Note : byte-exact comparisons start from fresh contexts,
because output of linked blocks at fast levels depends on context history
(pre-existing behavior, unrelated to capacity).
These tests fail on current dev.
`dstCapacity` is the name of the parameter this error refers to.
The former enum name remains available as an alias
(LZ4F_ERROR_dstMaxSize_tooSmall), so existing code still compiles.
LZ4F_getErrorName() now reports "ERROR_dstCapacity_tooSmall".
