Following #1304, Wireshark implements random access within LZ4 frames :
a first pass, directed by hints, records the position of each block start,
both within the frame and within decoded data.
Decoding later resumes from any block start, using a reset state :
the saved frame header is decoded with LZ4F_getFrameInfo(),
the last 64 KB of decoded data are provided with LZ4F_decompress_usingDict()
(linked blocks only), and checksums are skipped for the rest of the frame.
testRandomAccess() checks this recipe, for all combinations of
block mode, block checksum, and content checksum, including an uncompressed block.
It also checks that hints always point to the end of next block header
(or to the end of frame once endMark is consumed),
whatever the amount of input and output provided,
and that LZ4F_getFrameInfo() invoked mid-frame provides the same hint.
Note : frames declaring their content size can't be resumed this way.
LZ4F_decompress() hints were wrong in 2 cases, for frames with block checksums:
- while collecting a compressed block (dstage_storeCBlock),
the block checksum was counted twice, since tmpInTarget already includes it :
the hint exceeded the end of next block header by 4 bytes.
This also affected LZ4F_getFrameInfo() invoked mid-frame.
- when input ended in the middle of the checksum of an uncompressed block,
the hint was not set at all, and remained 1.
The single-thread lz4 CLI reads exactly the hinted amount,
and discards leftover input at the end of a frame.
With block checksums, no content checksum, and a last block compressed > 64 KB,
it read and discarded the beginning of the next frame,
silently truncating the output of concatenated frames.
Added this scenario to test-lz4-frame-concatenation.sh.
LZ4_loadDict() always advances currentOffset by 64 KB,
but for a dictionary smaller than HASH_UNIT, it returned early,
leaving the table marked as cleared.
Since LZ4_compress_fast_continue() clears the table of a stream without history
starting with a large block (e13b5416), this state made a difference :
LZ4_prepareTable() trusted the cleared marker, kept currentOffset,
and the output differed from a new stream,
and from the same dictionary attached with LZ4_attach_dictionary().
The table type is now set before the early return,
like LZ4_attach_dictionary() does when it advances currentOffset.
Normal dictionaries (>= HASH_UNIT) are unchanged.
LZ4_loadDict() ignores dictionaries smaller than HASH_UNIT.
A large first block must then give the same output as a new stream,
directly or through LZ4_attach_dictionary().
Fails on dev : after LZ4_loadDict(), the output differs.
The fuzzer can detect it too, as a difference between
LZ4_compress_fast_continue() using extDict and using extDictCtx (seed 9016, cycle 8086).
freestanding.c only provides _start for x86_64 Linux targets,
while tests/Makefile decided to add -ffreestanding -nostdlib based on uname.
Whenever host and target disagree (CC="gcc -m32", cross-compilers),
the binary is linked without any entry point and crashes.
Query the compiler's preprocessor instead, using the same condition
as the guard in freestanding.c, so that both decisions are always consistent.
fix#1464
The freestanding test is only valid for x86_64 Linux targets,
but tests/Makefile selects it based on the host (uname),
while freestanding.c selects its code path based on the compiler's target.
When the two disagree, the test binary is linked with -nostdlib
but without any _start, and crashes.
Add CI coverage for both mismatch scenarios:
- 32-bit target on 64-bit host (CC="$CC -m32")
- non-x86_64 cross-compiler on x86_64 host (all QEMU targets)
These tests are expected to fail until the Makefile is fixed (#1464).
- fullUpdate is now a field of cParams_t, set in k_clTable (level 12 only),
instead of being derived from cLevel at the call site,
so that all parameters of a level are defined in one place
- updated a stale comment in LZ4HC_FindLongerMatch()
No output change.
- heap / stack allocation of the opt[] table moved into a small wrapper,
LZ4HC_compress_optimal(), so that LZ4HC_compress_optimal_internal()
can return directly like the other parsers:
no more retval / _return_label, and a single #if LZ4HC_HEAPMODE block
- the fillOutput overflow is handled at the only LZ4HC_encodeSequence() call site:
no more ovml / ovoff / _dest_overflow
- sufficient_len is now an int (sufficientLen), set once,
no more (size_t) casts in comparisons
Output is unchanged (all levels, limitedOutput, fillOutput, heap mode).
LZ4MID_compress(), LZ4HC_compress_hashChain() and LZ4HC_compress_optimal()
each had their own copy of the same 2 tail sections:
- encoding the final run of literals (truncated in fillOutput mode)
- in fillOutput mode, trying to encode the sequence which doesn't fit,
with a shortened match length
They are now LZ4HC_encodeLastLiterals() and LZ4HC_encodeLastSequence_fillOutput().
Both use the same @oend convention as LZ4HC_encodeSequence()
(end of dst, minus LASTLITERALS in fillOutput mode),
so callers no longer need to restore oend before the final literals:
oend is now a const, set once.
Output is unchanged (all levels, limitedOutput, fillOutput, dictionaries).
LZ4HC_addMatch() started by trying to improve prices of the 3 positions
following @cur with literals. This update never triggers:
when it runs, position cur+k is always already priced
at or below the "continue with literals" price, either by
- the same literal continuation (initial or trailing literals,
computed with the same formula => exact tie, ~64% of evaluations), or
- an overlapping match (>= 3 bytes over its start, mostly flat with length),
which is cheaper than 4+ literals.
Prices also never increase once a position is reached,
so these bounds remain valid until cur is processed.
Literal length cost transitions (14->15, 269->270, ...) only happen
in the initial literal run (within a window, literal runs are <= 3),
where they add the same extra byte to both sides => tie.
Measured: 0 update out of > 33 billion evaluations
on synthetic LZ-like data (including inputs targeting
literal length and match length cost transitions),
plus enwik9, linux sources, arm64 binaries and silesia,
at levels 10-12, with or without favorDecSpeed,
with or without dictionary, with 4 KB, 64 KB and 4 MB blocks.
It was also inconsistent: after a match, it extended the literal run
which preceded that match (opt[cur].litlen) instead of starting a new one.
A corrected version (restarting the run after a match)
never triggers either (0 update out of > 10 billion evaluations).
Removing it simplifies the price model:
- litlen is now always the nb of literals since the last match (0 after a match)
- a single formula prices any match (LZ4HC_runStartPrice()),
no more special case for opt[cur].mlen == 1
- the last position is always a match
Output is unchanged on all tested samples.
Compression speed is improved by ~2-4% (gcc, x64, levels 10-12).
Pure restructuring of LZ4HC_compress_optimal(), producing identical output:
- LZ4HC_setLiteral() / LZ4HC_setMatch() replace hand-written 4-field updates
- LZ4HC_addMatch() updates prices from a match found at position cur.
The initial setup was the same step applied at position 0,
so it now uses the same function, removing a duplicated block,
including a second copy of the trailing literals completion.
- the 2 "immediate encoding" paths and the regular one
all go through the same encode: section,
removing one LZ4HC_encodeSequence() + overflow call site,
and the last_match_pos = cur + 1 trick.
- the reverse traversal stops at position 0, and is documented
- merged the 2 consecutive if (fullUpdate) blocks
state_attachDictRoundTrip() and state_attachDictHCRoundTrip() used the result
of LZ4_createStream() / LZ4_createStreamHC() without checking it.
On allocation failure, the NULL stream was dereferenced inside lz4.c / lz4hc.c,
showing up as a library crash instead of a failed assert in the harness.
Same issue as #1813, in the dictionary paths.
Synchronized mode : decoding needs the exact decompressed size of every block, including the last one.
Decoding may write up to dst + dstCapacity, over data still referenced.
Decoding ring larger than the encoding one : the margin must be maxBlockSize + 64 bytes, instead of maxBlockSize.
Since v1.9.0, with dstCapacity > block size, a block ending with a wide copy can write
up to 63 bytes beyond its end (LZ4_wildCopy64), over the oldest data a later block may still reference.
Same margin as LZ4_DECODER_RING_BUFFER_SIZE().
Worst case, with dstCapacity = maxBlockSize : decoding restarts at the beginning of the ring,
a block ends with a wide copy writing beyond its end, a 1-byte block follows,
then a block references the oldest byte allowed by the encoder's ring.
With the formerly documented margin (maxBlockSize), the output is corrupted since v1.9.0.
LZ4_compress_fast_continue() tolerates input overwriting the beginning of the dictionary
(ring buffer restarting from its beginning) : it keeps only the intact end of the dictionary.
Other overlaps were ignored : input at the same place as the dictionary, covering it entirely,
or covering its end. The dictionary then still pointed at overwritten data,
and the compressed output decoded into wrong data (e.g. with an input ring smaller than 2 blocks).
Any overlap now keeps only the part of the dictionary beyond input, possibly none,
like LZ4_compress_HC_continue(). Output is unchanged without overlap,
and when only the beginning of the dictionary is overwritten.
The new block overwrites the previous one (its dictionary) at the beginning, at the same place,
entirely, or at the end. Output decoded from a separate buffer must be correct.
Fails on dev : LZ4_compress_fast_continue() only handles an overwritten beginning.
A ring buffer of LZ4_decoderRingBufferSize(maxBlockSize) must decode any valid stream.
When decoding restarts at the beginning of the ring, with maxBlockSize-1 bytes left at its end,
the oldest history still referenced is only margin+2 bytes ahead of the write position.
A copy writing beyond its end can overwrite it.
The margin of 14 was sized for the decoder of v1.8.x, which writes up to 16 bytes beyond a copy.
The fast decoding loop (v1.9.0+) writes up to 31 bytes (LZ4_wildCopy32), requiring a margin of 29,
and up to 63 bytes on aarch64 (LZ4_wildCopy64), requiring 61.
The margin is now 64, and a static assert ties it to the widest copy.
Buffers sized with the former macro remain too small : applications must be recompiled.
LZ4_decompress_fast_continue() is not affected (exact copies only).
Worst case : decoding restarts at the beginning of the ring with maxBlockSize-1 bytes left at its end,
then a wide copy (long literals or long match) is followed by a match at maximum distance.
The copy writes beyond its end, over the oldest history, still referenced by the match.
Fails on dev : on x86, the margin must be 29 bytes (LZ4_wildCopy32) instead of 14.
The Makefile and meson enable multithreading when pthread is available,
and Windows enables it by default (programs/lz4conf.h).
CMake never did : on Linux, macOS or BSD, its lz4 binary was single-threaded.
CMake now detects pthread (find_package(Threads)) for lz4 and lz4c.
The new option LZ4_BUILD_MULTITHREAD (default ON) disables it, Windows included.
silesia.tar, lz4 -1 : 0.89 s => 0.25 s, same output size.
On a new (cleared) state, LZ4_attach_dictionary() sets currentOffset to 64 KB,
but leaves tableType == clearedTable.
LZ4_prepareTable() then trusts the table to be fresh, and skips its checks :
the next LZ4_compress_fast_extState_fastReset() uses a byU16 table with indexes >= 64 KB.
The block finds no match (4000 -> 4017 bytes instead of 87 in a simple repro),
and debug builds hit an assert in LZ4_putIndexOnHash().
The output remains valid. Present since at least v1.9.0.
LZ4_attach_dictionary() now marks the table as byU32, which it is :
all its entries designate index 0, below the 64 KB offset.
In LZ4F, this happens with a fast frame using a CDict and no data,
on a context just (re)initialized (e.g. switching from HC),
followed by a frame with independent blocks.
`frametest -s259` fails since #1828, which reuses one context across configurations.
Attaching a dictionary to a new state, then compressing with one-shot fast reset,
must give the same output as a new state.
It currently doesn't : the block finds no match, and debug builds hit an assert.
state_create() asserted state.cstream a second time instead of the
new state.cstreamHC, so a failed LZ4_createStreamHC() went unnoticed and
state_reset() then passed NULL to LZ4_resetStreamHC_fast().
Fixes#1813
When a block becomes an external dictionary, LZ4HC_setExternalDict() inserts its positions up to end-3 :
the last 3 positions can't be hashed. Their chainTable entries kept whatever was there :
zeroes on a new context, stale deltas from previous streams on a used one.
The optimal parser reads them (chainSwap, levels >= 10),
so its output depended on the context's history.
For example, an LZ4F frame with linked blocks after an HC frame : 66930 vs 66919 bytes.
These entries are now cleared.
New contexts : same output size and speed (silesia.tar, one LZ4F frame, linked 64 KB blocks, levels 10 and 12).
Found by `frametest -s34`, failing since #1828, which reuses one context across configurations.
testReproducibleFrames() now also covers level 10 (optimal parser).
It currently fails : after an HC frame, a level 10 frame with linked blocks
can differ from a new context's (66929 vs 66919 bytes).
Basic tests take ~1 s longer.
LZ4_loadDictHC() fully re-initializes the HC state, using LZ4_initStreamHC(),
which also reset favorDecSpeed. Only the compression level was preserved.
Similarly, when a stream attached to a CDict starts with a block > 4 KB,
LZ4HC_compress_generic_dictCtx() copies the dictionary state
over the working state, which overwrote favorDecSpeed.
Both now preserve favorDecSpeed.
Impact on LZ4F, levels >= 10 with favorDecSpeed :
LZ4F_compressBegin*() sets favorDecSpeed before loading the dictionary,
so frames started with LZ4F_compressBegin_usingDict()
or LZ4F_compressBegin_usingCDict() ignored it.
- CLI, linked blocks (-BD) : since v1.10.0, each chunk after the first one
starts with LZ4F_compressBegin_usingDict() (64 KB prefix), even with -T1,
so --favor-decSpeed only applied to the first chunk
(enwik9, -12 -BD : +16 KB instead of +4.7 MB).
- CLI, dictionary (-D) : --favor-decSpeed was ignored (already in v1.9.2).
Frames were valid, but didn't honor the requested preference.
frametest : at level 10, for linked and independent blocks,
a frame started with LZ4F_compressBegin_usingDict()
or LZ4F_compressBegin_usingCDict() differs depending on favorDecSpeed,
and decodes correctly.
The same check without dictionary ensures the test input
is sensitive to favorDecSpeed.
This test fails on current dev :
- LZ4_loadDictHC() fully re-initializes the HC state,
which also resets favorDecSpeed.
- with a CDict, the first block copies the dictionary state
over the working state, which overwrites favorDecSpeed.
LZ4_resetStream_fast() keeps the hash table, since it doesn't know the stream's first block size.
Reusing a table means rejecting its stale entries,
a poorly predictable branch, which costs more than clearing the table beyond a small input size.
It also made the output depend on the state's history :
a cleared table tests position 0 for unused hash slots, a used table rejects its stale entries.
LZ4_compress_fast_continue() now clears the table when a stream without history
(no dictionary) starts with a block >= LZ4_CLEAR_TABLE_THRESHOLD,
the threshold already used by one-shot fast reset (LZ4_HASHTABLESIZE / 16, 1 KB by default).
Such streams are faster, and produce the same output as a new stream.
Smaller first blocks, and streams using a dictionary, are unchanged.
LZ4F is unchanged : its linked frames already start from a cleared table.
LZ4_resetStream_fast() + LZ4_compress_fast_continue() on a reused state,
MB/s, AMD EPYC Genoa, gcc 11 -O3 (aligned code, both builds) :
first block silesia.tar enwik9
default table (LZ4_MEMORY_USAGE=14), threshold 1 KB :
< 1 KB unchanged unchanged
1 KB 572 -> 665 (+16%) 388 -> 520 (+34%)
2 KB 479 -> 667 (+39%) 315 -> 500 (+59%)
4 KB 403 -> 655 (+63%) 257 -> 478 (+86%)
8 KB 371 -> 643 (+73%) 238 -> 462 (+94%)
16 KB 396 -> 633 (+60%) 264 -> 443 (+68%)
64 KB 501 -> 620 (+24%) 341 -> 419 (+23%)
other tables, beyond their threshold :
4 KB table (256 B) +12% to +102% +7% to +133%
64 KB table (4 KB) +10% to +46% +20% to +56%
256 KB table (16 KB) +2% to +25% +7% to +26%
- unit test : a 64 KB block, compressed on a new state,
then as a new stream on the same (now used) state, must be identical,
including after dropping history with LZ4_saveDict(stream, buffer, 0).
The input starts with a 4-byte sequence which reappears right after,
followed by a different 5th byte.
- fuzzer : a state keeps a history across cycles;
each block starts a new stream on it with LZ4_resetStream_fast(),
and blocks >= 64 KB must match the output of a new stream.
64 KB is beyond the table clearing threshold, whatever LZ4_MEMORY_USAGE.
These tests fail on current dev :
LZ4_resetStream_fast() keeps the table, and a used table rejects its stale entries,
while a cleared table tests position 0 for unused hash slots.
The small-dst tests compared frames byte by byte,
so each compression started from a new context,
since output could depend on the context's history.
Frames no longer depend on the context's history,
so these tests now reuse a single context,
including after failed operations.
This extends their coverage to context reuse.
Compare its output with LZ4_compress_fast_extState(), which fully initializes the state.
Whether the fast reset clears the table or reuses it,
the output must be the same :
below 64 KB, the table uses a 4-byte hash,
so reusing a table can't miss a match that a cleared table would find.
Beyond, the table is always cleared.
This guards a change of the threshold deciding to clear the table.
At fast levels, linked frames started with LZ4_resetStream_fast(),
which keeps the previous hash table, and shifts indexes by 64 KB.
The resulting frame depended on the context's history.
A cleared table (new context, or after an HC frame) treats unused hash slots
as candidates at position 0, which sometimes match,
while a used table rejects its stale entries.
On 64-bit, the 5-byte hash lets the former find 4-byte matches the latter misses.
Both frames are valid, but the same data could be compressed differently,
for example by `lz4 -BD -m` in single-thread mode.
Linked frames without CDict now start with LZ4_initStream(), clearing the table.
Frames no longer depend on the context's history,
and remain identical to the ones produced by a new context.
Frames using a CDict, and independent blocks, were already reproducible,
and are unchanged.
It's also faster, except for tiny frames :
rejecting stale entries is a poorly predictable branch,
which costs more than clearing the table (16 KB by default).
The frame size is unknown when it starts, so this bets it's not tiny.
LZ4F streaming with a reused context, linked 64 KB blocks, default level,
MB/s, AMD EPYC Genoa, gcc 11 -O3 :
frame silesia.tar enwik9
64 B 597 -> 303 (-49%) 585 -> 310 (-47%)
256 B 627 -> 541 (-14%) 535 -> 496 (-7%)
1 KB 503 -> 625 (+24%) 368 -> 515 (+40%)
4 KB 370 -> 625 (+69%) 253 -> 485 (+92%)
16 KB 363 -> 599 (+65%) 261 -> 442 (+70%)
64 KB 471 -> 593 (+26%) 346 -> 426 (+23%)
256 KB 533 -> 569 (+7%) 392 -> 417 (+6%)
1 MB 555 -> 565 (+2%) 406 -> 414 (+2%)
8 MB 566 -> 567 412 -> 413
Compression ratio is very slightly better (4 KB frames : 1.6792 -> 1.6794).
Compress the same input with a new context, then with a reused one
after various histories (fast linked frame, HC frame, independent blocks,
or simply the previous configuration), and require identical frames.
Covers fast, mid and HC levels, LZ4F_compressFrame_usingCDict()
and streaming by chunks, linked and independent blocks, autoFlush,
with and without CDict.
The input starts with a 4-byte sequence which reappears right after,
followed by a different 5th byte.
This test fails on current dev :
at fast levels with linked blocks, a frame compressed with a reused context
can differ from the one compressed with a new context (both are valid).
