The block format section did not say how the match copy is performed.
Overlapping matches (matchlength > offset) were only mentioned in the
"Safe decoding" implementation note, framed as a safety concern rather
than as part of the format definition, which confused readers
implementing a decoder for the first time.
State explicitly, right after matchlength decoding, that match bytes
are copied one at a time in increasing order, so matchlength may
exceed offset.
Fixes#1794
Since 2c484419ef, the Linux Kernel is able to discern the end of a legacy LZ4 frame by the presence of four 0 bytes, parsed as a B.CSize of 0 in the decoder. This is originally intended to make it mix well with grub initrd concatenation behavior (which uses a clause in the initramfs format about allowing arbitrary zero paddings), but I think it is a pretty good idea for any application limited to the legacy format in general.
Adds additional record-keeping to track the live allocation size of a context
using a custom memory allocator.
This could be extended to more scenarios (e.g., streaming) if desired.
Note: effectively limited to using the dictionary once for now,
as opposed to once per block when blocks are independent
(no impact when blocks are linked: dictionary is supposed to be used once anyway)
Also :
- clarifies that default lz4frame block size is 64 KB
- refactor LZ4 Frame spec, dictionary paragraph
- updated manual
* replace assert with test for LZ4F_uncompressedUpdate
* update documentation to incldue correct docstring
* remove unecessary entry point
* remove compress_linked_block_mode from fuzzing test
Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
new method `uncompressed_update` allows to insert blocks without
compression into the lz4 stream.
The usage is documented in the frameCompress example
Signed-off-by: Alexander Mohr <alexander.m.mohr@mercedes-benz.com>
to specifically call attention to offset==0 scenario,
which is invalid, and could lead to some form of security issue
if incorrectly dealt with (ignored).
EndMark, the 4-bytes value indicating the end of frame,
must be `0x00000000`.
Previously, it was just mentioned as a `0-size` block.
But such definition could encompass uncompressed blocks of size 0,
with a header of value `0x80000000`.
But the intention was to also support uncompressed empty blocks.
They could be used as a keep-alive signal.
Note that compressed empty blocks are already supported,
it's just that they have a size 1 instead of 0 (for the `0` token).
Unfortunately, the decoder implementation was also wrong,
and would also interpret a `0x80000000` block header as an endMark.
This issue evaded detection so far simply because
this situation never happens, as LZ4Frame always issues
a clean 0x00000000 value as a endMark.
It also does not flush empty blocks.
This is fixed in this PR.
The decoder can now deal with empty uncompressed blocks,
and do not confuse them with EndMark.
The specification is also clarified.
Finally, FrameTest is updated to randomly insert empty blocks during fuzzing.
make it possible to generate LZ4-compressed block
with a controlled maximum offset (necessarily <= 65535).
This could be useful for compatibility with decoders
using a very limited memory budget (<64 KB).
Answer #154
- promoted LZ4_resetStream_fast() to stable
- moved LZ4_resetStream() into deprecate, but without triggering a compiler warning
- update all sources to no longer rely on LZ4_resetStream()
note : LZ4_initStream() proposal is slightly different :
it's able to initialize any buffer, provided that it's large enough.
To this end, it accepts a void*, and returns an LZ4_stream_t*.
