f4f7a9882c
A ring buffer of LZ4_DECODER_RING_BUFFER_SIZE(maxBlockSize) is allocated statically, so its size is compiled into applications.16d695b1enlarged its margin from 14 to 64 bytes to fit the fast loop's wide copies, leaving applications compiled with the former macro exposed to corrupted output. The decoder now also supports the former margin, without advertising it. When decoding restarts at the beginning of the ring, the oldest referenced history lies in the external dictionary, ahead of the write position, at a constant distance. When this distance is shorter than FASTLOOP_SAFE_DISTANCE, the block is decoded by the safe loop, which writes at most 16 bytes beyond a copy, like the v1.8.x decoder the 14-byte margin was sized for. This only concerns rings with less than 64 bytes of margin, hence never the current macro, for the blocks following such a restart, until 64 KB have been decoded. Other decoders are unaffected : the test is resolved at compile time outside of usingExtDict. Applications compiled with older headers decode correctly with the new library, without recompiling. The static assert now checks what correctness requires, i.e. the safe loop's margin, which the former macro satisfies : on this point, lz4.c compiles again with an older lz4.h (#1863). The fuzzer's worst-case ring test now also runs with the former ring size : it fails without this change, and passes with it. The other ring condition (decoder ring larger than the encoder's by maxBlockSize + 64 bytes,2f449bb3) is unchanged : the decoder can't know the encoder's ring size.
LZ4 - Extremely fast compression
LZ4 is a lossless compression algorithm, providing compression speed > 500 MB/s per core, scalable with multi-core CPUs. It features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.
Compression speed can be tuned dynamically, by selecting an "acceleration" factor which trades compression ratio for faster speed. On the other end, a high compression derivative, LZ4_HC, is also provided, trading CPU time for improved compression ratio. All versions feature the same decompression speed.
LZ4 is also compatible with dictionary compression,
both at API and CLI (-D) levels.
It can ingest any input file as dictionary, though only the final 64 KB are used.
This capability can be combined with the Zstandard Dictionary Builder,
in order to drastically improve compression performance on small files.
The LZ4 library is provided as open-source software using the BSD 2-Clause license.
The lz4 command line utility is provided under the GPL-2.0-or-later license.
Benchmarks
The benchmark uses lzbench, from @inikep compiled with GCC v8.2.0 on Linux 64-bits (Ubuntu 4.18.0-17). The reference system uses a Core i7-9700K CPU @ 4.9GHz (w/ turbo boost). Benchmark evaluates the compression of reference Silesia Corpus in single-thread mode.
| Compressor | Factor | Compression | Decompression |
|---|---|---|---|
| memcpy | 1.000 | 13700 MB/s | 13700 MB/s |
| LZ4 default (v1.9.0) | 2.101 | 780 MB/s | 4970 MB/s |
| LZO 2.09 | 2.108 | 670 MB/s | 860 MB/s |
| QuickLZ 1.5.0 | 2.238 | 575 MB/s | 780 MB/s |
| Snappy 1.1.4 | 2.091 | 565 MB/s | 1950 MB/s |
| Zstandard 1.4.0 -1 | 2.883 | 515 MB/s | 1380 MB/s |
| LZF v3.6 | 2.073 | 415 MB/s | 910 MB/s |
| zlib deflate 1.2.11 -1 | 2.730 | 100 MB/s | 415 MB/s |
| LZ4 HC -9 (v1.9.0) | 2.721 | 41 MB/s | 4900 MB/s |
| zlib deflate 1.2.11 -6 | 3.099 | 36 MB/s | 445 MB/s |
Installation
make
make install # this command may require root permissions
LZ4's Makefile supports standard Makefile conventions,
including staged installs, redirection, or command redefinition.
It is compatible with parallel builds (-j#).
Alternative build systems (CMake, Meson, Visual Studio) are provided in build/.
Building LZ4 - Using vcpkg
You can download and install LZ4 using the vcpkg dependency manager:
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install lz4
The LZ4 port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.
Documentation
The raw LZ4 block compression format is detailed within lz4_Block_format.
Arbitrarily long files or data streams are compressed using multiple blocks, for streaming requirements. These blocks are organized into a frame, defined in lz4_Frame_format. Interoperable versions of LZ4 must also respect the frame format.
The library API is documented in its headers: lz4.h (block API), lz4hc.h (high compression) and lz4frame.h (frame API).
Other source versions
Beyond the C reference source, many contributors have created versions of lz4 in multiple languages (Java, C#, Python, Perl, Ruby, etc.). A list of known source ports is maintained on the LZ4 Homepage.
Packaging status
Most distributions are bundled with a package manager
which allows easy installation of both the liblz4 library
and the lz4 command line interface.
Special Thanks
- Takayuki Matsuoka, aka @t-mat, for exceptional first-class support throughout the lifetime of this project