compression into small dst capacities (begin, frame, streaming), reproducible frames,
random access, LZ4F_resumeFrame(), LZ4F_seek() and truncated files.
frametest.c no longer contains unit tests : it's a fuzzer.
UT_release() releases a resource before the end of the test, for loops ;
UT_fillSample() and UT_fillNoise() fill existing buffers.
A short read of the frame header is a truncated file, not a read error :
report it like the other truncated frames, as a format not recognized,
including when nothing follows the magic number (formerly error 71).
Read errors (ferror()) still end with errors 71 and 72.
The visible outcome is unchanged for partial headers : dev already reported
them as not recognized, after decoding uninitialized bytes.
So the regression test runs under valgrind (make -C tests test-mem, in CI) :
--list on headers truncated in the first read (5 bytes)
and in the second one (10 bytes, within the content size).
Without the fix, valgrind reports the uninitialized bytes in both cases.
NEWS line.
frametest.c's unitTests() now only invokes its testXxx() functions.
decompress-partial.c and decompress-partial-usingDict.c become 2 tests of the lz4 suite ;
decompress-partial.c remains in tests/compat/cmake/, to check builds against an installed liblz4.
unit/ (unit tests), fuzz/ (randomized tests), cli/ (command line tests), compat/ (ABI, versions,
installation, freestanding, build environment), bench/, tools/ (datagen, checkFrame), data/.
Make targets and the location of test programs are unchanged.
33 tests, in 2 new suites : lz4 (decoding of invalid input, one-shot and streaming compression)
and lz4hc (streaming, dictionaries, destSize). Tests depending on the HC level run on both levels
previously selected by fuzzer.
UT_sample(), KB, MB and ARRAY_SIZE() are now provided by unittest.h, for all suites.
Each test is a function with its own context : resources are released at the end of the test,
even when it fails, and the random generator restarts from the same seed.
A failed assertion ends the test, the runner continues with the next one.
First suite : 18 lz4frame unit tests, moved from frametest.c
LZ4F_getMaxNbWorkers() (static linking only, experimental like nbWorkers) :
0 when liblz4 is built without LZ4F_MULTITHREAD, otherwise the cap of nbWorkers,
now 1024 on 64-bit systems and 128 on 32-bit ones (was 200).
lz4 : build macro LZ4_NBWORKERS_MAX removed ; worker threads of frame and legacy
formats are capped by LZ4F_getMaxNbWorkers(), 1 without liblz4 multi-threading
(restores a cap for the legacy format on POSIX). lz4 -V reports "multithread"
only when both lz4 and liblz4 support it ; -T# warning and -v note accordingly.
VS2022 _test.bat checks the version line, catching a CLI built without LZ4F_MULTITHREAD
(same output, so no other observable difference).
frameMT : validateMaxNbWorkers ; test builds use LZ4F_NBWORKERS_MAX=128.
Prototype, not intended for a PR as is.
- lz4 CLI : the automatic nb of workers (no -T, or -T0) is limited by a memory budget
(4 GB, 1 GB on 32-bit systems), estimated from liblz4's job slots, input buffers and
LZ4 states, plus the CLI's buffers (within ~5% of actual allocations when all are used).
It's also limited to the nb of jobs of each file, when its size is known
(ex: 3 threads for a 10 MB file, instead of 145 on a 166-core machine).
-9 on 844 MB, 166 cores : unchanged auto (145 workers, ~3 GB budget estimate).
- --memory=# (suffixes K, M, G) and environment variable LZ4_MEMORY set the budget ;
an explicitly set budget also limits an explicit -T#, as does the default one on 32-bit
systems. A budget too small for 1 worker triggers a warning.
- lz4 CLI : no more fixed maximum nb of workers for frames (LZ4_NBWORKERS_MAX now only
bounds the legacy format's thread pool) ; liblz4 keeps a sanity limit, raised to 1024
(200 on 32-bit systems, where it keeps LZ4F_compressBound() within address space),
overridable at compile time (tests : 200, keeping the capped test case affordable).
- liblz4 without LZ4F_MULTITHREAD (ex: default static library) : independent blocks take
the single-threaded path (same output, no job buffers, no extra copies) ; linked blocks
keep jobs processed serially (same output as multi-threaded, ~3% slower at level 1).
- liblz4 : frames cut into jobs no longer allocate tmpBuff (-4.1 MB, LZ4F_cctx_size()).
- liblz4 : LZ4F_uncompressedUpdate() with workers now cuts blocks as single-threaded mode :
an incomplete last block waits for next uncompressed input (it was written immediately,
producing different blocks for consecutive uncompressed invocations). With independent
blocks, output is now identical to nbWorkers == 0 in all cases, and doesn't depend on
multi-threading support. tests/frameMT.c compares uncompressed segments with
single-threaded mode too. Full digest reference : 899dd7e3d21cad9a.
- Documented in man (-T#, --memory, LZ4_MEMORY, LZ4_NBWORKERS), programs/README.md, lz4frame.h.
Validated : full tests MT / no-MT (same digest), ASAN+UBSAN, TSAN, make test, wine,
CLI == dev CLI, lz4-nomt == lz4, memory budget scenarios, modification scenarios,
build policies, ~88k libFuzzer executions.
Prototype, not intended for a PR as is.
- FORWARD_IF_ERROR() used to evaluate its argument twice ; applied to LZ4F_mt_begin(),
an allocation failure re-ran it, and LZ4F_compressBegin() reported success without
starting the frame. Found by allocation failure injection.
Fixed in dev by #1877 (single evaluation) ; FORWARD_IF_ERROR_MT() now follows the same pattern.
- tests/frameMT.c :
- each allocation fails in turn (one-shot, streaming copy / stableSrc, levels 1 and 9) :
clean allocation_failed or correct output, no leak, context remains usable ;
- LZ4F_write() / LZ4F_read() with workers : same frame as one-shot ;
- quick mode (-q), step labels, FRAMEMT_DUMP diagnostics ;
- CDict now created after its dictionary is filled (found by ASAN).
- make test runs test-frameMT-quick ; CI sanitizers : ASan and TSan steps.
- ossfuzz : compress_frame_stream_fuzzer employs different nbWorkers for reference and test,
comparing whole frames when output lags behind input ; round_trip_frame_fuzzer : random nbWorkers.
- LZ4F_MT_JOBSIZE_* can be overridden (testing only), to cross job boundaries with small inputs.
- lz4frame.h : per-invocation output depends on scheduling (without autoFlush).
Validated : full tests MT / no-MT (same digest), ASAN+UBSAN, TSAN, make test,
mingw-w64 build under wine (same digest as Linux), ~160k libFuzzer executions.
Prototype, not intended for a PR as is.
- LZ4F_preferences_t.nbWorkers (replaces reserved[0], ABI preserved):
LZ4F_compressFrame() and LZ4F_compressFrame_usingCDict() cut input into jobs,
compressed by up to nbWorkers threads (inline fork-join, threads created per call).
Output only depends on parameters, never on nbWorkers (>= 1) nor scheduling;
identical to single-threaded with independent blocks, identical to the CLI -T output.
- LZ4F_MULTITHREAD build macro (default 0) : without it, jobs run serially, same output.
- build policy (zstd-like) : dynamic library multi-threaded by default, static library not
(make lib-mt / lib-nomt / MULTITHREAD=, cmake LZ4_SHARED/STATIC_MULTITHREAD,
meson shared-/static-multithread, VS2022 liblz4-dll).
- internal LZ4F_compressBlocks_internal() extracted from LZ4F_compressUpdateImpl().
- tests/frameMT.c (make -C tests test-frameMT) : validation + bench mode.
Known gaps : Win32 thread path never compiled; no persistent thread pool; no streaming.
C++ forbids including a header within a linkage specification.
With clang C++ modules (-fmodules), including lz4.h or lz4hc.h failed :
<stdint.h> was imported within extern "C".
All public headers (lz4.h, lz4hc.h, lz4frame.h, lz4file.h) now include
their dependencies before opening extern "C".
New test : make -C tests test-cxx-modules (requires clang),
run in CI with clang on Linux, and on macOS.
Fixes#1600. Based on #1815 by @sergioperezcheco.
Firefox stores its .jsonlz4 / .mozlz4 files (session, bookmark backups,
search engines) as "mozLz40\0", the decompressed size (LE32),
then a single LZ4 block extending to the end of the file
(see MozLZ4 in Firefox's xpcom/ioutils/IOUtils.cpp).
`lz4 -d`, `lz4 -t` and `lz4cat` now recognize this format,
and decode it like Firefox's reader :
- the decompressed size is only used as decoding capacity
- empty content has no block (Firefox doesn't write one)
Allocations are bounded : sizes > LZ4_MAX_INPUT_SIZE are rejected,
as they can't be compressed into a single block,
and input is read up to LZ4_COMPRESSBOUND(size), the largest valid block.
Data starting with the first half of the magic only ("mozL") keeps
the previous behavior : `lz4cat` passes it through unchanged,
including the bytes read to check the second half.
Decoding only : lz4 doesn't produce this non-standard format.
Based on #1259 by @UnitedMarsupials, rewritten for current dev
(no mmap, bounded allocations, full magic check, tests).
freestanding.c only provides _start for x86_64 Linux targets,
while tests/Makefile decided to add -ffreestanding -nostdlib based on uname.
Whenever host and target disagree (CC="gcc -m32", cross-compilers),
the binary is linked without any entry point and crashes.
Query the compiler's preprocessor instead, using the same condition
as the guard in freestanding.c, so that both decisions are always consistent.
fix#1464
test_custom_block_sizes.sh was never invoked by any build target,
so the regression of custom block sizes (-B#, with # >= 32)
in multithreaded builds went unnoticed.
Replace it by test-lz4-blocksize.sh, part of test-lz4-essentials (make check).
It verifies the actual size of each block with checkFrame,
across all block size IDs, thread counts (-T1 vs -T4 must be identical),
concatenated frames, inputs below and above the 4 MB job size, stdin,
linked blocks (fast and HC), and block checksums.
This test fails on current dev.
Ubuntu patches uname -p to return the processor type:
https://git.launchpad.net/ubuntu/+source/coreutils/tree/debian/patches/80_fedora_sysinfo.patch?h=ubuntu/oracular
and Fedora also did before Fedora 38:
cd953e11dd
but in general, uname -p returns "unknown". Use uname -m instead.
Fixes build error:
/usr/bin/ld: /tmp/ccgv9E4e.o: in function `_start':
freestanding.c:(.text+0x6255a): multiple definition of `_start'; /usr/lib/gcc/x86_64-redhat-linux/14/../../../../lib64/crt1.o:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status
When $(PATH) contains ' ' (space, 0x20), this prefix causes error.
Also we don't need reconfiguring PATH for each shell script, we don't need them.
This changeset fixes issue #1195
Freestanding test is intended to run on only Linux x86_64 platform.
Because it uses Linux x86_64 syscall directly to avoid any dependency to the standard library.
This changeset adds platform checking code to tests/Makefile to avoid unintended
error in non-Linux x86_64 platforms.
This changeset resolves issue #1186
Specifically this adds support for the following options:
- LZ4_ALIGN_TEST
- LZ4_STATIC_LINKING_ONLY_DISABLE_MEMORY_ALLOCATION
- LZ4_DISTANCE_MAX
- LZ4_FAST_DEC_LOOP
- LZ4_FORCE_SW_BITCOUNT
- LZ4_FREESTANDING
- LZ4_USER_MEMORY_FUNCTIONS
- compiling ossfuzz targets
- compiling more test targets
- registering some tests
where `--test -m files*.lz4` would create a bunch of `files*` files.
`--test` should not create any file.
It turns out the bug was already fixed when #1164 was filled,
but it also shows that the CI system was blind to such issues when it happened.
The new test was run with `v1.9.3` and triggered an error, as expected.
note : it's unlikely to improve speed, as in most cases I/O is slower than lz4 decompression,
but maybe in extreme scenarios, it might show a difference.
