33 tests, in 2 new suites : lz4 (decoding of invalid input, one-shot and streaming compression)
and lz4hc (streaming, dictionaries, destSize). Tests depending on the HC level run on both levels
previously selected by fuzzer.
UT_sample(), KB, MB and ARRAY_SIZE() are now provided by unittest.h, for all suites.
Each test is a function with its own context : resources are released at the end of the test,
even when it fails, and the random generator restarts from the same seed.
A failed assertion ends the test, the runner continues with the next one.
First suite : 18 lz4frame unit tests, moved from frametest.c
The DLL was compiled directly from sources, without the objects' cache keyed by flags :
after make lib-nomt, make lib-mt kept the DLL without threads (and conversely).
It now uses c_dynamic_library(), as the Unix shared library, with the multi-threading flags
in the cache key ; the resource object is an additional dependency. The import library
is the same for both variants (same exports). With BUILD_SHARED=no, liblz4 builds nothing, as before.
Checked with mingw-w64 : MULTITHREAD=0 / 1 / back to 1 (cached) switch the DLL's thread imports ;
a program linked to the multi-threaded DLL compresses with nbWorkers=4 under wine.
LZ4F_getMaxNbWorkers() (static linking only, experimental like nbWorkers) :
0 when liblz4 is built without LZ4F_MULTITHREAD, otherwise the cap of nbWorkers,
now 1024 on 64-bit systems and 128 on 32-bit ones (was 200).
lz4 : build macro LZ4_NBWORKERS_MAX removed ; worker threads of frame and legacy
formats are capped by LZ4F_getMaxNbWorkers(), 1 without liblz4 multi-threading
(restores a cap for the legacy format on POSIX). lz4 -V reports "multithread"
only when both lz4 and liblz4 support it ; -T# warning and -v note accordingly.
VS2022 _test.bat checks the version line, catching a CLI built without LZ4F_MULTITHREAD
(same output, so no other observable difference).
frameMT : validateMaxNbWorkers ; test builds use LZ4F_NBWORKERS_MAX=128.
LZ4F_compressBegin*() validates its arguments first (dictSize included) :
rejecting them leaves a frame in progress untouched, jobs in flight included
(contract of #1817, tested by frametest). Past validation, the frame in progress
is abandoned before cctx is modified : an allocation failure invalidates it
(previously, cStage remained 1 with an inconsistent cctx, also with nbWorkers == 0).
lz4frame.h : LZ4F_srcStableUntilFlush lifetime and LZ4F_compressBegin() failures clarified.
Without autoFlush, LZ4F_compressUpdate() defers completed jobs not fitting into dst,
instead of failing : it only fails when output must be emitted (all job slots busy).
VS2022 : lz4.vcxproj x64 configurations define LZ4F_MULTITHREAD=1.
frameMT : validateDeferredEmission, validateBeginFailures.
Prototype, not intended for a PR as is.
programs/Makefile probed pthread with a fixed have_pthread.c in its directory
(pre-existing in dev) : concurrent invocations collided (24 / 192 concurrent probes
wrongly reported no threads, producing an lz4 CLI without multi-threading),
and read-only source trees made it fail.
- build/make/lz4defs.make : PTHREAD_PROBE, evaluated with the CC and ALLFLAGS of
the invoking Makefile, within a private temporary directory (mktemp -d).
- lib/Makefile and programs/Makefile employ it (lib/'s copy from previous commit removed).
Validated : probes of lib/ and programs/ (192 concurrent, read-only trees), full tests
MT / no-MT, ASAN+UBSAN, TSAN, make test, wine, CLI build (lz4 multithread, lz4-nomt not),
build policies, install scenarios + test_install.sh.
Prototype, not intended for a PR as is.
- persistent thread pool within LZ4F_cctx (pthread / Win32), replaces the per-call fork-join;
nbWorkers is now the nb of worker threads (the calling thread also runs jobs while waiting).
- streaming : LZ4F_compressUpdate() / LZ4F_flush() / LZ4F_compressEnd() cut input into jobs
(4 MB linked, 1 MB independent), at fixed positions of the content; outputs emitted in order;
input read in place with stableSrc or autoFlush, copied otherwise; autoFlush respected;
LZ4F_compressBound() covers jobs in flight; LZ4F_uncompressedUpdate() drains pending jobs.
- one-shot compression goes through the same engine.
- CLI : frame compression goes through LZ4F streaming with nbWorkers (output identical to
previous CLI, and lz4 == lz4-nomt); custom block sizes fall back to single-thread.
- build systems compile the CLI's liblz4 sources with LZ4F_MULTITHREAD.
- tests/frameMT.c : streaming validation (slicing, stableSrc, flushes, autoFlush,
uncompressed segments, exact bounds, abandoned frames).
Prototype, not intended for a PR as is.
- LZ4F_preferences_t.nbWorkers (replaces reserved[0], ABI preserved):
LZ4F_compressFrame() and LZ4F_compressFrame_usingCDict() cut input into jobs,
compressed by up to nbWorkers threads (inline fork-join, threads created per call).
Output only depends on parameters, never on nbWorkers (>= 1) nor scheduling;
identical to single-threaded with independent blocks, identical to the CLI -T output.
- LZ4F_MULTITHREAD build macro (default 0) : without it, jobs run serially, same output.
- build policy (zstd-like) : dynamic library multi-threaded by default, static library not
(make lib-mt / lib-nomt / MULTITHREAD=, cmake LZ4_SHARED/STATIC_MULTITHREAD,
meson shared-/static-multithread, VS2022 liblz4-dll).
- internal LZ4F_compressBlocks_internal() extracted from LZ4F_compressUpdateImpl().
- tests/frameMT.c (make -C tests test-frameMT) : validation + bench mode.
Known gaps : Win32 thread path never compiled; no persistent thread pool; no streaming.
- grammar: "a lossless compression algorithm", "multi-core CPUs",
"The LZ4 library ... the BSD 2-Clause license", "defined in"
- clarify that the acceleration factor tunes compression speed
- dictionary API / CLI links were pinned to v1.8.3:
point to the in-tree lib/lz4frame.h and programs/lz4.1.md instead
- Zstandard dictionary builder link: v1.3.5 -> dev
- mention the GPL-2.0-or-later license of the CLI, as stated in LICENSE
- switch homepage, zlib, zstd and Silesia links to https
- point to build/ for CMake, Meson and Visual Studio builds
- vcpkg instructions: ./vcpkg, not ./vcpkg.exe, after bootstrap-vcpkg.sh
- list the API headers in the Documentation section
build/README.md: VS2022 solution outputs to build\VS2022\bin, not build\VS2010\bin
On Windows, TPool_jobsCompleted() only waited on an auto-reset event,
set when the pending job count drops to 0 :
- with no job submitted, the event is never set and the wait never ends.
`lz4 -l` on an empty file submits no write job,
and decoding the resulting stream (magic number only) submits no decoding job.
- ResetEvent() + increment in TPool_submitJob() race with
decrement + SetEvent() in a worker : the event can stay set while a job
is still pending, so the wait could end before the last write.
#1740 checks the pending job count before waiting, and after each wake up.
The Windows CI never ran lz4 on files :
_test.bat now compresses and decodes an empty file in legacy format.
The Makefile and meson enable multithreading when pthread is available,
and Windows enables it by default (programs/lz4conf.h).
CMake never did : on Linux, macOS or BSD, its lz4 binary was single-threaded.
CMake now detects pthread (find_package(Threads)) for lz4 and lz4c.
The new option LZ4_BUILD_MULTITHREAD (default ON) disables it, Windows included.
silesia.tar, lz4 -1 : 0.89 s => 0.25 s, same output size.
- Add conditional compilation flag for Debug builds
- Set LZ4_DEBUG=1 to enable assert() statements when debugging
- Use compiler-specific syntax: /DLZ4_DEBUG=1 for MSVC, -DLZ4_DEBUG=1 for GCC/Clang
- Ensure CMake 3.5+ compatibility using add_definitions()
- Apply globally to all targets for consistent debug behavior
Add missing DEPENDS lz4cli to symlink creation targets to ensure
the lz4cli executable is built before attempting to create symlinks.
This resolves 'No such file or directory' errors when using --parallel
builds where symlink targets could run before the executable target
was completed.
Fixes parallel build compatibility for Unix symlink creation.
- Set Release as default build type when none is specified
- Prevents undefined build behavior when running cmake without -DCMAKE_BUILD_TYPE
- Includes proper cache configuration with user-friendly description
- Provides dropdown options for cmake-gui users
This commit:
- Maintains CMake compatibility with older systems by keeping the minimum version at 3.5
- Sets the maximum policy version to 4.0.2 to ensure consistent behavior with newer CMake versions
- Adds a comment explaining the version range specification purpose
- Ensures future-proofing as newer CMake versions will use the 4.0.2 policy set
This commit improves the readability and maintainability of the CMake build system by:
1. Adding clear section headers with descriptive comments
2. Fixing indentation consistency throughout the file
3. Adding labels to endif() statements to match their corresponding if statements
4. Using visual separators (dashed lines) for better visual organization
These changes make the file easier to navigate without changing its functionality.
Update cmake minimum required version to work with CMake v4 and shut up about the deprecation warning (for cmake_minimum_required < 3.10) on CMake v3.31+
Regression in original introduction in commit
198e532300.
Every meson file in the repository must have this variable set, to an
appropriate per-file value, in order to cater to the split buildsystem
design where the build files are stored in a subdirectory away from the
source code.
If ossfuzz is enabled, this file is processed and the missing value will
come from some other file instead, which may not be correct.
- Using default values, ossfuzz is enabled but no other subdirs are, other
than of course lib, which has a coincidentally identical value, and
everything works.
- If contrib is enabled, it will set the per-file variable to a value
that is *not* coincidentally the same as ossfuzz.
- If examples is enabled, it will set the per-file variable to a value
that is coincidentally the same as ossfuzz... and do so later than
contrib.
So, if contrib is enabled but examples is not, then the ossfuzz
directory fails to process correctly and the build aborts. Fix this by
explicitly adding the correct per-file variable.
Fixes: https://github.com/lz4/lz4/issues/1559
Match the Makefile, which never sets LZ4_DEBUG for this test. If
LZ4_DEBUG is non-zero, lib/lz4.c #includes <assert.h>, the musl version of
which has a conflicting definition for __assert_fail:
https://www.openwall.com/lists/musl/2019/03/04/6
Fixes build failure:
../../../tests/freestanding.c:173:6: error: conflicting types for '__assert_fail'; have 'void(const char *, const char *, unsigned int, const char *)'
173 | void __assert_fail(const char * assertion, const char * file, unsigned int line, const char * function) {
| ^~~~~~~~~~~~~
In file included from ../../../tests/../lib/lz4.c:270,
from ../../../tests/freestanding.c:49:
/usr/include/assert.h:19:16: note: previous declaration of '__assert_fail' with type 'void(const char *, const char *, int, const char *)'
19 | _Noreturn void __assert_fail (const char *, const char *, int, const char *);
| ^~~~~~~~~~~~~
On Solaris/OpenIndiana hosts forcing (old) C99 means disabling POSIX
2001 functionality, resulting in errors like
In file included from ../../../net/ptah/export/gentoo/working-repos/lz4/programs/bench.c:39:
../../../net/ptah/export/gentoo/working-repos/lz4/programs/util.h: In function UTIL_getOpenFileSize’:
../../../net/ptah/export/gentoo/working-repos/lz4/programs/util.h:156:23: error: implicit declaration of function ‘fileno’ [-Wimplicit-function-declaration]
156 | # define UTIL_fileno fileno
| ^~~~~~
../../../net/ptah/export/gentoo/working-repos/lz4/programs/util.h:325:10: note: in expansion of macro ‘UTIL_fileno’
325 | fd = UTIL_fileno(file);
| ^~~~~~~~~~~
These can be fixed either by forcing a standard to be applied in programs/platform.h or by not forcing the compiler to use an old standard.
Since CMake and Makefile don't force C99 by default either, just drop it from
meson.build.
Signed-off-by: Fabian Groffen <grobian@gentoo.org>
The freestanding test is special as it requires reimplementation of
runtime/toolchain functionality. When building with clang where the
compiler is configured to enable the stack-protector by default,
building fails due to missing __stack_chk_fail symbols.
Thefore disable stack protection specifically for this test.
