LZ4F_getMaxNbWorkers() (static linking only, experimental like nbWorkers) :
0 when liblz4 is built without LZ4F_MULTITHREAD, otherwise the cap of nbWorkers,
now 1024 on 64-bit systems and 128 on 32-bit ones (was 200).
lz4 : build macro LZ4_NBWORKERS_MAX removed ; worker threads of frame and legacy
formats are capped by LZ4F_getMaxNbWorkers(), 1 without liblz4 multi-threading
(restores a cap for the legacy format on POSIX). lz4 -V reports "multithread"
only when both lz4 and liblz4 support it ; -T# warning and -v note accordingly.
VS2022 _test.bat checks the version line, catching a CLI built without LZ4F_MULTITHREAD
(same output, so no other observable difference).
frameMT : validateMaxNbWorkers ; test builds use LZ4F_NBWORKERS_MAX=128.
Prototype, not intended for a PR as is.
- lz4 CLI : the automatic nb of workers (no -T, or -T0) is limited by a memory budget
(4 GB, 1 GB on 32-bit systems), estimated from liblz4's job slots, input buffers and
LZ4 states, plus the CLI's buffers (within ~5% of actual allocations when all are used).
It's also limited to the nb of jobs of each file, when its size is known
(ex: 3 threads for a 10 MB file, instead of 145 on a 166-core machine).
-9 on 844 MB, 166 cores : unchanged auto (145 workers, ~3 GB budget estimate).
- --memory=# (suffixes K, M, G) and environment variable LZ4_MEMORY set the budget ;
an explicitly set budget also limits an explicit -T#, as does the default one on 32-bit
systems. A budget too small for 1 worker triggers a warning.
- lz4 CLI : no more fixed maximum nb of workers for frames (LZ4_NBWORKERS_MAX now only
bounds the legacy format's thread pool) ; liblz4 keeps a sanity limit, raised to 1024
(200 on 32-bit systems, where it keeps LZ4F_compressBound() within address space),
overridable at compile time (tests : 200, keeping the capped test case affordable).
- liblz4 without LZ4F_MULTITHREAD (ex: default static library) : independent blocks take
the single-threaded path (same output, no job buffers, no extra copies) ; linked blocks
keep jobs processed serially (same output as multi-threaded, ~3% slower at level 1).
- liblz4 : frames cut into jobs no longer allocate tmpBuff (-4.1 MB, LZ4F_cctx_size()).
- liblz4 : LZ4F_uncompressedUpdate() with workers now cuts blocks as single-threaded mode :
an incomplete last block waits for next uncompressed input (it was written immediately,
producing different blocks for consecutive uncompressed invocations). With independent
blocks, output is now identical to nbWorkers == 0 in all cases, and doesn't depend on
multi-threading support. tests/frameMT.c compares uncompressed segments with
single-threaded mode too. Full digest reference : 899dd7e3d21cad9a.
- Documented in man (-T#, --memory, LZ4_MEMORY, LZ4_NBWORKERS), programs/README.md, lz4frame.h.
Validated : full tests MT / no-MT (same digest), ASAN+UBSAN, TSAN, make test, wine,
CLI == dev CLI, lz4-nomt == lz4, memory budget scenarios, modification scenarios,
build policies, ~88k libFuzzer executions.
