lz4frame :
- LZ4F_uncompressedUpdate() accepts linked blocks (dev and v1.10.0 could silently corrupt such frames) :
single-threaded mode drops the LZ4 state's history when an uncompressed segment starts
(the decoder's history contains the uncompressed data) ; multi-threaded mode needs nothing,
jobs reload the 64 KB actually preceding them. LZ4F_compressBegin() now starts frames in compressed mode :
the mode of the previous frame's end was kept, which skipped the history drop (frame corrupted with a dictionary).
- Windows : worker stacks reserved instead of committed (STACK_SIZE_PARAM_IS_A_RESERVATION) ;
#error when <windows.h> was already included without the Vista API (amalgamated sources) ;
doc : no multi-threaded context use nor release under the loader lock (DllMain, DLL global objects).
lib/Makefile, programs/Makefile : threads detection skipped by goals building nothing threaded (clean, uninstall, ...),
run once per command (sub-makes inherit it) : make install probes once instead of 2-4 times, make clean never.
lz4 : explicit -T# also limited to the jobs of each file (-T1000 on a 10 MB file : 3 workers, RSS 298 -> 27 MB at -9),
and by the memory budget on all systems (-T99999 : 1024 liblz4 maximum, 331 threads within the 4 GB default budget) ;
an explicit nb of threads reduced triggers a warning (automatic : only with -v).
tests : frametest testUncompressedLinked (linked blocks, dictionary made of input, frames ending uncompressed),
mixed updates and small-dst scripts and fuzzer with linked blocks ; frameMT uncompressed segments with linked blocks ;
ossfuzz stream and uncompressed round-trip fuzzers with linked blocks ; test-lz4-multithread budget warnings.
LZ4F_getMaxNbWorkers() (static linking only, experimental like nbWorkers) :
0 when liblz4 is built without LZ4F_MULTITHREAD, otherwise the cap of nbWorkers,
now 1024 on 64-bit systems and 128 on 32-bit ones (was 200).
lz4 : build macro LZ4_NBWORKERS_MAX removed ; worker threads of frame and legacy
formats are capped by LZ4F_getMaxNbWorkers(), 1 without liblz4 multi-threading
(restores a cap for the legacy format on POSIX). lz4 -V reports "multithread"
only when both lz4 and liblz4 support it ; -T# warning and -v note accordingly.
VS2022 _test.bat checks the version line, catching a CLI built without LZ4F_MULTITHREAD
(same output, so no other observable difference).
frameMT : validateMaxNbWorkers ; test builds use LZ4F_NBWORKERS_MAX=128.
Prototype, not intended for a PR as is.
- lz4 CLI : the automatic nb of workers (no -T, or -T0) is limited by a memory budget
(4 GB, 1 GB on 32-bit systems), estimated from liblz4's job slots, input buffers and
LZ4 states, plus the CLI's buffers (within ~5% of actual allocations when all are used).
It's also limited to the nb of jobs of each file, when its size is known
(ex: 3 threads for a 10 MB file, instead of 145 on a 166-core machine).
-9 on 844 MB, 166 cores : unchanged auto (145 workers, ~3 GB budget estimate).
- --memory=# (suffixes K, M, G) and environment variable LZ4_MEMORY set the budget ;
an explicitly set budget also limits an explicit -T#, as does the default one on 32-bit
systems. A budget too small for 1 worker triggers a warning.
- lz4 CLI : no more fixed maximum nb of workers for frames (LZ4_NBWORKERS_MAX now only
bounds the legacy format's thread pool) ; liblz4 keeps a sanity limit, raised to 1024
(200 on 32-bit systems, where it keeps LZ4F_compressBound() within address space),
overridable at compile time (tests : 200, keeping the capped test case affordable).
- liblz4 without LZ4F_MULTITHREAD (ex: default static library) : independent blocks take
the single-threaded path (same output, no job buffers, no extra copies) ; linked blocks
keep jobs processed serially (same output as multi-threaded, ~3% slower at level 1).
- liblz4 : frames cut into jobs no longer allocate tmpBuff (-4.1 MB, LZ4F_cctx_size()).
- liblz4 : LZ4F_uncompressedUpdate() with workers now cuts blocks as single-threaded mode :
an incomplete last block waits for next uncompressed input (it was written immediately,
producing different blocks for consecutive uncompressed invocations). With independent
blocks, output is now identical to nbWorkers == 0 in all cases, and doesn't depend on
multi-threading support. tests/frameMT.c compares uncompressed segments with
single-threaded mode too. Full digest reference : 899dd7e3d21cad9a.
- Documented in man (-T#, --memory, LZ4_MEMORY, LZ4_NBWORKERS), programs/README.md, lz4frame.h.
Validated : full tests MT / no-MT (same digest), ASAN+UBSAN, TSAN, make test, wine,
CLI == dev CLI, lz4-nomt == lz4, memory budget scenarios, modification scenarios,
build policies, ~88k libFuzzer executions.
Prototype, not intended for a PR as is.
Since the previous checkpoint, any modification of a memory-mapped source (default mode)
was an error, so compressing a log file growing meanwhile, which used to work, failed.
A file only grown leaves its mapped part unchanged, and the frame is consistent.
When a modification is detected, the mapped content is hashed again (XXH32) and compared
with the frame's content checksum, computed while reading it :
- equal : the content read is unchanged (ex: log file growing meanwhile) : the output is
valid, holding the file as it was when opened : warning only, source kept with --rm ;
- different, or can't be verified (--no-frame-crc, or file shrunk) : error 54, output removed.
Unmodified files cost nothing more.
Documented in man (--[no-]mmap), -H help and lz4io.h.
Validated : growing log (default mode : valid output), rewrite of content already read
(error 54), rewrite ahead of reading (consistent : accepted), --no-frame-crc (error 54),
--no-mmap (always valid), also with a CLI built with ASAN+UBSAN ; make test, CLI == dev CLI.
Prototype, not intended for a PR as is.
- lz4 CLI : a source file modified during its compression is detected, comparing its size
and times (mtime, ctime) before and after, like tar ("file changed as we read it").
The source is kept, even with --rm.
- memory-mapped input is read several times (checksum, compression, history of next job) :
the frame can be inconsistent : error 54, output removed.
- otherwise (--no-mmap), input is read once : the frame holds the content read,
ex: up to the end of a log file growing meanwhile : warning only.
Documented in man (--[no-]mmap, --rm), -H help and lz4io.h.
- lz4frame.c : a missing or incomplete worker pool (ex: transient allocation failure) is
created again at next frame, instead of staying serial for the context's lifetime.
- lz4frame.c : abandoning a frame (error, LZ4F_compressBegin(), LZ4F_freeCompressionContext())
cancels jobs not started yet, instead of compressing them (and the calling thread helping) :
only running jobs are waited for. Level 12, 4 MB jobs : CPU spent abandoning a frame
1.58 s -> 0.80 s (2 workers), 4.03 s -> 3.50 s (8 workers).
- lz4 CLI : memory-mapped input released behind (2*nbWorkers+1) jobs of 4 MB + 64 KB of history,
instead of nbWorkers+2 chunks (fewer refaults : ~5-8% fewer minor faults at -9 -BD).
- tests/frameMT.c : validatePoolRetry (each absorbed allocation failure, then next frame
creates worker threads again ; fails with the previous lib).
Validated : full tests MT / no-MT, ASAN+UBSAN, TSAN, make test, wine, CLI == dev CLI,
in-place rewrite / growing log / truncation scenarios, ~88k libFuzzer executions.
Prototype, not intended for a PR as is.
A memory-mapped file truncated during its compression, or unreadable (I/O error),
raises SIGBUS in the thread reading it. Worker threads were blocking all signals,
so a fault in a worker killed the process, bypassing any handler.
- lz4frame.c : worker threads still block signals intended for the application,
but not synchronous ones (SIGBUS, SIGSEGV, SIGFPE, SIGILL) : the application's
handlers are invoked on faults (documented in lz4frame.h, nbWorkers).
- lz4 CLI : while a file is memory-mapped, a SIGBUS handler reports
"Error 53 : <file> : file truncated or unreadable during compression (memory-mapped) ;
retry with --no-mmap", then terminates the process (async-signal-safe : write, _exit).
The message is prepared beforehand ; when several threads fault simultaneously,
only the first one reports. Previous handler restored once unmapped.
- -H help, man page (--[no-]mmap) and lz4io.h updated.
Truncation test (silesia x4 truncated during -9 -T4) : error 53 with one message,
including --no-frame-crc (workers read the mapping first) ; with the former signal
mask, the process was killed by SIGBUS. --no-mmap : unchanged.
Output and speed unchanged. Validated : full tests MT / no-MT, ASAN+UBSAN, TSAN,
make test, wine (frameMT, lz4.exe == Linux CLI), CLI == dev CLI.
Prototype, not intended for a PR as is.
- regular files are memory-mapped (POSIX 64-bit, frames cut into jobs only) and given to
LZ4F_compressUpdate() with stableSrc : compression jobs read input in place, removing
two copies from the main thread (fread, then copy into job buffers).
- on Linux, input far behind jobs in flight is released (madvise MADV_DONTNEED) :
only a memory hint, released pages are read again from the file if still needed.
- --no-mmap disables it : a file truncated while being compressed makes lz4 crash (SIGBUS).
- output is identical with and without mmap.
- -T# : # worker threads ; the main thread also compresses while waiting (up to #+1 cores).
- custom block sizes (-B# other than 64KB / 256KB / 1MB / 4MB) remain single-threaded :
documented in -H help, man page (-T#, -B#), and a note with -v.
Level 1, -T16, silesia.tar : 82 ms (previous CLI 118 ms, --no-mmap 162 ms).
The automatic output name of decompression removed the .lz4 extension
with an obscure loop, comparing the last 5 characters (terminating zero
included) while zeroing them : it's now a plain suffix check.
`--fast=5x` was accepted as `--fast=5`: characters following the
number were silently ignored, unlike `--threads=5x` which is rejected.
`--threadsX 2` was accepted as `--threads 2`, since the long command
was matched as a prefix only.
Both are now rejected with the usage message.
cLevelLast defaulted to -10000, meaning "unset".
With --fast=# and # > 10000, cLevel is below this sentinel,
so -b benchmarked all levels from -# up to -10000:
--fast=10003 ran 4 benchmarks, --fast=2147483647 ~2 billions.
Use INT_MIN as the sentinel instead.
readU32FromChar() silently wrapped around on values larger than 32 bits,
and compression levels were then cast to int without a range check.
As a result, out-of-range arguments did something unrelated instead of
failing, for example:
lz4 --fast=3000000000 compressed at maximum (HC) level
lz4 -3000000000 used the fastest level (output larger than input)
lz4 -4294967305 used level 9
lz4 -B4294967300 used 64 KB blocks
lz4 -B4194305K used 1 KB blocks
readU32FromChar() now exits with an error when the value (including
K/M suffixes) overflows 32 bits, and compression level arguments
(-#, -e#, --fast=#) are rejected when larger than INT_MAX.
LZ4_CLEVEL and LZ4_NBWORKERS values that overflow are ignored with the
existing warning, like other invalid values.
Also fix the --fast=0 and --fast=-1 tests in test-lz4-basic.sh: they
used a misspelled file name (dg20K instead of dg20k), so lz4 failed
because the file was missing rather than because of the option.
fixes#1193.
Implicit `stdout` has been a behavior of the `lz4` cli since its early days,
but it ends up being confusing for users, which do not expect this behavior.
The feature is already deprecated, it has been issuing a warning message for several years.
It's time to cut it out.
early versions of clang fail on a heisenbug,
on what seems to be a buggy optimizer issue (disappears with -O0).
later versions have even more issue producing basic functioning executables.
this test seems hard to setup.
might not be worth the trouble.
The resource is being freed anyway, since this is an exit point from `main()`,
but it's proper to release the resource manually first,
and avoids triggering an automated detection tool.
