The manual says that -t neither creates nor removes files, and that
--rm deletes the source after a successful compression or
decompression. The CLI did not follow it for two cases :
- `lz4 -t --rm file.lz4` tested the file, then deleted it.
- With stdin as source, the output was complete, but remove("stdin")
failed and the command ended with "Remove error : stdin"
(exit code 50 when compressing, 45 when decompressing).
A file named `stdin` in the working directory would have been deleted.
One helper, LZ4IO_mustRemoveSrcFile(), now decides. It answers no in
test mode and for stdin, and both remove sites use it :
LZ4IO_finishCompressedFile() and LZ4IO_decompressSrcFile().
A regular source file is still removed.
dev (#1918) split file compression : LZ4IO_compressFilename_extRess() opens files and prepares preferences,
frame compressors only compress, and LZ4IO_finishCompressedFile() ends files for frame and legacy formats.
During the rebase, the branch's frame compression (liblz4 multi-threading, memory-mapped input,
detection of a source modified or replaced) kept its single function ; it now follows this split :
- LZ4IO_compressFrame() compresses, returning the frame tail (content checksum) ;
- LZ4IO_compressFilename_extRess() maps input, adjusts nbWorkers, then checks whether the source changed
(inconsistent memory-mapped output : removed, error 54) ;
- LZ4IO_finishCompressedFile() keeps a source modified or replaced during compression (--rm not honored),
and doesn't copy the attributes of a replaced one (legacy : never).
The processed size is 0 when a file can't be opened (dev fix, lost in the rebase).
lz4frame : worker states initialized by LZ4F_initLZ4State() (dev #1921), as the cctx's state.
Note : LZ4F_makeBlock() keeps explicit state parameters, instead of the cctx (dev #1921) : worker threads have no cctx.
lz4frame :
- LZ4F_uncompressedUpdate() accepts linked blocks (dev and v1.10.0 could silently corrupt such frames) :
single-threaded mode drops the LZ4 state's history when an uncompressed segment starts
(the decoder's history contains the uncompressed data) ; multi-threaded mode needs nothing,
jobs reload the 64 KB actually preceding them. LZ4F_compressBegin() now starts frames in compressed mode :
the mode of the previous frame's end was kept, which skipped the history drop (frame corrupted with a dictionary).
- Windows : worker stacks reserved instead of committed (STACK_SIZE_PARAM_IS_A_RESERVATION) ;
#error when <windows.h> was already included without the Vista API (amalgamated sources) ;
doc : no multi-threaded context use nor release under the loader lock (DllMain, DLL global objects).
lib/Makefile, programs/Makefile : threads detection skipped by goals building nothing threaded (clean, uninstall, ...),
run once per command (sub-makes inherit it) : make install probes once instead of 2-4 times, make clean never.
lz4 : explicit -T# also limited to the jobs of each file (-T1000 on a 10 MB file : 3 workers, RSS 298 -> 27 MB at -9),
and by the memory budget on all systems (-T99999 : 1024 liblz4 maximum, 331 threads within the 4 GB default budget) ;
an explicit nb of threads reduced triggers a warning (automatic : only with -v).
tests : frametest testUncompressedLinked (linked blocks, dictionary made of input, frames ending uncompressed),
mixed updates and small-dst scripts and fuzzer with linked blocks ; frameMT uncompressed segments with linked blocks ;
ossfuzz stream and uncompressed round-trip fuzzers with linked blocks ; test-lz4-multithread budget warnings.
LZ4F_compressBound(), without autoFlush :
- single-threaded : (srcSize / blockSize + 1) blocks, stored uncompressed in worst case, plus the frame footer.
It covers up to (blockSize-1) buffered bytes, a buffered block emitted separately (mode switch),
or remaining from an incomplete LZ4F_flush(). srcSize == 0 : one block + footer, so that LZ4F_flush() and
LZ4F_compressEnd() complete in a single invocation, strictly below the bound (v1.10.0 guarantee kept).
- nbWorkers >= 1 : the job replaces the block : ceil(srcSize / jobSize) jobs (1 for srcSize == 0) + footer,
independently of nbWorkers. Updates defer completed jobs not fitting into dstBuffer : output is only required
to start a job while all slots are busy. Flush and end emit jobs in flight in several invocations.
nbWorkers is new : no existing user depends on a single invocation there.
Multi-threaded LZ4F_uncompressedUpdate() : uncompressed segments are jobs storing blocks uncompressed ;
a mode switch submits the job being filled (same output as single-threaded mode). A single LZ4F_mt_update()
serves both updates : uncTail, LZ4F_mt_emitUncTail() and LZ4F_uncompressedUpdate_MT() are removed.
lz4 : output buffer sized by LZ4F_compressBound(chunk) (~4 MB, instead of 72 MB with 8 threads),
end of frame in a loop ; more threads within a memory budget (--memory=300M : 13 -> 23 threads) ;
releaseLag = (2*nbWorkers+1) jobs + 64 KB.
tests : frameMT (bound structure, flush / end loops), stream fuzzer (reference loops with nbWorkers).
LZ4F_flush() and LZ4F_compressEnd() : each invocation fills dstBuffer, or completes :
while it returns dstCapacity, output may remain, and the operation is invoked again, with any dstCapacity.
Output remains strictly below LZ4F_compressBound(0) (+1 for autoFlush), so that callers sizing
dstBuffer with the bound see no change : a single invocation completes.
- the first invocation sets the flush point : buffered input is compressed (all jobs waited for),
so input is no longer read after it returns (LZ4F_srcStableUntilFlush lifetime unchanged) ;
- single-threaded : a block not fitting is compressed into stageBuff (allocated on first need), same output ;
multi-threaded : the oldest job can be emitted partially (mt->emitted) ; frame footer staged (8 bytes) ;
- compressUpdate() / uncompressedUpdate() may follow an incomplete flush : remaining output comes first ;
until an end completes, only LZ4F_compressEnd() continues it (LZ4F_compressBegin() abandons it) ;
- dstCapacity 0 is rejected when output remains, without modifying the frame ;
- contentSize mismatch is detected when the end starts, before writing anything ;
- LZ4F_compressFrame*() keep failing with dstCapacity_tooSmall when the frame doesn't fit.
lz4file : LZ4F_writeClose() loops on LZ4F_compressEnd().
lz4 : releaseLag = LZ4F_compressBound(0) + 64 KB (bound of input retained within cctx).
tests : frametest (#1817 streaming probes, fuzzer), frameMT (validatePartialFlush, small dst, strict bound),
stream fuzzer : flush and end in pieces.
lz4hc levels >= 10 allocate a temporary buffer (LZ4HC_HEAPMODE) within worker threads :
documented in lz4frame.h (nbWorkers), lz4frame.c (pool) and lz4.c (LZ4_USER_MEMORY_FUNCTIONS
must then be thread-safe).
lz4 : the automatic nb of workers no longer shrinks from one file to the next
(cRess_t.nbWorkers high-water mark) : worker threads and buffers are kept
(-m on 300 files of 64 KB - 8 MB : 1156 -> 21 thread creations).
lz4 : with independent blocks, a file fitting into a single job is compressed by the calling
thread (nbWorkers 0, same output). Not with linked blocks : single-threaded output then depends
on stableSrc (last partial block referencing history in place), so memory-mapped input would differ.
lz4frame : without LZ4F_MULTITHREAD, LZ4F_compressBound() covers a single job slot
(jobs processed serially) : 4 MB instead of 68 MB with linked blocks and nbWorkers 8.
tests : frameMT validateCompressBound ; test-lz4-multithread : single-job files and -m.
LZ4F_getMaxNbWorkers() (static linking only, experimental like nbWorkers) :
0 when liblz4 is built without LZ4F_MULTITHREAD, otherwise the cap of nbWorkers,
now 1024 on 64-bit systems and 128 on 32-bit ones (was 200).
lz4 : build macro LZ4_NBWORKERS_MAX removed ; worker threads of frame and legacy
formats are capped by LZ4F_getMaxNbWorkers(), 1 without liblz4 multi-threading
(restores a cap for the legacy format on POSIX). lz4 -V reports "multithread"
only when both lz4 and liblz4 support it ; -T# warning and -v note accordingly.
VS2022 _test.bat checks the version line, catching a CLI built without LZ4F_MULTITHREAD
(same output, so no other observable difference).
frameMT : validateMaxNbWorkers ; test builds use LZ4F_NBWORKERS_MAX=128.
lz4io.c : the legacy compression pipeline no longer carries frame-only code
(ReadTracker.prefix and .xxh32, prefixSize parameter, CompressJobDesc.lastBlock).
xxhash.h remains included, for the content checksum of a memory-mapped source.
lz4frame.c, frameMT.c : nbWorkers == 1 means one worker thread, in addition to
the calling thread. Comments only, behavior unchanged.
Prototype, not intended for a PR as is.
- lz4 CLI : the automatic nb of workers (no -T, or -T0) is limited by a memory budget
(4 GB, 1 GB on 32-bit systems), estimated from liblz4's job slots, input buffers and
LZ4 states, plus the CLI's buffers (within ~5% of actual allocations when all are used).
It's also limited to the nb of jobs of each file, when its size is known
(ex: 3 threads for a 10 MB file, instead of 145 on a 166-core machine).
-9 on 844 MB, 166 cores : unchanged auto (145 workers, ~3 GB budget estimate).
- --memory=# (suffixes K, M, G) and environment variable LZ4_MEMORY set the budget ;
an explicitly set budget also limits an explicit -T#, as does the default one on 32-bit
systems. A budget too small for 1 worker triggers a warning.
- lz4 CLI : no more fixed maximum nb of workers for frames (LZ4_NBWORKERS_MAX now only
bounds the legacy format's thread pool) ; liblz4 keeps a sanity limit, raised to 1024
(200 on 32-bit systems, where it keeps LZ4F_compressBound() within address space),
overridable at compile time (tests : 200, keeping the capped test case affordable).
- liblz4 without LZ4F_MULTITHREAD (ex: default static library) : independent blocks take
the single-threaded path (same output, no job buffers, no extra copies) ; linked blocks
keep jobs processed serially (same output as multi-threaded, ~3% slower at level 1).
- liblz4 : frames cut into jobs no longer allocate tmpBuff (-4.1 MB, LZ4F_cctx_size()).
- liblz4 : LZ4F_uncompressedUpdate() with workers now cuts blocks as single-threaded mode :
an incomplete last block waits for next uncompressed input (it was written immediately,
producing different blocks for consecutive uncompressed invocations). With independent
blocks, output is now identical to nbWorkers == 0 in all cases, and doesn't depend on
multi-threading support. tests/frameMT.c compares uncompressed segments with
single-threaded mode too. Full digest reference : 899dd7e3d21cad9a.
- Documented in man (-T#, --memory, LZ4_MEMORY, LZ4_NBWORKERS), programs/README.md, lz4frame.h.
Validated : full tests MT / no-MT (same digest), ASAN+UBSAN, TSAN, make test, wine,
CLI == dev CLI, lz4-nomt == lz4, memory budget scenarios, modification scenarios,
build policies, ~88k libFuzzer executions.
Prototype, not intended for a PR as is.
Since the previous checkpoint, any modification of a memory-mapped source (default mode)
was an error, so compressing a log file growing meanwhile, which used to work, failed.
A file only grown leaves its mapped part unchanged, and the frame is consistent.
When a modification is detected, the mapped content is hashed again (XXH32) and compared
with the frame's content checksum, computed while reading it :
- equal : the content read is unchanged (ex: log file growing meanwhile) : the output is
valid, holding the file as it was when opened : warning only, source kept with --rm ;
- different, or can't be verified (--no-frame-crc, or file shrunk) : error 54, output removed.
Unmodified files cost nothing more.
Documented in man (--[no-]mmap), -H help and lz4io.h.
Validated : growing log (default mode : valid output), rewrite of content already read
(error 54), rewrite ahead of reading (consistent : accepted), --no-frame-crc (error 54),
--no-mmap (always valid), also with a CLI built with ASAN+UBSAN ; make test, CLI == dev CLI.
Prototype, not intended for a PR as is.
- lz4 CLI : a source file modified during its compression is detected, comparing its size
and times (mtime, ctime) before and after, like tar ("file changed as we read it").
The source is kept, even with --rm.
- memory-mapped input is read several times (checksum, compression, history of next job) :
the frame can be inconsistent : error 54, output removed.
- otherwise (--no-mmap), input is read once : the frame holds the content read,
ex: up to the end of a log file growing meanwhile : warning only.
Documented in man (--[no-]mmap, --rm), -H help and lz4io.h.
- lz4frame.c : a missing or incomplete worker pool (ex: transient allocation failure) is
created again at next frame, instead of staying serial for the context's lifetime.
- lz4frame.c : abandoning a frame (error, LZ4F_compressBegin(), LZ4F_freeCompressionContext())
cancels jobs not started yet, instead of compressing them (and the calling thread helping) :
only running jobs are waited for. Level 12, 4 MB jobs : CPU spent abandoning a frame
1.58 s -> 0.80 s (2 workers), 4.03 s -> 3.50 s (8 workers).
- lz4 CLI : memory-mapped input released behind (2*nbWorkers+1) jobs of 4 MB + 64 KB of history,
instead of nbWorkers+2 chunks (fewer refaults : ~5-8% fewer minor faults at -9 -BD).
- tests/frameMT.c : validatePoolRetry (each absorbed allocation failure, then next frame
creates worker threads again ; fails with the previous lib).
Validated : full tests MT / no-MT, ASAN+UBSAN, TSAN, make test, wine, CLI == dev CLI,
in-place rewrite / growing log / truncation scenarios, ~88k libFuzzer executions.
Prototype, not intended for a PR as is.
programs/Makefile probed pthread with a fixed have_pthread.c in its directory
(pre-existing in dev) : concurrent invocations collided (24 / 192 concurrent probes
wrongly reported no threads, producing an lz4 CLI without multi-threading),
and read-only source trees made it fail.
- build/make/lz4defs.make : PTHREAD_PROBE, evaluated with the CC and ALLFLAGS of
the invoking Makefile, within a private temporary directory (mktemp -d).
- lib/Makefile and programs/Makefile employ it (lib/'s copy from previous commit removed).
Validated : probes of lib/ and programs/ (192 concurrent, read-only trees), full tests
MT / no-MT, ASAN+UBSAN, TSAN, make test, wine, CLI build (lz4 multithread, lz4-nomt not),
build policies, install scenarios + test_install.sh.
Prototype, not intended for a PR as is.
A memory-mapped file truncated during its compression, or unreadable (I/O error),
raises SIGBUS in the thread reading it. Worker threads were blocking all signals,
so a fault in a worker killed the process, bypassing any handler.
- lz4frame.c : worker threads still block signals intended for the application,
but not synchronous ones (SIGBUS, SIGSEGV, SIGFPE, SIGILL) : the application's
handlers are invoked on faults (documented in lz4frame.h, nbWorkers).
- lz4 CLI : while a file is memory-mapped, a SIGBUS handler reports
"Error 53 : <file> : file truncated or unreadable during compression (memory-mapped) ;
retry with --no-mmap", then terminates the process (async-signal-safe : write, _exit).
The message is prepared beforehand ; when several threads fault simultaneously,
only the first one reports. Previous handler restored once unmapped.
- -H help, man page (--[no-]mmap) and lz4io.h updated.
Truncation test (silesia x4 truncated during -9 -T4) : error 53 with one message,
including --no-frame-crc (workers read the mapping first) ; with the former signal
mask, the process was killed by SIGBUS. --no-mmap : unchanged.
Output and speed unchanged. Validated : full tests MT / no-MT, ASAN+UBSAN, TSAN,
make test, wine (frameMT, lz4.exe == Linux CLI), CLI == dev CLI.
Prototype, not intended for a PR as is.
Historically, stableSrc only guarantees that the last 64 KB of input remain
present (linked blocks reference them instead of copying them). The MT engine
was reading whole jobs in place after LZ4F_compressUpdate() returns, silently
extending this contract : code correct with nbWorkers == 0 (buffer reused with
independent blocks, alternating buffers with linked blocks) became racy once
nbWorkers is enabled.
- lz4frame.h : enum LZ4F_stableSrc_t, the field remains unsigned (C++ compatibility) :
- LZ4F_srcUnstable (0) : default ;
- LZ4F_srcStable (1) : historical meaning, last 64 KB of input ;
- LZ4F_srcStableUntilFlush (2) : all of src remains until LZ4F_flush() / LZ4F_compressEnd()
returns, or the frame is abandoned : MT jobs read it in place.
- single-threaded behavior is unchanged : any non-zero value means stable.
- MT : only exactly LZ4F_srcStableUntilFlush enables in-place reading after return ;
otherwise job content is copied. autoFlush still reads in place (jobs complete within the call).
- one-shot compression and the CLI (memory-mapped input) employ LZ4F_srcStableUntilFlush.
- tests/frameMT.c : the three values ; with LZ4F_srcStable and workers, input is scrambled
after each invocation (a mutant reading it in place is detected, also by TSan).
- compress_frame_stream_fuzzer : stableSrc in 0..2.
Output unchanged (same digests), CLI mmap speed unchanged.
Validated : full tests MT / no-MT, ASAN+UBSAN, TSAN, make test, wine, CLI == dev CLI,
~88k libFuzzer executions.
Prototype, not intended for a PR as is.
- regular files are memory-mapped (POSIX 64-bit, frames cut into jobs only) and given to
LZ4F_compressUpdate() with stableSrc : compression jobs read input in place, removing
two copies from the main thread (fread, then copy into job buffers).
- on Linux, input far behind jobs in flight is released (madvise MADV_DONTNEED) :
only a memory hint, released pages are read again from the file if still needed.
- --no-mmap disables it : a file truncated while being compressed makes lz4 crash (SIGBUS).
- output is identical with and without mmap.
- -T# : # worker threads ; the main thread also compresses while waiting (up to #+1 cores).
- custom block sizes (-B# other than 64KB / 256KB / 1MB / 4MB) remain single-threaded :
documented in -H help, man page (-T#, -B#), and a note with -v.
Level 1, -T16, silesia.tar : 82 ms (previous CLI 118 ms, --no-mmap 162 ms).
Prototype, not intended for a PR as is.
- persistent thread pool within LZ4F_cctx (pthread / Win32), replaces the per-call fork-join;
nbWorkers is now the nb of worker threads (the calling thread also runs jobs while waiting).
- streaming : LZ4F_compressUpdate() / LZ4F_flush() / LZ4F_compressEnd() cut input into jobs
(4 MB linked, 1 MB independent), at fixed positions of the content; outputs emitted in order;
input read in place with stableSrc or autoFlush, copied otherwise; autoFlush respected;
LZ4F_compressBound() covers jobs in flight; LZ4F_uncompressedUpdate() drains pending jobs.
- one-shot compression goes through the same engine.
- CLI : frame compression goes through LZ4F streaming with nbWorkers (output identical to
previous CLI, and lz4 == lz4-nomt); custom block sizes fall back to single-thread.
- build systems compile the CLI's liblz4 sources with LZ4F_MULTITHREAD.
- tests/frameMT.c : streaming validation (slicing, stableSrc, flushes, autoFlush,
uncompressed segments, exact bounds, abandoned frames).
The automatic output name of decompression removed the .lz4 extension
with an obscure loop, comparing the last 5 characters (terminating zero
included) while zeroing them : it's now a plain suffix check.
Frame and legacy compression of multiple files each built the name of
the compressed file, with the same buffer management : initial
allocation, growth, strcpy + strcat of the suffix, and stdout special
case. They now share LZ4IO_compressedFileName(), and decompression of
multiple files shares its buffer management, LZ4IO_reserveFileName().
Single and multi-threaded frame compression duplicated the opening of
files, the preparation of preferences, and the end of file processing :
closing files, copying owner, permissions and modification time, --rm,
final status. Legacy compression had none of this end : `lz4 -l` ignored
`--rm`, and the compressed file didn't keep the source's attributes.
The end of file processing is now LZ4IO_finishCompressedFile(), shared by
frame and legacy compression, and LZ4IO_compressFilename_extRess() opens
files and prepares preferences for both frame compressors, which only
compress.
Also, LZ4IO_compressFilename_extRess() now reports a processed size of 0
when a file can't be opened, instead of leaving it uninitialized : with
-m, a missing file added a stale or garbage size to the final speed
summary (found by scan-build).
`--fast=5x` was accepted as `--fast=5`: characters following the
number were silently ignored, unlike `--threads=5x` which is rejected.
`--threadsX 2` was accepted as `--threads 2`, since the long command
was matched as a prefix only.
Both are now rejected with the usage message.
fseek() succeeds beyond end of file, so a skippable frame whose
declared size exceeds the remaining input was silently accepted
when reading from a file (`lz4 -t`, `lz4 -d` and `lz4 --list` all
reported success), while the same input from a pipe correctly failed
with "Error 43 : cannot skip skippable area".
fseek_u32() now seeks to the last byte of the area and reads it,
so truncation is detected the same way for files and pipes.
`lz4 -b -d` reserved 255x each frame's compressed size in the result
buffer, but capped that buffer at LZ4_MAX_INPUT_SIZE. With multiple
large files, later frames pointed past the buffer : 4 files of 4 MB
crash. The decode loop also recomputed each capacity with an INT_MAX
cap instead, so a frame decoding to more than LZ4_MAX_INPUT_SIZE wrote
out of bounds.
Decode-only output is not checked, so all frames now decode at the
start of the result buffer, each with its whole capacity. The capacity
is computed once, when the block table is laid out, and reused by the
decode loop.
Bug since v1.9.4. Based on #1770 by @arshsmith1
The multithreaded legacy decoder updated the decoded-byte counter
in its decoding thread, while the writer thread read it to display progress :
a data race, reported by ThreadSanitizer on any multi-block legacy stream.
The writer thread is now the only one to update and read it.
The final count is read after both threads complete.
Found by @agamal026 in #1890
The kernel's legacy decoder also stops at a zero Block Compressed Size,
and initramfs builders (mkinitcpio) append these four zero bytes
to delimit concatenated archives.
lz4 passed it to LZ4_decompress_safe() as an empty block,
and failed with "Error 64 : Decoding Failed".
A zero block size now ends the legacy frame,
in both legacy decoders and in --list,
and decoding continues with the next frame, if any.
It was never a valid block : a compressed block is at least 1 byte.
The format spec (v1.6.5) documents it as an extension :
encoders should not produce it, since older decoders reject it.
Fixes#1889, based on #1890 by @agamal026
BMK_benchMem() sized the compressed buffer with a margin of 1024 bytes
per block, computed as `maxNbBlocks * 1024` in 32-bit unsigned
arithmetic. With 4M blocks or more (e.g. `-B32` on a 128 MiB input),
the product wrapped around, and the buffer could be smaller than the
LZ4_compressBound() of all blocks, making the benchmark write past its end.
LZ4_compressBound() adds exactly 16 bytes per block, so the margin is
now `maxNbBlocks * 16`, computed in size_t.
Firefox stores its .jsonlz4 / .mozlz4 files (session, bookmark backups,
search engines) as "mozLz40\0", the decompressed size (LE32),
then a single LZ4 block extending to the end of the file
(see MozLZ4 in Firefox's xpcom/ioutils/IOUtils.cpp).
`lz4 -d`, `lz4 -t` and `lz4cat` now recognize this format,
and decode it like Firefox's reader :
- the decompressed size is only used as decoding capacity
- empty content has no block (Firefox doesn't write one)
Allocations are bounded : sizes > LZ4_MAX_INPUT_SIZE are rejected,
as they can't be compressed into a single block,
and input is read up to LZ4_COMPRESSBOUND(size), the largest valid block.
Data starting with the first half of the magic only ("mozL") keeps
the previous behavior : `lz4cat` passes it through unchanged,
including the bytes read to check the second half.
Decoding only : lz4 doesn't produce this non-standard format.
Based on #1259 by @UnitedMarsupials, rewritten for current dev
(no mmap, bounded allocations, full magic check, tests).
The legacy compressor used acceleration = -cLevel,
while lz4frame and bench use -cLevel + 1.
Legacy levels were shifted by one: `lz4 -l --fast=1` produced the same
output as `lz4 -l -1`, and `-l --fast=2` matched the frame `--fast=1`.
Use the same mapping, capped at LZ4_ACCELERATION_MAX
(avoids a signed overflow with --fast=2147483647).
Levels >= 0 are unchanged.
test-lz4-basic.sh: `-l --fast=1` must compress less than `-l -1`
(fails on dev), and `-l --fast=2147483647` runs.
Follow-up to "lz4frame: avoid signed overflow", per the #1824 review:
saturate at LZ4_ACCELERATION_MAX rather than INT_MAX,
since higher values make no difference.
LZ4_ACCELERATION_DEFAULT and LZ4_ACCELERATION_MAX move from lz4.c
to the LZ4_STATIC_LINKING_ONLY section of lz4.h, which lz4.c, lz4frame.c,
bench.c and frametest.c already include, so their values derive from it.
The LZ4_compress_fast() documentation already referred to both constants.
bench.c computed the acceleration the same way (-cLevel + 1),
so `lz4 -b --fast=2147483647` hit the same signed overflow:
apply the same cap there.
test-lz4-basic.sh: --fast=2147483647 must produce the same frame
as --fast=65536 (fails on dev, where acceleration wraps to the default),
and its benchmark runs (caught by the UBSan CI job before this fix).
lz4.1: --fast values above 65536 make no difference.
cLevelLast defaulted to -10000, meaning "unset".
With --fast=# and # > 10000, cLevel is below this sentinel,
so -b benchmarked all levels from -# up to -10000:
--fast=10003 ran 4 benchmarks, --fast=2147483647 ~2 billions.
Use INT_MIN as the sentinel instead.
readU32FromChar() silently wrapped around on values larger than 32 bits,
and compression levels were then cast to int without a range check.
As a result, out-of-range arguments did something unrelated instead of
failing, for example:
lz4 --fast=3000000000 compressed at maximum (HC) level
lz4 -3000000000 used the fastest level (output larger than input)
lz4 -4294967305 used level 9
lz4 -B4294967300 used 64 KB blocks
lz4 -B4194305K used 1 KB blocks
readU32FromChar() now exits with an error when the value (including
K/M suffixes) overflows 32 bits, and compression level arguments
(-#, -e#, --fast=#) are rejected when larger than INT_MAX.
LZ4_CLEVEL and LZ4_NBWORKERS values that overflow are ignored with the
existing warning, like other invalid values.
Also fix the --fast=0 and --fast=-1 tests in test-lz4-basic.sh: they
used a misspelled file name (dg20K instead of dg20k), so lz4 failed
because the file was missing rather than because of the option.
When the output filename buffer was too small,
LZ4IO_compressMultipleFilenames() and its _Legacy variant
reallocated it to ifnSize + 20, ignoring the suffix length.
A suffix longer than 19 bytes would overflow it in strcat().
Not reachable from the CLI, which only passes ".lz4".
Reported in #1773.
In both the _WIN32 and the POSIX variant of UTIL_prepareFileList(),
the buffer-growth branch computed
*bufEnd = *bufStart + newListSize;
before checking whether UTIL_realloc() succeeded. UTIL_realloc frees
the old block and returns NULL on failure, so the statement performs
pointer arithmetic on a null pointer (undefined behavior, C17 6.5.6)
and leaves *bufEnd holding a bogus non-NULL extent.
Found via cppcheck nullPointerArithmeticRedundantCheck warnings
(util.h:411 and util.h:468); confirmed end-to-end by building the real
lz4 CLI with clang -fsanitize=pointer-overflow and failing the growth
realloc via LD_PRELOAD: 'util.h:468:37: runtime error: applying
non-zero offset 16384 to null pointer'. With this patch the same run
completes with no sanitizer report.
On Windows, TPool_jobsCompleted() only waited on an auto-reset event,
set when the pending job count drops to 0 :
- with no job submitted, the event is never set and the wait never ends.
`lz4 -l` on an empty file submits no write job,
and decoding the resulting stream (magic number only) submits no decoding job.
- ResetEvent() + increment in TPool_submitJob() race with
decrement + SetEvent() in a worker : the event can stay set while a job
is still pending, so the wait could end before the last write.
#1740 checks the pending job count before waiting, and after each wake up.
The Windows CI never ran lz4 on files :
_test.bat now compresses and decodes an empty file in legacy format.
LZ4IO_decompressMultipleFilenames() always opened stdout as the shared
destination, even when the suffix is nulmark (test mode). As a result,
`lz4 -t -m` wrote decoded content to stdout, for every decoder which
does not skip writes in test mode (multithreaded decoders, legacy format).
Now, the shared destination is opened from the suffix (stdout or /dev/null),
and only when needed. Decoders no longer need to be aware of test mode.
The multithreaded compressor cut input into 4 MB jobs,
each compressed with a single LZ4F_compressUpdate() invocation,
and single-job inputs were compressed with LZ4F_compressFrame().
In both cases, blocks were cut at the block size ID maximum (64 KB minimum),
silently ignoring io_prefs->blockSize.
Now each job contains a whole number of blocks,
and is compressed one block per LZ4F_compressUpdate() invocation (stableSrc),
which is how LZ4F supports custom block sizes (with autoFlush).
Output buffers are sized accordingly.
Multithreading is preserved for custom block sizes,
and output for standard block sizes is unchanged.
Also: LZ4IO_defaultPreferences() now sets a blockSize consistent with blockSizeId
(it was 0, which the compressors cannot use).
The --rm option promises to remove the source only after successful decompression, but the recoverable trailing-data error path reaches source removal with a nonzero result. Keep the input whenever decoding reports failure, and cover the affected skippable-frame sequence with a regression.
