Frame and legacy compression of multiple files each built the name of
the compressed file, with the same buffer management : initial
allocation, growth, strcpy + strcat of the suffix, and stdout special
case. They now share LZ4IO_compressedFileName(), and decompression of
multiple files shares its buffer management, LZ4IO_reserveFileName().
Single and multi-threaded frame compression duplicated the opening of
files, the preparation of preferences, and the end of file processing :
closing files, copying owner, permissions and modification time, --rm,
final status. Legacy compression had none of this end : `lz4 -l` ignored
`--rm`, and the compressed file didn't keep the source's attributes.
The end of file processing is now LZ4IO_finishCompressedFile(), shared by
frame and legacy compression, and LZ4IO_compressFilename_extRess() opens
files and prepares preferences for both frame compressors, which only
compress.
Also, LZ4IO_compressFilename_extRess() now reports a processed size of 0
when a file can't be opened, instead of leaving it uninitialized : with
-m, a missing file added a stale or garbage size to the final speed
summary (found by scan-build).
`--fast=5x` was accepted as `--fast=5`: characters following the
number were silently ignored, unlike `--threads=5x` which is rejected.
`--threadsX 2` was accepted as `--threads 2`, since the long command
was matched as a prefix only.
Both are now rejected with the usage message.
fseek() succeeds beyond end of file, so a skippable frame whose
declared size exceeds the remaining input was silently accepted
when reading from a file (`lz4 -t`, `lz4 -d` and `lz4 --list` all
reported success), while the same input from a pipe correctly failed
with "Error 43 : cannot skip skippable area".
fseek_u32() now seeks to the last byte of the area and reads it,
so truncation is detected the same way for files and pipes.
`lz4 -b -d` reserved 255x each frame's compressed size in the result
buffer, but capped that buffer at LZ4_MAX_INPUT_SIZE. With multiple
large files, later frames pointed past the buffer : 4 files of 4 MB
crash. The decode loop also recomputed each capacity with an INT_MAX
cap instead, so a frame decoding to more than LZ4_MAX_INPUT_SIZE wrote
out of bounds.
Decode-only output is not checked, so all frames now decode at the
start of the result buffer, each with its whole capacity. The capacity
is computed once, when the block table is laid out, and reused by the
decode loop.
Bug since v1.9.4. Based on #1770 by @arshsmith1
The multithreaded legacy decoder updated the decoded-byte counter
in its decoding thread, while the writer thread read it to display progress :
a data race, reported by ThreadSanitizer on any multi-block legacy stream.
The writer thread is now the only one to update and read it.
The final count is read after both threads complete.
Found by @agamal026 in #1890
The kernel's legacy decoder also stops at a zero Block Compressed Size,
and initramfs builders (mkinitcpio) append these four zero bytes
to delimit concatenated archives.
lz4 passed it to LZ4_decompress_safe() as an empty block,
and failed with "Error 64 : Decoding Failed".
A zero block size now ends the legacy frame,
in both legacy decoders and in --list,
and decoding continues with the next frame, if any.
It was never a valid block : a compressed block is at least 1 byte.
The format spec (v1.6.5) documents it as an extension :
encoders should not produce it, since older decoders reject it.
Fixes#1889, based on #1890 by @agamal026
BMK_benchMem() sized the compressed buffer with a margin of 1024 bytes
per block, computed as `maxNbBlocks * 1024` in 32-bit unsigned
arithmetic. With 4M blocks or more (e.g. `-B32` on a 128 MiB input),
the product wrapped around, and the buffer could be smaller than the
LZ4_compressBound() of all blocks, making the benchmark write past its end.
LZ4_compressBound() adds exactly 16 bytes per block, so the margin is
now `maxNbBlocks * 16`, computed in size_t.
Firefox stores its .jsonlz4 / .mozlz4 files (session, bookmark backups,
search engines) as "mozLz40\0", the decompressed size (LE32),
then a single LZ4 block extending to the end of the file
(see MozLZ4 in Firefox's xpcom/ioutils/IOUtils.cpp).
`lz4 -d`, `lz4 -t` and `lz4cat` now recognize this format,
and decode it like Firefox's reader :
- the decompressed size is only used as decoding capacity
- empty content has no block (Firefox doesn't write one)
Allocations are bounded : sizes > LZ4_MAX_INPUT_SIZE are rejected,
as they can't be compressed into a single block,
and input is read up to LZ4_COMPRESSBOUND(size), the largest valid block.
Data starting with the first half of the magic only ("mozL") keeps
the previous behavior : `lz4cat` passes it through unchanged,
including the bytes read to check the second half.
Decoding only : lz4 doesn't produce this non-standard format.
Based on #1259 by @UnitedMarsupials, rewritten for current dev
(no mmap, bounded allocations, full magic check, tests).
The legacy compressor used acceleration = -cLevel,
while lz4frame and bench use -cLevel + 1.
Legacy levels were shifted by one: `lz4 -l --fast=1` produced the same
output as `lz4 -l -1`, and `-l --fast=2` matched the frame `--fast=1`.
Use the same mapping, capped at LZ4_ACCELERATION_MAX
(avoids a signed overflow with --fast=2147483647).
Levels >= 0 are unchanged.
test-lz4-basic.sh: `-l --fast=1` must compress less than `-l -1`
(fails on dev), and `-l --fast=2147483647` runs.
Follow-up to "lz4frame: avoid signed overflow", per the #1824 review:
saturate at LZ4_ACCELERATION_MAX rather than INT_MAX,
since higher values make no difference.
LZ4_ACCELERATION_DEFAULT and LZ4_ACCELERATION_MAX move from lz4.c
to the LZ4_STATIC_LINKING_ONLY section of lz4.h, which lz4.c, lz4frame.c,
bench.c and frametest.c already include, so their values derive from it.
The LZ4_compress_fast() documentation already referred to both constants.
bench.c computed the acceleration the same way (-cLevel + 1),
so `lz4 -b --fast=2147483647` hit the same signed overflow:
apply the same cap there.
test-lz4-basic.sh: --fast=2147483647 must produce the same frame
as --fast=65536 (fails on dev, where acceleration wraps to the default),
and its benchmark runs (caught by the UBSan CI job before this fix).
lz4.1: --fast values above 65536 make no difference.
cLevelLast defaulted to -10000, meaning "unset".
With --fast=# and # > 10000, cLevel is below this sentinel,
so -b benchmarked all levels from -# up to -10000:
--fast=10003 ran 4 benchmarks, --fast=2147483647 ~2 billions.
Use INT_MIN as the sentinel instead.
readU32FromChar() silently wrapped around on values larger than 32 bits,
and compression levels were then cast to int without a range check.
As a result, out-of-range arguments did something unrelated instead of
failing, for example:
lz4 --fast=3000000000 compressed at maximum (HC) level
lz4 -3000000000 used the fastest level (output larger than input)
lz4 -4294967305 used level 9
lz4 -B4294967300 used 64 KB blocks
lz4 -B4194305K used 1 KB blocks
readU32FromChar() now exits with an error when the value (including
K/M suffixes) overflows 32 bits, and compression level arguments
(-#, -e#, --fast=#) are rejected when larger than INT_MAX.
LZ4_CLEVEL and LZ4_NBWORKERS values that overflow are ignored with the
existing warning, like other invalid values.
Also fix the --fast=0 and --fast=-1 tests in test-lz4-basic.sh: they
used a misspelled file name (dg20K instead of dg20k), so lz4 failed
because the file was missing rather than because of the option.
When the output filename buffer was too small,
LZ4IO_compressMultipleFilenames() and its _Legacy variant
reallocated it to ifnSize + 20, ignoring the suffix length.
A suffix longer than 19 bytes would overflow it in strcat().
Not reachable from the CLI, which only passes ".lz4".
Reported in #1773.
In both the _WIN32 and the POSIX variant of UTIL_prepareFileList(),
the buffer-growth branch computed
*bufEnd = *bufStart + newListSize;
before checking whether UTIL_realloc() succeeded. UTIL_realloc frees
the old block and returns NULL on failure, so the statement performs
pointer arithmetic on a null pointer (undefined behavior, C17 6.5.6)
and leaves *bufEnd holding a bogus non-NULL extent.
Found via cppcheck nullPointerArithmeticRedundantCheck warnings
(util.h:411 and util.h:468); confirmed end-to-end by building the real
lz4 CLI with clang -fsanitize=pointer-overflow and failing the growth
realloc via LD_PRELOAD: 'util.h:468:37: runtime error: applying
non-zero offset 16384 to null pointer'. With this patch the same run
completes with no sanitizer report.
On Windows, TPool_jobsCompleted() only waited on an auto-reset event,
set when the pending job count drops to 0 :
- with no job submitted, the event is never set and the wait never ends.
`lz4 -l` on an empty file submits no write job,
and decoding the resulting stream (magic number only) submits no decoding job.
- ResetEvent() + increment in TPool_submitJob() race with
decrement + SetEvent() in a worker : the event can stay set while a job
is still pending, so the wait could end before the last write.
#1740 checks the pending job count before waiting, and after each wake up.
The Windows CI never ran lz4 on files :
_test.bat now compresses and decodes an empty file in legacy format.
LZ4IO_decompressMultipleFilenames() always opened stdout as the shared
destination, even when the suffix is nulmark (test mode). As a result,
`lz4 -t -m` wrote decoded content to stdout, for every decoder which
does not skip writes in test mode (multithreaded decoders, legacy format).
Now, the shared destination is opened from the suffix (stdout or /dev/null),
and only when needed. Decoders no longer need to be aware of test mode.
The multithreaded compressor cut input into 4 MB jobs,
each compressed with a single LZ4F_compressUpdate() invocation,
and single-job inputs were compressed with LZ4F_compressFrame().
In both cases, blocks were cut at the block size ID maximum (64 KB minimum),
silently ignoring io_prefs->blockSize.
Now each job contains a whole number of blocks,
and is compressed one block per LZ4F_compressUpdate() invocation (stableSrc),
which is how LZ4F supports custom block sizes (with autoFlush).
Output buffers are sized accordingly.
Multithreading is preserved for custom block sizes,
and output for standard block sizes is unchanged.
Also: LZ4IO_defaultPreferences() now sets a blockSize consistent with blockSizeId
(it was 0, which the compressors cannot use).
The --rm option promises to remove the source only after successful decompression, but the recoverable trailing-data error path reaches source removal with a nonzero result. Keep the input whenever decoding reports failure, and cover the affected skippable-frame sequence with a regression.
as cpus get faster, it's now possible for some of them to reach speeds > 10000 MB/s on single core.
Ensure that the display remains properly aligned in this case.
correctly skips the initialization part of the programs,
so that it's now on par with posix's clock().
In the future, would allow to measure cpu time between arbitrary points in the program execution, though for the time being, it's used for full session.
