`lz4 -b -d` reserved 255x each frame's compressed size in the result
buffer, but capped that buffer at LZ4_MAX_INPUT_SIZE. With multiple
large files, later frames pointed past the buffer : 4 files of 4 MB
crash. The decode loop also recomputed each capacity with an INT_MAX
cap instead, so a frame decoding to more than LZ4_MAX_INPUT_SIZE wrote
out of bounds.
Decode-only output is not checked, so all frames now decode at the
start of the result buffer, each with its whole capacity. The capacity
is computed once, when the block table is laid out, and reused by the
decode loop.
Bug since v1.9.4. Based on #1770 by @arshsmith1
BMK_benchMem() sized the compressed buffer with a margin of 1024 bytes
per block, computed as `maxNbBlocks * 1024` in 32-bit unsigned
arithmetic. With 4M blocks or more (e.g. `-B32` on a 128 MiB input),
the product wrapped around, and the buffer could be smaller than the
LZ4_compressBound() of all blocks, making the benchmark write past its end.
LZ4_compressBound() adds exactly 16 bytes per block, so the margin is
now `maxNbBlocks * 16`, computed in size_t.
Follow-up to "lz4frame: avoid signed overflow", per the #1824 review:
saturate at LZ4_ACCELERATION_MAX rather than INT_MAX,
since higher values make no difference.
LZ4_ACCELERATION_DEFAULT and LZ4_ACCELERATION_MAX move from lz4.c
to the LZ4_STATIC_LINKING_ONLY section of lz4.h, which lz4.c, lz4frame.c,
bench.c and frametest.c already include, so their values derive from it.
The LZ4_compress_fast() documentation already referred to both constants.
bench.c computed the acceleration the same way (-cLevel + 1),
so `lz4 -b --fast=2147483647` hit the same signed overflow:
apply the same cap there.
test-lz4-basic.sh: --fast=2147483647 must produce the same frame
as --fast=65536 (fails on dev, where acceleration wraps to the default),
and its benchmark runs (caught by the UBSan CI job before this fix).
lz4.1: --fast values above 65536 make no difference.
as cpus get faster, it's now possible for some of them to reach speeds > 10000 MB/s on single core.
Ensure that the display remains properly aligned in this case.
benchmark dictionary mode, implemented in #808, is incorrect.
It would desynchronize compression and decompression as soon as 2+ files are dictionary compressed.
Also : slightly improved traces system, to also include __LINE__ number
Support the -D command line option for running benchmarks. The
benchmark code was slightly restructured to factor out the calls
that need to be different for each benchmark scenario. Since there
are now 4 scenarios (all combinations of fast/HC and with/without
dictionary), the logic was getting somewhat convoluted otherwise.
This was done by extending the compressionParameters struct that
previously contained just a single function pointer. It now
contains 4 function pointers for init/reset/compress/cleanup,
with the related state. The functions get a pointer to the
structure as their first argument (inspired by C++), so that they
can access the state values in the struct.
it was a fairly complex scenario,
involving source files > 64K
and some extraordinary conditions related to specific layout of ranges of zeroes.
and only on level 9.
Previous method would produce too many time() invocations,
becoming a significant fraction of workload measured.
The new strategy is to use time() only once per batch,
and dynamically resize batch size so that each round lasts approximately 1 second.
This only matters for small inputs.
Measurement for large files (such as silesia.tar) are much less impacted
(though decoding speed is so fast that even medium-size files will notice an improvement).
The timer used was only accurate up to 0.01 seconds. This timer is accurate up to 1 ns.
It is a monotonic timer that measures the real time difference, not on CPU time.
Copied the benchmark code from 6ab4d5e904
