f4f7a9882c
A ring buffer of LZ4_DECODER_RING_BUFFER_SIZE(maxBlockSize) is allocated statically, so its size is compiled into applications.16d695b1enlarged its margin from 14 to 64 bytes to fit the fast loop's wide copies, leaving applications compiled with the former macro exposed to corrupted output. The decoder now also supports the former margin, without advertising it. When decoding restarts at the beginning of the ring, the oldest referenced history lies in the external dictionary, ahead of the write position, at a constant distance. When this distance is shorter than FASTLOOP_SAFE_DISTANCE, the block is decoded by the safe loop, which writes at most 16 bytes beyond a copy, like the v1.8.x decoder the 14-byte margin was sized for. This only concerns rings with less than 64 bytes of margin, hence never the current macro, for the blocks following such a restart, until 64 KB have been decoded. Other decoders are unaffected : the test is resolved at compile time outside of usingExtDict. Applications compiled with older headers decode correctly with the new library, without recompiling. The static assert now checks what correctness requires, i.e. the safe loop's margin, which the former macro satisfies : on this point, lz4.c compiles again with an older lz4.h (#1863). The fuzzer's worst-case ring test now also runs with the former ring size : it fails without this change, and passes with it. The other ring condition (decoder ring larger than the encoder's by maxBlockSize + 64 bytes,2f449bb3) is unchanged : the decoder can't know the encoder's ring size.
Programs and scripts for automated testing of LZ4
This directory contains the following programs and scripts:
datagen: Synthetic and parametrable data generator, for testsframetest: Test tool that checks lz4frame integrity on target platformfullbench: Precisely measure speed for each lz4 inner functionsfuzzer: Test tool, to check lz4 integrity on target platformtest-lz4-speed.py: script for testing lz4 speed difference between commitstest-lz4-versions.py: compatibility test between lz4 versions stored on Github
test-lz4-versions.py - script for testing lz4 interoperability between versions
This script creates versionsTest directory to which lz4 repository is cloned.
Then all tagged (released) versions of lz4 are compiled.
In the following step interoperability between lz4 versions is checked.
test-lz4-speed.py - script for testing lz4 speed difference between commits
This script creates speedTest directory to which lz4 repository is cloned.
Then it compiles all branches of lz4 and performs a speed benchmark for a given list of files (the testFileNames parameter).
After sleepTime (an optional parameter, default 300 seconds) seconds the script checks repository for new commits.
If a new commit is found it is compiled and a speed benchmark for this commit is performed.
The results of the speed benchmark are compared to the previous results.
If compression or decompression speed for one of lz4 levels is lower than lowerLimit (an optional parameter, default 0.98) the speed benchmark is restarted.
If second results are also lower than lowerLimit the warning e-mail is sent to recipients from the list (the emails parameter).
Additional remarks:
- To be sure that speed results are accurate the script should be run on a "stable" target system with no other jobs running in parallel
- Using the script with virtual machines can lead to large variations of speed results
- The speed benchmark is not performed until computers' load average is lower than
maxLoadAvg(an optional parameter, default 0.75) - The script sends e-mails using
mutt; ifmuttis not available it sends e-mails without attachments usingmail; if both are not available it only prints a warning
The example usage with two test files, one e-mail address, and with an additional message:
./test-lz4-speed.py "silesia.tar calgary.tar" "email@gmail.com" --message "tested on my laptop" --sleepTime 60
To run the script in background please use:
nohup ./test-lz4-speed.py testFileNames emails &
The full list of parameters:
positional arguments:
testFileNames file names list for speed benchmark
emails list of e-mail addresses to send warnings
optional arguments:
-h, --help show this help message and exit
--message MESSAGE attach an additional message to e-mail
--lowerLimit LOWERLIMIT
send email if speed is lower than given limit
--maxLoadAvg MAXLOADAVG
maximum load average to start testing
--lastCLevel LASTCLEVEL
last compression level for testing
--sleepTime SLEEPTIME
frequency of repository checking in seconds
License
All files in this directory are licensed under GPL-v2. See COPYING for details. The text of the license is also included at the top of each source file.