Yann Collet c0a1b631e6 review round 12: LZ4F_uncompressedUpdate() supports linked blocks ; CLI threads capped by file jobs and memory budget, with warnings ; Windows and Makefile fixes (checkpoint) ...

lz4frame :
- LZ4F_uncompressedUpdate() accepts linked blocks (dev and v1.10.0 could silently corrupt such frames) :
  single-threaded mode drops the LZ4 state's history when an uncompressed segment starts
  (the decoder's history contains the uncompressed data) ; multi-threaded mode needs nothing,
  jobs reload the 64 KB actually preceding them. LZ4F_compressBegin() now starts frames in compressed mode :
  the mode of the previous frame's end was kept, which skipped the history drop (frame corrupted with a dictionary).
- Windows : worker stacks reserved instead of committed (STACK_SIZE_PARAM_IS_A_RESERVATION) ;
  #error when <windows.h> was already included without the Vista API (amalgamated sources) ;
  doc : no multi-threaded context use nor release under the loader lock (DllMain, DLL global objects).
lib/Makefile, programs/Makefile : threads detection skipped by goals building nothing threaded (clean, uninstall, ...),
run once per command (sub-makes inherit it) : make install probes once instead of 2-4 times, make clean never.
lz4 : explicit -T# also limited to the jobs of each file (-T1000 on a 10 MB file : 3 workers, RSS 298 -> 27 MB at -9),
and by the memory budget on all systems (-T99999 : 1024 liblz4 maximum, 331 threads within the 4 GB default budget) ;
an explicit nb of threads reduced triggers a warning (automatic : only with -v).
tests : frametest testUncompressedLinked (linked blocks, dictionary made of input, frames ending uncompressed),
mixed updates and small-dst scripts and fuzzer with linked blocks ; frameMT uncompressed segments with linked blocks ;
ossfuzz stream and uncompressed round-trip fuzzers with linked blocks ; test-lz4-multithread budget warnings.

2026-10-22 12:58:19 -07:00

8.0 KiB

Raw Permalink Blame History

View Raw