lz4frame :
- LZ4F_uncompressedUpdate() accepts linked blocks (dev and v1.10.0 could silently corrupt such frames) :
single-threaded mode drops the LZ4 state's history when an uncompressed segment starts
(the decoder's history contains the uncompressed data) ; multi-threaded mode needs nothing,
jobs reload the 64 KB actually preceding them. LZ4F_compressBegin() now starts frames in compressed mode :
the mode of the previous frame's end was kept, which skipped the history drop (frame corrupted with a dictionary).
- Windows : worker stacks reserved instead of committed (STACK_SIZE_PARAM_IS_A_RESERVATION) ;
#error when <windows.h> was already included without the Vista API (amalgamated sources) ;
doc : no multi-threaded context use nor release under the loader lock (DllMain, DLL global objects).
lib/Makefile, programs/Makefile : threads detection skipped by goals building nothing threaded (clean, uninstall, ...),
run once per command (sub-makes inherit it) : make install probes once instead of 2-4 times, make clean never.
lz4 : explicit -T# also limited to the jobs of each file (-T1000 on a 10 MB file : 3 workers, RSS 298 -> 27 MB at -9),
and by the memory budget on all systems (-T99999 : 1024 liblz4 maximum, 331 threads within the 4 GB default budget) ;
an explicit nb of threads reduced triggers a warning (automatic : only with -v).
tests : frametest testUncompressedLinked (linked blocks, dictionary made of input, frames ending uncompressed),
mixed updates and small-dst scripts and fuzzer with linked blocks ; frameMT uncompressed segments with linked blocks ;
ossfuzz stream and uncompressed round-trip fuzzers with linked blocks ; test-lz4-multithread budget warnings.
Prototype, not intended for a PR as is.
programs/Makefile probed pthread with a fixed have_pthread.c in its directory
(pre-existing in dev) : concurrent invocations collided (24 / 192 concurrent probes
wrongly reported no threads, producing an lz4 CLI without multi-threading),
and read-only source trees made it fail.
- build/make/lz4defs.make : PTHREAD_PROBE, evaluated with the CC and ALLFLAGS of
the invoking Makefile, within a private temporary directory (mktemp -d).
- lib/Makefile and programs/Makefile employ it (lib/'s copy from previous commit removed).
Validated : probes of lib/ and programs/ (192 concurrent, read-only trees), full tests
MT / no-MT, ASAN+UBSAN, TSAN, make test, wine, CLI build (lz4 multithread, lz4-nomt not),
build policies, install scenarios + test_install.sh.
Prototype, not intended for a PR as is.
- persistent thread pool within LZ4F_cctx (pthread / Win32), replaces the per-call fork-join;
nbWorkers is now the nb of worker threads (the calling thread also runs jobs while waiting).
- streaming : LZ4F_compressUpdate() / LZ4F_flush() / LZ4F_compressEnd() cut input into jobs
(4 MB linked, 1 MB independent), at fixed positions of the content; outputs emitted in order;
input read in place with stableSrc or autoFlush, copied otherwise; autoFlush respected;
LZ4F_compressBound() covers jobs in flight; LZ4F_uncompressedUpdate() drains pending jobs.
- one-shot compression goes through the same engine.
- CLI : frame compression goes through LZ4F streaming with nbWorkers (output identical to
previous CLI, and lz4 == lz4-nomt); custom block sizes fall back to single-thread.
- build systems compile the CLI's liblz4 sources with LZ4F_MULTITHREAD.
- tests/frameMT.c : streaming validation (slicing, stableSrc, flushes, autoFlush,
uncompressed segments, exact bounds, abandoned frames).
also:
- target `all` also produces symlinks `unlz4` and `lz4cat`
- user-provided `CFLAGS` are appended, so have higher priority
- `-O3` is no longer erased when `CFLAGS` is provided via environment variables
- No more usage of non-standard `MOREFLAGS`
- Better centralized `clean` logic
+ as a consequence of noticing that Windows' Makefile doesn't necessarily $(EXT) suffix to properly delete executable
Moved a few other tests to Makefiles.inc. Other things might need to go there.
Made a test for symlink appropriateness. Windows can NOT handle them the same way Unix-like operating systems do (if at all). This is mostly the same as the Visual C projects.
embed version info into .dll and .exes that are redistributed.
make it possible to generate LZ4-compressed block
with a controlled maximum offset (necessarily <= 65535).
This could be useful for compatibility with decoders
using a very limited memory budget (<64 KB).
Answer #154
* Uninstall didn't remove the pkg-config correctly.
* Fix `mandir`
* Allow overriding either upper- or lower-case location variables, but
always use the lower case variables.
* Add test case that ensures overriding both upper- and lower-case
variables is the same, and that the directory is empty after uninstall.
instead of selected at compilation time depending on a macro.
This design makes it possible to have a single binary which supports both modes.
An advantageous side effect is that when doing `make; make install`
no additional binary is created during `make install`
(it used to create `lz4c`, because `make` would only build `lz4`)
