1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-29 11:21:22 +03:00
Commit Graph

2251 Commits

Author SHA1 Message Date
4fb0a77314 update -V documentation
to answer #3727 comment
2024-03-03 23:24:40 -08:00
e385c3dd46 Merge pull request #3753 from facebook/make2
minor Makefile refactoring
2024-03-03 19:13:00 -08:00
4d2bf7f0f2 removed sprintf usage from zstdcli.c
some static analyzers flag this standard C90 function as unsafe.
2024-02-24 23:03:40 -08:00
cc4530924b speed optimized version of targetCBlockSize
note that the size of individual compressed blocks will vary more wildly with this modification.
But it seems good enough for a first test, and fix the speed regression issue.
Further refinements can be attempted later.
2024-02-23 14:03:26 -08:00
68a232c591 benchmark more can test targetCBlockSize 2024-02-23 13:13:03 -08:00
621a263fb2 Merge pull request #3903 from gruenich/feature/reduce-scope-of-variables
Reduce scope of variables
2024-02-22 09:42:02 -08:00
7170f51dd2 fix include order 2024-02-20 23:36:04 -08:00
e62e15df19 fix clangbuild
notably -Wconversion and -Wdocumentation
2024-02-20 22:43:22 -08:00
588dfbcc97 fix c89 compatibility by removing snprintf()
note that this function has been in the code for a long while,
so why does it only start failing CI tests now ?
2024-02-20 19:24:50 -08:00
7a225c0c46 internal benchmark: can select size of generated synthetic sample 2024-02-20 15:47:09 -08:00
83598aa106 datagen generates lorem ipsum by default 2024-02-20 15:24:25 -08:00
7003c9905e increase word dictionary
for higher variety of messages.
Now, level 5 compresses better than level 4 (by a hair).
2024-02-20 13:27:36 -08:00
3dbd861b7d runtime weight distribution table
and made small words a bit more common.
2024-02-20 12:26:37 -08:00
5a1bb4a4e0 add question marks
and (slightly) longer sentences.
2024-02-20 00:37:21 -08:00
40874d4aea enriched vocabulary again
using real latin sentences from Cicero.

Compression ratio lower again, closer to "real" text,

now level 6 is way better than level 4.

level 5 is still lower than level 4,
but at least it's now higher than level 3.
2024-02-20 00:30:29 -08:00
1e046ce7fa increase vocabulary size
makes compression a bit less good,
hence a bit more comparable with real text (though still too easy to compress).
level 6 is now stronger than level 4, by a hair.
However, there is still a ratio dip at level 5.
2024-02-20 00:12:32 -08:00
b921f1aad6 Reduce scope of variables
This improves readability, keeps variables local, and
prevents the unintended use (e.g. typo) later on.
Found by Cppcheck (variableScope)
2024-02-11 22:00:03 +01:00
1f87c88ecf [Zstd] Less verbose log for patch mode. 2024-02-06 12:07:11 -08:00
d0b7da30e2 add a lorem ipsum generator
this generator replaces the statistical generator
for the general case when no statistic is requested.

Generated data features a compression level speed / ratio curve
which is more in line with expectation.
2024-01-29 15:00:32 -08:00
e6f4b46493 playTests.sh does no longer needs grep -E
it makes the test script more portable across posix systems
because `grep -E` is not guaranteed
while `grep` is fairly common.
2024-01-15 11:16:46 -08:00
8052cd0131 cli: better errors on arguent parsing 2023-12-18 13:59:33 +00:00
585aaa0ed3 Do not test WIN32, instead test _WIN32
To the best of my knowledge:
* `_WIN32` and `_WIN64` are defined by the compiler,
* `WIN32` and `WIN64` are defined by the user, to indicate whatever
  the user chooses them to indicate. They mean 32-bit and 64-bit Windows
  compilation by convention only.

See:
https://accu.org/journals/overload/24/132/wilson_2223/

Windows compilers in general, and MSVC in particular, have been defining
`_WIN32` and `_WIN64` for a long time, provably at least since Visual Studio
2015, and in practice as early as in the days of 16-bit Windows.

See:
https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=msvc-140
https://learn.microsoft.com/en-us/windows/win32/winprog64/the-tools

Tests used to be inconsistent, sometimes testing `_WIN32`, sometimes
`_WIN32` and `WIN32`. This brings consistency to Windows detection.
2023-09-23 19:03:18 +02:00
f4dbfce79c define LIB_SRCDIR and LIB_BINDIR 2023-09-12 13:46:03 -07:00
4edfaa93b7 default targets of lib/ and programs/ have different names
avoid risks on overlapping in case of include
2023-09-12 13:46:03 -07:00
839c7939e8 Added qnx in the posix test section of platform.h 2023-09-03 11:40:53 +02:00
78dbba76b8 Updated Makefiles for full MSYS2 and Cygwin installation and testing support.
They are Linux-like environments under Windows and have all the tools needed to support staged installation and testing.

Beware: this only affects the make build system.
2023-08-13 19:44:15 +02:00
4d267f3d4f fileio_asyncio: handle malloc fails in AIO_ReadPool_create 2023-07-21 12:17:03 +08:00
de6b46dfc8 Update fileio.c: fix build failure with enabled LTO
For some reasons when LTO is enabled, the compiler complains about statbuf variable not being correctly initialized, even though the variable has an assert != NULL just few lines below (FIO_getDictFileStat)

This is the fixed build failure:
x86_64-linux-gnu-gcc -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/libzstd-1.5.5+dfsg2-1 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -g -Werror -Wa,--noexecstack -Wdate-time -D_FORTIFY_SOURCE=2 -DXXH_NAMESPACE=ZSTD_ -DDEBUGLEVEL=1 -DZSTD_LEGACY_SUPPORT=5 -DZSTD_MULTITHREAD -DZSTD_GZCOMPRESS -DZSTD_GZDECOMPRESS -DZSTD_LZMACOMPRESS -DZSTD_LZMADECOMPRESS -DZSTD_LZ4COMPRESS -DZSTD_LZ4DECOMPRESS -DZSTD_LEGACY_SUPPORT=5  -c -MT obj/conf_086c46a51a716b674719b8acb8484eb8/zstdcli_trace.o -MMD -MP -MF obj/conf_086c46a51a716b674719b8acb8484eb8/zstdcli_trace.d -o obj/conf_086c46a51a716b674719b8acb8484eb8/zstdcli_trace.o zstdcli_trace.c
In function ‘UTIL_isRegularFileStat’,
    inlined from ‘UTIL_getFileSizeStat’ at util.c:524:10,
    inlined from ‘FIO_createDResources’ at fileio.c:2230:30:
util.c:209:12: error: ‘statbuf.st_mode’ may be used uninitialized [-Werror=maybe-uninitialized]
  209 |     return S_ISREG(statbuf->st_mode) != 0;
      |            ^
fileio.c: In function ‘FIO_createDResources’:
fileio.c:2223:12: note: ‘statbuf’ declared here
 2223 |     stat_t statbuf;
      |            ^
lto1: all warnings being treated as errors
2023-07-20 19:00:13 -04:00
0a794163f4 add makefile entry to build fat binary on macos 2023-04-13 18:43:06 -05:00
c28031df8f Add new line + [no-] to mmap-dict help output (#3601) 2023-04-06 13:01:58 -04:00
2e29728797 fix #3583
As reported by @georgmu,
the previous fix is undone by the later initialization.
Switch order, so that initialization is adjusted by special case.
2023-04-03 09:45:11 -07:00
9f58241dcc updated version number to v1.5.5
also : updated man pages
2023-03-31 23:02:08 -07:00
14d0cd5d69 do not add invocation of UTIL_isRegularFile() 2023-03-31 13:09:52 -07:00
5bf1359e3b fix decompression with -o writing into a block device
decompression features automatic support of sparse files,
aka a form of "compression" where entire blocks consists only of zeroes.
This only works for some compatible file systems (like ext4),
others simply ignore it (like afs).

Triggering this feature relies of `fseek()`.
But `fseek()` is not compatible with non-seekable devices, such as pipes.
Therefore it's disabled for pipes.

However, there are other objects which are not compatible with `fseek()`, such as block devices.

Changed the logic, so that `fseek()` (and therefore sparse write) is only automatically enabled on regular files.

Note that this automatic behavior can always be overridden by explicit commands `--sparse` and `--no-sparse`.

fix #3583
2023-03-31 11:29:16 -07:00
b2ad17a658 mmap for windows (#3557)
* mmap for windows

* remove enabling mmap for testing

* rename FIO dictionary initialization methods + un-const dictionary objects in free functions

* remove enabling mmap for testing

* initDict returns void, underlying setDictBuffer methods return the size of the set buffer

* fix comment
2023-03-28 19:44:53 -04:00
35c0c2075e minor doc update on --rsyncable
as requested by @devZer0.
fix #3567
2023-03-23 15:42:27 -06:00
283c228abe Merge pull request #3541 from felixhandte/fix-setvbuf-segfault
Avoid Segfault Caused by Calling `setvbuf()` on Null File Pointer
2023-03-09 13:54:11 -08:00
e769da1645 Merge pull request #3526 from facebook/bench_zstd_api
Simplify benchmark unit invocation API from CLI
2023-03-09 13:11:11 -08:00
e0fc9fd90b Merge pull request #3486 from daniellerozenblit/patch-from-low-memory-mode
Mmap large dictionaries in patch-from mode
2023-03-09 15:30:09 -05:00
c4c3e11958 Avoid Calling setvbuf() on Null File Pointer 2023-03-09 12:47:40 -05:00
70850eb72b assert to ensure that dict buffer type is valid 2023-03-08 16:54:57 -08:00
96e55c14f2 ability to disable mmap + struct to manage FIO dictionary 2023-03-08 08:06:10 -08:00
1e38e07b3d simplified BMK_benchFilesAdvanced() 2023-03-06 12:34:13 -08:00
9efc14804e minor: fixed zlib wrapper internal benchmark
another possibility could be to link it to programs/benchfn .
Not worth the effort.
2023-03-06 12:20:06 -08:00
db79219f70 simplify BMK_syntheticTest() 2023-03-06 12:15:22 -08:00
547794ef40 Fix typos found by codespell 2023-02-18 10:31:48 +01:00
1c42844668 Merge pull request #3479 from felixhandte/faster-file-ops
Use `f`-variants of `chmod()` and `chown()`
2023-02-16 13:07:34 -05:00
2d8afd9ce1 add manual flag to mmap dictionary 2023-02-14 09:42:23 -08:00
886de7bc04 Use correct types in LZMA comp/decomp (#3497)
Bytef and uInt are zlib types, not available when zlib is disabled

Fixes: 1598e6c634 ("Async write for decompression")
Fixes: cc0657f27d ("AsyncIO compression part 2 - added async read and asyncio to compression code (#3022)")
2023-02-13 16:30:56 -08:00
8a189b1b29 refactor dictionary file stat 2023-02-13 15:23:06 -08:00