Yann Collet
c3cba9d858
fixed silent conversion warnings in GZDECOMPRESS path
2017-02-02 17:12:50 -08:00
Yann Collet
8d8513fb64
fixed C constant restrictions
2017-01-30 14:37:08 -08:00
Yann Collet
3672d06d06
zstdmt : section size is set to be a minimum of overlapSize
...
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
2017-01-30 13:35:45 -08:00
Yann Collet
6be2337c26
added command --block-size=
...
for Multi-threading only.
alias : -B#
2017-01-30 11:17:26 -08:00
Yann Collet
5d9b894e46
Fixed status display for zstdmt
...
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.
This patch disables the displaying of % compression
when multi-threading is enabled.
It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.
There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
Yann Collet
717c65d690
Merge pull request #519 from inikep/dev11
...
Dev11
2017-01-26 14:23:44 -08:00
Yann Collet
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
Yann Collet
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37
fixed zstdmt cli freeze issue with large nb of threads
...
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski
5022a18d51
improved #232 fix
2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski
64fa2dbc5e
Fixed https://github.com/facebook/zstd/issues/232
2017-01-25 13:02:33 +01:00
Yann Collet
512cbe8c10
zstdmt cli and API allow selection of section sizes
...
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.
It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()
The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
Yann Collet
500014af49
zstd cli can now compress using multi-threading
...
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads() (fileio)
2017-01-19 17:04:28 -08:00
Przemyslaw Skibinski
7a8a03c20d
util.h: restore BSD license for Facebook Open-Source
2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski
97a258d71d
updated comments
2016-12-21 14:00:41 +01:00
Przemyslaw Skibinski
e679741b18
_CRT_SECURE_NO_WARNINGS moved to util.h
2016-12-21 13:47:11 +01:00
Przemyslaw Skibinski
2f6ccee6af
platform.h: removed Compiler Options
2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski
a35b9448ac
improved MinGW support
2016-12-21 11:18:45 +01:00
Przemyslaw Skibinski
20b089e53d
simplified zstdcli.c
2016-12-21 09:19:15 +01:00
Przemyslaw Skibinski
16ae6563a2
executables use new util.h and platform.h
2016-12-21 09:06:14 +01:00
Przemyslaw Skibinski
f8046b8e72
Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112
...
# Conflicts:
# appveyor.yml
2016-12-19 08:20:26 +01:00
Przemyslaw Skibinski
b866e72826
tools use platform.h
2016-12-16 14:24:01 +01:00
Przemyslaw Skibinski
c71e552b2e
fixed "strategy" in advanced compression parameters
2016-12-13 20:04:32 +01:00
Przemyslaw Skibinski
8349d675e0
fileio.c: support advanced compression parameters
2016-12-13 13:24:59 +01:00
Przemyslaw Skibinski
7c6bbc3298
updated FIO_passThrough
2016-12-05 18:31:14 +01:00
Przemyslaw Skibinski
6b508b1770
updated test-gzstd
2016-12-05 18:02:40 +01:00
Przemyslaw Skibinski
b493e3b3d3
introduced srcBufferLoaded
2016-12-05 17:39:38 +01:00
Przemyslaw Skibinski
3c69760275
improved FIO_decompressGzFrame
2016-12-05 15:58:23 +01:00
Przemyslaw Skibinski
8489f184f6
improved detection of -lz
2016-12-05 13:47:00 +01:00
Yann Collet
743b33f57e
fix zstdcat
2016-12-02 15:18:57 -08:00
Yann Collet
5bd4237beb
minor refactor
2016-12-02 12:40:57 -08:00
Przemyslaw Skibinski
690753ea1a
improved formatting in FIO_decompressSrcFile
2016-12-02 16:20:16 +01:00
Przemyslaw Skibinski
4e49580407
removed testing artifacts
2016-12-02 15:19:00 +01:00
Przemyslaw Skibinski
c5eebca128
rewritten FIO_decompressGzFile
2016-12-02 15:01:31 +01:00
Przemyslaw Skibinski
b0f2ef2119
improved gzip_* functions
2016-12-02 13:50:29 +01:00
Przemyslaw Skibinski
4b504f131a
added gzip_open, gzip_close, gzip_read
2016-12-02 13:11:39 +01:00
Przemyslaw Skibinski
daaf754539
detect stream with ungetc
2016-12-01 13:29:19 +01:00
Przemyslaw Skibinski
19aad42ee1
added FIO_decompressGzFile
2016-12-01 11:56:31 +01:00
Przemyslaw Skibinski
abfb51f5f2
gzstd: decompresses .gz files
2016-11-30 15:05:54 +01:00
Przemyslaw Skibinski
0e14675df2
fileio.c: detect .gz files
2016-11-30 13:34:21 +01:00
Yann Collet
cdff19c4b3
minor comment change
2016-11-11 17:26:54 -08:00
Yann Collet
0018ca28dc
zstd cli : displays total decoded size, even when a stream consists of multiple frames
2016-11-07 14:42:37 -08:00
Przemyslaw Skibinski
1f60eb6b17
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-11-03 11:38:04 +01:00
Przemyslaw Skibinski
26306fcacf
BMK_SetNbIterations renamed to BMK_SetNbSeconds
2016-11-03 11:38:01 +01:00
Yann Collet
179b19776f
fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h
...
Added : ZSTD_isFrame() in experimental section
2016-11-02 17:30:49 -07:00
Przemyslaw Skibinski
fcf22e3473
set permissions, access and modification times
2016-11-02 14:08:07 +01:00
Przemyslaw Skibinski
a42794df61
preserve file modification time
2016-11-02 13:08:39 +01:00
Yann Collet
d4cda27b63
new command -M#, to limit memory usage during decompression ( #403 )
2016-10-14 13:32:20 -07:00
Nick Terrell
3e82f2b61a
Remove references to GPLv2 license
...
License headers added to Makefiles were taken from `zstd/Makefile`.
2016-10-11 15:28:40 -07:00
Yann Collet
993060e0f2
cli : better adaptation to small files
2016-09-21 16:46:08 +02:00
