lib/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-07-30 22:23:13 +03:00
Code Activity

advanced decompression function replaces by normal streaming one

Browse Source 
advanced parameters compatible with ZSTD_decompressStream().
This commit is contained in:
Yann Collet
2018-12-04 10:28:36 -08:00
parent 6ced8f7c7c
commit 34e146f548
7 changed files with 107 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
doc/zstd_manual.html
View File

@ -248,13 +248,17 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx);
A ZSTD_CStream object is required to track streaming operation.
Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources.
ZSTD_CStream objects can be reused multiple times on consecutive compression operations.
It is recommended to re-use ZSTD_CStream in situations where many streaming operations will be achieved consecutively,
since it will play nicer with system's memory, by re-using already allocated memory.
Use one separate ZSTD_CStream per thread for parallel execution.
It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory.
Start a new compression by initializing ZSTD_CStream context.
Use ZSTD_initCStream() to start a new compression operation.
Use variants ZSTD_initCStream_usingDict() or ZSTD_initCStream_usingCDict() for streaming with dictionary (experimental section)
For parallel execution, use one separate ZSTD_CStream per thread.
note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing.
Parameters are sticky : when starting a new compression on the same context,
it will re-use the same sticky parameters as previous compression session.
It's recommended to initialize the context before every usage.
Use ZSTD_initCStream() to set the parameter to a selected compression level.
Use advanced API (ZSTD_CCtx_setParameter(), etc.) to set more detailed parameters.
Use ZSTD_compressStream() as many times as necessary to consume input stream.
The function will automatically update both `pos` fields within `input` and `output`.
@ -265,10 +269,10 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx);
If not, the caller must make some room to receive more compressed data,
typically by emptying output buffer, or allocating a new output buffer,
and then present again remaining input data.
@return : a size hint, preferred nb of bytes to use as input for next function call
or an error code, which can be tested using ZSTD_isError().
Note 1 : it's just a hint, to help latency a little, any other value will work fine.
Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize()
@return : a size hint, preferred nb of bytes to use as input for next function call
or an error code, which can be tested using ZSTD_isError().
Note 1 : it's just a hint, to help latency a little, any value will work fine.
Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize()
At any moment, it's possible to flush whatever data might remain stuck within internal buffer,
using ZSTD_flushStream(). `output->pos` will be updated.
@ -390,7 +394,9 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
ZSTD_btlazy2=6,
ZSTD_btopt=7,
ZSTD_btultra=8
</b>/* note : new strategies might be added in the future */<b>
</b>/* note : new strategies might be added in the future<b>
at this stage, only the order (from fast to strong) is guaranteed.
new strategy names may be introduced, pushing the maximum number upward */
} ZSTD_strategy;
</b></pre><BR>
<pre><b>typedef enum {
@ -512,7 +518,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
* ZSTD_p_forceMaxWindow
* ZSTD_p_forceAttachDict
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
* note : never use experimentalParam names directly
* note : never ever use experimentalParam? names directly
*/
ZSTD_p_experimentalParam1=500,
ZSTD_p_experimentalParam2=10,
@ -636,7 +642,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
const void* src, size_t srcSize);
</b><p> Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API.
- Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
- The function is always blocking.
- The function is always blocking, returns when compression is completed.
Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
@return : compressed size written into `dst` (<= `dstCapacity),
or an error code if it fails (which can be tested using ZSTD_isError()).
@ -648,24 +654,25 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
ZSTD_e_flush=1, </b>/* flush any data provided so far,<b>
* it creates (at least) one new block, that can be decoded immediately on reception;
* frame will continue: any future data can still reference previously compressed data, improving compression. */
ZSTD_e_end=2 </b>/* flush any remaining data and close current frame.<b>
* any additional data starts a new frame.
* each frame is independent (does not reference any content from previous frame). */
ZSTD_e_end=2 </b>/* flush any remaining data _and_ close current frame.<b>
* note that frame is only closed after compressed data is fully flushed (return value == 0).
* After that point, any additional data starts a new frame.
* note : each frame is independent (does not reference any content from previous frame). */
} ZSTD_EndDirective;
</b></pre><BR>
<pre><b>size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
ZSTD_outBuffer* output,
ZSTD_inBuffer* input,
ZSTD_EndDirective endOp);
</b><p> Behave about the same as ZSTD_compressStream, with additional control on end directive.
</b><p> Behaves about the same as ZSTD_compressStream, with additional control on end directive.
- Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
- Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode)
- outpot->pos must be <= dstCapacity, input->pos must be <= srcSize
- outpot->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
- In single-thread mode (default), function is blocking : it completes its job before returning to caller.
- In multi-thread mode, function is non-blocking : it just acquires a copy of input, and distribute job to internal worker threads,
and then immediately returns, just indicating that there is some data remaining to be flushed.
The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
- When nbWorkers==0 (default), function is blocking : it completes its job before returning to caller.
- When nbWorkers>=1, function is non-blocking : it just acquires a copy of input, and distributes jobs to internal worker threads,
and then immediately returns, just indicating that there is some data remaining to be flushed.
The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
- Exception : if the first call requests a ZSTD_e_end directive, the function delegates to ZSTD_compress2() which is always blocking.
- @return provides a minimum amount of data remaining to be flushed from internal buffers
or an error code, which can be tested using ZSTD_isError().
@ -679,12 +686,40 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
</b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
This protects a decoder context from reserving too much memory for itself (potential attack scenario).
This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
@return : 0, or an error code (which can be tested using ZSTD_isError()).
<pre><b>typedef enum {
ZSTD_d_windowLogMax=100, </b>/* Used to select a limit beyond which<b>
* the streaming API will refuse to allocate memory buffer
* in order to protect the host from unreasonable memory requirements.
* This parameter is only useful in streaming mode, as no internal buffer is allocated in single-pass mode.
* By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) */
</b>/* note : additional experimental parameters are also available<b>
* within the experimental section of the API.
* At the time of this writing, they include :
* ZSTD_p_format
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
* note : never ever use experimentalParam? names directly
*/
ZSTD_d_experimentalParam1=1000
} ZSTD_dParameter;
</b></pre><BR>
<pre><b>ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam); </b>/* not implemented yet */<b>
</b><p> All parameters must belong to an interval with lower and upper bounds,
otherwise they will either trigger an error or be automatically clamped.
@return : a structure, ZSTD_bounds, which contains
- an error status field, which must be tested using ZSTD_isError()
- both lower and upper bounds, inclusive
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int value); </b>/* not implemented yet */<b>
</b><p> Set one compression parameter, selected by enum ZSTD_dParameter.
All parameters have valid bounds. Bounds can be queried using ZSTD_dParam_getBounds().
Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
Setting a parameter is only possible during frame initialization (before starting decompression).
@return : an error code (which can be tested using ZSTD_isError()).
</p></pre><BR>
@ -711,7 +746,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
@result : 0, or an error code (which can be tested with ZSTD_isError()).
Note 1 : Currently, only one dictionary can be managed.
Referencing a new dictionary effectively "discards" any previous one.
Special : adding a NULL DDict means "return to no-dictionary mode".
Special: referencing a NULL DDict means "return to no-dictionary mode".
Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.
</p></pre><BR>
@ -722,16 +757,16 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
This is the reverse operation of ZSTD_CCtx_refPrefix(),
and must use the same prefix as the one used during compression.
Prefix is **only used once**. Reference is discarded at end of frame.
End of frame is reached when ZSTD_DCtx_decompress_generic() returns 0.
End of frame is reached when ZSTD_decompressStream() returns 0.
@result : 0, or an error code (which can be tested with ZSTD_isError()).
Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary
Note 2 : Prefix buffer is referenced. It **must** outlive decompression job.
Prefix buffer must remain unmodified up to the end of frame,
reached when ZSTD_DCtx_decompress_generic() returns 0.
reached when ZSTD_decompressStream() returns 0.
Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent).
Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode.
Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section)
Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
A fulldict prefix requires building tables, hence is more costly.
A full dictionary is more costly, as it requires building tables.
</p></pre><BR>
@ -743,17 +778,6 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
</p></pre><BR>
<pre><b>size_t ZSTD_decompress_generic(ZSTD_DCtx* dctx,
ZSTD_outBuffer* output,
ZSTD_inBuffer* input);
</b><p> Behave the same as ZSTD_decompressStream.
Decompression parameters cannot be changed once decompression is started.
@return : an error code, which can be tested using ZSTD_isError()
if >0, a hint, nb of expected input bytes for next invocation.
`0` means : a frame has just been fully decoded and flushed.
</p></pre><BR>
<a name="Chapter14"></a><h2>experimental API (static linking only)</h2><pre>
The following symbols and constants
are not planned to join "stable API" status in the near future.
@ -1152,6 +1176,15 @@ size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params);
how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?)
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
</b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
This protects a decoder context from reserving too much memory for itself (potential attack scenario).
This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
@return : 0, or an error code (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
</b><p> Instruct the decoder context about what kind of data to decode next.
This instruction is mandatory to decode data without a fully-formed header,
@ -1159,11 +1192,11 @@ size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params);
@return : 0, or an error code (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>size_t ZSTD_decompress_generic_simpleArgs (
<pre><b>size_t ZSTD_decompressStream_simpleArgs (
ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos);
</b><p> Same as ZSTD_decompress_generic(),
</b><p> Same as ZSTD_decompressStream(),
but using only integral types as arguments.
This can be helpful for binders from dynamic languages
which have troubles handling structures containing memory pointers.