diff --git a/programs/Makefile b/programs/Makefile index d738f40c2..dd6e7838a 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -24,9 +24,9 @@ # zstd : Command Line Utility, supporting gzip-like arguments # zstd32 : Same as zstd, but forced to compile in 32-bits mode # zstd_nolegacy : zstd without support of decompression of legacy versions -# zstd-small: minimal zstd without dictBuilder and bench -# zstd-compress: compressor-only version of zstd -# zstd-decompress: decompressor-only version of zstd +# zstd-small : minimal zstd without dictionary builder and benchmark +# zstd-compress : compressor-only version of zstd +# zstd-decompress : decompressor-only version of zstd # ########################################################################## DESTDIR?= diff --git a/programs/README.md b/programs/README.md new file mode 100644 index 000000000..f38e6c49b --- /dev/null +++ b/programs/README.md @@ -0,0 +1,93 @@ +zstd - Command Line Interface +================================ + +Command Line Interface (CLI) can be created using the `make` command without any additional parameters. +There are however other Makefile targets that create different variations of CLI: +- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd versions +- `zstd32` : Same as `zstd`, but forced to compile in 32-bits mode +- `zstd_nolegacy` : Same as `zstd` except of support for decompression of legacy zstd versions +- `zstd-small` : CLI optimized for minimal size; without dictionary builder, benchmark, and support for decompression of legacy zstd versions +- `zstd-compress` : compressor-only version of CLI; without dictionary builder, benchmark, and support for decompression of legacy zstd versions +- `zstd-decompress` : decompressor-only version of CLI; without dictionary builder, benchmark, and support for decompression of legacy zstd versions + + +#### Aggregation of parameters +CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`. + + +#### Dictionary builder in Command Line Interface +Zstd offers a training mode, which can be used to tune the algorithm for a selected +type of data, by providing it with a few samples. The result of the training is stored +in a file selected with the `-o` option (default name is `dictionary`), +which can be loaded before compression and decompression. + +Using a dictionary, the compression ratio achievable on small data improves dramatically. +These compression gains are achieved while simultaneously providing faster compression and decompression speeds. +Dictionary work if there is some correlation in a family of small data (there is no universal dictionary). +Hence, deploying one dictionary per type of data will provide the greater benefits. +Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm +will rely more and more on previously decoded content to compress the rest of the file. + +Usage of the dictionary builder and created dictionaries with CLI: +1) Create the dictionary : `zstd --train FullPathToTrainingSet/* -o dictionaryName` +2) Compress with dictionary: `zstd FILE -D dictionaryName` +3) Decompress with dictionary: `zstd --decompress FILE.zst -D dictionaryName` + + + +#### Benchmark in Command Line Interface +CLI includes in-memory compression benchmark module for zstd. +The benchmark is conducted using given filenames which are read into memory and joined together. +It makes benchmark more precise as it eliminates I/O overhead. +Many filenames can be supplied as multiple parameters, parameters with wildcards or +names of directories can be used as parameters with the `-r` option. + +The benchmark measures ratio, compressed size, compression and decompression speed. +One can select compression levels starting from `-b` and ending with `-e`. +The `-i` parameter selects minimal time used for each of tested levels. + + + +#### Usage of Command Line Interface +The full list of options can be obtained with `-h` or `-H` parameter: +``` +Usage : + zstd [args] [FILE(s)] [-o file] + +FILE : a filename + with no FILE, or when FILE is - , read standard input +Arguments : + -# : # compression level (1-19, default:3) + -d : decompression + -D file: use `file` as Dictionary + -o file: result stored into `file` (only if 1 input file) + -f : overwrite output without prompting +--rm : remove source file(s) after successful de/compression + -k : preserve source file(s) (default) + -h/-H : display help/long help and exit + +Advanced arguments : + -V : display Version number and exit + -v : verbose mode; specify multiple times to increase log level (default:2) + -q : suppress warnings; specify twice to suppress errors too + -c : force write to standard output, even if it is the console + -r : operate recursively on directories +--ultra : enable levels beyond 19, up to 22 (requires more memory) +--no-dictID : don't write dictID into header (dictionary compression) +--[no-]check : integrity check (default:enabled) +--test : test compressed file integrity +--[no-]sparse : sparse mode (default:enabled on file, disabled on stdout) + +Dictionary builder : +--train ## : create a dictionary from a training set of files + -o file : `file` is dictionary name (default: dictionary) +--maxdict ## : limit dictionary to specified size (default : 112640) + -s# : dictionary selectivity level (default: 9) +--dictID ## : force dictionary ID to specified value (default: random) + +Benchmark arguments : + -b# : benchmark file(s), using # compression level (default : 1) + -e# : test all compression levels from -bX to # (default: 1) + -i# : minimum evaluation time in seconds (default : 3s) + -B# : cut file into independent blocks of size # (default: no block) + ``` \ No newline at end of file diff --git a/projects/README.md b/projects/README.md index d724cdcba..6353623ba 100644 --- a/projects/README.md +++ b/projects/README.md @@ -4,11 +4,11 @@ projects for various integrated development environments (IDE) #### Included projects The following projects are included with the zstd distribution: -- cmake - CMake project contributed by Artyom Dymchenko -- VS2005 - Visual Studio 2005 project -- VS2008 - Visual Studio 2008 project -- VS2010 - Visual Studio 2010 project (which also works well with Visual Studio 2012, 2013, 2015) -- build - command line scripts prepared for Visual Studio compilation without IDE +- `cmake` - CMake project contributed by Artyom Dymchenko +- `VS2005` - Visual Studio 2005 project +- `VS2008` - Visual Studio 2008 project +- `VS2010` - Visual Studio 2010 project (which also works well with Visual Studio 2012, 2013, 2015) +- `build` - command line scripts prepared for Visual Studio compilation without IDE #### How to compile zstd with Visual Studio diff --git a/tests/README.md b/tests/README.md index 309638f91..72be1a9b3 100644 --- a/tests/README.md +++ b/tests/README.md @@ -2,24 +2,24 @@ programs and scripts for automated testing of zstd ================================ This directory contains the following programs and scripts: -- datagen : Synthetic and parametrable data generator, for tests -- fullbench : Precisely measure speed for each zstd inner functions -- fuzzer : Test tool, to check zstd integrity on target platform -- paramgrill : parameter tester for zstd -- test-zstd-speed.py : script for testing zstd speed difference between commits -- test-zstd-versions.py : compatibility test between zstd versions stored on Github (v0.1+) -- zbufftest : Test tool, to check ZBUFF integrity on target platform -- zstreamtest : Fuzzer test tool for zstd streaming API +- `datagen` : Synthetic and parametrable data generator, for tests +- `fullbench` : Precisely measure speed for each zstd inner functions +- `fuzzer` : Test tool, to check zstd integrity on target platform +- `paramgrill` : parameter tester for zstd +- `test-zstd-speed.py` : script for testing zstd speed difference between commits +- `test-zstd-versions.py` : compatibility test between zstd versions stored on Github (v0.1+) +- `zbufftest` : Test tool to check ZBUFF integrity on target platform +- `zstreamtest` : Fuzzer test tool for zstd streaming API -#### test-zstd-versions.py - script for testing zstd interoperability between versions +#### `test-zstd-versions.py` - script for testing zstd interoperability between versions This script creates `versionsTest` directory to which zstd repository is cloned. Then all taged (released) versions of zstd are compiled. In the following step interoperability between zstd versions is checked. -#### test-zstd-speed.py - script for testing zstd speed difference between commits +#### `test-zstd-speed.py` - script for testing zstd speed difference between commits This script creates `speedTest` directory to which zstd repository is cloned. Then it compiles all branches of zstd and performs a speed benchmark for a given list of files (the `testFileNames` parameter).