Assaf Gordon
20d7bce0f7
shuf: use reservoir-sampling for large or unknown sized inputs
...
Reservoir sampling optimizes selecting K random lines from large or
unknown-sized input: http://en.wikipedia.org/wiki/Reservoir_sampling
Note this also avoids reading any input when -n0 is specified.
* src/shuf.c (main): Use reservoir-sampling when the number of output
lines is known, and the input size is large or unknown.
(input_size): A new function to get the input size for regular files.
(read_input_reservoir_sampling): New function to read lines from input,
keeping only K lines in memory, replacing lines with decreasing prob.
(write_permuted_output_reservoir): New function to output reservoir.
* tests/misc/shuf-reservoir.sh: An expensive_ test using valgrind to
exercise the reservoir-sampling code.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
2013-03-25 20:07:14 +00:00
..
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-02-04 13:55:01 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-02-28 11:03:01 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-02-28 01:49:56 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-02-10 20:30:53 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-26 02:37:13 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-03-25 20:07:14 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-02-05 16:05:54 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-26 02:38:52 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-02-28 23:35:45 +00:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00
2013-01-01 04:51:20 +01:00