The default memory allocation with pipes was too passive/static,
resulting in not allocating enough memory to enable threading.
By dynamically reallocating the buffer when reading from
unknown sized inputs we better use available memory and threads.
$ time seq 10000000 -1 0 | sort-old >/dev/null
real 0m16.523s
user 0m16.900s
sys 0m0.167s
$ time seq 10000000 -1 0 | sort-old -S1G >/dev/null
real 0m12.263s
user 0m29.646s
sys 0m0.527s
$ time seq 10000000 -1 0 | sort-new >/dev/null
real 0m12.994s
user 0m31.266s
sys 0m0.716s
It also avoids the overhead of writing to temp files
for modestly sized inputs. For example the following
input would induce interaction with temp storage:
$ seq 125000 | wc -c
763895
* src/sort.c (sort_buffer_size): Rename to ...
(sort_buffer_policy): ... here, and adjust to set
an initial size and limit, rather than just a size.
(fillbuf): Add a POLICY parameter, and use that
to call maybe_growbuf() as needed.
(maybe_growbuf): Return true if POLICY dictates we
should grow the buffer, and try_growbuf() was
able to reallocate the larger buffer.
* tests/sort/sort-buffer-size.sh: Add a new test.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
Related to https://bugs.gnu.org/10877
* tests/misc/io-errors.sh: Add a check to ensure we diagose
writing to closed stdout. Also ensure we exit with failure
in other cases.
* tests/misc/write-errors.sh: Likewise.
I noticed a single failure on cfarm29 (Linux 6.12 ppc64le),
where the output was missing:
-tail: directory containing watched file was removed
-tail: inotify cannot be used, reverting to polling
-tail: 'dir/file' has appeared; following new file
* tests/tail/inotify-dir-recreate.sh: Leverage the new tail --debug
output to sync to a point where we know the inotify watches are in
place. Also normalize the file argument quoting a bit.
* NEWS: Mention the bug fix.
* src/comm.c (usage): Remove mention that FILE1 and FILE2 cannot both be
standard input.
(compare_files): Only close standard input once.
* doc/coreutils.texi (comm invocation): Document the behavior of
'comm - -' which is not portable to all implementations.
* tests/comm/dash-dash.sh: New file.
* tests/misc/comm.pl: Move to tests/comm/comm.pl.
* tests/local.mk (all_tests): Add the new test. Rename the existing
test.
* tests/misc/user.sh: Modify the "unshare -U logname" test, so that it
does not fail on glibc versions (< 2.28, >= 2.38) that do a fallback
lookup based on the tty.
* tests/date/date-locale-hour.sh: Just strip the char before the last x,
assuming it's going to be the newline output by `locale`.
dash doesn't support $'' within ${}.
* tests/misc/io-errors.sh: Allow a generic error string on musl since
the first line is emitted immediately instead of being buffered as
expected.
Reported by Bruno Haible.
* tests/dd/no-allocate.sh: Run getlimits_ to ensure
we have $SSIZE_MAX etc. available. Also give some buffer
for the configured ulimit to leave space for returns_ to work.
* tests/cp/sparse-perf.sh: Old Centos 7 can give EINVAL
from SEEK_DATA on sparse files being copied from /dev/shm.
Avoid this failure as it's not practical to fix.
* tests/date/date-locale-hour.sh: Ensure `locale date_fmt`
is propagated exactly, even when it contains trailing new lines,
as was seen with Serbian locales on Centos 7.
As noted in https://bugs.gnu.org/9089
ksh gives intermittent ECONNRESET errors with closed pipes.
This can be seen reliably on Linux with:
ksh -c 'yes | (sleep .1; head -n10 >/dev/null)'
* tests/misc/io-errors.sh: Avoid part of test on ksh.
* tests/misc/write-errors.sh: Likewise.
As described in commit v9.10-283-g5cb0cca3b
avoid using ulimit with $SHELL -c, and explict
invocation of the shell like that can require lots of memory,
often due to loading the locale archive.
* tests/basenc/bounded-memory.sh: Avoid $SHELL -c with ulimit.
* tests/cut/bounded-memory.sh: Likewise.
* tests/expand/bounded-memory.sh Likewise.
* tests/pr/bounded-memory.sh Likewise.
* tests/unexpand/bounded-memory.sh Likewise.
* tests/fold/fold-zero-width.sh: Bring the ulimit bound
down to the standard 6M over the determined limit.
* tests/misc/tty-eof.pl: Fix a race that commit v9.10-269-gf312af49a
made more apparent, and resulted in intermittent failure like:
"... (with input) didn't produce expected output".
Now Instead of waiting for overlapping echo and output,
just wait for EOF, and if received, use exp->before()
to inspect the accumulated output.
Do not compare only with the latest entry for given device id but also
all previously saved entries with the same id.
* src/df.c (struct devlist): Add next_same_dev struct member.
(filter_mount_list): Iterate over next_same_dev to find duplicates.
* tests/df/skip-duplicates.sh: Add test cases.
* NEWS: Mention the improvement.
https://redhat.atlassian.net/browse/RHEL-5649
Verified with:
make -j4 PREFERABLY_POSIX_SHELL=/bin/ksh SUBDIRS=. check
* tests/fold/fold-zero-width.sh: Don't timeout $SHELL -c ...
as the ulimit induces a failure in the subshell depending
on the order of the allocations it does. The main issue is
disparity between the probed ulimit and that needed by $SHELL -c.
Such subshells load the often very large locale archive, thus
if there are any allocations done after the now too low ulimit is set,
then the $SHELL command fails. Note we timeout fold rather than
the whole pipeline so any 124 exit status is propagated.
* tests/misc/responsive.sh: Test commands that should output immediately
upon receiving input, and that there is no unecessary buffering.
* cfg.mk: Avoid false failure in sc_prohibit_test_backticks.
* tests/local.mk: Reference the new test.
* tests/tty/tty-eof.pl: Remove command specific logic,
and adjust commands to support general input.
Also add cut -b, as cut_bytes has its own read loop.
* tests/date/date.pl: Set the max supported year to INT_MAX.
Most systems support INT_MAX+1900, but mktime() on OpenBSD 7.8
limits the passed tm_year to INT_MAX.
Reported by Bruno Haible.
* tests/numfmt/numfmt.pl: Move recently added test that depends
on long double support to the appropriately guarded set.
Also reduce the value to be in the definitely safe long double range.
Reported by Bruno Haible.
* tests/cat/splice.sh: Ensure splice is called multiple times
before we check specific invocation counts.
On Linux kernel 5.10 for example, splice from /dev/zero
returns EINVAL.
* tests/ls/quoting-utf8.sh: New test verifying that
--quoting-style=locale and --quoting-style=clocale use Unicode
left/right single quotation marks in UTF-8 locales, and that
embedded apostrophes and double quotes are not escaped when the
delimiters are different characters.
Also check C locale fallback to ASCII quotes.
* tests/local.mk: Reference the new test.
https://github.com/coreutils/coreutils/pull/243
With field delimiter = line delimiter we need to know
if there is any more data to be read, as field delimiter
in the last byte of the file is treated differently.
So reiterate the loop to ensure enough read()s to make
the appropriate determination.