Pick up mbrto{c32,wc} optimizations on UTF-8 on GLIBC.
Note configure.ac defines the required GNULIB_WCHAR_SINGLE_LOCALE.
This speeds up wc -m by 2.6x, when processing non ASCII chars,
and will similarly speed up per character processing
in the impending cut multi-byte implementation.
* NEWS: Mention the wc -m speed improvement.
This patch also fixes a bug where 'tac' would print a vague error on
some inputs:
$ seq 10000 | ./src/tac-prev > /dev/full
tac-prev: write error
$ seq 10000 | ./src/tac > /dev/full
tac: write error: No space left on device
In this case ferror (stdout) is true, but errno has been set back to
zero by a successful fclose (stdout) call.
* src/tac.c (output): Call write_error() if fwrite fails.
* tests/misc/io-errors.sh: Check that 'tac' prints a detailed write
error.
* NEWS: Mention the improvement.
* src/timeout.c (main): Save the process ID before creating a child
process. Check if the result of getppid is different than the saved
process ID instead of checking if it is 1.
* tests/timeout/init-parent.sh: New file.
* tests/local.mk (all_tests): Add the new test.
* NEWS: Mention the bug fix. Also mention that this change allows
'timeout' to work when reparented by a subreaper process instead of
init.
* src/dd.c (dd_copy): Increment the partial write count upon failure.
* tests/dd/partial-write.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/80583
* NEWS: It was ambiguous as to whether we quoted a range of
observered throughputs. Clarify this was the old and new
throughput on a single test system.
A good reference for the concepts used here is:
https://mazzo.li/posts/fast-pipes.html
We don't consider huge pages or busy loops here,
but use vmsplice(), and splice() to get significant speedups:
i7-5600U-laptop $ taskset 1 yes | taskset 2 pv > /dev/null
... [4.98GiB/s]
i7-5600U-laptop $ taskset 1 src/yes | taskset 2 pv > /dev/null
... [34.1GiB/s]
IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
... [11.6GiB/s]
IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
... [175GiB/s]
Also throughput to file (on BTRFS) was seen to increase significantly.
With a Fedora 43 laptop improving from 690MiB/s to 1.1GiB/s.
* bootstrap.conf: Ensure sys/uio.h is present.
This was an existing transitive dependency.
* m4/jm-macros.m4: Define HAVE_SPLICE appropriately.
We assume vmsplice() is available if splice() is as they
were introduced at the same time to Linux and glibc.
* src/yes.c (repeat_pattern): A new function to efficiently
duplicate a pattern in a buffer with memcpy calls that double in size.
This also makes the setup for the existing write() path more efficient.
(pipe_splice_size): A new function to increase the kernel pipe buffer
if possible, and use an appropriately sized buffer based on that (25%).
(splice_write): A new function to call vmplice() when outputting
to a pipe, and also splice() if outputting to a non-pipe.
* tests/misc/yes.sh: Verify the non-pipe output case,
(main): Adjust to always calling write on the minimal buffer first,
then trying vmsplice(), then falling back to write from bigger buffer.
and the vmsplice() fallback to write() case.
* NEWS: Mention the improvement.
* src/system.h (c32issep): A new function that is essentially
iswblank() on GLIBC platforms, and iswspace() with exceptions elsewhere.
* src/expand.c: Use it instead of c32isblank().
* src/fold.c: Likewise.
* src/join.c: Likewise.
* src/numfmt.c: Likewise.
* src/unexpand.c: Likewise.
* src/uniq.c: Likewise.
* NEWS: Mention the improvement.
* src/expand.c: Use mbbuf to support multi-byte input.
* src/unexpand.c: Likewise.
* tests/expand/mb.sh: New multi-byte test.
* tests/unexpand/mb.sh: Likewise.
* tests/local.mk: Reference new tests.
* NEWS: Mention the improvement.
* NEWS: Mention the improvement.
* src/install.c (enum copy_status): New type to let the caller know if
the copy was performed or skipped.
(copy_file): Return the new type instead of bool. Reduce variable scope.
(install_file_in_file): Only strip the file if the copy was
performed. Update the timestamps if the copy was skipped.
(main): Don't error when --compare and --preserve-timestamps are
combined.
* tests/install/install-C.sh: Add some test cases.
cksum --check is often the first interaction
users have with possibly untrusted downloads, so we should try
to be as defensive as possible when processing it.
Specifically we currently only escape \n characters in file names
presented in checksum files being parsed with cksum --check.
This gives some possibilty of dumping arbitrary data to the terminal
when checking downloads from an untrusted source.
This change gives these advantages:
1. Avoids dumping arbitrary data to vulnerable terminals
2. Avoids visual deception with ansi codes hiding checksum failures
3. More secure if users copy and paste file names from --check output
4. Simplifies programmatic parsing
Note this changes programmatic parsing, but given the original
format was so awkward to parse, I expect that's extremely rare.
I was not able to find example in the wild at least.
To parse the new format from from shell, you can do something like:
cksum -c checksums | while IFS= read -r line; do
case $line in
*': FAILED')
filename=$(eval "printf '%s' ${line%: FAILED}")
cp -v "$filename" /quarantine
;;
esac
done
This change also slightly reduces the size of the sum(1) utility.
This change also apples to md5sum, sha*sum, and b2sum.
* src/cksum.c (digest_check): Call quotef() instead of
cksum(1) specific quoting.
* tests/cksum/md5sum-bsd.sh: Adjust accordingly.
* doc/coreutils.texi (cksum general options): Describe the
shell quoting used for problematic file names.
* NEWS: Mention the change in behavior.
Reported by: Aaron Rainbolt
On signed char platforms, 0xFF was converted to -1
which matches MBBUF_EOF, causing fold to stop processing.
* NEWS: Mention the bug fix.
* gl/lib/mbbuf.h: Avoid sign extension on signed char platforms.
* tests/fold/fold-characters.sh: Adjust test case.
Reported at https://src.fedoraproject.org/rpms/coreutils/pull-request/20
* src/groups.c (main):
* src/id.c (main, print_stuff):
Don’t flush stdout before testing for write error.
Do the test only when in a loop, as a one-shot will
test for write error soon anyway.
* NEWS: Mention the fix.
* src/stat.c (main): Only check QUOTING_STYLE if there is a %N that is
not preceded by a percentage sign.
* tests/stat/stat-fmt.sh: Add some test cases.
* NEWS: Mention the improvement.
* src/id.c (print_stuff): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'id'.
* NEWS: Mention the improvement.
* src/groups.c (main): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'groups'.
This avoids the test harness being terminated like:
make[1]: *** [Makefile:24419: check-recursive] Hangup
make[3]: *** [Makefile:24668: check-TESTS] Hangup
make: *** [Makefile:24922: check] Hangup
make[2]: *** [Makefile:24920: check-am] Hangup
make[4]: *** [Makefile:24685: tests/misc/usage_vs_refs.log] Error 129
...
This happened sometimes when the tests were being run non interactively.
For example when run like:
setsid make TESTS="tests/timeout/timeout.sh \
tests/tail/overlay-headers.sh" SUBDIRS=. -j2 check
Note the race window can be made bigger by adding a sleep
after tail is stopped in overlay-headers.sh
The race can trigger the kernel to induce its job control
mechanism to prevent stuck processes.
I.e. where it sends SIGHUP + SIGCONT to a process group
when it determines that group may become orphaned,
and there are stopped processes in that group.
* tests/tail/overlay-headers.sh: Use setsid(1) to keep the stopped
tail process in a separate process group, thus avoiding any kernel
job control protection mechanism.
* tests/timeout/timeout.sh: Use setsid(1) to avoid the kernel
checking the main process group when sleep(1) is reparented.
Fixes https://bugs.gnu.org/80477
Here is an example of the performance improvement:
$ yes abcdefghijklmnopqrstuvwxyz | head -n 100000000 > input
$ time ./src/wc-prev -l < input
100000000
real 0m0.793s
user 0m0.630s
sys 0m0.162s
$ time ./src/wc -l < input
100000000
real 0m0.230s
user 0m0.065s
sys 0m0.164s
* NEWS: Mention the performance improvement.
* gnulib: Update to the latest commit.
* configure.ac: Check the the necessary intrinsics and functions.
* src/local.mk (noinst_LIBRARIES) [USE_NEON_WC_LINECOUNT]: Add
src/libwc_neon.a.
(src_libwc_neon_a_SOURCES, wc_neon_ldadd, src_libwc_neon_a_CFLAGS)
[USE_NEON_WC_LINECOUNT]: New variables.
(src_wc_LDADD) [USE_NEON_WC_LINECOUNT]: Add $(wc_neon_ldadd).
* src/wc.c [USE_NEON_WC_LINECOUNT]: Include sys/auxv.h and asm/hwcap.h.
(neon_supported) [USE_NEON_WC_LINECOUNT]: New function.
(wc_lines) [USE_NEON_WC_LINECOUNT]: Use neon_supported and
wc_lines_neon.
* src/wc.h (wc_lines_neon): Add declaration.
* src/wc_neon.c: New file.
* doc/coreutils.texi (Hardware Acceleration): Document the "-ASIMD"
hwcap and the variable used in ./configure to override detection of Neon
instructions.
* tests/wc/wc-cpu.sh: Also add "-ASIMD" to disable the use of Neon
instructions.
* NEWS: Mention the improvement.
* src/nl.c: Include mcel.h.
(DEFAULT_SECTION_DELIMITERS): Resize to fit 2 multi-byte characters.
(section_del_len): New variable.
(check_section): Compare against section_del_len instead of 2.
(main): Support multi-byte characters for the -d option.
* tests/nl/multibyte.sh: New file.
* tests/nl/nl.sh: New file, moved from tests/misc/nl.sh.
* tests/local.mk (all_tests): Add the new test. Adjust the existing
tests file name.
* cfg.mk (exclude_file_name_regexp--sc_space_tab): Adjust Adjust the
existing tests file name.
With dash as /bin/sh, you get the error
```
checking for sys/capability.h... yes
./configure: 95775: test: no: unexpected operator
checking for working fork... yes
```
* configure.ac: Use '=' in test for equality, not '==',
for POSIX shell compatibility.
* NEWS: Mention the build fix.
tail(1) could fail with an "Interrupted system call"
diagnostic, on some systems like Centos 5 (Linux 2.6.18).
This was seen with tests/tail/overlay-headers.sh
which sends SIGCONT, which should not induce a failure.
* src/tail.c (tail_forever_inotify): Retry the poll()
upon receiving a non terminating signal, and the syscall
is not automatically restarted by the system.
* NEWS: Mention the bug fix.
Reported by Bruno Haible.
This is a more standard mechanism to disable markup.
* src/system.h (oputs_): Logic change to honor TERM=dumb,
rather than HELP_NO_MARKUP=something.
* doc/coreutils.texi: Adjust the description for --help.
* man/local.mk: Ensure TERM is set to something,
so that man pages have links included.
* man/viewman: Just honor users $TERM.
* tests/misc/getopt_vs_usage.sh: Remove env var complication,
as TERM is unset automatically.
* tests/misc/usage_vs_refs.sh: Likewise.
* NEWS: Adjust the change in behavior note.
* src/ls.c (quote_name): Use ST (ESC \) rather than BEL,
as that's the only terminator mentioned in at least
ECMA-48, DEC STD 070, and EK-VT520-RM.
* NEWS: Mention the change in behavior.
* tests/ls/hyperlink.sh: Adjust accordingly.
Suggested by Egmont Koblinger.
Setup
$ ln -nsf src/coreutils foo
Before
$ ./foo; echo $?
foo: unknown program ‘foo’
Try './foo --help' for more information.
1
./foo --version; echo $?
coreutils (GNU coreutils) 9.9.172-01993
0
After
$ ./foo; echo $?
coreutils: unknown program 'foo'
1
$ ./foo --version; echo $?
coreutils: unknown program 'foo'
1
* src/coreutils.c (main): Don't process options if
we don't know they're intended for the multi-call binary.
Otherwise `foo --version` would return true, even though
foo was symlinked to the multi-call binary, but not supported.
* tests/misc/coreutils.sh: Add test cases.
* NEWS: Mention the change in behavior.
* src/paste.c (collapse_escapes): This is the central --delimiters
parsing function, so adjust to handle multi-byte chars with
mcel_scanz(). Populate a delimiters length array to support
characters of differing lengths.
(paste_serial): Use the delimiters length array to output
the appropriate delimiter.
(paste_parallel): Likewise.
* tests/paste/multi-byte.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
* gnulib: Update to latest mainly to pull in selinux module fixes.
* NEWS: Mention the build fix to honor --with-selinux building
stub chcon and runcon binaries on systems without libselinux.
* src/cksum.c (output_file, digest_check): Check if standard output has
it's error flag set after printing.
* tests/misc/write-errors.sh: Add a test case that would previously run
forever.
* NEWS: Mention the improvement. Reorder alphabetically.
The 'readlink' and 'realpath' programs have an uncommon case where they
can run for a very long time. When canonicalizing file names longer than
PATH_MAX, we have to call 'openat' for each directory up the tree until
we reach root which takes a long time. Here is an example of the current
behavior:
$ mkdir -p $(yes a/ | head -n $((32 * 1024)) | tr -d '\n')
$ while cd $(yes a/ | head -n 1024 | tr -d '\n'); do :; \
done 2>/dev/null
$ pwd | tr '/' '\n' | wc -l
32771
$ env time --format=%E readlink -f $(yes . | head -n 5) > /dev/full
readlink: write error: No space left on device
Command exited with non-zero status 1
0:59.72
$ env time --format=%E realpath $(yes . | head -n 5) > /dev/full
realpath: write error: No space left on device
Command exited with non-zero status 1
1:00.32
It is better to exit as soon as there is an error writing to standard
output:
$ env time --format=%E readlink -f $(yes . | head -n 5) > /dev/full
readlink: write error: No space left on device
Command exited with non-zero status 1
0:11.88
$ env time --format=%E realpath $(yes . | head -n 5) > /dev/full
realpath: write error: No space left on device
Command exited with non-zero status 1
0:12.04
* src/readlink.c (main): Check if standard output has it's error flag
set after printing a file name.
* src/realpath.c (process_path): Likewise.
* NEWS: Mention the improvement.
We disallow `cksum --tag --check` which is fine,
but the error should be consistent with md5sum,
and less confusing, as it currently mentions
"--binary" and "--text" which weren't specified.
We disallow `cksum --tag --text` which is fine,
but we should also disallow `cksum --text --tag`.
We should honor an explicit --binary (output *)
with this combination of options:
cksum --binary --tag --untagged -a md5 /dev/null
Note this also makes both of `cksum -a md5` and
`cksum --tag -a md5` consistently use binary mode
when reading from a tty on systems like MinGW
where O_BINARY is set.
* src/cksum.c (main): Adjust --text,--binary
and --tag,--untagged option processing.
* tests/cksum/cksum-a.sh: Add test cases.
* tests/cksum/cksum-c.sh: Likewise.
* NEWS: Mention the improvement.
Fixes https://github.com/coreutils/coreutils/issues/163
This aligns with `fold -w` and BSD `fmt` implementations.
* src/fmt.c (fmt_paragraph): Check len <= max_width.
* tests/fmt/width.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the change in behavior.
Addresses part of https://bugs.gnu.org/79497
Align the -t implementation with the Heirloom project.
* src/ptx.c (usage): Describe -t, and also mention
the default width is 72 when not used.
* doc/coreutils.texi (ptx invocation): Likewise.
(main): Override the default width if -t is specified.
* tests/ptx/ptx.pl: Add test cases.
* NEWS: Mention the change in behavior.
Commit v9.8-95-g4c0cf3864 intended to initialize
ext_start to src_pos, as was described at:
https://lists.gnu.org/r/coreutils/2025-11/msg00035.html
However ipos was inadvertently used, which is only
valid the first time through the loop.
* src/copy-file-data.c (lseek_copy): Use scan_inference->hole_start
only with the initial offset passed to lseek_copy().
* NEWS: Mention the bug fix.
Reported at https://github.com/coreutils/coreutils/issues/159
* src/dd.c (main): Reduce the scope of exit_status. Exit immediately if
ftruncate fails.
* tests/dd/fail-ftruncate-fstat.sh: New test.
* tests/local.mk (all_tests): Add the new test.
* NEWS: Mention the bug fix.