Commit Graph
5435 Commits
Author SHA1 Message Date
Pádraig Brady ea6a7ba547 cut: optimize when no delimiter in input
This is about 20x faster.
Note we only do the delimiter search once per chunk,
and it's usually quick as delimiters wouldn't be too far
into the a chunk if present, so we don't bother
to cache the found delimiter.
2026-04-05 13:15:56 +01:00
Pádraig Brady 7d017f83bc tests: cut: ensure multi-byte delimiter is rejected in uni-byte locales
tests/cut/cut.pl: Check the appropriate diagnostic is presented.
2026-04-05 13:15:56 +01:00
Pádraig Brady f6b3055f74 cut: merge cut_fields and cut_fields_bytesearch
TODO: See why this is much slower:
time LC_ALL=C.UTF-8 src/cut -f1 -dc as.in > /dev/null
2026-04-05 13:15:56 +01:00
Pádraig Brady 801686242e cut: faster utf8 processing
TODO: improve to use bounded memory where possible
2026-04-05 13:15:56 +01:00
Pádraig Brady a14ac29629 cut: support -F as an alias for -f -w -O ' '
To improve compatibility with toybox/busybox scripts.
2026-04-05 13:15:56 +01:00
Pádraig Brady eb1f057746 cut: support --whitespace-delimited=trimmed
Support ignoring leading and trailing whitespace.
E.g. this matches awk's default field splitting mode.

* src/cut.c
* tests/cut/cut.pl: Add test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 77ccacb9a7 cut: support -O as an alias for --output-delimiter
To improve compatibility with toybox/busybox scripts.

* doc/coreutils.texi (cut invocation): Add -O description.
* src/cut.c: Support -O as well as --output-delimiter
* tests/cut/cut.pl: Adjust one case to use -O.
2026-04-05 13:15:56 +01:00
Pádraig Brady 57110d8bae cut: implement -n to avoid outputting partial characters
Both the i18n patch and FreeBSD/macOS support this option.
They do differ in behavior somewhat as the i18n patch
may output more bytes than requested.

  $ printf '\xc3\xa9b\n' | i18n-cut -n -b1
  é

There is also a bug in the i18n patch with multi-byte
at the start of a line:

  $ printf '\xc3\xa9b\n' | i18n-cut -n -b1-2
  éb

We follow the FreeBSD behavior since it seems more
useful to have -b be a hard limit, rather than a soft limit.
This also reduces the possibility of duplicate character output
with separate cut invocations with non overlapping byte ranges.

* src/cut.c (cut_bytes_no_split): A new function
similar to cut_characters, to handle multi-byte characters
with byte limit semantics.
* tests/cut/cut.pl: Add test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady caf1e91266 tests: cut: add a test for divergence from i18n patch
* tests/cut/cut.pl: We don't fall back to byte mode
upon invalid uni-byte delimiter.
2026-04-05 13:15:56 +01:00
Pádraig Brady bed93d46f8 tests: cut: add case currently failing for coreutils-i18n patch
* tests/cut/cut.pl: Test for extraneous character output with:
printf 'aéb\n' | cut -s -d 'é' -f1 | od -tx1
2026-04-05 13:15:56 +01:00
Pádraig Brady 1cdc079860 tests: cut: check multi-byte output delimiter
* tests/cut/cut.pl: Add a test case.
2026-04-05 13:15:56 +01:00
Pádraig Brady 74047ec55e cut: adjust error message to be less specific
* src/cut.c (main): Cater for both misplaced -w and -d.
2026-04-05 13:15:56 +01:00
Pádraig Brady 19aa72b4ea cut: implement -w,--whitespace-delimited
* src/cut.c (cut_fields_ws): A new function handling both
uni-byte and multi-byte cases.
* tests/cut/cut.pl: Add a test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 32f1de5b4f cut: support single byte -d with GB18030 input
* src/cut.c
* tests/cut/mb-non-utf8.sh
* tests/local.mk
2026-04-05 13:15:55 +01:00
Pádraig Brady a021b0b698 cut: support multi-byte field delimiters
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady 97703386e6 cut: support multi-byte input with -c
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady e3c7dc2b03 tests: cut: ensure no unecessary buffering
* tests/misc/write-errors.sh: Ensure we write output when possible.
2026-04-05 13:15:55 +01:00
oech3 3558839bdd tests: dd: ensure memory exhaustion is handled gracefully
* tests/dd/no-allocate.sh: Ensure we exit 1 upon mem allocation failure.
Also check other buffer size edge cases.
https://github.com/uutils/coreutils/issues/11436
https://github.com/uutils/coreutils/issues/11580
https://github.com/coreutils/coreutils/pull/235
2026-04-01 16:54:12 +01:00
Pádraig Brady 178c48154d tests: dd: avoid false failure with no controlling terminal
* tests/dd/misc.sh: test -w /dev/tty is not a strong enough check,
we need to actually open /dev/tty to ensure it's available.
It's not available under setsid for example.
2026-04-01 13:31:57 +01:00
oech3 ee5092971e tests: dd: check that erroneous seeks are not done in output
* tests/dd/misc.sh: Add test case for of=/dev/tty.
The same occurs for /dev/stdout, but that varies
in the test hardness so is best avoided.
https://github.com/coreutils/coreutils/pull/234
2026-03-31 12:09:26 +01:00
oech3 368bfc7cb0 tests: coreutils: ensure empty arg is diagnosed
* tests/misc/coreutils.sh: Add a test case.
https://github.com/coreutils/coreutils/pull/232
2026-03-30 13:43:58 +01:00
oech3 fcccbab0fb tests: env/env.sh: improve portability
* tests/env/env.sh: Make more portable by avoiding references to our
build dir,  and avoiding names that may cause false matches in
multi-call binaries.
https://github.com/coreutils/coreutils/pull/216
2026-03-27 21:15:18 +00:00
Pádraig Brady 0ec2340c08 od: suppress address output on read error
We don't output an address for `od missing` or `od --strings .`,
so be consistent and suppress the address for `od .`.

* src/od.c (dump): Only output an address if no errors
or the offset is non zero.
2026-03-25 17:46:37 +00:00
oech3 cc3dd92f84 tests: od: ensure -j1 /dev/null succeeds
Users may be using this to convert bases.

* tests/od/od-j.sh: Add a test case.
https://github.com/coreutils/coreutils/pull/228
2026-03-25 17:46:16 +00:00
Collin Funk 7b4bbaa074 tests: truncate: don't rely on errno being EISDIR
* tests/truncate/multiple-files.sh: Only check that an error is printed
instead of an exact message.
Reported by Bruno Haible.
2026-03-24 19:02:21 -07:00
oech3 c495cbcfe2 tests: yes: support more zero-copy related syscalls
* tests/misc/yes.sh: Disable other related zero-copy syscalls
to ensure better testing of future or other implementations.
https://github.com/coreutils/coreutils/pull/227
2026-03-24 20:35:01 +00:00
Sylvestre Ledru 4fd2de166a tests: cut: add test for -z with NUL delimiter and -s flag
* tests/cut/cut.pl (zerot-7): New test.
Identified https://github.com/uutils/coreutils/pull/11394
https://github.com/coreutils/coreutils/pull/226
2026-03-23 12:11:49 +00:00
Sylvestre Ledru 939def75f3 tests: tr: add test for invalid character class name
* tests/tr/tr.pl (invalid-class): New test.
Identified : https://github.com/uutils/coreutils/pull/11398
https://github.com/coreutils/coreutils/pull/225
2026-03-23 12:11:35 +00:00
Collin Funk 26b5e3d3cf maint: fix an incomplete sentence
* tests/pwd/argument.sh: Fix the test description.
Reported by G. Branden Robinson.
2026-03-22 13:32:52 -07:00
Collin Funk 35a1c88f48 tests: pwd: test the behavior when given an argument
* tests/pwd/argument.sh: New file.
* tests/local.mk (all_tests): Add the new test.
2026-03-21 22:16:59 -07:00
Collin Funk 6a37187a5f tests: rm: fix a test that would sometimes hang
* tests/rm/dash-hint.sh: Add the file name argument to grep, as I
intended when adding this test.
2026-03-21 12:19:21 -07:00
Collin Funk 30a5cbec0e tac: promptly diagnose write errors
This patch also fixes a bug where 'tac' would print a vague error on
some inputs:

    $ seq 10000 | ./src/tac-prev > /dev/full
    tac-prev: write error
    $ seq 10000 | ./src/tac > /dev/full
    tac: write error: No space left on device

In this case ferror (stdout) is true, but errno has been set back to
zero by a successful fclose (stdout) call.

* src/tac.c (output): Call write_error() if fwrite fails.
* tests/misc/io-errors.sh: Check that 'tac' prints a detailed write
error.
* NEWS: Mention the improvement.
2026-03-21 12:10:59 -07:00
Pádraig Brady 4b680f8392 tests: support checking for specific write errors
* tests/misc/io-errors.sh: Support checkout for a specific error
in commands that don't run indefinitely.  Currently all the explicitly
listed commands output a specific error and do not need to be tagged.
2026-03-21 12:43:38 +00:00
Collin Funk f679990df2 tests: nl: check that all files are processed
* tests/nl/multiple-files.sh: New file.
* tests/local.mk (all_tests): Add the new test.
2026-03-20 19:46:04 -07:00
Collin Funk 5ec45a1aa5 test: truncate: improve the test added in the previous commit
* tests/truncate/multiple-files.sh: Check that nothing is printed to
standard output and that standard error has the correct error.
2026-03-19 23:17:47 -07:00
Collin Funk 71c17d6f2e tests: truncate: check that all files are processed
* tests/truncate/multiple-files.sh: New file.
* tests/local.mk (all_tests): Add the new test.
2026-03-19 22:56:52 -07:00
Pádraig Brady 58d88d2435 tests: dd: fix false failure on NetBSD 10
* tests/dd/partial-write.sh: Skip the test if
nothing written at all, as was seen on NetBSD 10.
Reported by Bruno Haible.
2026-03-16 22:34:58 +00:00
Pádraig Brady c1346460d1 tests: ls: fix false failure on FreeBSD
* tests/ls/non-utf8-hidden.sh: Avoid sorting in ls, to avoid:
ls: cannot compare file names ...: Illegal byte sequence
seen on FreeBSD 14.
Reported by Bruno Haible.
2026-03-16 22:25:42 +00:00
Collin Funk e644eea122 timeout: don't exit immediately if the parent is the init process
* src/timeout.c (main): Save the process ID before creating a child
process. Check if the result of getppid is different than the saved
process ID instead of checking if it is 1.
* tests/timeout/init-parent.sh: New file.
* tests/local.mk (all_tests): Add the new test.
* NEWS: Mention the bug fix. Also mention that this change allows
'timeout' to work when reparented by a subreaper process instead of
init.
2026-03-13 20:37:10 -07:00
Pádraig Brady 711fa8f9a5 dd: always diagnose partial writes on write failure
* src/dd.c (dd_copy): Increment the partial write count upon failure.
* tests/dd/partial-write.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/80583
2026-03-12 21:19:33 +00:00
Collin Funk ba85544a19 tests: rm: check for hints when running 'rm -foo'
* tests/rm/dash-hint.sh: New file.
* tests/local.mk (all_tests): Add the new test.
2026-03-10 21:39:22 -07:00
Pádraig Brady 2b1c059e6a yes: use a zero-copy implementation via (vm)splice
A good reference for the concepts used here is:
https://mazzo.li/posts/fast-pipes.html
We don't consider huge pages or busy loops here,
but use vmsplice(), and splice() to get significant speedups:

  i7-5600U-laptop $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [4.98GiB/s]
  i7-5600U-laptop $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [34.1GiB/s]

  IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [11.6GiB/s]
  IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [175GiB/s]

Also throughput to file (on BTRFS) was seen to increase significantly.
With a Fedora 43 laptop improving from 690MiB/s to 1.1GiB/s.

* bootstrap.conf: Ensure sys/uio.h is present.
This was an existing transitive dependency.
* m4/jm-macros.m4: Define HAVE_SPLICE appropriately.
We assume vmsplice() is available if splice() is as they
were introduced at the same time to Linux and glibc.
* src/yes.c (repeat_pattern): A new function to efficiently
duplicate a pattern in a buffer with memcpy calls that double in size.
This also makes the setup for the existing write() path more efficient.
(pipe_splice_size): A new function to increase the kernel pipe buffer
if possible, and use an appropriately sized buffer based on that (25%).
(splice_write): A new function to call vmplice() when outputting
to a pipe, and also splice() if outputting to a non-pipe.
* tests/misc/yes.sh: Verify the non-pipe output case,
(main): Adjust to always calling write on the minimal buffer first,
then trying vmsplice(), then falling back to write from bigger buffer.
and the vmsplice() fallback to write() case.
* NEWS: Mention the improvement.
2026-03-10 18:13:34 +00:00
Pádraig Brady a4cf72f5a7 tests: expand: fix false failure on various systems
* tests/expand/mb.sh: Use $LOCALE_FR_UTF8 rather than
hardcoding "en_US.UTF-8".
* tests/unexpand/mb.sh: Likewise.
Reported by Bruno Haible.
2026-03-09 21:08:40 +00:00
Lukáš Zaoral 2b92c16d26 expand,unexpand: support multi-byte input
* src/expand.c: Use mbbuf to support multi-byte input.
* src/unexpand.c: Likewise.
* tests/expand/mb.sh: New multi-byte test.
* tests/unexpand/mb.sh: Likewise.
* tests/local.mk: Reference new tests.
* NEWS: Mention the improvement.
2026-03-07 12:39:37 +00:00
Collin Funk b7ff7e7c2c install: allow the combination of --compare and --preserve-timestamps
* NEWS: Mention the improvement.
* src/install.c (enum copy_status): New type to let the caller know if
the copy was performed or skipped.
(copy_file): Return the new type instead of bool. Reduce variable scope.
(install_file_in_file): Only strip the file if the copy was
performed. Update the timestamps if the copy was skipped.
(main): Don't error when --compare and --preserve-timestamps are
combined.
* tests/install/install-C.sh: Add some test cases.
2026-03-04 18:42:48 -08:00
Pádraig Brady b3fe24213e cksum: use more defensive escaping for --check
cksum --check is often the first interaction
users have with possibly untrusted downloads, so we should try
to be as defensive as possible when processing it.

Specifically we currently only escape \n characters in file names
presented in checksum files being parsed with cksum --check.
This gives some possibilty of dumping arbitrary data to the terminal
when checking downloads from an untrusted source.
This change gives these advantages:

  1. Avoids dumping arbitrary data to vulnerable terminals
  2. Avoids visual deception with ansi codes hiding checksum failures
  3. More secure if users copy and paste file names from --check output
  4. Simplifies programmatic parsing

Note this changes programmatic parsing, but given the original
format was so awkward to parse, I expect that's extremely rare.
I was not able to find example in the wild at least.
To parse the new format from from shell, you can do something like:

  cksum -c checksums | while IFS= read -r line; do
    case $line in
      *': FAILED')
        filename=$(eval "printf '%s' ${line%: FAILED}")
        cp -v "$filename" /quarantine
        ;;
    esac
  done

This change also slightly reduces the size of the sum(1) utility.
This change also apples to md5sum, sha*sum, and b2sum.

* src/cksum.c (digest_check): Call quotef() instead of
cksum(1) specific quoting.
* tests/cksum/md5sum-bsd.sh: Adjust accordingly.
* doc/coreutils.texi (cksum general options): Describe the
shell quoting used for problematic file names.
* NEWS: Mention the change in behavior.
Reported by: Aaron Rainbolt
2026-03-04 22:17:39 +00:00
Pádraig Brady e24372e6d0 maint: tests: refactor uses of bad_unicode()
* init.cfg: Use 0xFF rather than 0xC3 everywhere.
* tests/fold/fold-characters.sh: Reuse bad_unicode().
* tests/tac/tac-locale.sh: Likewise.
2026-03-04 17:57:54 +00:00
Pádraig Brady a85e9182b1 fold: fix output truncation with 0xFF bytes in input
On signed char platforms, 0xFF was converted to -1
which matches MBBUF_EOF, causing fold to stop processing.

* NEWS: Mention the bug fix.
* gl/lib/mbbuf.h: Avoid sign extension on signed char platforms.
* tests/fold/fold-characters.sh: Adjust test case.
Reported at https://src.fedoraproject.org/rpms/coreutils/pull-request/20
2026-03-04 17:45:57 +00:00
Sylvestre Ledru 6c6bb37e2f tests: date: add timezone conversion test
*tests/date/date.pl: Add the test case.
Add test case for https://github.com/uutils/coreutils/issues/10800
to verify `date -u -d '10:30 UTC-05'` converts to 15:30 UTC.
2026-03-04 13:22:46 +00:00
Sylvestre Ledru 452bf39162 tests: date: add edge cases for modifiers
* tests/date/date.pl: Add the test case.
Add test cases for https://github.com/uutils/coreutils/issues/10957
2026-03-04 13:20:33 +00:00