Commit Graph
1884 Commits
Author SHA1 Message Date
Pádraig Brady 38cf97c1ff sort: use more dynamic memory allocation with pipes
The default memory allocation with pipes was too passive/static,
resulting in not allocating enough memory to enable threading.
By dynamically reallocating the buffer when reading from
unknown sized inputs we better use available memory and threads.

  $ time seq 10000000 -1 0 | sort-old >/dev/null
  real	0m16.523s
  user	0m16.900s
  sys	0m0.167s

  $ time seq 10000000 -1 0 | sort-old -S1G >/dev/null
  real	0m12.263s
  user	0m29.646s
  sys	0m0.527s

  $ time seq 10000000 -1 0 | sort-new >/dev/null
  real	0m12.994s
  user	0m31.266s
  sys	0m0.716s

It also avoids the overhead of writing to temp files
for modestly sized inputs. For example the following
input would induce interaction with temp storage:

  $ seq 125000 | wc -c
  763895

* src/sort.c (sort_buffer_size): Rename to ...
(sort_buffer_policy): ... here, and adjust to set
an initial size and limit, rather than just a size.
(fillbuf): Add a POLICY parameter, and use that
to call maybe_growbuf() as needed.
(maybe_growbuf): Return true if POLICY dictates we
should grow the buffer, and try_growbuf() was
able to reallocate the larger buffer.
* tests/sort/sort-buffer-size.sh: Add a new test.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
Related to https://bugs.gnu.org/10877
2026-04-29 13:16:21 +01:00
Collin Funk 4c4afad943 pinky: don't print output in the wrong order when fully buffered
* NEWS: Mention the bug fix.
* src/pinky.c (cat_file): Prefer streams to file descriptors when
writing to standard output.
2026-04-28 18:28:25 -07:00
Paul Eggert d64e35a8a4 uniq: fix read overrun with -w
Problem reported by Michał Majchrowicz.
* src/uniq.c (find_field): Fix typo.
* tests/uniq/uniq.pl (add_z_variants): Test for the bug.
2026-04-28 11:25:41 -07:00
Collin Funk c5ddd417aa comm: don't close standard input twice
* NEWS: Mention the bug fix.
* src/comm.c (usage): Remove mention that FILE1 and FILE2 cannot both be
standard input.
(compare_files): Only close standard input once.
* doc/coreutils.texi (comm invocation): Document the behavior of
'comm - -' which is not portable to all implementations.
* tests/comm/dash-dash.sh: New file.
* tests/misc/comm.pl: Move to tests/comm/comm.pl.
* tests/local.mk (all_tests): Add the new test. Rename the existing
test.
2026-04-22 19:12:44 -07:00
Pádraig Brady ac7e111873 maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2026-04-20 14:17:01 +01:00
Pádraig Brady c01fd163a4 version 9.11
* NEWS: Record release date.
2026-04-20 13:41:57 +01:00
Paul Eggert 271d3ff5f9 doc: NEWS item for who systemd fix 2026-04-19 11:50:56 -07:00
Lukáš Zaoral f6fda635bd df: improve detection of duplicate entries
Do not compare only with the latest entry for given device id but also
all previously saved entries with the same id.

* src/df.c (struct devlist): Add next_same_dev struct member.
(filter_mount_list): Iterate over next_same_dev to find duplicates.
* tests/df/skip-duplicates.sh: Add test cases.
* NEWS: Mention the improvement.
https://redhat.atlassian.net/browse/RHEL-5649
2026-04-15 12:56:16 +01:00
Collin Funk 457f88513a cat: use splice if operating on pipes or if copy_file_range fails
On a AMD Ryzen 7 3700X system:

    $ timeout 10 taskset 1 ./src/cat-prev /dev/zero \
        | taskset 2 pv -r > /dev/null
    [1.67GiB/s]
    $ timeout 10 taskset 1 ./src/cat /dev/zero \
        | taskset 2 pv -r > /dev/null
    [9.03GiB/s]

On a Power10 system:

    $ taskset 1 ./src/yes | timeout 10 taskset 2 ./src/cat-prev \
        | taskset 3 pv -r > /dev/null
    [12.9GiB/s]
    $ taskset 1 ./src/yes | timeout 10 taskset 2 ./src/cat \
            | taskset 3 pv -r > /dev/null
    [81.8GiB/s]

* NEWS: Mention the improvement.
* src/cat.c: Include isapipe.h, splice.h, and unistd--.h.
(splice_cat): New function.
(main): Use it.
* src/local.mk (noinst_HEADERS): Add src/splice.h.
* src/splice.h: New file, based on definitions from src/yes.c.
* src/yes.c: Include splice.h.
(pipe_splice_size): Use increase_pipe_size from src/splice.h.
(SPLICE_PIPE_SIZE): Remove definition, moved to src/splice.h.
* tests/cat/splice.sh: New file, based on some tests in
tests/misc/yes.sh.
* tests/local.mk (all_tests): Add the new test.
2026-04-06 21:57:07 -07:00
Pádraig Brady a325c99781 doc: document cut(1) multi-byte and interface consolidation
This patch set updates cut(1) to be multi-byte aware.
It also reduces interface divergence across implementations.

multi-byte awareness was added to the existing -c, n, and -d options.
Also considered for compatibility are the -w, -F, and -O options,
as these are present on at least two other common implementations.

= Interface / New functionality =

    macOS,  i18n, uutils, Toybox, Busybox, GNU
-c    x      x       x      x        x      x
-n    x      x                              x
-w    x              x                      x
-F                          x        x      x
-O                          x        x      x

-c is needed anyway as specified by all, including POSIX.
-n is needed also as specified by i18n/macOS/POSIX
-w is somewhat less important, but seeing as it's
on two other common platforms (and its functionality is
provided on two more), providing it is worthwhile for compat.
-F and -O are really just aliases to other options
so trivial to add, and probably worthwhile for compatibility.

Interface / functionality notes:

There is a slight divergence between -n implementations.
There was already a difference between FreeBSD and i18n, and
we've aligned with the more sensible FreeBSD implementation.
Note the i18n -n implementation is otherwise buggy in any case,
so I doubt this will be a practical compatibility concern.
Actually -n is specified by POSIX, and it matches FreeBSD.
Specifically our -n will not output a character unless the
byte range encompasses _the end_ of the multi-byte character.
I.e. the -b is a limit that is not passed, and thus ensures
we don't output overlapping characters for separate cut
invocations that do not have overlapping byte ranges.

-d <regex> from toybox is not implemented.
That's edge case functionality IMHO and not well suited to cut(1).
This functionality is supported by awk, and regex functionality
is best restricted to awk I think.

cut is a significant part of the i18n patch, so it will be good
to avoid that downstream divergence.  Unfortunately there were
no tests with the cut i18n implementation.
Note the i18n cut implementation used fread() as so was
not reponsive to new data < BUFSIZ, whereas this implementation
uses read() and thus is responsive to data as it becomes available.

= Performance =

General performance notes:

We prefer byte searching (with -d) as that can be much faster
than character by character processing, and it's supported
on single byte and UTF-8 charsets.  We also use byte searching
with -w on uni-byte locales.
This was seen to give up to 100x perf increase over the i18n patch.

Where we do use per character processing, we avoid conversion to
wide char when processing ASCII data (mcel provides this optimization).
This was seen to give a 14x performance increase over the i18n patch.

We prefer memchr() and strstr() as these are tuned for specific
platforms on glibc, even if memchr2() or memmem()
are algorithmically better.

We maintain the important memory behavior
of only buffering when necessary.

Performance testing:

There are _lots_ of combinations and optimziation opportunities.
I performance tested this patch set with the following setup:

$ yes | head -n10M > sl.in
$ yes $(yes eeeaae | head -n10K | paste -s -d,) | head -n10K > ll.in
$ yes $(yes eeeaae | head -n9 | paste -s -d,) | head -n1M > as.in
$ yes $(yes éééááé | head -n9 | paste -s -d,) | head -n1M \
  > mb.in

$ for type in sl ll as mb; do
    cat $type.in >/dev/null;
    for imp in '' src/; do  # '' maps to the system i18n ver on Fedora
      echo ============ "${imp:-i18n}" $type ==============;
      for d in -d, -dc -d, -dç -w -b -c; do
        fields='-f1 -f10 -f100'
        test "$d" = "-b" && { fields='-b1 -b10 -b100'; d=''; }
        test "$d" = "-c" && { fields='-c1 -c10 -c100'; d=''; }
        for f in $fields; do
          for loc in C C.UTF-8; do
            # SKip -b for UTF-8 as no different
            test "$loc" = C.UTF-8 && echo "$f" | grep -q -- -b \
             && continue
            # Skip multi-byte delimiter for C and not allowed
            test "$loc" = C && test $(echo -n "$d" | wc -c) -ge 4 \
             && continue
            LC_ALL=$loc ${imp}cut $f $d /dev/null 2>/dev/null &&
            hyperfine -m2 -M4 \
             "LC_ALL=$loc ${imp}cut $f $d $type.in >/dev/null" ||
            printf 'Benchmark 1: %s\n  unsupported\n\n' \
             "LC_ALL=$loc ${imp}cut $f $d $type.in >/dev/null"
          done;
        done;
      done;
    done;
  done

After a little post-processing of the results, we get:

-- cut-i18n

| command         |       sl |       ll |       as |       mb |
| --------------- | -------- | -------- | -------- | -------- |
| C -f1 -d,       |  66.3 ms |  1.605 s | 145.9 ms | 366.4 ms |
| UTF8 -f1 -d,    |  65.8 ms |  1.593 s | 145.8 ms | 370.0 ms |
| C -f10 -d,      | 301.4 ms |  1.590 s | 161.8 ms | 126.7 ms |
| UTF8 -f10 -d,   | 303.5 ms |  1.599 s | 161.8 ms | 124.6 ms |
| C -f100 -d,     | 300.6 ms |  1.596 s | 162.1 ms | 126.7 ms |
| UTF8 -f100 -d,  | 301.3 ms |  1.595 s | 162.0 ms | 124.9 ms |
| C -f1 -dc       |  66.6 ms |  1.845 s | 179.1 ms | 365.7 ms |
| UTF8 -f1 -dc    |  73.8 ms |  1.878 s | 179.1 ms | 363.1 ms |
| C -f10 -dc      | 300.7 ms | 349.8 ms |  76.0 ms | 125.3 ms |
| UTF8 -f10 -dc   | 300.4 ms | 347.2 ms |  75.7 ms | 124.8 ms |
| C -f100 -dc     | 300.1 ms | 348.1 ms |  76.5 ms | 125.5 ms |
| UTF8 -f100 -dc  | 300.8 ms | 348.7 ms |  76.4 ms | 125.8 ms |
| UTF8 -f1 -d,   | 563.5 ms | 21.775 s |  1.963 s |  1.665 s |
| UTF8 -f10 -d,  | 833.6 ms | 20.504 s |  2.022 s |  1.612 s |
| UTF8 -f100 -d, | 825.2 ms | 20.448 s |  2.009 s |  1.616 s |
| UTF8 -f1 -dç    | 563.7 ms | 21.827 s |  1.964 s |  2.319 s |
| UTF8 -f10 -dç   | 825.3 ms | 21.713 s |  2.011 s |  2.248 s |
| UTF8 -f100 -dç  | 831.6 ms | 20.505 s |  2.019 s |  2.276 s |
| C -f1 -w        |        - |        - |        - |        - |
| UTF8 -f1 -w     |        - |        - |        - |        - |
| C -f10 -w       |        - |        - |        - |        - |
| UTF8 -f10 -w    |        - |        - |        - |        - |
| C -f100 -w      |        - |        - |        - |        - |
| UTF8 -f100 -w   |        - |        - |        - |        - |
| C -b1           |  60.8 ms |  1.596 s | 154.8 ms | 313.7 ms |
| C -b10          |  51.6 ms |  1.594 s | 154.3 ms | 310.8 ms |
| C -b100         |  51.4 ms |  1.594 s | 153.0 ms | 312.2 ms |
| C -c1           |  60.7 ms |  1.597 s | 153.8 ms | 313.0 ms |
| UTF8 -c1        | 526.5 ms | 14.662 s |  1.362 s |  1.573 s |
| C -c10          |  51.8 ms |  1.591 s | 153.3 ms | 311.4 ms |
| UTF8 -c10       | 436.9 ms | 14.450 s |  1.336 s |  1.563 s |
| C -c100         |  51.0 ms |  1.593 s | 152.7 ms | 313.2 ms |
| UTF8 -c100      | 426.7 ms | 14.429 s |  1.344 s |  1.551 s |

-- src/cut

| command         |       sl |       ll |       as |       mb |
| --------------- | -------- | -------- | -------- | -------- |
| C -f1 -d,       |   4.6 ms | 108.2 ms |  45.4 ms |  24.2 ms |
| UTF8 -f1 -d,    |   4.8 ms | 108.4 ms |  45.4 ms |  24.5 ms |
| C -f10 -d,      |   4.5 ms | 109.3 ms | 123.7 ms |  24.3 ms |
| UTF8 -f10 -d,   |   4.9 ms | 114.1 ms | 124.1 ms |  24.5 ms |
| C -f100 -d,     |   4.7 ms | 119.2 ms | 124.1 ms |  24.5 ms |
| UTF8 -f100 -d,  |   4.8 ms | 120.0 ms | 125.1 ms |  24.5 ms |
| C -f1 -dc       |   4.4 ms | 120.5 ms |  11.9 ms |  24.1 ms |
| UTF8 -f1 -dc    |   4.9 ms | 120.5 ms |  12.1 ms |  24.6 ms |
| C -f10 -dc      |   4.7 ms | 125.3 ms |  11.8 ms |  24.1 ms |
| UTF8 -f10 -dc   |   4.8 ms | 126.7 ms |  12.0 ms |  24.4 ms |
| C -f100 -dc     |   4.6 ms | 127.0 ms |  11.9 ms |  24.3 ms |
| UTF8 -f100 -dc  |   4.7 ms | 126.4 ms |  12.0 ms |  24.4 ms |
| UTF8 -f1 -d,   |   6.0 ms | 169.4 ms |  15.6 ms |  67.4 ms |
| UTF8 -f10 -d,  |   6.1 ms | 173.9 ms |  15.6 ms | 237.2 ms |
| UTF8 -f100 -d, |   6.1 ms | 174.0 ms |  15.6 ms | 237.8 ms |
| UTF8 -f1 -dç    |   6.3 ms | 170.8 ms |  15.7 ms |  32.2 ms |
| UTF8 -f10 -dç   |   6.0 ms | 172.9 ms |  15.9 ms |  32.1 ms |
| UTF8 -f100 -dç  |   6.7 ms | 173.1 ms |  15.5 ms |  32.3 ms |
| C -f1 -w        | 159.6 ms | 170.1 ms |  69.1 ms |  98.9 ms |
| UTF8 -f1 -w     | 128.1 ms |  2.525 s | 246.5 ms |  1.086 s |
| C -f10 -w       | 183.3 ms | 199.2 ms |  74.6 ms | 105.0 ms |
| UTF8 -f10 -w    | 130.3 ms |  2.659 s | 276.5 ms |  1.099 s |
| C -f100 -w      | 183.8 ms | 202.5 ms |  74.1 ms | 103.6 ms |
| UTF8 -f100 -w   | 130.1 ms |  2.663 s | 276.6 ms |  1.097 s |
| C -b1           |  65.0 ms | 110.2 ms |  22.4 ms |  35.6 ms |
| C -b10          |  48.7 ms | 109.6 ms |  24.2 ms |  36.7 ms |
| C -b100         |  48.7 ms | 110.6 ms |  19.0 ms |  36.6 ms |
| C -c1           |  65.8 ms | 109.5 ms |  22.4 ms |  35.6 ms |
| UTF8 -c1        |  63.2 ms |  1.130 s | 116.9 ms | 610.2 ms |
| C -c10          |  48.7 ms | 109.8 ms |  24.3 ms |  36.8 ms |
| UTF8 -c10       |  39.7 ms |  1.133 s | 118.7 ms | 610.0 ms |
| C -c100         |  48.3 ms | 110.7 ms |  18.9 ms |  36.7 ms |
| UTF8 -c100      |  39.4 ms |  1.141 s | 115.0 ms | 598.8 ms |

In summary, compared to the i18n patch we're now as fast in all cases,
and much faster in most cases.

We can see the -f byte searching performing well,
being 120x faster in the no matching delimiter case,
to at least 3x faster in the matching delimiter case.

When we resort to per character processing we also compare well,
being 14x faster in the ASCII processing case
(due to mcel short-circuiting the wide char conversion).
Note the processing mb.in results above also show a 2x win
in per character processing cases, but the i18n patch would have
also picked that win up as it's achieved separately to this patch set:
https://lists.gnu.org/r/coreutils/2026-03/msg00117.html
2026-04-06 15:52:58 +01:00
Pádraig Brady 57c87043f6 cut,fold,expand,unexpand: ensure we process all available characters
* gl/lib/mbbuf.h: Adjust mbbuf_fill() to process full characters
in the slop at the end of a read().  Previously valid characters
in the last MCEL_LEN_MAX bytes were ignored until the next read().
* src/cut.c (cut_fields_bytesearch): Adjust to the new naming.
* NEWS: Mention the fold(1) responsiveness fix, which was
improved with the change from fread() to read(),
and completed with this patch.
2026-04-06 15:52:56 +01:00
Pádraig Brady 1204b29bab build: update to latest gnulib
Pick up mbrto{c32,wc} optimizations on UTF-8 on GLIBC.
Note configure.ac defines the required GNULIB_WCHAR_SINGLE_LOCALE.
This speeds up wc -m by 2.6x, when processing non ASCII chars,
and will similarly speed up per character processing
in the impending cut multi-byte implementation.
* NEWS: Mention the wc -m speed improvement.
2026-04-05 13:11:21 +01:00
Collin Funk 30a5cbec0e tac: promptly diagnose write errors
This patch also fixes a bug where 'tac' would print a vague error on
some inputs:

    $ seq 10000 | ./src/tac-prev > /dev/full
    tac-prev: write error
    $ seq 10000 | ./src/tac > /dev/full
    tac: write error: No space left on device

In this case ferror (stdout) is true, but errno has been set back to
zero by a successful fclose (stdout) call.

* src/tac.c (output): Call write_error() if fwrite fails.
* tests/misc/io-errors.sh: Check that 'tac' prints a detailed write
error.
* NEWS: Mention the improvement.
2026-03-21 12:10:59 -07:00
Collin Funk e644eea122 timeout: don't exit immediately if the parent is the init process
* src/timeout.c (main): Save the process ID before creating a child
process. Check if the result of getppid is different than the saved
process ID instead of checking if it is 1.
* tests/timeout/init-parent.sh: New file.
* tests/local.mk (all_tests): Add the new test.
* NEWS: Mention the bug fix. Also mention that this change allows
'timeout' to work when reparented by a subreaper process instead of
init.
2026-03-13 20:37:10 -07:00
Pádraig Brady 711fa8f9a5 dd: always diagnose partial writes on write failure
* src/dd.c (dd_copy): Increment the partial write count upon failure.
* tests/dd/partial-write.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/80583
2026-03-12 21:19:33 +00:00
Pádraig Brady 6382995b0a doc: clarify a recent NEWS item
* NEWS: It was ambiguous as to whether we quoted a range of
observered throughputs.  Clarify this was the old and new
throughput on a single test system.
2026-03-12 20:05:58 +00:00
Collin Funk 0da738fe40 doc: NEWS: adjust 'wc -l' aarch64 benchmark after recent commit
After commit e0190a9d1 (wc: improve aarch64 Neon optimization for
'wc -l', 2026-03-09), on a Ampere eMAG machine:

    $ yes | head -n 10000000000 > input
    $ (time ./src/wc -l input)
    10000000000 input

    real	0m3.447s
    user	0m1.533s
    sys	0m1.913s
    $ (export GLIBC_TUNABLES='glibc.cpu.hwcaps=-ASIMD,-AVX2,-AVX512F'; \
       time ./src/wc -l input)
    10000000000 input

    real	0m15.758s
    user	0m14.039s
    sys	0m1.720s

* NEWS: Mention the improved benchmark.
2026-03-10 23:16:09 -07:00
Pádraig Brady 2b1c059e6a yes: use a zero-copy implementation via (vm)splice
A good reference for the concepts used here is:
https://mazzo.li/posts/fast-pipes.html
We don't consider huge pages or busy loops here,
but use vmsplice(), and splice() to get significant speedups:

  i7-5600U-laptop $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [4.98GiB/s]
  i7-5600U-laptop $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [34.1GiB/s]

  IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [11.6GiB/s]
  IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [175GiB/s]

Also throughput to file (on BTRFS) was seen to increase significantly.
With a Fedora 43 laptop improving from 690MiB/s to 1.1GiB/s.

* bootstrap.conf: Ensure sys/uio.h is present.
This was an existing transitive dependency.
* m4/jm-macros.m4: Define HAVE_SPLICE appropriately.
We assume vmsplice() is available if splice() is as they
were introduced at the same time to Linux and glibc.
* src/yes.c (repeat_pattern): A new function to efficiently
duplicate a pattern in a buffer with memcpy calls that double in size.
This also makes the setup for the existing write() path more efficient.
(pipe_splice_size): A new function to increase the kernel pipe buffer
if possible, and use an appropriately sized buffer based on that (25%).
(splice_write): A new function to call vmplice() when outputting
to a pipe, and also splice() if outputting to a non-pipe.
* tests/misc/yes.sh: Verify the non-pipe output case,
(main): Adjust to always calling write on the minimal buffer first,
then trying vmsplice(), then falling back to write from bigger buffer.
and the vmsplice() fallback to write() case.
* NEWS: Mention the improvement.
2026-03-10 18:13:34 +00:00
Pádraig Brady 5ca27c1929 all: use more consistent blank character determination
* src/system.h (c32issep): A new function that is essentially
iswblank() on GLIBC platforms, and iswspace() with exceptions elsewhere.
* src/expand.c: Use it instead of c32isblank().
* src/fold.c: Likewise.
* src/join.c: Likewise.
* src/numfmt.c: Likewise.
* src/unexpand.c: Likewise.
* src/uniq.c: Likewise.
* NEWS: Mention the improvement.
2026-03-10 16:30:52 +00:00
Lukáš Zaoral 2b92c16d26 expand,unexpand: support multi-byte input
* src/expand.c: Use mbbuf to support multi-byte input.
* src/unexpand.c: Likewise.
* tests/expand/mb.sh: New multi-byte test.
* tests/unexpand/mb.sh: Likewise.
* tests/local.mk: Reference new tests.
* NEWS: Mention the improvement.
2026-03-07 12:39:37 +00:00
Collin Funk b7ff7e7c2c install: allow the combination of --compare and --preserve-timestamps
* NEWS: Mention the improvement.
* src/install.c (enum copy_status): New type to let the caller know if
the copy was performed or skipped.
(copy_file): Return the new type instead of bool. Reduce variable scope.
(install_file_in_file): Only strip the file if the copy was
performed. Update the timestamps if the copy was skipped.
(main): Don't error when --compare and --preserve-timestamps are
combined.
* tests/install/install-C.sh: Add some test cases.
2026-03-04 18:42:48 -08:00
Pádraig Brady b3fe24213e cksum: use more defensive escaping for --check
cksum --check is often the first interaction
users have with possibly untrusted downloads, so we should try
to be as defensive as possible when processing it.

Specifically we currently only escape \n characters in file names
presented in checksum files being parsed with cksum --check.
This gives some possibilty of dumping arbitrary data to the terminal
when checking downloads from an untrusted source.
This change gives these advantages:

  1. Avoids dumping arbitrary data to vulnerable terminals
  2. Avoids visual deception with ansi codes hiding checksum failures
  3. More secure if users copy and paste file names from --check output
  4. Simplifies programmatic parsing

Note this changes programmatic parsing, but given the original
format was so awkward to parse, I expect that's extremely rare.
I was not able to find example in the wild at least.
To parse the new format from from shell, you can do something like:

  cksum -c checksums | while IFS= read -r line; do
    case $line in
      *': FAILED')
        filename=$(eval "printf '%s' ${line%: FAILED}")
        cp -v "$filename" /quarantine
        ;;
    esac
  done

This change also slightly reduces the size of the sum(1) utility.
This change also apples to md5sum, sha*sum, and b2sum.

* src/cksum.c (digest_check): Call quotef() instead of
cksum(1) specific quoting.
* tests/cksum/md5sum-bsd.sh: Adjust accordingly.
* doc/coreutils.texi (cksum general options): Describe the
shell quoting used for problematic file names.
* NEWS: Mention the change in behavior.
Reported by: Aaron Rainbolt
2026-03-04 22:17:39 +00:00
Pádraig Brady a85e9182b1 fold: fix output truncation with 0xFF bytes in input
On signed char platforms, 0xFF was converted to -1
which matches MBBUF_EOF, causing fold to stop processing.

* NEWS: Mention the bug fix.
* gl/lib/mbbuf.h: Avoid sign extension on signed char platforms.
* tests/fold/fold-characters.sh: Adjust test case.
Reported at https://src.fedoraproject.org/rpms/coreutils/pull-request/20
2026-03-04 17:45:57 +00:00
Pádraig Brady 46aef09d64 build: update gnulib to latest
* NEWS: Mention the more encompassing remoteness check for df.
* po/POTFILES.in: Add new lib/cygpath.c dependency.
2026-02-28 12:16:23 +00:00
Paul Eggert d41e1375cb id: avoid unnecessary buffer flushing
* src/groups.c (main):
* src/id.c (main, print_stuff):
Don’t flush stdout before testing for write error.
Do the test only when in a loop, as a one-shot will
test for write error soon anyway.
2026-02-27 16:18:32 -08:00
Collin Funk 803bfa01e1 stat: don't check QUOTING_STYLE when --printf %%N is used
* NEWS: Mention the fix.
* src/stat.c (main): Only check QUOTING_STYLE if there is a %N that is
not preceded by a percentage sign.
* tests/stat/stat-fmt.sh: Add some test cases.
2026-02-26 21:20:37 -08:00
Collin Funk 4195e36664 id: promptly diagnose write errors
* NEWS: Mention the improvement.
* src/id.c (print_stuff): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'id'.
2026-02-26 19:07:21 -08:00
Collin Funk 80b229be71 groups: promptly diagnose write errors
* NEWS: Mention the improvement.
* src/groups.c (main): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'groups'.
2026-02-26 19:05:32 -08:00
Pádraig Brady 8fab3c6d30 tests: fix "Hangup" termination of non-interactive runs
This avoids the test harness being terminated like:
  make[1]: *** [Makefile:24419: check-recursive] Hangup
  make[3]: *** [Makefile:24668: check-TESTS] Hangup
  make: *** [Makefile:24922: check] Hangup
  make[2]: *** [Makefile:24920: check-am] Hangup
  make[4]: *** [Makefile:24685: tests/misc/usage_vs_refs.log] Error 129
  ...

This happened sometimes when the tests were being run non interactively.
For example when run like:

  setsid make TESTS="tests/timeout/timeout.sh \
   tests/tail/overlay-headers.sh" SUBDIRS=. -j2 check

Note the race window can be made bigger by adding a sleep
after tail is stopped in overlay-headers.sh

The race can trigger the kernel to induce its job control
mechanism to prevent stuck processes.
I.e. where it sends SIGHUP + SIGCONT to a process group
when it determines that group may become orphaned,
and there are stopped processes in that group.

* tests/tail/overlay-headers.sh: Use setsid(1) to keep the stopped
tail process in a separate process group, thus avoiding any kernel
job control protection mechanism.
* tests/timeout/timeout.sh: Use setsid(1) to avoid the kernel
checking the main process group when sleep(1) is reparented.
Fixes https://bugs.gnu.org/80477
2026-02-24 19:25:58 +00:00
Collin Funk 89735ea830 shuf: avoid locking standard output when using --input-range
Here is the throughput before this patch:

    # write_permuted_numbers
    $ ./src/shuf-prev -i 0-100000000 | pv -r > /dev/null
    [ 153MiB/s]

    # write_random_numbers
    $ timeout 10 ./src/shuf-prev -i 0-100000 -r | pv -r > /dev/null
    [78.6MiB/s]

Here is the throughput after this patch:

    # write_permuted_numbers
    $ timeout 10 ./src/shuf -i 0-100000000 | pv -r > /dev/null
    [ 308MiB/s]

    # write_random_numbers
    $ timeout 10 ./src/shuf -i 0-100000 -r | pv -r > /dev/null
    [ 196MiB/s]

* NEWS: Mention the performance improvement.
* src/shuf.c (write_permuted_numbers, write_random_numbers): Prefer
fputs and fputc which may be unlocked over printf which locks standard
output.
2026-02-21 19:20:07 -08:00
Collin Funk 6a3dde5dd2 wc: add aarch64 Neon optimization for wc -l
Here is an example of the performance improvement:

    $ yes abcdefghijklmnopqrstuvwxyz | head -n 100000000 > input
    $ time ./src/wc-prev -l < input
    100000000

    real	0m0.793s
    user	0m0.630s
    sys	0m0.162s
    $ time ./src/wc -l < input
    100000000

    real	0m0.230s
    user	0m0.065s
    sys	0m0.164s

* NEWS: Mention the performance improvement.
* gnulib: Update to the latest commit.
* configure.ac: Check the the necessary intrinsics and functions.
* src/local.mk (noinst_LIBRARIES) [USE_NEON_WC_LINECOUNT]: Add
src/libwc_neon.a.
(src_libwc_neon_a_SOURCES, wc_neon_ldadd, src_libwc_neon_a_CFLAGS)
[USE_NEON_WC_LINECOUNT]: New variables.
(src_wc_LDADD) [USE_NEON_WC_LINECOUNT]: Add $(wc_neon_ldadd).
* src/wc.c [USE_NEON_WC_LINECOUNT]: Include sys/auxv.h and asm/hwcap.h.
(neon_supported) [USE_NEON_WC_LINECOUNT]: New function.
(wc_lines) [USE_NEON_WC_LINECOUNT]: Use neon_supported and
wc_lines_neon.
* src/wc.h (wc_lines_neon): Add declaration.
* src/wc_neon.c: New file.
* doc/coreutils.texi (Hardware Acceleration): Document the "-ASIMD"
hwcap and the variable used in ./configure to override detection of Neon
instructions.
* tests/wc/wc-cpu.sh: Also add "-ASIMD" to disable the use of Neon
instructions.
2026-02-18 20:09:44 -08:00
Pádraig Brady 8a0d9b889f tests: pwd: ensure our getcwd fallback is tested
* tests/pwd/pwd-long.sh: Avoid the getcwd syscall,
to ensure our fallback getcwd code is exercised.
* NEWS: Mention the recent bug fix in pwd.
2026-02-17 00:41:50 +00:00
Collin Funk 6bbf04db63 nl: support multi-byte section delimiters
* NEWS: Mention the improvement.
* src/nl.c: Include mcel.h.
(DEFAULT_SECTION_DELIMITERS): Resize to fit 2 multi-byte characters.
(section_del_len): New variable.
(check_section): Compare against section_del_len instead of 2.
(main): Support multi-byte characters for the -d option.
* tests/nl/multibyte.sh: New file.
* tests/nl/nl.sh: New file, moved from tests/misc/nl.sh.
* tests/local.mk (all_tests): Add the new test. Adjust the existing
tests file name.
* cfg.mk (exclude_file_name_regexp--sc_space_tab): Adjust Adjust the
existing tests file name.
2026-02-08 18:45:24 -08:00
Gabriel e2c7db6c70 date: add tests and NEWS for dd.mm.yy support
* NEWS: Mention the new feature.
* tests/date/date.pl: Add test cases.
2026-02-08 16:25:44 +00:00
Sam James a72ad1216d build: fix --enable-single-binary=hardlinks with dash
With dash as /bin/sh, you get the error
```
checking for sys/capability.h... yes
./configure: 95775: test: no: unexpected operator
checking for working fork... yes
```

* configure.ac: Use '=' in test for equality, not '==',
for POSIX shell compatibility.
* NEWS: Mention the build fix.
2026-02-08 11:18:34 +00:00
Collin Funk 2cf491412c doc: NEWS: mention the recent kill documentation fix
* NEWS: Mention that the kill documentation now has the anchors expected
by 'kill --help'.
2026-02-06 16:06:28 -08:00
Pádraig Brady cc64b63e99 maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2026-02-04 12:20:32 +00:00
Pádraig Brady 89b2cd58ac version 9.10
* NEWS: Record release date.
2026-02-04 11:45:38 +00:00
Pádraig Brady ebb856730c tail: fix EINTR handling on older systems
tail(1) could fail with an "Interrupted system call"
diagnostic, on some systems like Centos 5 (Linux 2.6.18).
This was seen with tests/tail/overlay-headers.sh
which sends SIGCONT, which should not induce a failure.

* src/tail.c (tail_forever_inotify): Retry the poll()
upon receiving a non terminating signal, and the syscall
is not automatically restarted by the system.
* NEWS: Mention the bug fix.
Reported by Bruno Haible.
2026-01-24 20:00:32 +00:00
Collin Funk be0a448133 doc: NEWS: add a missing quotation mark
* NEWS: Add a closing quotation mark.
2026-01-23 20:15:57 -08:00
Pádraig Brady 343cae4aed doc: NEWS: minor adjustments
* NEWS: Fix typos, and move item to more appropriate section.
2026-01-22 16:13:26 +00:00
Pádraig Brady 87219034b7 doc: use TERM=dumb rather than HELP_NO_MARKUP to disable markup
This is a more standard mechanism to disable markup.

* src/system.h (oputs_): Logic change to honor TERM=dumb,
rather than HELP_NO_MARKUP=something.
* doc/coreutils.texi: Adjust the description for --help.
* man/local.mk: Ensure TERM is set to something,
so that man pages have links included.
* man/viewman: Just honor users $TERM.
* tests/misc/getopt_vs_usage.sh: Remove env var complication,
as TERM is unset automatically.
* tests/misc/usage_vs_refs.sh: Likewise.
* NEWS: Adjust the change in behavior note.
2026-01-22 11:42:47 +00:00
Pádraig Brady 28c29d103e stat,tail: sync with latest Linux file systems
* src/stat.c (human_fstype): Add "guest-memfd".
* NEWS: Mention the improvement.
2026-01-21 16:29:03 +00:00
Pádraig Brady dfeb32f6e2 ls: --hyperlink: switch to more standard delimiters
* src/ls.c (quote_name): Use ST (ESC \) rather than BEL,
as that's the only terminator mentioned in at least
ECMA-48, DEC STD 070, and EK-VT520-RM.
* NEWS: Mention the change in behavior.
* tests/ls/hyperlink.sh: Adjust accordingly.

Suggested by Egmont Koblinger.
2026-01-21 14:05:48 +00:00
Pádraig Brady 7dfe08316d doc: NEWS: describe the --help markup configuration options
* NEWS: Mention build and runtime config options.
2026-01-21 14:05:48 +00:00
Pádraig Brady 331c5f6857 build: update to latest gnulib
* gnulib: Update to latest.
* NEWS: Mention the bug fix.
https://github.com/coreutils/coreutils/issues/176
2026-01-19 17:57:59 +00:00
Pádraig Brady 6b32a0bbde all: with multi-call binary, only process options for known names
Setup
  $ ln -nsf src/coreutils foo

Before
  $ ./foo; echo $?
  foo: unknown program ‘foo’
  Try './foo --help' for more information.
  1
  ./foo --version; echo $?
  coreutils (GNU coreutils) 9.9.172-01993
  0

After
  $ ./foo; echo $?
  coreutils: unknown program 'foo'
  1
  $ ./foo --version; echo $?
  coreutils: unknown program 'foo'
  1

* src/coreutils.c (main): Don't process options if
we don't know they're intended for the multi-call binary.
Otherwise `foo --version` would return true, even though
foo was symlinked to the multi-call binary, but not supported.
* tests/misc/coreutils.sh: Add test cases.
* NEWS: Mention the change in behavior.
2026-01-15 15:54:29 +00:00
Pádraig Brady 495e3528d7 fmt: correctly diagnose read errors
* src/fmt.c (fmt): Pass errno to error() so that
the particular error is diagnosed.
* NEWS: Mention the improvement.
2026-01-13 15:08:03 +00:00
Pádraig Brady 06d228043f paste: support multi-byte delimiters
* src/paste.c (collapse_escapes): This is the central --delimiters
parsing function, so adjust to handle multi-byte chars with
mcel_scanz().  Populate a delimiters length array to support
characters of differing lengths.
(paste_serial): Use the delimiters length array to output
the appropriate delimiter.
(paste_parallel): Likewise.
* tests/paste/multi-byte.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
2026-01-13 14:32:01 +00:00
Pádraig Brady 204dc3bce4 build: update gnulib submodule to latest
* gnulib: Update to latest mainly to pull in selinux module fixes.
* NEWS: Mention the build fix to honor --with-selinux building
stub chcon and runcon binaries on systems without libselinux.
2026-01-11 12:54:17 +00:00