Commit Graph
9471 Commits
Author SHA1 Message Date
Pádraig Brady 74047ec55e cut: adjust error message to be less specific
* src/cut.c (main): Cater for both misplaced -w and -d.
2026-04-05 13:15:56 +01:00
Pádraig Brady 19aa72b4ea cut: implement -w,--whitespace-delimited
* src/cut.c (cut_fields_ws): A new function handling both
uni-byte and multi-byte cases.
* tests/cut/cut.pl: Add a test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 32f1de5b4f cut: support single byte -d with GB18030 input
* src/cut.c
* tests/cut/mb-non-utf8.sh
* tests/local.mk
2026-04-05 13:15:55 +01:00
Pádraig Brady 94ddf45a60 cut: support single byte -d that may be part of multi-byte
Note this is a slight divergence from the i18n patch
as that switched to uni-byte for any single byte delimiter
that is not valid multi-byte.

That results in possibly splitting in the middle of
a valid multi-byte character.

Instead we only split on a single byte when they're
not part of a multi-byte character.

* src/cut.c
2026-04-05 13:15:55 +01:00
Pádraig Brady a021b0b698 cut: support multi-byte field delimiters
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady 97703386e6 cut: support multi-byte input with -c
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady fb78200249 maint: cut: refactor output calls
* src/cut.c (cut_fields): Refactor calls to fwrite() and putchar()
2026-04-05 13:15:55 +01:00
Pádraig Brady a04a1054f8 doc: cut: reorder --complement alphabetically in help
* src/cut.c (usage): Move placement of --comlement description.
* doc/coreutils.texi (cut invocation): Likewise.
2026-04-05 13:15:55 +01:00
Pádraig Brady 1173ebb7c8 doc: cut: clarify description of -b and -c
* src/cut.c (usage): State the arguments are positions,
in case users may think they were values.
2026-04-05 13:15:55 +01:00
Collin Funk 38fc6bde64 basename: avoid duplicate strlen calls on the suffix
$ ltrace -c ./src/basename-prev -s a $(seq 100000) > /dev/null
    % time     seconds  usecs/call     calls      function
    ------ ----------- ----------- --------- --------------------
     50.00   30.030316          75    400000 strlen
    [...]
    $ ltrace -c ./src/basename -s a $(seq 100000) > /dev/null
    % time     seconds  usecs/call     calls      function
    ------ ----------- ----------- --------- --------------------
     42.88   22.413953          74    300001 strlen
    [...]

* src/basename.c (remove_suffix, perform_basename): Add a length
argument for the suffix and use it instead of strlen.
(main): Calculate the suffix length. Refactor code to avoid calling
perform_basename in multiple places.
2026-04-04 14:55:56 -07:00
Paul Eggert b64f9bfe4f date: simplify -u by not calling putenv
* src/date.c (TZSET): Remove; no longer needed.
(main): Simplify -u’s implementation by passing "UTC0" to tzalloc,
rather than by setting TZ in the environment and then calling getenv.
The old way of doing things dates back to before we had tzalloc.
* configure.ac (LOCALTIME_CACHE): Remove; no longer needed.
2026-04-02 18:54:35 -07:00
Collin Funk 4cd0644472 date: avoid calling putenv multiple times unnecessarily
Adding environment variables can become quite expensive in some
admittedly unlikely situations.

    $ for i in $(seq 10000); do export A$i=A$i; done
    $ time ./src/date-prev -u $(yes -- -u | head -n 100000)
    Sun Mar 29 01:59:49 AM UTC 2026

    real	0m3.753s
    user	0m3.684s
    sys	0m0.050s
    $ time ./src/date -u $(yes -- -u | head -n 100000)
    Sun Mar 29 02:00:00 AM UTC 2026

    real	0m0.061s
    user	0m0.022s
    sys	0m0.045s

* src/date.c (main): Only add TZ=UTC0 to the environment once.
2026-03-28 18:57:49 -07:00
Collin Funk 753fa26918 maint: remove unnecessary return statements
* src/env.c (initialize_signals): Remove return at the end of the
function.
* src/who.c (print_runlevel): Likewise.
2026-03-28 12:48:38 -07:00
Collin Funk eec7b2a131 who: avoid locking standard output for each user with the -q option
* src/who (list_entries_who): Prefer putchar and fputs to printf.
Simplify separator tracking.
2026-03-28 12:45:14 -07:00
Pádraig Brady 0ec2340c08 od: suppress address output on read error
We don't output an address for `od missing` or `od --strings .`,
so be consistent and suppress the address for `od .`.

* src/od.c (dump): Only output an address if no errors
or the offset is non zero.
2026-03-25 17:46:37 +00:00
Collin Funk 430822d5f7 maint: remove some unnecessary casts
* src/sort.c (begfield, limfield): Remove size_t casts.
2026-03-23 19:32:21 -07:00
Chris Down 67b8aaae8d sort: speed up keyed field sorting significantly using memchr
When sort is invoked with an explicit field separator with `-t SEP`,
begfield() and limfield() scan for the separator to locate boundaries.
Right now the implementation there uses a loop that iterates over bytes
one by one, which is not ideal since we must scan past many bytes of
non-separator data one byte at a time.

Let's replace each of these loops with memchr(). On glibc systems,
memchr() uses SIMD to scan 16 bytes per step (NEON on aarch64) or 32
bytes per step (AVX2 on x86_64), rather than 1 byte at a time, so any
field longer than a handful of bytes stands to benefit quite
significantly.

Using the following input data:

  awk 'BEGIN {
      srand(42)
      for (i = 1; i <= 500000; i++)
          printf "%*d,%*d,%d\n", 4+int(rand()*9), 0,
                                 4+int(rand()*9), 0, int(rand()*10000)
  }' > short_csv_500k

  awk 'BEGIN {
      for (i = 1; i <= 500000; i++)
          printf "%100d,%100d,%d\n", 0, 0, int(rand()*10000)
  }' > wide_csv_500k

One can benchmark with:

  hyperfine --warmup 10 --runs 50 \
    "LC_ALL=C sort_before -t, -k3,3n short_csv_500k > /dev/null" \
    "LC_ALL=C sort_after -t, -k3,3n short_csv_500k > /dev/null"

  hyperfine --warmup 10 --runs 50 \
    "LC_ALL=C sort_before -t, -k3,3n wide_csv_500k > /dev/null" \
    "LC_ALL=C sort_after -t, -k3,3n wide_csv_500k > /dev/null"

  hyperfine --warmup 10 --runs 50 \
    "LC_ALL=C sort_before wide_csv_500k > /dev/null" \
    "LC_ALL=C sort_after wide_csv_500k > /dev/null"

The results on i9-14900HX x86_64 with -O2:

  sort -t, -k3,3n (500K lines, 4-12 byte short fields):
    Before: 123.1 ms    After: 108.1 ms    (-12.2%)

  sort -t, -k3,3n (500K lines, 100 byte wide fields):
    Before: 243.5 ms    After: 165.9 ms    (-31.9%)

  sort (default, no -k, 500K lines):
    Before: 141.6 ms    After: 141.8 ms    (unchanged)

And on M1 Pro aarch64 with -O2:

  sort -t, -k3,3n (500K lines, 4-12 byte short fields):
    Before: 98.0 ms     After: 92.3 ms     (-5.8%)

  sort -t, -k3,3n (500K lines, 100 byte wide fields):
    Before: 240.8 ms    After: 183.0 ms    (-24.0%)

  sort (default, no -k, 500K lines):
    Before: 145.6 ms    After: 145.6 ms    (unchanged)

Looking at profiling, the improvement is larger on x86_64 in these runs
because glibc's memchr uses AVX2 to scan 32 bytes per step versus 16
bytes per step with NEON on aarch64.
2026-03-23 11:49:02 +00:00
Collin Funk 4bc336b7ce tac: avoid unnecessary standard output buffering
This has removes a tiny amount of overhead:

    $ seq 10000000 > input
    $ perf stat -e cpu-clock --repeat 1000 taskset 1 ./src/tac \
        input 2>&1 > /dev/null | grep -F 'seconds time'
              0.095707 +- 0.000223 seconds time elapsed  ( +-  0.23% )
    $ perf stat -e cpu-clock --repeat 1000 taskset 1 ./src/tac-prev \
        input 2>&1 > /dev/null | grep -F 'seconds time'
             0.1009378 +- 0.0000995 seconds time elapsed  ( +-  0.10% )

* src/tac.c (output): Use full_write instead of fread since we already
buffer the output ourselves.
2026-03-21 15:50:33 -07:00
Collin Funk 30a5cbec0e tac: promptly diagnose write errors
This patch also fixes a bug where 'tac' would print a vague error on
some inputs:

    $ seq 10000 | ./src/tac-prev > /dev/full
    tac-prev: write error
    $ seq 10000 | ./src/tac > /dev/full
    tac: write error: No space left on device

In this case ferror (stdout) is true, but errno has been set back to
zero by a successful fclose (stdout) call.

* src/tac.c (output): Call write_error() if fwrite fails.
* tests/misc/io-errors.sh: Check that 'tac' prints a detailed write
error.
* NEWS: Mention the improvement.
2026-03-21 12:10:59 -07:00
Collin Funk f421d01128 sort,split,yes: ensure pipe and pipe2 don't open standard descriptors
* bootstrap.conf (gnulib_modules): Add pipe2-safer.
* cfg.mk (sc_require_unistd_safer): New rule for 'make syntax-check'.
* gl/lib/fd-reopen.c: Include unistd--.h instead of unistd.h.
* src/sort.c: Include unistd--.h.
* src/split.c: Likewise.
* src/yes.c: Likewise.
2026-03-17 23:08:37 -07:00
Collin Funk 8fc17919a3 maint: tee: remove an affirm call to silence coverity
* src/iopoll.c (write_wait): Don't check that an unsigned integer is
always great than or equal to zero since that is always true.
2026-03-16 15:04:24 -07:00
Collin Funk 9b29c9c899 wc: make sure input buffer for neon 'wc -l' is aligned
* src/wc_neon.c (wc_lines_neon): Use alignas.
2026-03-16 12:20:38 -07:00
Collin Funk 9b855166ed tee: prefer file descriptors over streams
We disable buffering on the streams anyways, so we were effectively
calling the write system call previously despite using streams.

* src/iopoll.h (fclose_wait, fwrite_wait): Remove declarations.
(close_wait, write_wait): Add declarations.
* src/iopoll.c (fwait_for_nonblocking_write, fclose_wait, fwrite_wait):
Remove functions.
(wait_for_nonblocking_write): New function based on
fwait_for_nonblocking_write.
(close_wait): New function based on fclose_wait.
(write_wait): New function based on fwrite_wait.
* src/tee.c: Include fcntl--.h. Don't include stdio--.h.
(get_next_out): Operate on file descriptors instead of streams.
(fail_output): Likewise. Remove clearerr call since we no longer call
fwrite on stdout.
(tee_files): Operate on file descriptors instead of streams. Remove
calls to setvbuf.
2026-03-15 15:32:08 -07:00
Collin Funk e644eea122 timeout: don't exit immediately if the parent is the init process
* src/timeout.c (main): Save the process ID before creating a child
process. Check if the result of getppid is different than the saved
process ID instead of checking if it is 1.
* tests/timeout/init-parent.sh: New file.
* tests/local.mk (all_tests): Add the new test.
* NEWS: Mention the bug fix. Also mention that this change allows
'timeout' to work when reparented by a subreaper process instead of
init.
2026-03-13 20:37:10 -07:00
Pádraig Brady 711fa8f9a5 dd: always diagnose partial writes on write failure
* src/dd.c (dd_copy): Increment the partial write count upon failure.
* tests/dd/partial-write.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/80583
2026-03-12 21:19:33 +00:00
Pádraig Brady e016498d6f maint: adjust to placate coverity
* src/system.h (c32issep): Adjust to more standard layout.
2026-03-10 20:14:42 +00:00
Pádraig Brady 2b1c059e6a yes: use a zero-copy implementation via (vm)splice
A good reference for the concepts used here is:
https://mazzo.li/posts/fast-pipes.html
We don't consider huge pages or busy loops here,
but use vmsplice(), and splice() to get significant speedups:

  i7-5600U-laptop $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [4.98GiB/s]
  i7-5600U-laptop $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [34.1GiB/s]

  IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [11.6GiB/s]
  IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [175GiB/s]

Also throughput to file (on BTRFS) was seen to increase significantly.
With a Fedora 43 laptop improving from 690MiB/s to 1.1GiB/s.

* bootstrap.conf: Ensure sys/uio.h is present.
This was an existing transitive dependency.
* m4/jm-macros.m4: Define HAVE_SPLICE appropriately.
We assume vmsplice() is available if splice() is as they
were introduced at the same time to Linux and glibc.
* src/yes.c (repeat_pattern): A new function to efficiently
duplicate a pattern in a buffer with memcpy calls that double in size.
This also makes the setup for the existing write() path more efficient.
(pipe_splice_size): A new function to increase the kernel pipe buffer
if possible, and use an appropriately sized buffer based on that (25%).
(splice_write): A new function to call vmplice() when outputting
to a pipe, and also splice() if outputting to a non-pipe.
* tests/misc/yes.sh: Verify the non-pipe output case,
(main): Adjust to always calling write on the minimal buffer first,
then trying vmsplice(), then falling back to write from bigger buffer.
and the vmsplice() fallback to write() case.
* NEWS: Mention the improvement.
2026-03-10 18:13:34 +00:00
Pádraig Brady 5ca27c1929 all: use more consistent blank character determination
* src/system.h (c32issep): A new function that is essentially
iswblank() on GLIBC platforms, and iswspace() with exceptions elsewhere.
* src/expand.c: Use it instead of c32isblank().
* src/fold.c: Likewise.
* src/join.c: Likewise.
* src/numfmt.c: Likewise.
* src/unexpand.c: Likewise.
* src/uniq.c: Likewise.
* NEWS: Mention the improvement.
2026-03-10 16:30:52 +00:00
Pádraig Brady 3ef107fa12 cksum: fix tagged output on 32 bit platforms
Fix an unreleased issue due to the recent change
to using idx_t in commit v9.10-91-g02983e493

* src/cksum.c (output_file): Cast the idx_t before passing to printf.
2026-03-10 16:18:29 +00:00
Collin Funk e0190a9d1b wc: improve aarch64 Neon optimization for 'wc -l'
$ yes abcdefghijklmnopqrstuvwxyz | head -n 200000000 > input
    $ time ./src/wc-prev -l input
    200000000 input

    real	0m1.240s
    user	0m0.456s
    sys	0m0.784s
    $ time ./src/wc -l input
    200000000 input

    real	0m0.936s
    user	0m0.141s
    sys	0m0.795s

* configure.ac: Use unsigned char for the buffer to avoid potential
compiler warnings. Check for the functions being used in src/wc_neon.c
after this patch.
* src/wc_neon.c (wc_lines_neon): Use vreinterpretq_s8_u8 to convert 0xff
into -1 instead of bitwise AND instructions into convert it into 1.
Perform the pairwise addition and lane extraction once every 8192 bytes
instead of once every 64 bytes.
Thanks to Lasse Collin for spotting this and reviewing a draft of this
patch.
2026-03-09 20:06:07 -07:00
Pádraig Brady 0422f0fd4b build: update to latest gnulib
* src/ls.c: Adjust for renamed acl permissions member.
2026-03-09 13:14:54 +00:00
Lukáš Zaoral 2b92c16d26 expand,unexpand: support multi-byte input
* src/expand.c: Use mbbuf to support multi-byte input.
* src/unexpand.c: Likewise.
* tests/expand/mb.sh: New multi-byte test.
* tests/unexpand/mb.sh: Likewise.
* tests/local.mk: Reference new tests.
* NEWS: Mention the improvement.
2026-03-07 12:39:37 +00:00
Weixie Cui 052eed069f maint: shred: fix typo in comment
* src/shred.c: Fix "then" -> "than" in comment.
2026-03-06 20:39:38 -08:00
Weixie Cui e71ed47088 maint: dd: fix typo in comment
* src/dd.c: Fix "that that" -> "that the" in comment.
2026-03-06 13:27:52 +00:00
Collin Funk 90d2b511e4 maint: touch: reduce variable scope
* src/touch.c (main): Declare variables where they are used instead of
at the start of the function.
2026-03-04 23:40:03 -08:00
Collin Funk e2e78f9e8e maint: chown,chgrp: reduce variable scope
* src/chown-core.c (describe_change, restricted_chown)
(change_file_owner, chown_files): Declare variables where they are used
instead of at the start of the function.
* src/chown.c (main): Likewise.
2026-03-04 23:34:45 -08:00
Collin Funk b7ff7e7c2c install: allow the combination of --compare and --preserve-timestamps
* NEWS: Mention the improvement.
* src/install.c (enum copy_status): New type to let the caller know if
the copy was performed or skipped.
(copy_file): Return the new type instead of bool. Reduce variable scope.
(install_file_in_file): Only strip the file if the copy was
performed. Update the timestamps if the copy was skipped.
(main): Don't error when --compare and --preserve-timestamps are
combined.
* tests/install/install-C.sh: Add some test cases.
2026-03-04 18:42:48 -08:00
Pádraig Brady b3fe24213e cksum: use more defensive escaping for --check
cksum --check is often the first interaction
users have with possibly untrusted downloads, so we should try
to be as defensive as possible when processing it.

Specifically we currently only escape \n characters in file names
presented in checksum files being parsed with cksum --check.
This gives some possibilty of dumping arbitrary data to the terminal
when checking downloads from an untrusted source.
This change gives these advantages:

  1. Avoids dumping arbitrary data to vulnerable terminals
  2. Avoids visual deception with ansi codes hiding checksum failures
  3. More secure if users copy and paste file names from --check output
  4. Simplifies programmatic parsing

Note this changes programmatic parsing, but given the original
format was so awkward to parse, I expect that's extremely rare.
I was not able to find example in the wild at least.
To parse the new format from from shell, you can do something like:

  cksum -c checksums | while IFS= read -r line; do
    case $line in
      *': FAILED')
        filename=$(eval "printf '%s' ${line%: FAILED}")
        cp -v "$filename" /quarantine
        ;;
    esac
  done

This change also slightly reduces the size of the sum(1) utility.
This change also apples to md5sum, sha*sum, and b2sum.

* src/cksum.c (digest_check): Call quotef() instead of
cksum(1) specific quoting.
* tests/cksum/md5sum-bsd.sh: Adjust accordingly.
* doc/coreutils.texi (cksum general options): Describe the
shell quoting used for problematic file names.
* NEWS: Mention the change in behavior.
Reported by: Aaron Rainbolt
2026-03-04 22:17:39 +00:00
Paul Eggert 69c0b63bea du: fflush after outputting a line
* src/du.c (print_size): Resurrect the fflush call, since there
can be significant delay between output lines.
2026-02-28 19:13:42 -08:00
Collin Funk 09a5449ff2 du: avoid locking and flushing standard output
This results in a noticeable increase in performance:

    $ yes /dev/null | head -n 10000000 | tr '\n' '\0' \
        | time --format=%E ./src/du-prev -l --files0-from=- > /dev/null
    0:20.40
    $ yes /dev/null | head -n 10000000 | tr '\n' '\0' \
        | time --format=%E ./src/du -l --files0-from=- > /dev/null
    0:16.57

* src/du.c (print_size): Prefer putchar and fputs which may be unlocked
unlike printf. Prefer ferror to fflush.
2026-02-27 22:22:20 -08:00
Paul Eggert d53203b316 stat: handle %%%N too
* src/stat.c (main): Fix incorrect counting of '%'s before 'N'.
* tests/stat/stat-fmt.sh: Test for the bug.
2026-02-27 16:50:36 -08:00
Paul Eggert d41e1375cb id: avoid unnecessary buffer flushing
* src/groups.c (main):
* src/id.c (main, print_stuff):
Don’t flush stdout before testing for write error.
Do the test only when in a loop, as a one-shot will
test for write error soon anyway.
2026-02-27 16:18:32 -08:00
Paul Eggert 02983e4935 cksum: prefer signed int
* src/cksum.c (min_digest_line_length, digest_hex_bytes)
(digest_length, md5_sum_stream, sha1_sum_stream)
(sha224_sum_stream, sha256_sum_stream, sha384_sum_stream)
(sha512_sum_stream, sha2_sum_stream, sha3_sum_stream)
(blake2b_sum_stream, sm3_sum_stream, problematic_chars)
(filename_unescape, valid_digits, bsd_split_3)
(algorithm_from_tag, split_3, digest_file, output_file)
(b64_equal, hex_equal, digest_check, main):
* src/cksum_avx2.c (cksum_avx2):
* src/cksum_avx512.c (cksum_avx512):
* src/cksum_crc.c (cksum_fp_t, cksum_slice8, crc_sum_stream)
(crc32b_sum_stream, output_crc):
* src/cksum_pclmul.c (cksum_pclmul):
* src/cksum_vmull.c (cksum_vmull):
* src/sum.c (bsd_sum_stream, sysv_sum_stream, output_bsd, output_sysv):
Prefer signed to unsigned int where either will do.
This allows better checking with -fsanitize=undefined.
It should also help simplify future patches, so that they
needn’t worry whether comparisons like ‘i < len - 2’ will misbehave.
2026-02-27 16:18:32 -08:00
Collin Funk 803bfa01e1 stat: don't check QUOTING_STYLE when --printf %%N is used
* NEWS: Mention the fix.
* src/stat.c (main): Only check QUOTING_STYLE if there is a %N that is
not preceded by a percentage sign.
* tests/stat/stat-fmt.sh: Add some test cases.
2026-02-26 21:20:37 -08:00
Collin Funk 4195e36664 id: promptly diagnose write errors
* NEWS: Mention the improvement.
* src/id.c (print_stuff): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'id'.
2026-02-26 19:07:21 -08:00
Collin Funk 80b229be71 groups: promptly diagnose write errors
* NEWS: Mention the improvement.
* src/groups.c (main): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'groups'.
2026-02-26 19:05:32 -08:00
Collin Funk 89735ea830 shuf: avoid locking standard output when using --input-range
Here is the throughput before this patch:

    # write_permuted_numbers
    $ ./src/shuf-prev -i 0-100000000 | pv -r > /dev/null
    [ 153MiB/s]

    # write_random_numbers
    $ timeout 10 ./src/shuf-prev -i 0-100000 -r | pv -r > /dev/null
    [78.6MiB/s]

Here is the throughput after this patch:

    # write_permuted_numbers
    $ timeout 10 ./src/shuf -i 0-100000000 | pv -r > /dev/null
    [ 308MiB/s]

    # write_random_numbers
    $ timeout 10 ./src/shuf -i 0-100000 -r | pv -r > /dev/null
    [ 196MiB/s]

* NEWS: Mention the performance improvement.
* src/shuf.c (write_permuted_numbers, write_random_numbers): Prefer
fputs and fputc which may be unlocked over printf which locks standard
output.
2026-02-21 19:20:07 -08:00
Collin Funk 992da4574a maint: printf: prefer static initialization
* src/printf.c (main): Remove unnecessary initialization.
2026-02-19 21:42:35 -08:00
Collin Funk 395fcca795 maint: fmt: prefer static initialization
* src/fmt.c (prefix, max_width): Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:40:44 -08:00
Collin Funk 6bf6d52e54 maint: sort: prefer static initialization
* src/sort.c (main): Remove unnecessary initialization.
2026-02-19 21:37:15 -08:00