Commit Graph
9449 Commits
Author SHA1 Message Date
Collin Funk 9b855166ed tee: prefer file descriptors over streams
We disable buffering on the streams anyways, so we were effectively
calling the write system call previously despite using streams.

* src/iopoll.h (fclose_wait, fwrite_wait): Remove declarations.
(close_wait, write_wait): Add declarations.
* src/iopoll.c (fwait_for_nonblocking_write, fclose_wait, fwrite_wait):
Remove functions.
(wait_for_nonblocking_write): New function based on
fwait_for_nonblocking_write.
(close_wait): New function based on fclose_wait.
(write_wait): New function based on fwrite_wait.
* src/tee.c: Include fcntl--.h. Don't include stdio--.h.
(get_next_out): Operate on file descriptors instead of streams.
(fail_output): Likewise. Remove clearerr call since we no longer call
fwrite on stdout.
(tee_files): Operate on file descriptors instead of streams. Remove
calls to setvbuf.
2026-03-15 15:32:08 -07:00
Collin Funk e644eea122 timeout: don't exit immediately if the parent is the init process
* src/timeout.c (main): Save the process ID before creating a child
process. Check if the result of getppid is different than the saved
process ID instead of checking if it is 1.
* tests/timeout/init-parent.sh: New file.
* tests/local.mk (all_tests): Add the new test.
* NEWS: Mention the bug fix. Also mention that this change allows
'timeout' to work when reparented by a subreaper process instead of
init.
2026-03-13 20:37:10 -07:00
Pádraig Brady 711fa8f9a5 dd: always diagnose partial writes on write failure
* src/dd.c (dd_copy): Increment the partial write count upon failure.
* tests/dd/partial-write.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/80583
2026-03-12 21:19:33 +00:00
Pádraig Brady e016498d6f maint: adjust to placate coverity
* src/system.h (c32issep): Adjust to more standard layout.
2026-03-10 20:14:42 +00:00
Pádraig Brady 2b1c059e6a yes: use a zero-copy implementation via (vm)splice
A good reference for the concepts used here is:
https://mazzo.li/posts/fast-pipes.html
We don't consider huge pages or busy loops here,
but use vmsplice(), and splice() to get significant speedups:

  i7-5600U-laptop $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [4.98GiB/s]
  i7-5600U-laptop $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [34.1GiB/s]

  IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
  ... [11.6GiB/s]
  IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
  ... [175GiB/s]

Also throughput to file (on BTRFS) was seen to increase significantly.
With a Fedora 43 laptop improving from 690MiB/s to 1.1GiB/s.

* bootstrap.conf: Ensure sys/uio.h is present.
This was an existing transitive dependency.
* m4/jm-macros.m4: Define HAVE_SPLICE appropriately.
We assume vmsplice() is available if splice() is as they
were introduced at the same time to Linux and glibc.
* src/yes.c (repeat_pattern): A new function to efficiently
duplicate a pattern in a buffer with memcpy calls that double in size.
This also makes the setup for the existing write() path more efficient.
(pipe_splice_size): A new function to increase the kernel pipe buffer
if possible, and use an appropriately sized buffer based on that (25%).
(splice_write): A new function to call vmplice() when outputting
to a pipe, and also splice() if outputting to a non-pipe.
* tests/misc/yes.sh: Verify the non-pipe output case,
(main): Adjust to always calling write on the minimal buffer first,
then trying vmsplice(), then falling back to write from bigger buffer.
and the vmsplice() fallback to write() case.
* NEWS: Mention the improvement.
2026-03-10 18:13:34 +00:00
Pádraig Brady 5ca27c1929 all: use more consistent blank character determination
* src/system.h (c32issep): A new function that is essentially
iswblank() on GLIBC platforms, and iswspace() with exceptions elsewhere.
* src/expand.c: Use it instead of c32isblank().
* src/fold.c: Likewise.
* src/join.c: Likewise.
* src/numfmt.c: Likewise.
* src/unexpand.c: Likewise.
* src/uniq.c: Likewise.
* NEWS: Mention the improvement.
2026-03-10 16:30:52 +00:00
Pádraig Brady 3ef107fa12 cksum: fix tagged output on 32 bit platforms
Fix an unreleased issue due to the recent change
to using idx_t in commit v9.10-91-g02983e493

* src/cksum.c (output_file): Cast the idx_t before passing to printf.
2026-03-10 16:18:29 +00:00
Collin Funk e0190a9d1b wc: improve aarch64 Neon optimization for 'wc -l'
$ yes abcdefghijklmnopqrstuvwxyz | head -n 200000000 > input
    $ time ./src/wc-prev -l input
    200000000 input

    real	0m1.240s
    user	0m0.456s
    sys	0m0.784s
    $ time ./src/wc -l input
    200000000 input

    real	0m0.936s
    user	0m0.141s
    sys	0m0.795s

* configure.ac: Use unsigned char for the buffer to avoid potential
compiler warnings. Check for the functions being used in src/wc_neon.c
after this patch.
* src/wc_neon.c (wc_lines_neon): Use vreinterpretq_s8_u8 to convert 0xff
into -1 instead of bitwise AND instructions into convert it into 1.
Perform the pairwise addition and lane extraction once every 8192 bytes
instead of once every 64 bytes.
Thanks to Lasse Collin for spotting this and reviewing a draft of this
patch.
2026-03-09 20:06:07 -07:00
Pádraig Brady 0422f0fd4b build: update to latest gnulib
* src/ls.c: Adjust for renamed acl permissions member.
2026-03-09 13:14:54 +00:00
Lukáš Zaoral 2b92c16d26 expand,unexpand: support multi-byte input
* src/expand.c: Use mbbuf to support multi-byte input.
* src/unexpand.c: Likewise.
* tests/expand/mb.sh: New multi-byte test.
* tests/unexpand/mb.sh: Likewise.
* tests/local.mk: Reference new tests.
* NEWS: Mention the improvement.
2026-03-07 12:39:37 +00:00
Weixie Cui 052eed069f maint: shred: fix typo in comment
* src/shred.c: Fix "then" -> "than" in comment.
2026-03-06 20:39:38 -08:00
Weixie Cui e71ed47088 maint: dd: fix typo in comment
* src/dd.c: Fix "that that" -> "that the" in comment.
2026-03-06 13:27:52 +00:00
Collin Funk 90d2b511e4 maint: touch: reduce variable scope
* src/touch.c (main): Declare variables where they are used instead of
at the start of the function.
2026-03-04 23:40:03 -08:00
Collin Funk e2e78f9e8e maint: chown,chgrp: reduce variable scope
* src/chown-core.c (describe_change, restricted_chown)
(change_file_owner, chown_files): Declare variables where they are used
instead of at the start of the function.
* src/chown.c (main): Likewise.
2026-03-04 23:34:45 -08:00
Collin Funk b7ff7e7c2c install: allow the combination of --compare and --preserve-timestamps
* NEWS: Mention the improvement.
* src/install.c (enum copy_status): New type to let the caller know if
the copy was performed or skipped.
(copy_file): Return the new type instead of bool. Reduce variable scope.
(install_file_in_file): Only strip the file if the copy was
performed. Update the timestamps if the copy was skipped.
(main): Don't error when --compare and --preserve-timestamps are
combined.
* tests/install/install-C.sh: Add some test cases.
2026-03-04 18:42:48 -08:00
Pádraig Brady b3fe24213e cksum: use more defensive escaping for --check
cksum --check is often the first interaction
users have with possibly untrusted downloads, so we should try
to be as defensive as possible when processing it.

Specifically we currently only escape \n characters in file names
presented in checksum files being parsed with cksum --check.
This gives some possibilty of dumping arbitrary data to the terminal
when checking downloads from an untrusted source.
This change gives these advantages:

  1. Avoids dumping arbitrary data to vulnerable terminals
  2. Avoids visual deception with ansi codes hiding checksum failures
  3. More secure if users copy and paste file names from --check output
  4. Simplifies programmatic parsing

Note this changes programmatic parsing, but given the original
format was so awkward to parse, I expect that's extremely rare.
I was not able to find example in the wild at least.
To parse the new format from from shell, you can do something like:

  cksum -c checksums | while IFS= read -r line; do
    case $line in
      *': FAILED')
        filename=$(eval "printf '%s' ${line%: FAILED}")
        cp -v "$filename" /quarantine
        ;;
    esac
  done

This change also slightly reduces the size of the sum(1) utility.
This change also apples to md5sum, sha*sum, and b2sum.

* src/cksum.c (digest_check): Call quotef() instead of
cksum(1) specific quoting.
* tests/cksum/md5sum-bsd.sh: Adjust accordingly.
* doc/coreutils.texi (cksum general options): Describe the
shell quoting used for problematic file names.
* NEWS: Mention the change in behavior.
Reported by: Aaron Rainbolt
2026-03-04 22:17:39 +00:00
Paul Eggert 69c0b63bea du: fflush after outputting a line
* src/du.c (print_size): Resurrect the fflush call, since there
can be significant delay between output lines.
2026-02-28 19:13:42 -08:00
Collin Funk 09a5449ff2 du: avoid locking and flushing standard output
This results in a noticeable increase in performance:

    $ yes /dev/null | head -n 10000000 | tr '\n' '\0' \
        | time --format=%E ./src/du-prev -l --files0-from=- > /dev/null
    0:20.40
    $ yes /dev/null | head -n 10000000 | tr '\n' '\0' \
        | time --format=%E ./src/du -l --files0-from=- > /dev/null
    0:16.57

* src/du.c (print_size): Prefer putchar and fputs which may be unlocked
unlike printf. Prefer ferror to fflush.
2026-02-27 22:22:20 -08:00
Paul Eggert d53203b316 stat: handle %%%N too
* src/stat.c (main): Fix incorrect counting of '%'s before 'N'.
* tests/stat/stat-fmt.sh: Test for the bug.
2026-02-27 16:50:36 -08:00
Paul Eggert d41e1375cb id: avoid unnecessary buffer flushing
* src/groups.c (main):
* src/id.c (main, print_stuff):
Don’t flush stdout before testing for write error.
Do the test only when in a loop, as a one-shot will
test for write error soon anyway.
2026-02-27 16:18:32 -08:00
Paul Eggert 02983e4935 cksum: prefer signed int
* src/cksum.c (min_digest_line_length, digest_hex_bytes)
(digest_length, md5_sum_stream, sha1_sum_stream)
(sha224_sum_stream, sha256_sum_stream, sha384_sum_stream)
(sha512_sum_stream, sha2_sum_stream, sha3_sum_stream)
(blake2b_sum_stream, sm3_sum_stream, problematic_chars)
(filename_unescape, valid_digits, bsd_split_3)
(algorithm_from_tag, split_3, digest_file, output_file)
(b64_equal, hex_equal, digest_check, main):
* src/cksum_avx2.c (cksum_avx2):
* src/cksum_avx512.c (cksum_avx512):
* src/cksum_crc.c (cksum_fp_t, cksum_slice8, crc_sum_stream)
(crc32b_sum_stream, output_crc):
* src/cksum_pclmul.c (cksum_pclmul):
* src/cksum_vmull.c (cksum_vmull):
* src/sum.c (bsd_sum_stream, sysv_sum_stream, output_bsd, output_sysv):
Prefer signed to unsigned int where either will do.
This allows better checking with -fsanitize=undefined.
It should also help simplify future patches, so that they
needn’t worry whether comparisons like ‘i < len - 2’ will misbehave.
2026-02-27 16:18:32 -08:00
Collin Funk 803bfa01e1 stat: don't check QUOTING_STYLE when --printf %%N is used
* NEWS: Mention the fix.
* src/stat.c (main): Only check QUOTING_STYLE if there is a %N that is
not preceded by a percentage sign.
* tests/stat/stat-fmt.sh: Add some test cases.
2026-02-26 21:20:37 -08:00
Collin Funk 4195e36664 id: promptly diagnose write errors
* NEWS: Mention the improvement.
* src/id.c (print_stuff): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'id'.
2026-02-26 19:07:21 -08:00
Collin Funk 80b229be71 groups: promptly diagnose write errors
* NEWS: Mention the improvement.
* src/groups.c (main): Call fflush for each listed user to check for
write errors.
* tests/misc/io-errors.sh: Add an invocation of 'groups'.
2026-02-26 19:05:32 -08:00
Collin Funk 89735ea830 shuf: avoid locking standard output when using --input-range
Here is the throughput before this patch:

    # write_permuted_numbers
    $ ./src/shuf-prev -i 0-100000000 | pv -r > /dev/null
    [ 153MiB/s]

    # write_random_numbers
    $ timeout 10 ./src/shuf-prev -i 0-100000 -r | pv -r > /dev/null
    [78.6MiB/s]

Here is the throughput after this patch:

    # write_permuted_numbers
    $ timeout 10 ./src/shuf -i 0-100000000 | pv -r > /dev/null
    [ 308MiB/s]

    # write_random_numbers
    $ timeout 10 ./src/shuf -i 0-100000 -r | pv -r > /dev/null
    [ 196MiB/s]

* NEWS: Mention the performance improvement.
* src/shuf.c (write_permuted_numbers, write_random_numbers): Prefer
fputs and fputc which may be unlocked over printf which locks standard
output.
2026-02-21 19:20:07 -08:00
Collin Funk 992da4574a maint: printf: prefer static initialization
* src/printf.c (main): Remove unnecessary initialization.
2026-02-19 21:42:35 -08:00
Collin Funk 395fcca795 maint: fmt: prefer static initialization
* src/fmt.c (prefix, max_width): Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:40:44 -08:00
Collin Funk 6bf6d52e54 maint: sort: prefer static initialization
* src/sort.c (main): Remove unnecessary initialization.
2026-02-19 21:37:15 -08:00
Collin Funk 96e39300b1 maint: df: prefer static initialization
* src/df.c (human_output_opts, grand_fsu): Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:34:03 -08:00
Collin Funk a24b3075c3 maint: split: prefer static initialization
* src/split.c (outbase, infile): Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:28:40 -08:00
Collin Funk 26c92a62c6 maint: tail: prefer static initialization
* src/tail.c (count_lines, line_end): Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:24:54 -08:00
Collin Funk cc87809aae maint: csplit: prefer static initialization
* src/csplit.c (prefix, remove_files): Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:22:17 -08:00
Collin Funk dc0baaf4de maint: comm: prefer static initialization
* src/comm.c (only_file_1, only_file_2, both): Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:18:41 -08:00
Collin Funk ecf3f98eec maint: wc: prefer static initialization
* src/wc.c (main): Remove unnecessary initializations.
2026-02-19 21:11:27 -08:00
Collin Funk 85dde4c2d9 maint: tac: prefer static initialization
* src/tac.c (separator, separator_ends_record, sentinel_length):
Initialize variables.
(main): Remove unnecessary initializations.
2026-02-19 21:08:43 -08:00
Collin Funk fea1dcadf6 maint: join: prefer static initialization
* src/join.c (print_pairables): Initialize variable.
(main): Remove unnecessary initializations.
2026-02-19 21:05:11 -08:00
Collin Funk 425dd90b88 maint: cut: prefer static initialization
* src/cut.c (main): Remove unnecessary initializations.
2026-02-19 21:01:10 -08:00
Collin Funk 82a20cd6f2 maint: seq: prefer static initialization
* src/seq.c (separator): Initialize variable.
(main): Remove unnecessary initializations.
2026-02-19 20:53:20 -08:00
Collin Funk ef72d0edd2 maint: paste: prefer static initialization
* src/paste.c (main): Remove unnecessary initializations.
2026-02-19 20:49:40 -08:00
Collin Funk 70e4e2d43a maint: install: prefer static initialization
* src/install.c (main): Remove unnecessary initializations.
2026-02-19 20:45:54 -08:00
Collin Funk beb9392206 maint: chmod: prefer static initialization
* src/chmod.c (main): Remove unnecessary initializations.
2026-02-19 20:39:41 -08:00
Collin Funk 5b29c9c04f maint: head: prefer static initialization
* src/head.c (line_end): Initialize variable.
(main): Remove unnecessary initializations.
2026-02-19 20:36:56 -08:00
Collin Funk 97b7213118 maint: ln: prefer static initialization
* src/ln.c (main): Remove unnecessary initializations.
2026-02-19 20:36:47 -08:00
Collin Funk 4bb3309b72 maint: nl: prefer static initialization
* src/nl.c (main): Remove unnecessary initialization.
2026-02-19 20:36:40 -08:00
Collin Funk 1197e54553 maint: touch: prefer static initialization
* src/touch.c (main): Remove unnecessary initializations.
2026-02-19 20:36:31 -08:00
Collin Funk f009f68e3b maint: fold: prefer static initialization
* src/fold.c (main): Remove unnecessary initializations.
2026-02-19 20:36:08 -08:00
Collin Funk a398d52b65 maint: tee: prefer static initialization
* src/tee.c (main): Remove unnecessary initializations.
2026-02-19 20:35:49 -08:00
Collin Funk 66d34782b9 maint: rmdir: prefer static initialization
* src/rmdir.c (main): Remove unnecessary initialization.
2026-02-19 20:35:35 -08:00
Collin Funk 300ef2e63b maint: tty: prefer static initialization
* src/tty.c (main): Remove unnecessary initialization.
2026-02-19 20:35:19 -08:00
Collin Funk 6a3dde5dd2 wc: add aarch64 Neon optimization for wc -l
Here is an example of the performance improvement:

    $ yes abcdefghijklmnopqrstuvwxyz | head -n 100000000 > input
    $ time ./src/wc-prev -l < input
    100000000

    real	0m0.793s
    user	0m0.630s
    sys	0m0.162s
    $ time ./src/wc -l < input
    100000000

    real	0m0.230s
    user	0m0.065s
    sys	0m0.164s

* NEWS: Mention the performance improvement.
* gnulib: Update to the latest commit.
* configure.ac: Check the the necessary intrinsics and functions.
* src/local.mk (noinst_LIBRARIES) [USE_NEON_WC_LINECOUNT]: Add
src/libwc_neon.a.
(src_libwc_neon_a_SOURCES, wc_neon_ldadd, src_libwc_neon_a_CFLAGS)
[USE_NEON_WC_LINECOUNT]: New variables.
(src_wc_LDADD) [USE_NEON_WC_LINECOUNT]: Add $(wc_neon_ldadd).
* src/wc.c [USE_NEON_WC_LINECOUNT]: Include sys/auxv.h and asm/hwcap.h.
(neon_supported) [USE_NEON_WC_LINECOUNT]: New function.
(wc_lines) [USE_NEON_WC_LINECOUNT]: Use neon_supported and
wc_lines_neon.
* src/wc.h (wc_lines_neon): Add declaration.
* src/wc_neon.c: New file.
* doc/coreutils.texi (Hardware Acceleration): Document the "-ASIMD"
hwcap and the variable used in ./configure to override detection of Neon
instructions.
* tests/wc/wc-cpu.sh: Also add "-ASIMD" to disable the use of Neon
instructions.
2026-02-18 20:09:44 -08:00