Commit Graph
31351 Commits
Author SHA1 Message Date
Pádraig Brady 801686242e cut: faster utf8 processing
TODO: improve to use bounded memory where possible
2026-04-05 13:15:56 +01:00
Pádraig Brady a14ac29629 cut: support -F as an alias for -f -w -O ' '
To improve compatibility with toybox/busybox scripts.
2026-04-05 13:15:56 +01:00
Pádraig Brady 74f6692aa0 maint: cut: refactor buffered and ordinary field scanning
* src/cut.c: Merge scan_mb_field and read_mb_field_to_buffer
2026-04-05 13:15:56 +01:00
Pádraig Brady eb1f057746 cut: support --whitespace-delimited=trimmed
Support ignoring leading and trailing whitespace.
E.g. this matches awk's default field splitting mode.

* src/cut.c
* tests/cut/cut.pl: Add test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 77ccacb9a7 cut: support -O as an alias for --output-delimiter
To improve compatibility with toybox/busybox scripts.

* doc/coreutils.texi (cut invocation): Add -O description.
* src/cut.c: Support -O as well as --output-delimiter
* tests/cut/cut.pl: Adjust one case to use -O.
2026-04-05 13:15:56 +01:00
Pádraig Brady 0ae17ffd99 doc: cut: adjust for multi-byte support
* doc/coreutils.texi (cut invocation): Remove the note about
-c being the same as -b.
2026-04-05 13:15:56 +01:00
Pádraig Brady f644b4ca53 cut: refactor multi-byte updates
* src/cut.c: 160 fewer lines

Helpers extracted (replacing repeated inline patterns):
- write_line_delim(), write_pending_line_delim(), reset_item_line()
  - line boundary code used by cut_bytes{,no_split}, cut_characters
- write_selected_item()
  - output-delimiter + write logic used by all three byte/char functions
- reset_field_line()
  - field line reset used by cut_fields_mb_any

Field functions unified via cut_fields_mb_any(stream, whitespace_mode):
- struct mbfield_parser encapsulates the whitespace vs.
  fixed-delimiter state (saved char, mode flag)
- mbfield_get_char() - dispatches to saved-char or direct read
- mbfield_terminator()
  - returns FIELD_{DATA,DELIMETER,LINE_DELIMITER} based on mode
- read_mb_field_to_buffer()
  - replaces the two duplicated first-field buffering loops
- scan_mb_field(mbbuf, parser, pending, write_field)
  - replaces the four duplicated field scan loops
  (print+skip × two modes) with a single function and a write_field bool
- cut_fields_mb and cut_fields_ws are now trivial wrappers
2026-04-05 13:15:56 +01:00
Pádraig Brady 57110d8bae cut: implement -n to avoid outputting partial characters
Both the i18n patch and FreeBSD/macOS support this option.
They do differ in behavior somewhat as the i18n patch
may output more bytes than requested.

  $ printf '\xc3\xa9b\n' | i18n-cut -n -b1
  é

There is also a bug in the i18n patch with multi-byte
at the start of a line:

  $ printf '\xc3\xa9b\n' | i18n-cut -n -b1-2
  éb

We follow the FreeBSD behavior since it seems more
useful to have -b be a hard limit, rather than a soft limit.
This also reduces the possibility of duplicate character output
with separate cut invocations with non overlapping byte ranges.

* src/cut.c (cut_bytes_no_split): A new function
similar to cut_characters, to handle multi-byte characters
with byte limit semantics.
* tests/cut/cut.pl: Add test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady caf1e91266 tests: cut: add a test for divergence from i18n patch
* tests/cut/cut.pl: We don't fall back to byte mode
upon invalid uni-byte delimiter.
2026-04-05 13:15:56 +01:00
Pádraig Brady bed93d46f8 tests: cut: add case currently failing for coreutils-i18n patch
* tests/cut/cut.pl: Test for extraneous character output with:
printf 'aéb\n' | cut -s -d 'é' -f1 | od -tx1
2026-04-05 13:15:56 +01:00
Pádraig Brady 1cdc079860 tests: cut: check multi-byte output delimiter
* tests/cut/cut.pl: Add a test case.
2026-04-05 13:15:56 +01:00
Pádraig Brady 74047ec55e cut: adjust error message to be less specific
* src/cut.c (main): Cater for both misplaced -w and -d.
2026-04-05 13:15:56 +01:00
Pádraig Brady 19aa72b4ea cut: implement -w,--whitespace-delimited
* src/cut.c (cut_fields_ws): A new function handling both
uni-byte and multi-byte cases.
* tests/cut/cut.pl: Add a test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 32f1de5b4f cut: support single byte -d with GB18030 input
* src/cut.c
* tests/cut/mb-non-utf8.sh
* tests/local.mk
2026-04-05 13:15:55 +01:00
Pádraig Brady 94ddf45a60 cut: support single byte -d that may be part of multi-byte
Note this is a slight divergence from the i18n patch
as that switched to uni-byte for any single byte delimiter
that is not valid multi-byte.

That results in possibly splitting in the middle of
a valid multi-byte character.

Instead we only split on a single byte when they're
not part of a multi-byte character.

* src/cut.c
2026-04-05 13:15:55 +01:00
Pádraig Brady a021b0b698 cut: support multi-byte field delimiters
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady 97703386e6 cut: support multi-byte input with -c
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady fb78200249 maint: cut: refactor output calls
* src/cut.c (cut_fields): Refactor calls to fwrite() and putchar()
2026-04-05 13:15:55 +01:00
Pádraig Brady e3c7dc2b03 tests: cut: ensure no unecessary buffering
* tests/misc/write-errors.sh: Ensure we write output when possible.
2026-04-05 13:15:55 +01:00
Pádraig Brady a04a1054f8 doc: cut: reorder --complement alphabetically in help
* src/cut.c (usage): Move placement of --comlement description.
* doc/coreutils.texi (cut invocation): Likewise.
2026-04-05 13:15:55 +01:00
Pádraig Brady 1173ebb7c8 doc: cut: clarify description of -b and -c
* src/cut.c (usage): State the arguments are positions,
in case users may think they were values.
2026-04-05 13:15:55 +01:00
Pádraig Brady 1204b29bab build: update to latest gnulib
Pick up mbrto{c32,wc} optimizations on UTF-8 on GLIBC.
Note configure.ac defines the required GNULIB_WCHAR_SINGLE_LOCALE.
This speeds up wc -m by 2.6x, when processing non ASCII chars,
and will similarly speed up per character processing
in the impending cut multi-byte implementation.
* NEWS: Mention the wc -m speed improvement.
2026-04-05 13:11:21 +01:00
Collin Funk 38fc6bde64 basename: avoid duplicate strlen calls on the suffix
$ ltrace -c ./src/basename-prev -s a $(seq 100000) > /dev/null
    % time     seconds  usecs/call     calls      function
    ------ ----------- ----------- --------- --------------------
     50.00   30.030316          75    400000 strlen
    [...]
    $ ltrace -c ./src/basename -s a $(seq 100000) > /dev/null
    % time     seconds  usecs/call     calls      function
    ------ ----------- ----------- --------- --------------------
     42.88   22.413953          74    300001 strlen
    [...]

* src/basename.c (remove_suffix, perform_basename): Add a length
argument for the suffix and use it instead of strlen.
(main): Calculate the suffix length. Refactor code to avoid calling
perform_basename in multiple places.
2026-04-04 14:55:56 -07:00
Paul Eggert b64f9bfe4f date: simplify -u by not calling putenv
* src/date.c (TZSET): Remove; no longer needed.
(main): Simplify -u’s implementation by passing "UTC0" to tzalloc,
rather than by setting TZ in the environment and then calling getenv.
The old way of doing things dates back to before we had tzalloc.
* configure.ac (LOCALTIME_CACHE): Remove; no longer needed.
2026-04-02 18:54:35 -07:00
Paul Eggert bb51268465 build: update gnulib submodule to latest 2026-04-01 14:48:25 -07:00
Paul Eggert 3fb7dc8e28 maint: avoid sigaction lock overhead
* configure.ac (GNULIB_SIGACTION_SINGLE_THREAD):
Define to avoid unnecessary locking in Gnulib sigaction.  See:
https://lists.gnu.org/r/bug-gnulib/2026-04/msg00008.html
2026-04-01 14:48:25 -07:00
Paul Eggert 8a6cb56817 maint: avoid Gnulib modules mbiter, mbiterf
* bootstrap.conf (avoided_gnulib_modules): Avoid mbiter and
mbiterf, for the same reason we avoid mbuiter and mbuiterf: these
modules are not needed because (due to mcel-prefer) we use mcel in
preference to mbiter/mbiterf/mbuiter/mbuiterf.
2026-04-01 12:32:47 -07:00
Paul Eggert afe3ce9cd6 build: update gnulib submodule to latest 2026-04-01 12:32:47 -07:00
oech3 3558839bdd tests: dd: ensure memory exhaustion is handled gracefully
* tests/dd/no-allocate.sh: Ensure we exit 1 upon mem allocation failure.
Also check other buffer size edge cases.
https://github.com/uutils/coreutils/issues/11436
https://github.com/uutils/coreutils/issues/11580
https://github.com/coreutils/coreutils/pull/235
2026-04-01 16:54:12 +01:00
Pádraig Brady 178c48154d tests: dd: avoid false failure with no controlling terminal
* tests/dd/misc.sh: test -w /dev/tty is not a strong enough check,
we need to actually open /dev/tty to ensure it's available.
It's not available under setsid for example.
2026-04-01 13:31:57 +01:00
oech3 ee5092971e tests: dd: check that erroneous seeks are not done in output
* tests/dd/misc.sh: Add test case for of=/dev/tty.
The same occurs for /dev/stdout, but that varies
in the test hardness so is best avoided.
https://github.com/coreutils/coreutils/pull/234
2026-03-31 12:09:26 +01:00
oech3 368bfc7cb0 tests: coreutils: ensure empty arg is diagnosed
* tests/misc/coreutils.sh: Add a test case.
https://github.com/coreutils/coreutils/pull/232
2026-03-30 13:43:58 +01:00
Collin Funk 4cd0644472 date: avoid calling putenv multiple times unnecessarily
Adding environment variables can become quite expensive in some
admittedly unlikely situations.

    $ for i in $(seq 10000); do export A$i=A$i; done
    $ time ./src/date-prev -u $(yes -- -u | head -n 100000)
    Sun Mar 29 01:59:49 AM UTC 2026

    real	0m3.753s
    user	0m3.684s
    sys	0m0.050s
    $ time ./src/date -u $(yes -- -u | head -n 100000)
    Sun Mar 29 02:00:00 AM UTC 2026

    real	0m0.061s
    user	0m0.022s
    sys	0m0.045s

* src/date.c (main): Only add TZ=UTC0 to the environment once.
2026-03-28 18:57:49 -07:00
Collin Funk 753fa26918 maint: remove unnecessary return statements
* src/env.c (initialize_signals): Remove return at the end of the
function.
* src/who.c (print_runlevel): Likewise.
2026-03-28 12:48:38 -07:00
Collin Funk eec7b2a131 who: avoid locking standard output for each user with the -q option
* src/who (list_entries_who): Prefer putchar and fputs to printf.
Simplify separator tracking.
2026-03-28 12:45:14 -07:00
Collin Funk c8b296683f doc: tty: mention the removal of the -s option from POSIX
* doc/coreutils.texi (tty invocation): Mention that POSIX.1-2001 removed
the -s option and that portable scripts can redirect standard out to
/dev/null instead.
2026-03-27 22:45:56 -07:00
oech3 fcccbab0fb tests: env/env.sh: improve portability
* tests/env/env.sh: Make more portable by avoiding references to our
build dir,  and avoiding names that may cause false matches in
multi-call binaries.
https://github.com/coreutils/coreutils/pull/216
2026-03-27 21:15:18 +00:00
Pádraig Brady 0ec2340c08 od: suppress address output on read error
We don't output an address for `od missing` or `od --strings .`,
so be consistent and suppress the address for `od .`.

* src/od.c (dump): Only output an address if no errors
or the offset is non zero.
2026-03-25 17:46:37 +00:00
oech3 cc3dd92f84 tests: od: ensure -j1 /dev/null succeeds
Users may be using this to convert bases.

* tests/od/od-j.sh: Add a test case.
https://github.com/coreutils/coreutils/pull/228
2026-03-25 17:46:16 +00:00
Collin Funk 7b4bbaa074 tests: truncate: don't rely on errno being EISDIR
* tests/truncate/multiple-files.sh: Only check that an error is printed
instead of an exact message.
Reported by Bruno Haible.
2026-03-24 19:02:21 -07:00
oech3 c495cbcfe2 tests: yes: support more zero-copy related syscalls
* tests/misc/yes.sh: Disable other related zero-copy syscalls
to ensure better testing of future or other implementations.
https://github.com/coreutils/coreutils/pull/227
2026-03-24 20:35:01 +00:00
Collin Funk 430822d5f7 maint: remove some unnecessary casts
* src/sort.c (begfield, limfield): Remove size_t casts.
2026-03-23 19:32:21 -07:00
Sylvestre Ledru 4fd2de166a tests: cut: add test for -z with NUL delimiter and -s flag
* tests/cut/cut.pl (zerot-7): New test.
Identified https://github.com/uutils/coreutils/pull/11394
https://github.com/coreutils/coreutils/pull/226
2026-03-23 12:11:49 +00:00
Sylvestre Ledru 939def75f3 tests: tr: add test for invalid character class name
* tests/tr/tr.pl (invalid-class): New test.
Identified : https://github.com/uutils/coreutils/pull/11398
https://github.com/coreutils/coreutils/pull/225
2026-03-23 12:11:35 +00:00
Chris Down 67b8aaae8d sort: speed up keyed field sorting significantly using memchr
When sort is invoked with an explicit field separator with `-t SEP`,
begfield() and limfield() scan for the separator to locate boundaries.
Right now the implementation there uses a loop that iterates over bytes
one by one, which is not ideal since we must scan past many bytes of
non-separator data one byte at a time.

Let's replace each of these loops with memchr(). On glibc systems,
memchr() uses SIMD to scan 16 bytes per step (NEON on aarch64) or 32
bytes per step (AVX2 on x86_64), rather than 1 byte at a time, so any
field longer than a handful of bytes stands to benefit quite
significantly.

Using the following input data:

  awk 'BEGIN {
      srand(42)
      for (i = 1; i <= 500000; i++)
          printf "%*d,%*d,%d\n", 4+int(rand()*9), 0,
                                 4+int(rand()*9), 0, int(rand()*10000)
  }' > short_csv_500k

  awk 'BEGIN {
      for (i = 1; i <= 500000; i++)
          printf "%100d,%100d,%d\n", 0, 0, int(rand()*10000)
  }' > wide_csv_500k

One can benchmark with:

  hyperfine --warmup 10 --runs 50 \
    "LC_ALL=C sort_before -t, -k3,3n short_csv_500k > /dev/null" \
    "LC_ALL=C sort_after -t, -k3,3n short_csv_500k > /dev/null"

  hyperfine --warmup 10 --runs 50 \
    "LC_ALL=C sort_before -t, -k3,3n wide_csv_500k > /dev/null" \
    "LC_ALL=C sort_after -t, -k3,3n wide_csv_500k > /dev/null"

  hyperfine --warmup 10 --runs 50 \
    "LC_ALL=C sort_before wide_csv_500k > /dev/null" \
    "LC_ALL=C sort_after wide_csv_500k > /dev/null"

The results on i9-14900HX x86_64 with -O2:

  sort -t, -k3,3n (500K lines, 4-12 byte short fields):
    Before: 123.1 ms    After: 108.1 ms    (-12.2%)

  sort -t, -k3,3n (500K lines, 100 byte wide fields):
    Before: 243.5 ms    After: 165.9 ms    (-31.9%)

  sort (default, no -k, 500K lines):
    Before: 141.6 ms    After: 141.8 ms    (unchanged)

And on M1 Pro aarch64 with -O2:

  sort -t, -k3,3n (500K lines, 4-12 byte short fields):
    Before: 98.0 ms     After: 92.3 ms     (-5.8%)

  sort -t, -k3,3n (500K lines, 100 byte wide fields):
    Before: 240.8 ms    After: 183.0 ms    (-24.0%)

  sort (default, no -k, 500K lines):
    Before: 145.6 ms    After: 145.6 ms    (unchanged)

Looking at profiling, the improvement is larger on x86_64 in these runs
because glibc's memchr uses AVX2 to scan 32 bytes per step versus 16
bytes per step with NEON on aarch64.
2026-03-23 11:49:02 +00:00
Collin Funk 26b5e3d3cf maint: fix an incomplete sentence
* tests/pwd/argument.sh: Fix the test description.
Reported by G. Branden Robinson.
2026-03-22 13:32:52 -07:00
Collin Funk 35a1c88f48 tests: pwd: test the behavior when given an argument
* tests/pwd/argument.sh: New file.
* tests/local.mk (all_tests): Add the new test.
2026-03-21 22:16:59 -07:00
Collin Funk 4bc336b7ce tac: avoid unnecessary standard output buffering
This has removes a tiny amount of overhead:

    $ seq 10000000 > input
    $ perf stat -e cpu-clock --repeat 1000 taskset 1 ./src/tac \
        input 2>&1 > /dev/null | grep -F 'seconds time'
              0.095707 +- 0.000223 seconds time elapsed  ( +-  0.23% )
    $ perf stat -e cpu-clock --repeat 1000 taskset 1 ./src/tac-prev \
        input 2>&1 > /dev/null | grep -F 'seconds time'
             0.1009378 +- 0.0000995 seconds time elapsed  ( +-  0.10% )

* src/tac.c (output): Use full_write instead of fread since we already
buffer the output ourselves.
2026-03-21 15:50:33 -07:00
Collin Funk 6a37187a5f tests: rm: fix a test that would sometimes hang
* tests/rm/dash-hint.sh: Add the file name argument to grep, as I
intended when adding this test.
2026-03-21 12:19:21 -07:00
Collin Funk 30a5cbec0e tac: promptly diagnose write errors
This patch also fixes a bug where 'tac' would print a vague error on
some inputs:

    $ seq 10000 | ./src/tac-prev > /dev/full
    tac-prev: write error
    $ seq 10000 | ./src/tac > /dev/full
    tac: write error: No space left on device

In this case ferror (stdout) is true, but errno has been set back to
zero by a successful fclose (stdout) call.

* src/tac.c (output): Call write_error() if fwrite fails.
* tests/misc/io-errors.sh: Check that 'tac' prints a detailed write
error.
* NEWS: Mention the improvement.
2026-03-21 12:10:59 -07:00