Support ignoring leading and trailing whitespace.
E.g. this matches awk's default field splitting mode.
* src/cut.c
* tests/cut/cut.pl: Add test cases.
To improve compatibility with toybox/busybox scripts.
* doc/coreutils.texi (cut invocation): Add -O description.
* src/cut.c: Support -O as well as --output-delimiter
* tests/cut/cut.pl: Adjust one case to use -O.
* src/cut.c: 160 fewer lines
Helpers extracted (replacing repeated inline patterns):
- write_line_delim(), write_pending_line_delim(), reset_item_line()
- line boundary code used by cut_bytes{,no_split}, cut_characters
- write_selected_item()
- output-delimiter + write logic used by all three byte/char functions
- reset_field_line()
- field line reset used by cut_fields_mb_any
Field functions unified via cut_fields_mb_any(stream, whitespace_mode):
- struct mbfield_parser encapsulates the whitespace vs.
fixed-delimiter state (saved char, mode flag)
- mbfield_get_char() - dispatches to saved-char or direct read
- mbfield_terminator()
- returns FIELD_{DATA,DELIMETER,LINE_DELIMITER} based on mode
- read_mb_field_to_buffer()
- replaces the two duplicated first-field buffering loops
- scan_mb_field(mbbuf, parser, pending, write_field)
- replaces the four duplicated field scan loops
(print+skip × two modes) with a single function and a write_field bool
- cut_fields_mb and cut_fields_ws are now trivial wrappers
Both the i18n patch and FreeBSD/macOS support this option.
They do differ in behavior somewhat as the i18n patch
may output more bytes than requested.
$ printf '\xc3\xa9b\n' | i18n-cut -n -b1
é
There is also a bug in the i18n patch with multi-byte
at the start of a line:
$ printf '\xc3\xa9b\n' | i18n-cut -n -b1-2
éb
We follow the FreeBSD behavior since it seems more
useful to have -b be a hard limit, rather than a soft limit.
This also reduces the possibility of duplicate character output
with separate cut invocations with non overlapping byte ranges.
* src/cut.c (cut_bytes_no_split): A new function
similar to cut_characters, to handle multi-byte characters
with byte limit semantics.
* tests/cut/cut.pl: Add test cases.
Note this is a slight divergence from the i18n patch
as that switched to uni-byte for any single byte delimiter
that is not valid multi-byte.
That results in possibly splitting in the middle of
a valid multi-byte character.
Instead we only split on a single byte when they're
not part of a multi-byte character.
* src/cut.c
Pick up mbrto{c32,wc} optimizations on UTF-8 on GLIBC.
Note configure.ac defines the required GNULIB_WCHAR_SINGLE_LOCALE.
This speeds up wc -m by 2.6x, when processing non ASCII chars,
and will similarly speed up per character processing
in the impending cut multi-byte implementation.
* NEWS: Mention the wc -m speed improvement.
* src/date.c (TZSET): Remove; no longer needed.
(main): Simplify -u’s implementation by passing "UTC0" to tzalloc,
rather than by setting TZ in the environment and then calling getenv.
The old way of doing things dates back to before we had tzalloc.
* configure.ac (LOCALTIME_CACHE): Remove; no longer needed.
* bootstrap.conf (avoided_gnulib_modules): Avoid mbiter and
mbiterf, for the same reason we avoid mbuiter and mbuiterf: these
modules are not needed because (due to mcel-prefer) we use mcel in
preference to mbiter/mbiterf/mbuiter/mbuiterf.
* tests/dd/misc.sh: test -w /dev/tty is not a strong enough check,
we need to actually open /dev/tty to ensure it's available.
It's not available under setsid for example.
* tests/dd/misc.sh: Add test case for of=/dev/tty.
The same occurs for /dev/stdout, but that varies
in the test hardness so is best avoided.
https://github.com/coreutils/coreutils/pull/234
Adding environment variables can become quite expensive in some
admittedly unlikely situations.
$ for i in $(seq 10000); do export A$i=A$i; done
$ time ./src/date-prev -u $(yes -- -u | head -n 100000)
Sun Mar 29 01:59:49 AM UTC 2026
real 0m3.753s
user 0m3.684s
sys 0m0.050s
$ time ./src/date -u $(yes -- -u | head -n 100000)
Sun Mar 29 02:00:00 AM UTC 2026
real 0m0.061s
user 0m0.022s
sys 0m0.045s
* src/date.c (main): Only add TZ=UTC0 to the environment once.
* doc/coreutils.texi (tty invocation): Mention that POSIX.1-2001 removed
the -s option and that portable scripts can redirect standard out to
/dev/null instead.
* tests/env/env.sh: Make more portable by avoiding references to our
build dir, and avoiding names that may cause false matches in
multi-call binaries.
https://github.com/coreutils/coreutils/pull/216
We don't output an address for `od missing` or `od --strings .`,
so be consistent and suppress the address for `od .`.
* src/od.c (dump): Only output an address if no errors
or the offset is non zero.
When sort is invoked with an explicit field separator with `-t SEP`,
begfield() and limfield() scan for the separator to locate boundaries.
Right now the implementation there uses a loop that iterates over bytes
one by one, which is not ideal since we must scan past many bytes of
non-separator data one byte at a time.
Let's replace each of these loops with memchr(). On glibc systems,
memchr() uses SIMD to scan 16 bytes per step (NEON on aarch64) or 32
bytes per step (AVX2 on x86_64), rather than 1 byte at a time, so any
field longer than a handful of bytes stands to benefit quite
significantly.
Using the following input data:
awk 'BEGIN {
srand(42)
for (i = 1; i <= 500000; i++)
printf "%*d,%*d,%d\n", 4+int(rand()*9), 0,
4+int(rand()*9), 0, int(rand()*10000)
}' > short_csv_500k
awk 'BEGIN {
for (i = 1; i <= 500000; i++)
printf "%100d,%100d,%d\n", 0, 0, int(rand()*10000)
}' > wide_csv_500k
One can benchmark with:
hyperfine --warmup 10 --runs 50 \
"LC_ALL=C sort_before -t, -k3,3n short_csv_500k > /dev/null" \
"LC_ALL=C sort_after -t, -k3,3n short_csv_500k > /dev/null"
hyperfine --warmup 10 --runs 50 \
"LC_ALL=C sort_before -t, -k3,3n wide_csv_500k > /dev/null" \
"LC_ALL=C sort_after -t, -k3,3n wide_csv_500k > /dev/null"
hyperfine --warmup 10 --runs 50 \
"LC_ALL=C sort_before wide_csv_500k > /dev/null" \
"LC_ALL=C sort_after wide_csv_500k > /dev/null"
The results on i9-14900HX x86_64 with -O2:
sort -t, -k3,3n (500K lines, 4-12 byte short fields):
Before: 123.1 ms After: 108.1 ms (-12.2%)
sort -t, -k3,3n (500K lines, 100 byte wide fields):
Before: 243.5 ms After: 165.9 ms (-31.9%)
sort (default, no -k, 500K lines):
Before: 141.6 ms After: 141.8 ms (unchanged)
And on M1 Pro aarch64 with -O2:
sort -t, -k3,3n (500K lines, 4-12 byte short fields):
Before: 98.0 ms After: 92.3 ms (-5.8%)
sort -t, -k3,3n (500K lines, 100 byte wide fields):
Before: 240.8 ms After: 183.0 ms (-24.0%)
sort (default, no -k, 500K lines):
Before: 145.6 ms After: 145.6 ms (unchanged)
Looking at profiling, the improvement is larger on x86_64 in these runs
because glibc's memchr uses AVX2 to scan 32 bytes per step versus 16
bytes per step with NEON on aarch64.
This patch also fixes a bug where 'tac' would print a vague error on
some inputs:
$ seq 10000 | ./src/tac-prev > /dev/full
tac-prev: write error
$ seq 10000 | ./src/tac > /dev/full
tac: write error: No space left on device
In this case ferror (stdout) is true, but errno has been set back to
zero by a successful fclose (stdout) call.
* src/tac.c (output): Call write_error() if fwrite fails.
* tests/misc/io-errors.sh: Check that 'tac' prints a detailed write
error.
* NEWS: Mention the improvement.