This change is performance neutral as tested with:
$ yes $(yes eeeaae | head -n10K | paste -s -d,) | head -n10K > ll.in
$ hyperfine src/cut{,.orig}' -f10 -w ll.in >/dev/null'
* gl/lib/mbbuf.h (mbbuf_fill): Repeat the mbbuf_topup(), i.e. the read()
if the previous run didn't read enough. Note we add the explicit
common case short-circuit to avoid a 20% regression
for the test case above, with GCC 16.2 with -O3 -flto.
* tests/cut/mb-short-read.sh: A new test file.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Discovered by Anthropic's OSS Scanner, as bug ANT-2026-JSCMJCN2.
These are the most common input encodings so worth optimizing for.
Performance was tested with GCC 16 on an i7-5600U with
CFLAGS='-march=native -O3 -flto' with the following setup:
$ yes | head -n10M > sl.in
$ yes $(yes eeeaae | head -n10K | paste -s -d' ') | head -n10K > llw.in
$ yes $(yes eeeaae | head -n9 | paste -s -d' ') | head -n1M > asw.in
$ yes $(yes éééááé | head -n9 | paste -s -d$'\xe2\x80\x83') |
head -n1M > mbw.in
for type in sl llw asw mbw; do
cat $type.in >/dev/null;
for imp in '-before' '-after'; do
echo ============ "${imp:-base}" $type ==============;
for d in -c; do
fields='-f1 -f10 -f100'
test "$d" = "-c" && { fields='-c1 -c10 -c100000'; d=''; }
for f in $fields; do
for loc in C.UTF-8; do
# Skip -b for UTF-8 as no different
test "$loc" = C.UTF-8 && echo "$f" | grep -q -- -b \
&& continue
# Skip multi-byte delimiter for C and not allowed
test "$loc" = C && test $(echo -n "$d" | wc -c) -ge 4 \
&& continue
bin=src/cut${imp}
LC_ALL=$loc $bin $f $d /dev/null 2>/dev/null &&
hyperfine --warmup=2 -m2 -M6 \
"LC_ALL=$loc $bin $f $d $type.in >/dev/null" ||
printf 'Benchmark 1: %s\n unsupported\n\n' \
"LC_ALL=$loc $bin $f $d $type.in >/dev/null"
done;
done;
done;
done;
done
After a little post-processing of the results, we get:
-- src/cut-before
| command | sl | llw | asw | mbw |
| --------------- | -------- | -------- | -------- | -------- |
| UTF8 -c1 | 70.7 ms | 103.5 ms | 26.9 ms | 48.6 ms |
| UTF8 -c10 | 55.1 ms | 103.9 ms | 48.6 ms | 127.3 ms |
| UTF8 -c100000 | 54.7 ms | 1.585 s | 156.9 ms | 555.9 ms |
-- src/cut-after
| command | sl | llw | asw | mbw |
| --------------- | -------- | -------- | -------- | -------- |
| UTF8 -c1 | 90.1 ms | 103.6 ms | 28.0 ms | 50.9 ms |
| UTF8 -c10 | 39.7 ms | 103.2 ms | 40.0 ms | 127.5 ms |
| UTF8 -c100000 | 39.0 ms | 104.4 ms | 18.5 ms | 30.3 ms |
* bootstrap.conf: Explicitly depend on unistr/u8-check.
* src/cut.c (cut_characters_mode): Quickly skip over unselected
ASCII/UTF-8 data.
* tests/cut/cut.pl: Add test cases.
Performance was tested with GCC 16 on an i7-5600U with
CFLAGS='-march=native -O3 -flto' with the following setup:
$ yes | head -n10M > sl.in
$ yes $(yes eeeaae | head -n10K | paste -s -d' ') | head -n10K > llw.in
$ yes $(yes eeeaae | head -n9 | paste -s -d' ') | head -n1M > asw.in
$ yes $(yes éééááé | head -n9 | paste -s -d$'\xe2\x80\x83') |
head -n1M > mbw.in
for type in sl llw asw mbw; do
cat $type.in >/dev/null;
for imp in '-before' '-after'; do
echo ============ "${imp:-base}" $type ==============;
for d in -c; do
fields='-f1 -f10 -f100'
test "$d" = "-c" && { fields='-c1 -c10 -c100000'; d=''; }
for f in $fields; do
for loc in C.UTF-8; do
# Skip -b for UTF-8 as no different
test "$loc" = C.UTF-8 && echo "$f" | grep -q -- -b \
&& continue
# Skip multi-byte delimiter for C and not allowed
test "$loc" = C && test $(echo -n "$d" | wc -c) -ge 4 \
&& continue
bin=src/cut${imp}
LC_ALL=$loc $bin $f $d /dev/null 2>/dev/null &&
hyperfine --warmup=2 -m2 -M6 \
"LC_ALL=$loc $bin $f $d $type.in >/dev/null" ||
printf 'Benchmark 1: %s\n unsupported\n\n' \
"LC_ALL=$loc $bin $f $d $type.in >/dev/null"
done;
done;
done;
done;
done
After a little post-processing of the results, we get:
-- src/cut-before
| command | sl | llw | asw | mbw |
| ------------- | -------- | -------- | -------- | -------- |
| UTF8 -c1 | 60.7 ms | 1.244 s | 123.5 ms | 515.2 ms |
| UTF8 -c10 | 39.3 ms | 1.248 s | 127.4 ms | 526.0 ms |
| UTF8 -c100000 | 39.4 ms | 1.242 s | 121.4 ms | 514.3 ms |
-- src/cut-after
| command | sl | llw | asw | mbw |
| ------------- | -------- | -------- | -------- | -------- |
| UTF8 -c1 | 70.9 ms | 105.5 ms | 27.0 ms | 47.6 ms |
| UTF8 -c10 | 55.7 ms | 105.7 ms | 48.2 ms | 126.8 ms |
| UTF8 -c100000 | 55.2 ms | 1.596 s | 158.8 ms | 555.1 ms |
* src/cut.c (cut_characters_mode): Shortcut the line delimiter
search if finished processing characters.
* tests/cut/cut.pl: Add a test case.
There were no significant performance changes from the test cases
in commit a325c9978, but adding additional whitespace delimited cases
shows significant improvement. Below, with -f1, typical space delimited
input shows 4x more throughput, while the long line test shows 17x more.
Performance was tested with GCC 16 on an i7-5600U with
CFLAGS='-march=native -O3 -flto' with the following setup:
$ yes | head -n10M > sl.in
$ yes $(yes eeeaae | head -n10K | paste -s -d' ') | head -n10K > llw.in
$ yes $(yes eeeaae | head -n9 | paste -s -d' ') | head -n1M > asw.in
$ yes $(yes éééááé | head -n9 | paste -s -d$'\xe2\x80\x83') |
head -n1M > mbw.in
for type in sl llw asw mbw; do
cat $type.in >/dev/null;
for imp in '-before' '-after'; do
echo ============ "${imp:-base}" $type ==============;
for d in -w; do
fields='-f1 -f10 -f100'
test "$d" = "-b" && { fields='-b1 -b10 -b100'; d=''; }
test "$d" = "-c" && { fields='-c1 -c10 -c100'; d=''; }
for f in $fields; do
for loc in C.UTF-8; do
# Skip -b for UTF-8 as no different
test "$loc" = C.UTF-8 && echo "$f" | grep -q -- -b \
&& continue
# Skip multi-byte delimiter for C and not allowed
test "$loc" = C && test $(echo -n "$d" | wc -c) -ge 4 \
&& continue
bin=src/cut${imp}
LC_ALL=$loc $bin $f $d /dev/null 2>/dev/null &&
hyperfine --warmup=2 -m2 -M6 \
"LC_ALL=$loc $bin $f $d $type.in >/dev/null" ||
printf 'Benchmark 1: %s\n unsupported\n\n' \
"LC_ALL=$loc $bin $f $d $type.in >/dev/null"
done;
done;
done;
done;
done
After a little post-processing of the results, we get:
-- src/cut-before
| command | sl | llw | asw | mbw |
| --------------- | -------- | -------- | -------- | -------- |
| UTF8 -f1 -w | 96.2 ms | 1.916 s | 190.6 ms | 906.5 ms |
| UTF8 -f10 -w | 104.6 ms | 1.921 s | 187.7 ms | 845.6 ms |
| UTF8 -f100 -w | 104.4 ms | 1.916 s | 187.4 ms | 846.4 ms |
-- src/cut-after
| command | sl | llw | asw | mbw |
| --------------- | -------- | -------- | -------- | -------- |
| UTF8 -f1 -w | 88.7 ms | 112.4 ms | 43.5 ms | 146.3 ms |
| UTF8 -f10 -w | 95.2 ms | 114.4 ms | 193.5 ms | 848.4 ms |
| UTF8 -f100 -w | 94.8 ms | 130.6 ms | 194.4 ms | 848.1 ms |
* src/cut.c (scan_mb_line_end): A new function to skip
to EOL once we have exhausted the field selection.
(cut_fields_mb_any): Only skip after recognizing a field delimiter.
* tests/cut/cut.pl: Add test cases.
* tests/cut/mb-non-utf8.sh: Likewise.
* NEWS: Mention the improvement.
* tests/cut/cut.pl: Add cases verifying that in the C locale -c and
-b -n select a single byte instead of decoding a multi-byte sequence.
Link: https://github.com/coreutils/coreutils/pull/316
* src/cut.c (delim_bytes): Add a spare byte so a maximum-length
multi-byte delimiter stays NUL terminated for the strstr needle in
find_field_delim. Note in 9.11 this buffer was larger (16 bytes)
and so didn't not have a buffer over-read issue.
(main): Ensure DELIM_BYTES is NUL terminated after each --delimeter.
Also explicitly writing the NUL allows for better testing by
triggering overflow issues in default builds, if the buffer
is not sized correctly.
* NEWS: Mention the bug fix.
* tests/cut/cut.pl: Add test cases.
Reported at https://github.com/coreutils/coreutils/pull/286
As described in commit v9.10-283-g5cb0cca3b
avoid using ulimit with $SHELL -c, and explict
invocation of the shell like that can require lots of memory,
often due to loading the locale archive.
* tests/basenc/bounded-memory.sh: Avoid $SHELL -c with ulimit.
* tests/cut/bounded-memory.sh: Likewise.
* tests/expand/bounded-memory.sh Likewise.
* tests/pr/bounded-memory.sh Likewise.
* tests/unexpand/bounded-memory.sh Likewise.
* tests/fold/fold-zero-width.sh: Bring the ulimit bound
down to the standard 6M over the determined limit.
With field delimiter = line delimiter we need to know
if there is any more data to be read, as field delimiter
in the last byte of the file is treated differently.
So reiterate the loop to ensure enough read()s to make
the appropriate determination.
* src/cut.c (bytesearch_field_delim_ok): Expand the range
of bytes that can be simply searched for. 0xF5-0xFF can't
appear in valid UTF-8 characters, and so may be used as
delimiters in UTF-8 input, so it's worth optimizing for.
* tests/cut/cut.pl: Add a test case (mainly as documentation).
This is about 20x faster.
Note we only do the delimiter search once per chunk,
and it's usually quick as delimiters wouldn't be too far
into the a chunk if present, so we don't bother
to cache the found delimiter.
Support ignoring leading and trailing whitespace.
E.g. this matches awk's default field splitting mode.
* src/cut.c
* tests/cut/cut.pl: Add test cases.
To improve compatibility with toybox/busybox scripts.
* doc/coreutils.texi (cut invocation): Add -O description.
* src/cut.c: Support -O as well as --output-delimiter
* tests/cut/cut.pl: Adjust one case to use -O.
Both the i18n patch and FreeBSD/macOS support this option.
They do differ in behavior somewhat as the i18n patch
may output more bytes than requested.
$ printf '\xc3\xa9b\n' | i18n-cut -n -b1
é
There is also a bug in the i18n patch with multi-byte
at the start of a line:
$ printf '\xc3\xa9b\n' | i18n-cut -n -b1-2
éb
We follow the FreeBSD behavior since it seems more
useful to have -b be a hard limit, rather than a soft limit.
This also reduces the possibility of duplicate character output
with separate cut invocations with non overlapping byte ranges.
* src/cut.c (cut_bytes_no_split): A new function
similar to cut_characters, to handle multi-byte characters
with byte limit semantics.
* tests/cut/cut.pl: Add test cases.
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
ulimit -v is generally not supported with ASAN, giving errors like:
"ReserveShadowMemoryRange failed while trying to map 0x... bytes.
Perhaps you're using ulimit -v"
* tests/cp/link-heap.sh: Mention ASAN as a possible reason for skipping.
* tests/csplit/csplit-heap.sh: Likewise.
* tests/cut/cut-huge-range.sh: Likewise.
* tests/dd/no-allocate.sh: Likewise.
* tests/printf/printf-surprise.sh: Likewise.
* tests/rm/many-dir-entries-vs-OOM.sh: Likewise.
* tests/head/head-c.sh: Only skip the part of the test needing ulimit.
* tests/split/line-bytes.sh: Likewise.
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
Also, diagnose the '-' in "cut -f -" as an invalid range, rather
than interpreting it as the unlimited range, "1-".
* NEWS: Mention these changes.
* src/cut.c (set_fields): Don't interpret an accumulator "value"
of 0 as an unspecified range endpoint.
Give better diagnostics.
Adjust a comment so that it is true also for 64-bit size_t.
* tests/cut/Test.pm: Add tests for the above.
* Transform all Makefile.am files so that when running "make check",
CU_TEST_NAME is set to the name of the test. This is so that when I
run valgrind-enabled (--log-file-qualifier=CU_TEST_NAME) "make check"
on the entire package it is more convenient to map a leak or error
found in a valgrind log file back to the offending test.
Use this command:
(echo tests/Makefile.am.in; find tests -name Makefile.am) \
|xargs perl -pi -e '/^(\s*)PATH=...VG_PATH_PREFIX/ and ' \
-e 'print $1,q|CU_TEST_NAME=`basename $(abs_srcdir)`,$$tst |,"\\\n"'
Remove from CVS, since ./bootstrap generates them automatically.
* .cvsignore: Add INSTALL, Makefile.in, *.cache, *.lineno, *.log.
Remove more-specific entries. This catches files like configure.lineno.
* man/.cvsignore: Add Makefile.in.
* src/.cvsignore: Add Makefile.in.
Remove .version, dir.c, install, mvdir, stamp-v, vdir.c, version.c.
For .cvsignore file under the tests directory:
Add Makefile.in. Sort entries if necessary. Remove *.I, *.E,
*.X, *.O, *-tests, build-script, mk-script if they're never
created in this directory.
* build-aux/.cvsignore: New file.
* doc/.cvsignore: Add Makefile.in, coreutils.html, coreutils.pdf,
coreutils.ps, coreutils.tps. Remove coreutils.cm (dunno what it
is, but the makefile doesn't mention it). Remove coreutils.info
as it is subsumed by coreutils.info*.
* lib/.cvsignore: Add Makefile.in, getdate.tab.h.
Remove stat.c, sysexit.h.
* m4/.cvsignore: Remove Makefile, dev-ino.m4, glibc2.m4, intdiv0.m4,
inttypes-h.m4, inttypes-pri.m4, jm-glibc-io.m4, lcmessage.m4,
lock.m4, printf-posix.m4, same-inode.m4, size_max.m4,
uintmax_t.m4, ulonglong.m4, visibility.m4, xsize.m4. Many of
these are put in automatically by ../bootstrap, but that's a
bootstrap bug that I plan to fix shortly.
* po/.cvsignore: More ../bootstrap-related fixes, plus remove old
cruft. Add *.po, LINGUAS, Makevars. Remove *.cat, *.msg,
cat-id-dbl.c, messages.mo, stamp-cat-id.