Commit Graph
9543 Commits
Author SHA1 Message Date
Pádraig Brady b60a159fdc unexpand: fix heap overflow
* src/unexpand.c (unexpand): Use xinmalloc() to gracefully
handle overflow.  Also use the runtime locale specific MB_CUR_MAX
rather than the worst case MB_LEN_MAX.
* tests/unexpand/mb.sh: Add a test case that fails in a default
glibc build with either MB_CUR_MAX or MB_LEN_MAX.
* NEWS: Mention the bug fix.
Reported by Michał Majchrowicz.
2026-04-29 13:16:43 +01:00
Pádraig Brady 00cd91288c unexpand: consolidate error messages
* src/unexpand.c (main): Use the same error
as in parse_tab_stops().
2026-04-29 13:16:21 +01:00
Pádraig Brady dc723665b5 cut: use MCEL_LEN_MAX rather than MB_LEN_MAX
* src/cut.c: We use the mcel interface to populate these arrays,
so restrict to the mcel supported max.  This is more efficient
as generally stateful encodings are not used or supported.
2026-04-29 13:16:21 +01:00
Pádraig Brady 38cf97c1ff sort: use more dynamic memory allocation with pipes
The default memory allocation with pipes was too passive/static,
resulting in not allocating enough memory to enable threading.
By dynamically reallocating the buffer when reading from
unknown sized inputs we better use available memory and threads.

  $ time seq 10000000 -1 0 | sort-old >/dev/null
  real	0m16.523s
  user	0m16.900s
  sys	0m0.167s

  $ time seq 10000000 -1 0 | sort-old -S1G >/dev/null
  real	0m12.263s
  user	0m29.646s
  sys	0m0.527s

  $ time seq 10000000 -1 0 | sort-new >/dev/null
  real	0m12.994s
  user	0m31.266s
  sys	0m0.716s

It also avoids the overhead of writing to temp files
for modestly sized inputs. For example the following
input would induce interaction with temp storage:

  $ seq 125000 | wc -c
  763895

* src/sort.c (sort_buffer_size): Rename to ...
(sort_buffer_policy): ... here, and adjust to set
an initial size and limit, rather than just a size.
(fillbuf): Add a POLICY parameter, and use that
to call maybe_growbuf() as needed.
(maybe_growbuf): Return true if POLICY dictates we
should grow the buffer, and try_growbuf() was
able to reallocate the larger buffer.
* tests/sort/sort-buffer-size.sh: Add a new test.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
Related to https://bugs.gnu.org/10877
2026-04-29 13:16:21 +01:00
Collin Funk 4c4afad943 pinky: don't print output in the wrong order when fully buffered
* NEWS: Mention the bug fix.
* src/pinky.c (cat_file): Prefer streams to file descriptors when
writing to standard output.
2026-04-28 18:28:25 -07:00
Paul Eggert d64e35a8a4 uniq: fix read overrun with -w
Problem reported by Michał Majchrowicz.
* src/uniq.c (find_field): Fix typo.
* tests/uniq/uniq.pl (add_z_variants): Test for the bug.
2026-04-28 11:25:41 -07:00
Pádraig Brady 31572ed49e doc: apply man page formatting suggestions from manpage-l10n
* src/basenc.c: Add (1) to base64 and base32 references.
* src/cksum.c. Likewise for cksum references.
* src/echo.c: Use "bell" in descriptions rather than BEL.
* src/printf.c: Likewise.
* src/id.c: Separate -u,-U,-G so marked up appropriately.
Fixes https://bugs.gnu.org/80904
2026-04-25 19:04:12 +01:00
Bruno Haible c70c45eb88 doc: remove redundant "Report translation bugs to" in --help
* configure.ac: Define PACKAGE_L10N_BUGREPORT.
* man/local.mk (run_help2man): Set the IN_HELP2MAN environment variable.
* src/system.h (emit_ancillary_info): Don't emit "Report any translation
bugs to" line; this is already done by emit_bug_reporting_address() in
version-etc.c.
Fixes https://bugs.gnu.org/80886
2026-04-25 12:26:12 +01:00
Pádraig Brady 4bf753a801 dd: avoid recursive parsing of multipliers
* src/dd.c (parse_integer): Use iterative rather than recursive parsing,
to avoid potential stack overflow.
* tests/dd/bytes.sh: Add a test case.
https://github.com/coreutils/coreutils/issues/254
2026-04-23 13:58:46 +01:00
Collin Funk c5ddd417aa comm: don't close standard input twice
* NEWS: Mention the bug fix.
* src/comm.c (usage): Remove mention that FILE1 and FILE2 cannot both be
standard input.
(compare_files): Only close standard input once.
* doc/coreutils.texi (comm invocation): Document the behavior of
'comm - -' which is not portable to all implementations.
* tests/comm/dash-dash.sh: New file.
* tests/misc/comm.pl: Move to tests/comm/comm.pl.
* tests/local.mk (all_tests): Add the new test. Rename the existing
test.
2026-04-22 19:12:44 -07:00
Pádraig Brady 3cbe0491ba yes: make operation independent of pipe size
* src/yes.c (splice_write): Always drain what we've written
to an internal pipe, so there is no possibility of vmsplice() blocking.
I.e., be defensive in the case that fcntl() fails, and
our default buffer size (currently 16kiB) is larger than the pipe.
https://github.com/coreutils/coreutils/issues/253
2026-04-17 16:15:08 +01:00
Pádraig Brady 5597a275c2 build: fix build failure on AIX
* m4/jm-macros.m4: AIX has a splice() function for TCP,
so check for vmsplice() instead.
* src/splice.h: Define HAVE_SPLICE if vmsplice available.
Reported by Bruno Haible.
2026-04-16 22:05:13 +01:00
Pádraig Brady 852c056889 cksum: fix --length validation on 32 bit platforms
Fix an unreleased issue due to the recent change
to using idx_t in commit v9.10-91-g02983e493

* src/cksum.c (main): Limit the possible return to
the range supported by idx_t.
Reported by Bruno Haible.
2026-04-16 20:40:34 +01:00
Lukáš Zaoral f6fda635bd df: improve detection of duplicate entries
Do not compare only with the latest entry for given device id but also
all previously saved entries with the same id.

* src/df.c (struct devlist): Add next_same_dev struct member.
(filter_mount_list): Iterate over next_same_dev to find duplicates.
* tests/df/skip-duplicates.sh: Add test cases.
* NEWS: Mention the improvement.
https://redhat.atlassian.net/browse/RHEL-5649
2026-04-15 12:56:16 +01:00
Collin Funk 79a6da3be7 mktemp: prefer rmdir and unlink to remove
This avoids the following behavior:

    $ strace -e silence=exit -e trace=unlink,rmdir \
        mktemp -d > /dev/full
    unlink("/tmp/tmp.ZBuPmS9ZGD") = -1 EISDIR (Is a directory)
    rmdir("/tmp/tmp.ZBuPmS9ZGD")  = 0
    mktemp: write error: No space left on device

In the above invocation we know that we created a directory, so we
should not remove a regular file that must have been created by another
process:

    $ strace -e silence=exit -e trace=unlink,rmdir \
        ./src/mktemp -d > /dev/full
    rmdir("/tmp/tmp.hGbME1HmJr") = 0
    mktemp: write error: No space left on device

* src/mktemp.c (main): Prefer rmdir and unlink depending on whether we
created a directory or regular file.
* bootstrap.conf (gnulib_modules): Remove the remove module.
2026-04-11 09:58:13 -07:00
Collin FunkandPádraig Brady 9a3479f1ea cat: avoid redundant pipe creation and resizing
* src/cat.c (splice_cat): Don't bother resizing input as it generally
doesn't help perf, and also save an fstat per input. Don't close the
intermediate pipe once created, unless there is an error reading from
it.

Co-authored-by: Pádraig Brady <P@draigBrady.com>
2026-04-10 22:44:22 -07:00
Collin Funk f634724b8f env: avoid locking standard output for each printed variable
* src/env.c (main): Use fputs and putchar instead of printf.
2026-04-09 21:59:12 -07:00
Collin Funk 77ba5a9815 printenv: avoid locking standard output for each printed variable
* src/printenv.c (main): Use fputs and putchar instead of printf.
2026-04-09 21:53:28 -07:00
Pádraig Brady d0fa892c16 maint: remove last remaining assert()
* src/split.c (bytes_chunk_extract): Prefer affirm to assert,
as it allows for better static checking when compiling with -DNDEBUG.
2026-04-09 21:36:43 +01:00
Pádraig Brady d27e48c122 maint: cat: avoid coverity NULL dreference warning
* src/cat.c (ensure_buf_size): Affirm we won't return NULL;
2026-04-09 21:25:27 +01:00
Pádraig Brady 3131fbf0e1 cat: avoid memory allocation per file
* src/cat.c (main): Only resize the allocated buffer when needed,
which avoids per file heap manipulation and mmap/munmap syscalls.
2026-04-09 16:39:12 +01:00
Pádraig Brady e0a641674a cat: fix splice() from empty input
* src/cat.c (splice_cat): Ensure we don't retry a read() after
splice() completes, as this is significant on a tty.
2026-04-09 16:39:06 +01:00
Collin Funk bd5d98e8a7 maint: touch: prefer timespec_cmp
* src/touch.c (main): Use timespec_cmp instead of comparing each member
of the timespec.
2026-04-08 22:34:13 -07:00
Pádraig Brady 2e7fc39ec1 maint: cut: avoid discarded-qualifiers warnings
Seen on GCC 15.2.1 with GLIBC 2.43 on Arch
Not seen on GCC 15.2.1 on GLIBC 2.42 on Fedora

* src/cut.c (search_bytes): Cast the return from memchr()
to avoid const propagation.
(find_field_delim): Adjust the return from strstr() similarly.
https://github.com/coreutils/coreutils/issues/244
2026-04-07 14:57:18 +01:00
Collin Funk 457f88513a cat: use splice if operating on pipes or if copy_file_range fails
On a AMD Ryzen 7 3700X system:

    $ timeout 10 taskset 1 ./src/cat-prev /dev/zero \
        | taskset 2 pv -r > /dev/null
    [1.67GiB/s]
    $ timeout 10 taskset 1 ./src/cat /dev/zero \
        | taskset 2 pv -r > /dev/null
    [9.03GiB/s]

On a Power10 system:

    $ taskset 1 ./src/yes | timeout 10 taskset 2 ./src/cat-prev \
        | taskset 3 pv -r > /dev/null
    [12.9GiB/s]
    $ taskset 1 ./src/yes | timeout 10 taskset 2 ./src/cat \
            | taskset 3 pv -r > /dev/null
    [81.8GiB/s]

* NEWS: Mention the improvement.
* src/cat.c: Include isapipe.h, splice.h, and unistd--.h.
(splice_cat): New function.
(main): Use it.
* src/local.mk (noinst_HEADERS): Add src/splice.h.
* src/splice.h: New file, based on definitions from src/yes.c.
* src/yes.c: Include splice.h.
(pipe_splice_size): Use increase_pipe_size from src/splice.h.
(SPLICE_PIPE_SIZE): Remove definition, moved to src/splice.h.
* tests/cat/splice.sh: New file, based on some tests in
tests/misc/yes.sh.
* tests/local.mk (all_tests): Add the new test.
2026-04-06 21:57:07 -07:00
Pádraig Brady 7064be3061 build: cut: fix compilation error on non C23 compilers
* src/cut.c (main): Add curly brackets around variable
declaration in case label.
Reported by Bruno Haible.
2026-04-06 22:37:15 +01:00
Paul Eggert cff1fa2239 maint: simplify c32issep
* src/system.h (c32issep): Avoid unnecessary ‘!!’.
2026-04-06 11:15:10 -07:00
Paul Eggert 4c6cf6043a maint: cut: refactor delimiter handling
* src/cut.c: Use mcel_scanz() to parse in all cases,
and avoid redundant storage of delimiter_length and
the single byte delim.
2026-04-06 15:52:58 +01:00
Pádraig Brady 1a44a25808 cut: -f: fix handling of multi-byte delimiters that span buffers
* src/cut.c (cut_fields_bytesearch): Ensure up to delim_bytes -1
is left for the next refill.
* tests/cut/cut.pl: Add a test case.
2026-04-06 15:52:58 +01:00
Pádraig Brady 57c87043f6 cut,fold,expand,unexpand: ensure we process all available characters
* gl/lib/mbbuf.h: Adjust mbbuf_fill() to process full characters
in the slop at the end of a read().  Previously valid characters
in the last MCEL_LEN_MAX bytes were ignored until the next read().
* src/cut.c (cut_fields_bytesearch): Adjust to the new naming.
* NEWS: Mention the fold(1) responsiveness fix, which was
improved with the change from fread() to read(),
and completed with this patch.
2026-04-06 15:52:56 +01:00
Pádraig Brady 1b4b8104d3 cut: -b: avoid function calls in hot loop
$ time LC_ALL=C src/cut-before -b1 sl.in >/dev/null
  real	0m0.115s

  $ time LC_ALL=C src/cut-after -b1 sl.in >/dev/null
  real	0m0.076s

* src/cut.c (cut_bytes): Hoist the fileno() invariant outside the loop.
Avoid memchr for very short lines.
(search_bytes): Similar to copy_bytes() and write_bytes() helpers.
Note adding code to probe 3 or 4 bytes resulted in worse register
allocation.  I.e. slower operation even if the input was only 2 bytes.
2026-04-05 13:15:56 +01:00
Pádraig Brady 374ff00c36 cut: fix logic issue with field delim in last byte of buffer
With field delimiter = line delimiter we need to know
if there is any more data to be read, as field delimiter
in the last byte of the file is treated differently.
So reiterate the loop to ensure enough read()s to make
the appropriate determination.
2026-04-05 13:15:56 +01:00
Pádraig Brady d757d32f86 cut: ensure responsive input processing
* gl/lib/mbbuf.h (fill_buf): Switch from fread() to read()
as the former retries read() internally to fill the buffer.
* src/cut.c: Adjust accordingly, and avoid getc() interface entirely.
* bootstrap.h: Depend explicitly on fseterr.  This is already depended
on transitively, so should not introduce new build portability issues.
2026-04-05 13:15:56 +01:00
Pádraig Brady 8daf2a91be maint: cut: rename line_in to bytes_in
* src/cut.c: We're not reading a line, rather a buffer of bytes.
Suggested by Collin Funk.
2026-04-05 13:15:56 +01:00
Pádraig Brady 76edef14d8 cut: make the dependency on memchr2 explicit
* bootstrap.conf: Remove now unused getndelim2, add memchr2.
* src/cut.c: Remove now unused getndelim2.h.
2026-04-05 13:15:56 +01:00
Pádraig Brady 890cf82593 cut: combine cut_bytes_no_split and cut_characters
per character based so merge.
2026-04-05 13:15:56 +01:00
Pádraig Brady 25f0702eaa maint: cut: various code cleanups and comments
* src/cut.c: Document some functions, and remove extraneous
 abstractions.
2026-04-05 13:15:56 +01:00
Pádraig Brady 5d339d583d cut: support no delimiter match fast path with -s
* src/cut.c (cut_fields_bytesearch): Just skip the data with -s.
2026-04-05 13:15:56 +01:00
Pádraig Brady 032ecdee9b maint: cut: cleanup context management for byte search
* src/cut.c: Hoist at_eof into context so we're not
querying it multiple times.  Also add a helper
to explicitly init bytesearch_context.
2026-04-05 13:15:56 +01:00
Pádraig Brady 24571c41f3 cut: optimize UTF-8 input with 0xF5-0xFF delimiters
* src/cut.c (bytesearch_field_delim_ok): Expand the range
of bytes that can be simply searched for. 0xF5-0xFF can't
appear in valid UTF-8 characters, and so may be used as
delimiters in UTF-8 input, so it's worth optimizing for.
* tests/cut/cut.pl: Add a test case (mainly as documentation).
2026-04-05 13:15:56 +01:00
Pádraig Brady 7ceb4c348b maint: cut: simplify mbbuf_fill
We can only byte search with uni-byte or utf-8.
utf-8 implicitly can't false match a delimiter at buffer boundary.
So don't worry about finding the exact utf8 boundary at end of buffer,
rather just ensuring the buffer always starts with a valid character
(by ensuring MCEL_LEN_MAX-1 moved to start of buffer on each refill).
2026-04-05 13:15:56 +01:00
Pádraig Brady 5dd1a0bc77 maint: refactor is_utf8_charset helper to system.h
* src/cut.c: Move from here.
* src/numfmt.c: Likeise.
* src/system.h: To here.
2026-04-05 13:15:56 +01:00
Pádraig Brady 36751ab993 cut: optimize per character memcpy
$ time src/cut-before -f10 -w ll.in >/dev/null
real	0m4.309s

$ time src/cut-after -f10 -w ll.in >/dev/null
real	0m3.136s

* src/cut.c (cut_bytes): Add a new helper that avoids
the memcpy call in the common case of adding characters to a buffer.
2026-04-05 13:15:56 +01:00
Pádraig Brady 4ae2275952 cut: refactor skip_line_remainder logic
Simplify and optimize field exhaustion logic:

$ time LC_ALL=C src/cut-before -f1 -dc as.in >/dev/null
real	0m0.057s

$ time LC_ALL=C src/cut-after -f1 -dc as.in >/dev/null
real	0m0.023s

* src/cut.c (cut_fields_bytesearch): Refactor.
2026-04-05 13:15:56 +01:00
Pádraig Brady 30c19a1279 maint: cut simplify cut_fields_bytesearch
refactor line delimiter output,
and resetting of parse record state.
2026-04-05 13:15:56 +01:00
Pádraig Brady 1da3ccced2 cut: enable fast path for all delimiter lengths
1. Removed !have_pending_line from the fast path condition.
This is safe because:
  - field_1_n_bytes == 0 already ensures we haven't started
    buffering field 1 content
  - The fast path correctly continues any pending partial line
    by writing raw bytes including the completing \n
2. Added have_pending_line = false after the fast path write,
since all lines up to the last \n are now complete.

$ time src/cut.before -f1 -dç sl.in >/dev/null
real	0m0.081s
$ time src/cut.after -f1 -dç sl.in >/dev/null
real	0m0.012s

$ time src/cut.before -f10 -dç sl.in >/dev/null
real	0m0.081s
$ time src/cut.after -f10 -dç sl.in >/dev/null
real	0m0.012s
2026-04-05 13:15:56 +01:00
Pádraig Brady 7a124eb585 cut: optimize -f with -d longer than lines
$ time src/cut.before -f1 -dç sl.in >/dev/null
real	0m0.157s

$ time src/cut.after -f1 -dç sl.in >/dev/null
real	0m0.084s
2026-04-05 13:15:56 +01:00
Pádraig Brady e73dbeb5ab cut: optimize -b for short lines
For a 40% performance increase it's worth reinstating the simple
original cut_bytes() which avoids data copying and function calls.
Once a longer line is encountered we defer to the buffered variant.

$ time src/cut.before -b2 sl.in  >/dev/null
real	0m0.101s

$ time src/cut.after -b2 sl.in  >/dev/null
real	0m0.060s
2026-04-05 13:15:56 +01:00
Pádraig Brady f5cab8e30c cut: optimize per character field scanning
Use specialized loops rather than branching per character,
giving a 28% increase.

$ time src/cut -f1 -w ll.in >/dev/null
real	0m7.199s

$ time src/cut -f1 -w ll.in >/dev/null
real	0m5.204s
2026-04-05 13:15:56 +01:00
Pádraig Brady a78ef2b65c cut: prefer c_isblank() to c32issep()
12% perf increase with:

$ time src/cut -f2 -w ll.in >/dev/null
real	0m6.469s

$ time src/cut -f2 -w ll.in >/dev/null
real	0m5.689s
2026-04-05 13:15:56 +01:00