Commit Graph
247 Commits
Author SHA1 Message Date
Pádraig Brady a16d56d60c cut: optimize -b by avoiding per byte iteration
Always memchr(line_delim) which is fast and allows:

- skipping whole segments when the next selected byte is beyond them
- skipping unselected prefixes in bulk
- writing contiguous selected spans in bulk

This wins for lines >= 4 characters,
but is slower lines <= 3 characters, especially if selecting bytes 1-3.
That is unusual though.
2026-04-05 13:15:56 +01:00
Pádraig Brady ea6a7ba547 cut: optimize when no delimiter in input
This is about 20x faster.
Note we only do the delimiter search once per chunk,
and it's usually quick as delimiters wouldn't be too far
into the a chunk if present, so we don't bother
to cache the found delimiter.
2026-04-05 13:15:56 +01:00
Pádraig Brady 700ffc51a1 cut: optimize -w for uni-byte case
* src/cut.c: Limit search to SPACE and TAB
2026-04-05 13:15:56 +01:00
Pádraig Brady c1d7b492c6 doc: cut: document the -w option
* src/cut.c (usage): Mention blank characters are used to separate.
* doc/coreutils.texi (cut invocation): Likewise.  Also describe
the 'trimmed' argument and the relation to -F.
2026-04-05 13:15:56 +01:00
Pádraig Brady cf25ef286a cut: refactor find_bytesearch_field_terminator to be stateful
Allows better/simpler avoidance of repeated line/delim scans

TODO: speed up our really slow cut_fields_mb_any.
Compare for example:
  time src/cut -w  -f1 ll.in >/dev/null #14s
  time src/cut -d, -f1 ll.in >/dev/null #.1s
Could adjust so that LC_ALL=C does memchr2(space,tab) ?
2026-04-05 13:15:56 +01:00
Pádraig Brady 0adb7c6edd cut: avoid repeated searchs for line_delim in the multi-byte delim case
TODO: Refactor all this into find_bytesearch_field_terminator.
Also handle in the delim_length==1 case.
2026-04-05 13:15:56 +01:00
Pádraig Brady d9825aa9b1 cut: refactor all byte search to find_bytesearch_field_terminator
TODO: Perhaps also add search only fields mode
to avoid rescans of very long lines
2026-04-05 13:15:56 +01:00
Pádraig Brady 6250b59ef9 cut: optimize -f when finished processing fields for a line
TODO: simplify and compare perf
2026-04-05 13:15:56 +01:00
Pádraig Brady 352a396a16 cut: optimize -f for fhe common case of single byte delimiters
* TODO: perf comparison
2026-04-05 13:15:56 +01:00
Pádraig Brady 2a6b36ff5b cut: optimize -d '?' in UTF-8 case
ensure all ascii delims are processed with byte search in UTF-8
2026-04-05 13:15:56 +01:00
Pádraig Brady f6b3055f74 cut: merge cut_fields and cut_fields_bytesearch
TODO: See why this is much slower:
time LC_ALL=C.UTF-8 src/cut -f1 -dc as.in > /dev/null
2026-04-05 13:15:56 +01:00
Pádraig Brady b3ef6231bd cut: refactor -f to byte search and character processing
Not sure about this at all.
Only worthwhile if can also remove cut_fields_line_delim.

Refactored src/cut.c so the old UTF-8 byte-search path is now the
general byte-search field engine for all safe byte-search cases:
ordinary single-byte delimiters and valid UTF-8 delimiters. The old
cut_fields path did not go away completely; it is now
cut_fields_line_delim and is used only when the field delimiter equals
the record delimiter, because -d $'\n' and -z -d '' have different
semantics from normal line-based field splitting.

As part of that, I also folded the duplicated “start selected field”
logic into a shared helper, and renamed the byte-search helpers to match
their broader use. The current dispatcher in src/cut.c is now:
whitespace parser, then line- delimiter field mode, then byte-search
field mode, then the decoded multibyte parser.
2026-04-05 13:15:56 +01:00
Pádraig Brady 16b1ff40ae cut: fix 25% perf regression mentioned in previous change
* src/cut.c: Prefer strstr() over memmem() as the former is
optimized per platforma on GLIBC, where as the latter is only
optimized for S390 as of glibc-2.42

TODO: look at merging cut_fields and cut_fields_mb_utf8
as they're both byte search
2026-04-05 13:15:56 +01:00
Pádraig Brady 26028fb2c6 cut: use bounded memory in utf8 mode when possible
TODO: See why a bit slower than old code

$ time src/cut.old -f1 -dç mb.in >/dev/null
real	0m0.136s
user	0m0.096s
sys	0m0.039s

$ time src/cut.new -f1 -dç mb.in >/dev/null
real	0m0.170s
user	0m0.139s
sys	0m0.030s
2026-04-05 13:15:56 +01:00
Pádraig Brady 801686242e cut: faster utf8 processing
TODO: improve to use bounded memory where possible
2026-04-05 13:15:56 +01:00
Pádraig Brady a14ac29629 cut: support -F as an alias for -f -w -O ' '
To improve compatibility with toybox/busybox scripts.
2026-04-05 13:15:56 +01:00
Pádraig Brady 74f6692aa0 maint: cut: refactor buffered and ordinary field scanning
* src/cut.c: Merge scan_mb_field and read_mb_field_to_buffer
2026-04-05 13:15:56 +01:00
Pádraig Brady eb1f057746 cut: support --whitespace-delimited=trimmed
Support ignoring leading and trailing whitespace.
E.g. this matches awk's default field splitting mode.

* src/cut.c
* tests/cut/cut.pl: Add test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 77ccacb9a7 cut: support -O as an alias for --output-delimiter
To improve compatibility with toybox/busybox scripts.

* doc/coreutils.texi (cut invocation): Add -O description.
* src/cut.c: Support -O as well as --output-delimiter
* tests/cut/cut.pl: Adjust one case to use -O.
2026-04-05 13:15:56 +01:00
Pádraig Brady f644b4ca53 cut: refactor multi-byte updates
* src/cut.c: 160 fewer lines

Helpers extracted (replacing repeated inline patterns):
- write_line_delim(), write_pending_line_delim(), reset_item_line()
  - line boundary code used by cut_bytes{,no_split}, cut_characters
- write_selected_item()
  - output-delimiter + write logic used by all three byte/char functions
- reset_field_line()
  - field line reset used by cut_fields_mb_any

Field functions unified via cut_fields_mb_any(stream, whitespace_mode):
- struct mbfield_parser encapsulates the whitespace vs.
  fixed-delimiter state (saved char, mode flag)
- mbfield_get_char() - dispatches to saved-char or direct read
- mbfield_terminator()
  - returns FIELD_{DATA,DELIMETER,LINE_DELIMITER} based on mode
- read_mb_field_to_buffer()
  - replaces the two duplicated first-field buffering loops
- scan_mb_field(mbbuf, parser, pending, write_field)
  - replaces the four duplicated field scan loops
  (print+skip × two modes) with a single function and a write_field bool
- cut_fields_mb and cut_fields_ws are now trivial wrappers
2026-04-05 13:15:56 +01:00
Pádraig Brady 57110d8bae cut: implement -n to avoid outputting partial characters
Both the i18n patch and FreeBSD/macOS support this option.
They do differ in behavior somewhat as the i18n patch
may output more bytes than requested.

  $ printf '\xc3\xa9b\n' | i18n-cut -n -b1
  é

There is also a bug in the i18n patch with multi-byte
at the start of a line:

  $ printf '\xc3\xa9b\n' | i18n-cut -n -b1-2
  éb

We follow the FreeBSD behavior since it seems more
useful to have -b be a hard limit, rather than a soft limit.
This also reduces the possibility of duplicate character output
with separate cut invocations with non overlapping byte ranges.

* src/cut.c (cut_bytes_no_split): A new function
similar to cut_characters, to handle multi-byte characters
with byte limit semantics.
* tests/cut/cut.pl: Add test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 74047ec55e cut: adjust error message to be less specific
* src/cut.c (main): Cater for both misplaced -w and -d.
2026-04-05 13:15:56 +01:00
Pádraig Brady 19aa72b4ea cut: implement -w,--whitespace-delimited
* src/cut.c (cut_fields_ws): A new function handling both
uni-byte and multi-byte cases.
* tests/cut/cut.pl: Add a test cases.
2026-04-05 13:15:56 +01:00
Pádraig Brady 32f1de5b4f cut: support single byte -d with GB18030 input
* src/cut.c
* tests/cut/mb-non-utf8.sh
* tests/local.mk
2026-04-05 13:15:55 +01:00
Pádraig Brady 94ddf45a60 cut: support single byte -d that may be part of multi-byte
Note this is a slight divergence from the i18n patch
as that switched to uni-byte for any single byte delimiter
that is not valid multi-byte.

That results in possibly splitting in the middle of
a valid multi-byte character.

Instead we only split on a single byte when they're
not part of a multi-byte character.

* src/cut.c
2026-04-05 13:15:55 +01:00
Pádraig Brady a021b0b698 cut: support multi-byte field delimiters
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady 97703386e6 cut: support multi-byte input with -c
* src/cut.c
* tests/cut/cut.pl
2026-04-05 13:15:55 +01:00
Pádraig Brady fb78200249 maint: cut: refactor output calls
* src/cut.c (cut_fields): Refactor calls to fwrite() and putchar()
2026-04-05 13:15:55 +01:00
Pádraig Brady a04a1054f8 doc: cut: reorder --complement alphabetically in help
* src/cut.c (usage): Move placement of --comlement description.
* doc/coreutils.texi (cut invocation): Likewise.
2026-04-05 13:15:55 +01:00
Pádraig Brady 1173ebb7c8 doc: cut: clarify description of -b and -c
* src/cut.c (usage): State the arguments are positions,
in case users may think they were values.
2026-04-05 13:15:55 +01:00
Collin Funk 425dd90b88 maint: cut: prefer static initialization
* src/cut.c (main): Remove unnecessary initializations.
2026-02-19 21:01:10 -08:00
Pádraig Brady 2dad24adc0 doc: all: use option highlighting and more standard alignment
Adjust all utils to:
  - Have a separate translation per option
  - Use either oputs() or oprintf(), instead of fputs() or printf().
  - Use more standard alignment as discussed in previous commits.
  - Various tweaks to descriptions.
2026-01-21 13:51:40 +00:00
Pádraig Brady 71ea30a742 maint: prefer NULL to nullptr
NULL is best for C as discussed at:
https://bugs.gnu.org/66221#53

* bootstrap.conf: Remove dependency on nullptr.
* s/nullptr/NULL/.
2026-01-18 15:55:38 +00:00
Collin Funk 73d3a49f45 maint: run 'make update-copyright' 2026-01-01 10:56:16 -08:00
Paul Eggert ee367bd38d maint: STREQ → streq
Use new Gnulib streq function instead of rolling our own macro.
* bootstrap.conf (gnulib_modules): Add stringeq.
* src/rm.c (main): Don’t assume streq is a macro that expands to (...),
as it is now a function.
* src/system.h:
* tests/df/no-mtab-status.sh, tests/df/skip-duplicates.sh:
(STREQ): Remove.  All uses replaced by streq.
2025-09-17 12:20:24 -07:00
Pádraig Brady 28b176085f maint: update all copyright year number ranges
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...

* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
2025-01-01 09:33:08 +00:00
Pádraig Brady a966dcdb69 maint: update all copyright year number ranges
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...

* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
2024-01-01 13:27:23 +00:00
Dragan Simic 73ba9f71e1 maint: minor comment cleanups
* src/cut.c: Adjust a few comments slightly, simply to have their
trailing whitespace the same as in the majority of the comments.
2023-08-01 15:35:44 +01:00
Dragan Simic 7783df863a cut: promptly diagnose write errors, continued
* src/cut.c: Complete the error-handling improvements started in
commit e0a4a60af5, by adding a couple of remaining checks for putchar().
While there, sprinkle a few rather useful comments, and perform a few
small code cleanups, to make the code and the comments more uniform
and more conformant to the official coding style.  Also make the help
message slightly more uniform.
2023-08-01 15:34:50 +01:00
Pádraig Brady e0a4a60af5 cut: promptly diagnose write errors
* src/cut.c (cut_bytes): Diagnose errors from fwrite() and putchar().
(cut_fields): Likewise.
* tests/misc/write-errors.sh: Enable the test for cut,
and augment to cover both cut_bytes() and cut_fields().
* NEWS: Mention the improvement.
2023-07-17 11:28:36 +01:00
Paul Eggert 17aaba6a41 maint: pacify GCC bug#109613 better
* src/cut.c (cut_file):
* src/nl.c (nl_file): Pacify GCC Bug#109613 in a better way, by
narrowing the coverage of the ‘assume’ so that bugs in the
no-longer-covered part are not masked.
2023-07-01 11:51:16 -07:00
Paul Eggert 6d61667d0d maint: go back to using ‘error’
Now that Gnulib’s ‘error’ module does proper static checking
for not returning, we need no longer use the ‘die’ macro.
This makes code easier to read for people that are used to ‘error’.
* cfg.mk (error_fns, exclude_file_name_regexp): Remove ‘die’.
(sc_die_EXIT_FAILURE): Remove.
* src/die.h: Remove.  All includes removed.  All calls to ‘die’
changed back to calls to ‘error’.
* src/install.c (get_ids): Use quoteaf (problem found with
make syntax-check).
* src/system.h: Include error.h, since some of our macros call ‘error’.
Stop including error.h elsewhere.
2023-07-01 11:51:16 -07:00
Paul Eggert 478055dc30 maint: improve static and dynamic checking
This modernizes the source code somewhat, to take advantage
of advances in GCC over the years, and Gnulib’s ‘assure’ module.
Include assure.h in files that now need it.
Do not include assert.h directly; it’s no longer needed.
* bootstrap.conf (gnulib_modules): Add ‘assure’.
* gl/lib/randread.c (randread_error):
* src/chmod.c (describe_change):
* src/chown-core.c (describe_change):
* src/cp.c (decode_preserve_arg):
* src/head.c (diagnose_copy_fd_failure):
* src/ls.c (parse_ls_color):
* src/od.c (decode_one_format):
* src/split.c (main):
* src/test.c (binary_operator, posixtest):
Prefer affirm to abort, since it has better diagnostics in the
normal case and better performance with -DNDEBUG.
* gl/lib/xdectoint.c, src/die.h: Include stddef.h, for unreachable.
* gl/lib/xdectoint.c: Do not include verify.h; no longer needed.
* gl/lib/xdectoint.c (__xnumtoint):
* src/die.h (die):
Prefer C23 unreachable () to assume (false).
* gl/lib/xfts.c (xfts_open):
* src/basenc.c (base32hex_encode):
* src/copy.c (abandon_move, copy_internal, valid_options):
* src/cut.c (cut_fields):
* src/df.c (alloc_field, decode_output_arg, get_dev):
* src/du.c (process_file, main):
* src/echo.c (usage):
* src/factor.c (udiv_qrnnd, mod2, gcd2_odd, factor_insert_large)
(mulredc2, factor_using_pollard_rho, isqrt2, div_smallq)
(factor_using_squfof):
* src/iopoll.c (iopoll_internal, fwrite_wait):
* src/join.c (add_field):
* src/ls.c (dev_ino_pop, main, gobble_file, sort_files):
* src/mv.c (do_move):
* src/od.c (decode_format_string, read_block, dump, main):
* src/remove.c (rm):
* src/rm.c (main):
* src/sort.c (stream_open):
* src/split.c (next_file_name, lines_chunk_split):
* src/stdbuf.c (main):
* src/stty.c (set_speed):
* src/tac-pipe.c (line_ptr_decrement, line_ptr_increment):
* src/touch.c (touch):
* src/tr.c (find_bracketed_repeat, get_next)
(validate_case_classes, get_spec_stats, string2_extend, main):
* src/tsort.c (search_item, tsort):
* src/wc.c (main):
Prefer affirm to assert, as it allows for better static
checking when compiling with -DNDEBUG.
* src/chown-core.c (change_file_owner):
* src/df.c (get_field_list):
* src/expr.c (printv, null, tostring, toarith, eval2):
* src/ls.c (time_type_to_statx, calc_req_mask, get_funky_string)
(print_long_format):
* src/numfmt.c (simple_strtod_fatal):
* src/od.c (decode_one_format):
* src/stty.c (mode_type_flag):
* src/tail.c (xlseek):
* src/tr.c (is_char_class_member, get_next, get_spec_stats)
(string2_extend):
Prefer unreachable () to abort () or assert (false) when merely
pacifying the compiler, e.g., in a switch statement on an enum
where all cases are covered.
* src/copy.c (valid_options): Now returns void; the bool was useless.
Caller no longer needs to assert.
* src/csplit.c (find_line):
* src/expand-common.c (next_file):
* src/shred.c (incname):
* src/sort.c (main):
* src/tr.c (append_normal_char, append_range, append_char_class)
(append_repeated_char, append_equiv_class):
* src/tsort.c (search_item):
Omit assert, since the hardware will check for us.
* src/df.c (header_mode): Now the enum type it should have been.
* src/du.c (process_file):
* src/ls.c (assert_matching_dev_ino):
* src/tail.c (valid_file_spec):
* src/tr.c (validate_case_classes):
Mark defns with MAYBE_UNUSED if they’re not used when -DNDEBUG.
* src/factor.c (prime_p, prime2_p, mp_prime_p): Now ATTRIBUTE_PURE.
Prefer affirm to error+abort.  No need to translate this diagnostic.
* src/fmt.c (get_paragraph):
* src/stty.c (display_changed, display_all, sane_mode):
* src/who.c (idle_string):
Prefer assume to assert, since the goal is merely pacification
and assert doesn’t pacify anyway if -DNDEBUG is used.
* src/join.c (decode_field_spec):
Omit unreachable abort.
* src/ls.c (assert_matching_dev_ino, main):
* src/tr.c (get_next):
Prefer assure to assert, since the check is relatively expensive
and won’t help static analysis.
* src/ls.c (main):
Prefer static_assert to assert of a constant expression.
(format_inode): Redo to make it clear that buflen doesn’t matter,
and that buf must have a certain number of bytes.  All callers changed.
This pacifies -Wformat-overflow.
* src/od.c (decode_one_format):
Omit an assert that tested for obviously undefined behavior,
as the compiler could optimize it away anyway.
* src/od.c (decode_one_format, decode_format_string):
Prefer ATTRIBUTE_NONNULL to runtime checking.
* src/stat.c: Do not include <stddef.h> since system.h does that now.
* src/sync.c (sync_arg):
Prefer unreachable () to assert (true), which was a typo.
* src/system.h: Include stddef.h, for unreachable.
* src/tail.c (xlseek): Simplify by relying on ‘error’ to exit.
2023-07-01 11:51:15 -07:00
Paul Eggert 16b5ca6e0d maint: prefer C23-style nullptr
* bootstrap.conf (gnulib_modules): Add nullptr.
In code, prefer nullptr to NULL where either will do.
2023-06-29 15:29:29 -07:00
Pádraig Brady 76e1200eeb build: pacify GCC 13 with -flto
* src/cut.c (cut_file): Explicitly mark STREAM as nonnull to avoid
-Werror=null-dereference.
* src/nl.c (nl_file): Likewise.
2023-05-19 11:32:14 +01:00
Pádraig Brady 01755d36e7 maint: update all copyright year number ranges
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...

* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
2023-01-01 14:50:15 +00:00
Jim Meyering 7e6c39cc02 maint: remove unnecessary inclusion of hash.h
* src/cut.c: Don't include hash.h. The implementation was
changed not to need that in v8.21-43-g3e466ad05.
2022-06-23 08:31:20 -07:00
Pádraig Brady 253a46d09f maint: cut: avoid exporting recently added variable
* src/cut.c: Make output_delimiter_default static,
as identified by `make syntax-check`.
2022-01-31 22:00:12 +00:00
Paul Eggert 423fed3dd8 cut: simplify and remove an IF_LINT
* src/cut.c (enum operating_mode, operating_mode)
(output_delimiter_specified, cut_stream):
Remove; no longer needed.
(output_delimiter_default): New static var.  Code can now
use ‘output_delimiter_string != output_delimiter_default’
instead of ‘output_delimiter_specified’.
(cut_file): New arg CUT_STREAM.  Caller changed.
(main): Simplify.  Coalesce duplicate code.  Redo to avoid need
for IF_LINT, or for the static var.  No need to xstrdup optarg.
2022-01-31 12:07:39 -08:00
Paul Eggert 22a0f9c32e cut: simplify -fsanitize=leak pacification
* src/set-fields.c (reset_fields): Remove, as it’s not needed for
-fsanitize=leak even when ‘lint’ is defined.  All uses removed.
2022-01-31 12:07:39 -08:00