* gl/lib/mbbuf.h: Adjust mbbuf_fill() to process full characters
in the slop at the end of a read(). Previously valid characters
in the last MCEL_LEN_MAX bytes were ignored until the next read().
* src/cut.c (cut_fields_bytesearch): Adjust to the new naming.
* NEWS: Mention the fold(1) responsiveness fix, which was
improved with the change from fread() to read(),
and completed with this patch.
$ time LC_ALL=C src/cut-before -b1 sl.in >/dev/null
real 0m0.115s
$ time LC_ALL=C src/cut-after -b1 sl.in >/dev/null
real 0m0.076s
* src/cut.c (cut_bytes): Hoist the fileno() invariant outside the loop.
Avoid memchr for very short lines.
(search_bytes): Similar to copy_bytes() and write_bytes() helpers.
Note adding code to probe 3 or 4 bytes resulted in worse register
allocation. I.e. slower operation even if the input was only 2 bytes.
With field delimiter = line delimiter we need to know
if there is any more data to be read, as field delimiter
in the last byte of the file is treated differently.
So reiterate the loop to ensure enough read()s to make
the appropriate determination.
* gl/lib/mbbuf.h (fill_buf): Switch from fread() to read()
as the former retries read() internally to fill the buffer.
* src/cut.c: Adjust accordingly, and avoid getc() interface entirely.
* bootstrap.h: Depend explicitly on fseterr. This is already depended
on transitively, so should not introduce new build portability issues.
This is for consistency with other implementations and since the
interface separates -b and -c it might in future support -g (graphemes).
Normalizing content with a filter seems like the most appropriate
approach anyway, as there are various normalizations possible including
case etc. rather than baking that into every tool
* doc/coreutils.texi (cut invocation): Add back the -d description,
and adjust for multi-byte support, and expand on specifying a NUL
delimitier, and detail the behavior when the delimiter matches
the line delimiter.
* src/cut.c (bytesearch_field_delim_ok): Expand the range
of bytes that can be simply searched for. 0xF5-0xFF can't
appear in valid UTF-8 characters, and so may be used as
delimiters in UTF-8 input, so it's worth optimizing for.
* tests/cut/cut.pl: Add a test case (mainly as documentation).
* doc/coreutils.texi (cut invocation): State explicitly that
-s --whitespace-delimited=trimmed will suppress lines that
do not have field separating blanks.
We can only byte search with uni-byte or utf-8.
utf-8 implicitly can't false match a delimiter at buffer boundary.
So don't worry about finding the exact utf8 boundary at end of buffer,
rather just ensuring the buffer always starts with a valid character
(by ensuring MCEL_LEN_MAX-1 moved to start of buffer on each refill).
$ time src/cut-before -f10 -w ll.in >/dev/null
real 0m4.309s
$ time src/cut-after -f10 -w ll.in >/dev/null
real 0m3.136s
* src/cut.c (cut_bytes): Add a new helper that avoids
the memcpy call in the common case of adding characters to a buffer.
Simplify and optimize field exhaustion logic:
$ time LC_ALL=C src/cut-before -f1 -dc as.in >/dev/null
real 0m0.057s
$ time LC_ALL=C src/cut-after -f1 -dc as.in >/dev/null
real 0m0.023s
* src/cut.c (cut_fields_bytesearch): Refactor.
1. Removed !have_pending_line from the fast path condition.
This is safe because:
- field_1_n_bytes == 0 already ensures we haven't started
buffering field 1 content
- The fast path correctly continues any pending partial line
by writing raw bytes including the completing \n
2. Added have_pending_line = false after the fast path write,
since all lines up to the last \n are now complete.
$ time src/cut.before -f1 -dç sl.in >/dev/null
real 0m0.081s
$ time src/cut.after -f1 -dç sl.in >/dev/null
real 0m0.012s
$ time src/cut.before -f10 -dç sl.in >/dev/null
real 0m0.081s
$ time src/cut.after -f10 -dç sl.in >/dev/null
real 0m0.012s
For a 40% performance increase it's worth reinstating the simple
original cut_bytes() which avoids data copying and function calls.
Once a longer line is encountered we defer to the buffered variant.
$ time src/cut.before -b2 sl.in >/dev/null
real 0m0.101s
$ time src/cut.after -b2 sl.in >/dev/null
real 0m0.060s
Use specialized loops rather than branching per character,
giving a 28% increase.
$ time src/cut -f1 -w ll.in >/dev/null
real 0m7.199s
$ time src/cut -f1 -w ll.in >/dev/null
real 0m5.204s
This is quite significant:
yes abcdfeg | head -n1MB > big-file
$ time src/cut-before -b1,3 big-file >/dev/null
real 0m0.050s
$ time src/cut-after -b1,3 big-file >/dev/null
real 0m0.029s
Always memchr(line_delim) which is fast and allows:
- skipping whole segments when the next selected byte is beyond them
- skipping unselected prefixes in bulk
- writing contiguous selected spans in bulk
This wins for lines >= 4 characters,
but is slower lines <= 3 characters, especially if selecting bytes 1-3.
That is unusual though.
This is about 20x faster.
Note we only do the delimiter search once per chunk,
and it's usually quick as delimiters wouldn't be too far
into the a chunk if present, so we don't bother
to cache the found delimiter.
* src/cut.c (usage): Mention blank characters are used to separate.
* doc/coreutils.texi (cut invocation): Likewise. Also describe
the 'trimmed' argument and the relation to -F.
Allows better/simpler avoidance of repeated line/delim scans
TODO: speed up our really slow cut_fields_mb_any.
Compare for example:
time src/cut -w -f1 ll.in >/dev/null #14s
time src/cut -d, -f1 ll.in >/dev/null #.1s
Could adjust so that LC_ALL=C does memchr2(space,tab) ?
Not sure about this at all.
Only worthwhile if can also remove cut_fields_line_delim.
Refactored src/cut.c so the old UTF-8 byte-search path is now the
general byte-search field engine for all safe byte-search cases:
ordinary single-byte delimiters and valid UTF-8 delimiters. The old
cut_fields path did not go away completely; it is now
cut_fields_line_delim and is used only when the field delimiter equals
the record delimiter, because -d $'\n' and -z -d '' have different
semantics from normal line-based field splitting.
As part of that, I also folded the duplicated “start selected field”
logic into a shared helper, and renamed the byte-search helpers to match
their broader use. The current dispatcher in src/cut.c is now:
whitespace parser, then line- delimiter field mode, then byte-search
field mode, then the decoded multibyte parser.
* src/cut.c: Prefer strstr() over memmem() as the former is
optimized per platforma on GLIBC, where as the latter is only
optimized for S390 as of glibc-2.42
TODO: look at merging cut_fields and cut_fields_mb_utf8
as they're both byte search
TODO: See why a bit slower than old code
$ time src/cut.old -f1 -dç mb.in >/dev/null
real 0m0.136s
user 0m0.096s
sys 0m0.039s
$ time src/cut.new -f1 -dç mb.in >/dev/null
real 0m0.170s
user 0m0.139s
sys 0m0.030s
* gl/lib/mbbuf.h: To safely search a bounded buffer,
without needed to have unbounded memory with getndelim2.
TODO: rename mbbuf_utf8_safe_prefix to mbbuf_fill_utf8
Support ignoring leading and trailing whitespace.
E.g. this matches awk's default field splitting mode.
* src/cut.c
* tests/cut/cut.pl: Add test cases.
To improve compatibility with toybox/busybox scripts.
* doc/coreutils.texi (cut invocation): Add -O description.
* src/cut.c: Support -O as well as --output-delimiter
* tests/cut/cut.pl: Adjust one case to use -O.