Fix an unreleased issue due to the recent change
to using idx_t in commit v9.10-91-g02983e493
* src/cksum.c (main): Limit the possible return to
the range supported by idx_t.
Reported by Bruno Haible.
Do not compare only with the latest entry for given device id but also
all previously saved entries with the same id.
* src/df.c (struct devlist): Add next_same_dev struct member.
(filter_mount_list): Iterate over next_same_dev to find duplicates.
* tests/df/skip-duplicates.sh: Add test cases.
* NEWS: Mention the improvement.
https://redhat.atlassian.net/browse/RHEL-5649
This avoids the following behavior:
$ strace -e silence=exit -e trace=unlink,rmdir \
mktemp -d > /dev/full
unlink("/tmp/tmp.ZBuPmS9ZGD") = -1 EISDIR (Is a directory)
rmdir("/tmp/tmp.ZBuPmS9ZGD") = 0
mktemp: write error: No space left on device
In the above invocation we know that we created a directory, so we
should not remove a regular file that must have been created by another
process:
$ strace -e silence=exit -e trace=unlink,rmdir \
./src/mktemp -d > /dev/full
rmdir("/tmp/tmp.hGbME1HmJr") = 0
mktemp: write error: No space left on device
* src/mktemp.c (main): Prefer rmdir and unlink depending on whether we
created a directory or regular file.
* bootstrap.conf (gnulib_modules): Remove the remove module.
* src/cat.c (splice_cat): Don't bother resizing input as it generally
doesn't help perf, and also save an fstat per input. Don't close the
intermediate pipe once created, unless there is an error reading from
it.
Co-authored-by: Pádraig Brady <P@draigBrady.com>
Seen on GCC 15.2.1 with GLIBC 2.43 on Arch
Not seen on GCC 15.2.1 on GLIBC 2.42 on Fedora
* src/cut.c (search_bytes): Cast the return from memchr()
to avoid const propagation.
(find_field_delim): Adjust the return from strstr() similarly.
https://github.com/coreutils/coreutils/issues/244
* gl/lib/mbbuf.h: Adjust mbbuf_fill() to process full characters
in the slop at the end of a read(). Previously valid characters
in the last MCEL_LEN_MAX bytes were ignored until the next read().
* src/cut.c (cut_fields_bytesearch): Adjust to the new naming.
* NEWS: Mention the fold(1) responsiveness fix, which was
improved with the change from fread() to read(),
and completed with this patch.
$ time LC_ALL=C src/cut-before -b1 sl.in >/dev/null
real 0m0.115s
$ time LC_ALL=C src/cut-after -b1 sl.in >/dev/null
real 0m0.076s
* src/cut.c (cut_bytes): Hoist the fileno() invariant outside the loop.
Avoid memchr for very short lines.
(search_bytes): Similar to copy_bytes() and write_bytes() helpers.
Note adding code to probe 3 or 4 bytes resulted in worse register
allocation. I.e. slower operation even if the input was only 2 bytes.
With field delimiter = line delimiter we need to know
if there is any more data to be read, as field delimiter
in the last byte of the file is treated differently.
So reiterate the loop to ensure enough read()s to make
the appropriate determination.
* gl/lib/mbbuf.h (fill_buf): Switch from fread() to read()
as the former retries read() internally to fill the buffer.
* src/cut.c: Adjust accordingly, and avoid getc() interface entirely.
* bootstrap.h: Depend explicitly on fseterr. This is already depended
on transitively, so should not introduce new build portability issues.
* src/cut.c (bytesearch_field_delim_ok): Expand the range
of bytes that can be simply searched for. 0xF5-0xFF can't
appear in valid UTF-8 characters, and so may be used as
delimiters in UTF-8 input, so it's worth optimizing for.
* tests/cut/cut.pl: Add a test case (mainly as documentation).
We can only byte search with uni-byte or utf-8.
utf-8 implicitly can't false match a delimiter at buffer boundary.
So don't worry about finding the exact utf8 boundary at end of buffer,
rather just ensuring the buffer always starts with a valid character
(by ensuring MCEL_LEN_MAX-1 moved to start of buffer on each refill).
$ time src/cut-before -f10 -w ll.in >/dev/null
real 0m4.309s
$ time src/cut-after -f10 -w ll.in >/dev/null
real 0m3.136s
* src/cut.c (cut_bytes): Add a new helper that avoids
the memcpy call in the common case of adding characters to a buffer.
Simplify and optimize field exhaustion logic:
$ time LC_ALL=C src/cut-before -f1 -dc as.in >/dev/null
real 0m0.057s
$ time LC_ALL=C src/cut-after -f1 -dc as.in >/dev/null
real 0m0.023s
* src/cut.c (cut_fields_bytesearch): Refactor.
1. Removed !have_pending_line from the fast path condition.
This is safe because:
- field_1_n_bytes == 0 already ensures we haven't started
buffering field 1 content
- The fast path correctly continues any pending partial line
by writing raw bytes including the completing \n
2. Added have_pending_line = false after the fast path write,
since all lines up to the last \n are now complete.
$ time src/cut.before -f1 -dç sl.in >/dev/null
real 0m0.081s
$ time src/cut.after -f1 -dç sl.in >/dev/null
real 0m0.012s
$ time src/cut.before -f10 -dç sl.in >/dev/null
real 0m0.081s
$ time src/cut.after -f10 -dç sl.in >/dev/null
real 0m0.012s
For a 40% performance increase it's worth reinstating the simple
original cut_bytes() which avoids data copying and function calls.
Once a longer line is encountered we defer to the buffered variant.
$ time src/cut.before -b2 sl.in >/dev/null
real 0m0.101s
$ time src/cut.after -b2 sl.in >/dev/null
real 0m0.060s
Use specialized loops rather than branching per character,
giving a 28% increase.
$ time src/cut -f1 -w ll.in >/dev/null
real 0m7.199s
$ time src/cut -f1 -w ll.in >/dev/null
real 0m5.204s
This is quite significant:
yes abcdfeg | head -n1MB > big-file
$ time src/cut-before -b1,3 big-file >/dev/null
real 0m0.050s
$ time src/cut-after -b1,3 big-file >/dev/null
real 0m0.029s
Always memchr(line_delim) which is fast and allows:
- skipping whole segments when the next selected byte is beyond them
- skipping unselected prefixes in bulk
- writing contiguous selected spans in bulk
This wins for lines >= 4 characters,
but is slower lines <= 3 characters, especially if selecting bytes 1-3.
That is unusual though.
This is about 20x faster.
Note we only do the delimiter search once per chunk,
and it's usually quick as delimiters wouldn't be too far
into the a chunk if present, so we don't bother
to cache the found delimiter.
* src/cut.c (usage): Mention blank characters are used to separate.
* doc/coreutils.texi (cut invocation): Likewise. Also describe
the 'trimmed' argument and the relation to -F.
Allows better/simpler avoidance of repeated line/delim scans
TODO: speed up our really slow cut_fields_mb_any.
Compare for example:
time src/cut -w -f1 ll.in >/dev/null #14s
time src/cut -d, -f1 ll.in >/dev/null #.1s
Could adjust so that LC_ALL=C does memchr2(space,tab) ?