This helps identify the outliers for long filenames, and also produces
a more compact display of columns when listing a directory with many
entries of various widths.
* src/ls.c (sort_type, sort_types, sort_width): New sort_width sort
type.
(sort_args): Add "width" sort arg.
(cmp_width, fileinfo_width): New sort function and helper for file name
width.
(quote_name_width): Add function prototype declaration.
(usage): Document --sort=width option.
* doc/coreutils.texi: Document --sort=width option.
* tests/ls/sort-width-option.sh: New test for --sort=width option.
* tests/local.mk: Reference new test.
* NEWS: Mention the new feature.
The diacrit module is obsolete, and ptx’s use of it is obsolete
too; it assumes an 8-bit locale (not that common these days) and
that TeX cannot process the 8-bit characters (nowadays, it can).
* NEWS, doc/coreutils.texi (Charset selection in ptx): Document this.
* bootstrap.conf (gnulib_modules): Remove diacrit.
* src/ptx.c: Do not include diacrit.h.
(print_field, fix_output_parameters): Remove obsolete support
for 8-bit diacritics.
* src/cksum.c: (main): Use getopt_long to parse options,
and handle the new --debug option.
(pclmul_supported): Diagnose various failures and attempts.
* NEWS: Mention the new option.
GNU/Linux is unusual here in that rmdir("symlink/") returns ENOTDIR,
whereas Solaris and FreeBSD at least, will follow the symlink
and remove the target directory. We don't make the behavior
on Linux kernels consistent, but at least clarify
the confusing error message.
* src/rmdir (main): Output a specific error message for the above case.
(remove_parents): In the error message, don't assume intermediate paths
are directories, as they could be symlinks.
* tests/rmdir/symlink-errors.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
This regression was introduced in commit COREUTILS-6_8-58-g553d347d3
* src/pr.c (init_parameters): Process tabs for multiple columns.
* tests/pr/pr-tests.pl: Add test cases.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/46422
- \r\n is common a line end combination
- catting such a file without options causes it to display normally
- overwriting the first char with $, loses info
* src/cat.c (cat): Convert \r preceeding a \n to ^M.
* tests/misc/cat-E.sh: New test.
* tests/local.mk: Reference new test.
* tests/misc/cat-proc.sh: Fix typo.
* doc/coreutils.texi (cat invocation): Mention the new behavior.
* NEWS: Mention the improvement.
This functionality regressed with the adjustments
in commit v8.25-4-g62e7af032
* src/split.c (bytes_chunk_extract): Account for already read data
when seeking into the file.
* tests/split/b-chunk.sh: Use the hidden ---io-blksize option,
to test this functionality.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/46048
Problem reported by David McCall (Bug#45886).
I introduced this problem when fixing Bug#14371.
* NEWS: Mention the fix.
* src/mkdir.c (struct mkdir_options): New members umask_ancestor,
umask_self, replacing umask_value.
(make_ancestor): Use them when temporarily adjusting umask.
(main): Set them, and set the umask to umask_self instead
of leaving it alone.
* tests/mkdir/perm.sh (tests): Add test case for bug.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
A 100MB file improves from 2.50s to 1.80s on a Sparc T5220
A 100MB file improves from 0.54s to 0.13s on an i3-2310M
* bootstrap.conf: Explicitly depend on byteswap,
since now used directly by coreutils.
* src/cksum.c (cksum): Process in multiples of 8 bytes.
(main): Adjust for generation of expanded crctab.
* src/cksum.h: Split now larger crctab to separate header.
* src/local.mk: Reference the new header.
* NEWS: Mention the improvement.
* src/nl.c (main): Enforce the POSIX specified
behavior of assuming ':' is specified after a single
character argument to -d.
* tests/misc/nl.sh: Add a test case.
* NEWS: Mention the bug fix.
When removing a directory fails for some reason, and that directory
is empty, the rm_fts code gets the return value of the excise call
confused with the return value of its earlier call to prompt,
causing fts_skip_tree to be called again and the next file
that rm would otherwise have deleted to survive.
* src/remove.c (rm_fts): Ensure we only skip a single fts entry,
when processing empty dirs. I.e. only skip the entry
having successfully removed it.
* tests/rm/empty-immutable-skip.sh: New root-only test.
* tests/local.mk: Add it.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/44883
This crash was identified by Cyber Independent Testing Lab:
https://cyber-itl.org/2020/10/28/citl-7000-defects.html
and was introduced with commit v8.5-163-g3f48829c2
* src/tr.c (validate_case_classes): Don't apply these
extra case alignment checks in the --complement case,
which is even more restrictive as to the contents of SET2.
* tests/misc/tr-case-class.sh: Add a test case,
for a large SET1, which caused the length adjustment
in validate_case_classes to underflow and trigger the assert.
* NEWS: Mention the bug fix.
This crash was identified by Cyber Independent Testing Lab:
https://cyber-itl.org/2020/10/28/citl-7000-defects.html
and was introduced with commit v6.9.90-11-g4245876e2
* src/ls.c (gobble_file): Ensure scontext is initialized
in the case where files are not statable.
* tests/ls/selinux-segfault.sh: Renamed from proc-selinux-segfault.sh,
and added test case for broken symlinks.
* tests/local.mk: Adjust for the renamed test.
* NEWS: Mention the bug fix.
* src/csplit.c (process_regexp): Process the line suppression
in all invocations so that the last match is suppressed.
Previously with a non infinite match count,
the last regex pattern was not suppressed.
* NEWS: Mention the bug fix.
* tests/misc/csplit-suppress-matched.pl: Add a test case.
Fixes https://bugs.gnu.org/42764
If it works, prefer lseek with SEEK_DATA and SEEK_HOLE to FIEMAP,
as lseek is simpler and more portable (will be in next POSIX).
Problem reported in 2011 by Jeff Liu (Bug#8061).
* NEWS: Mention this.
* src/copy.c (lseek_copy) [SEEK_HOLE]: New function.
(enum scantype): New constants ERROR_SCANTYPE, LSEEK_SCANTYPE.
(union scan_inference): New type.
(infer_scantype): Last arg is now union scan_inference *,
not struct extent_scan *. All callers changed.
Prefer SEEK_HOLE to FIEMAP if both work, since
SEEK_HOLE is simpler and more portable.
(copy_reg): Do the fdadvise after initial scan, in case the scan
fails. Report an error if the initial scan fails.
(copy_reg) [SEEK_HOLE]: Use lseek_copy if scantype says so.
* src/stat.c (human_fstype): Add case for the 'vboxsf' file system type
which is used for VirtualBox Shared Folders mounted in VirtualBox guest
VMs.
* NEWS: Mention the Improvement.
Fixes https://bugs.gnu.org/41935
Have the `ls` `--classify` option take an optional argument for when to
classify ("always", "auto", "never"), just like the optional argument
for `--color`. When the optional argument is not specified, default to
"always" for backwards compatibility.
* src/ls.c (usage): Update help text.
(decode_switches): Support an optional argument for --classify.
* tests/ls/classify.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the new feature.
Pull in a fix for FTS to avoid a crash when traversing a heavily
changed XFS file system:
> fts: remove NOSTAT_LEAF_OPTIMIZATION
* NEWS (Bug fixes): Mention the fix.
* gnulib: Update to latest.
* bootstrap: Sync from gnulib/build-aux/bootstrap.
Discussed at:
<https://lists.gnu.org/r/bug-gnulib/2020-04/msg00068.html>
* NEWS: Mention this.
* src/ls.c: Do not include <sys/sycall.h>
(print_dir): Don't worry about whether the directory is removed.
* tests/ls/removed-directory.sh: Adjust to match new (i.e., old)
behavior.
* configure.ac: Reenable distribution of gzip-compressed
tarballs, for Guix bootstrapping reasons as discussed at:
https://lists.gnu.org/r/coreutils/2020-02/msg00042.html
* THANKS.in: Remove me, as now a committer.
* NEWS (Build-related): Mention this.
If the current directory has been removed, then "ls" confusingly
produced no output and no error message, indistinguishable from
running on an empty directory.
* src/ls.c (print_dir): Report ENOENT on GNU/Linux if readdir
finds no directory entries at all, not even "." or "..",
and a recheck with the getdents syscall returns ENOENT.
We recheck with getdents() as POSIX states that
"The directory entries for dot and dot-dot are optional".
* tests/ls/removed-directory.sh: New file.
* tests/local.mk (all_tests): Add new test.
* NEWS: Mention the change in behavior.
Reported by Owen Thomas.
strcoll() is only significant to uniq(1) if it returns 0,
and it generally only does so with buggy locales or mismatched
locales and data. Some systems may have strcoll()
return 0 for equivalent normalized unicode forms,
but for consistency across platforms strcoll() is avoided.
The various cases are defined in the new test.
This is consistent with newer POSIX standards as discussed at:
https://www.austingroupbugs.net/view.php?id=963
* src/uniq.c: s/xstrcoll/memcmp/.
* tests/local.mk: Reference the new test.
* tests/misc/uniq-collate.sh: Add a new test.
* NEWS: Mention the change in behavior.
Fixes https://bugs.gnu.org/38627
Since v6.10-21-ged5c4e7 `rmdir --ignore-fail-on-non-empty`
had reversed the failure status for directories that failed
to be removed for permissions reasons. I.E. it would have
returned a failure status for such non empty dirs, and vice versa.
* src/rmdir.c (errno_may_be_non_empty): Rename from the
more confusing errno_may_be_empty(), and remove the EEXIST
case (specific to Solaris), which is moot here since
handled in errno_rmdir_non_empty().
(ignorable_failure): Fix the logic error so that
_non_ empty dirs are deemed to have ignorable failures.
(main): Fix clobbering of errno by is_empty_dir().
(remove_parents): Likewise.
* tests/rmdir/ignore.sh: Add a test case.
* THANKS.in: Add reporter who fixed the errno handling.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/39364
* src/ls.c (usage): Reorganize help for --time,
and add description for --time=birth.
(do_statx): Store btime in mtime if available.
(get_stat_btime): A new function to read the creation time
from the appropriate stat structure member.
(cmp_btime): A new function to compare birth time.
(print_long_format): Output '?' when birth time unavailable.
* doc/coreutils.texi: Document --time={birth,creation}.
* tests/local.mk: Reference the new test.
* tests/ls/birthtime.sh: Add a new test.
* NEWS: Mention the new feature.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
This addresses a longstanding "update all callers" FIXME in
lib/xstrtol.c, by having programs check that numbers do not
have unknown suffixes. The problem was also reported for
'shuf' by my student Maggie Huang while reimplementing a shuf
subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
This patch also improves the portability of the code to unusual
platforms where ULONG_MAX < SIZE_MAX.
* NEWS: Mention user-visible changes.
* src/chgrp.c (parse_group):
* src/chroot.c (parse_additional_groups):
* src/du.c (main):
* src/install.c (get_ids):
* src/join.c (string_to_join_field):
* src/ls.c (decode_switches):
* src/md5sum.c (split_3):
* src/shuf.c (main):
* src/sort.c (specify_nthreads):
* src/uniq.c (size_opt, main):
Use uintmax_t instead of unsigned long, for portability
to oddball platforms where unsigned long is not wide enough.
* src/du.c (main):
* src/expr.c (mpz_init_set_str) [!HAVE_GMP]:
* src/install.c (get_ids):
* src/ls.c (decode_switches):
* src/mknod.c (main):
* src/ptx.c (main):
* src/shuf.c (main):
* src/sort.c (specify_nmerge, specify_nthreads):
Reject numbers with suffixes.
* src/md5sum.c (split_3): Simplify.
‘shuf -r -n 0 file’ would mistakenly read from standard input.
Problem reported by my student Jingnong Qu while reimplementing a
shuf subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
* NEWS: Mention the fix. Also, ASCIIfy a previous item.
* src/shuf.c (main): Fix bug.
* tests/misc/shuf.sh: Add a test case for the bug.
statx allows ls to indicate interest in only certain inode metadata.
This is potentially a win on networked/clustered/distributed
file systems. In cases where we'd have to do a full, heavyweight stat()
call we can now do a much lighter statx() call.
As a real-world example, consider a file system like CephFS where one
client is actively writing to a file and another client does an
ls --color in the same directory. --color means that we need to fetch
the mode of the file.
Doing that with a stat() call means that we have to fetch the size and
mtime in addition to the mode. The MDS in that situation will have to
revoke caps in order to ensure that it has up-to-date values to report,
which disrupts the writer.
This has a measurable affect on performance. I ran a fio sequential
write test on one cephfs client and had a second client do "ls --color"
in a tight loop on the directory that held the file:
Baseline -- no activity on the second client:
WRITE: bw=76.7MiB/s (80.4MB/s), 76.7MiB/s-76.7MiB/s (80.4MB/s-80.4MB/s),
io=4600MiB (4824MB), run=60016-60016msec
Without this patch series, we see a noticable performance hit:
WRITE: bw=70.4MiB/s (73.9MB/s), 70.4MiB/s-70.4MiB/s (73.9MB/s-73.9MB/s),
io=4228MiB (4433MB), run=60012-60012msec
With this patch series, we gain most of that ground back:
WRITE: bw=75.9MiB/s (79.6MB/s), 75.9MiB/s-75.9MiB/s (79.6MB/s-79.6MB/s),
io=4555MiB (4776MB), run=60019-60019msec
* src/stat.c: move statx to stat struct conversion to new header...
* src/statx.h: ...here.
* src/ls.c: Add wrapper functions for stat/lstat/fstat calls,
and add variants for when we are only interested in specific info.
Add statx-enabled functions and set the request mask based on the
output format and what values are needed.
* NEWS: Mention the Improvement.
Under certain circumstances seq prints an extra line when the output
format has custom format with characters following the printed numbers:
$ seq -f "%g " 1000000 1000000
1e+06
1e+06
This is due to the "print_extra_number" logic using strings to determine
whether a 'extra number' is needed, but only one string was trimmed
when using a custom printf format.
Prompted by https://lists.gnu.org/r/coreutils/2019-08/msg00001.html
* NEWS: Mention fix.
* src/seq.c (print_numbers): Trim the 'x0_str' string before comparing
it to the previous 'x_str' string.
* tests/misc/seq-extra-number.sh: Add this scenario.
* tests/local.mk (all_tests): Add new test.
Problem reported by Szőts Ákos (Bug#36291).
* NEWS: Mention this.
* src/od.c (skip): Try fseek even on files that do not have usable
sizes, falling back on fread if fseek fails.
* src/stat.c: Drop statbuf argument from out_epoch_sec().
Use statx() rather than [lf]stat() where available,
so a separate call is not required to get birth time.
Set STATX_* mask bits only for things we want to print,
which can be more efficient on some file systems.
Add a new --cache= command-line option that sets the appropriate hint
flags in the statx call. These are primarily used with network
file systems to indicate what level of cache coherency is desired.
The new option is available unconditionally for better portability,
and ignored where not implemented.
* doc/coreutils.texi: Add documention for --cached.
* man/stat.x (SEE ALSO): Mention statx().
* NEWS: Mention the new feature.