29030 Commits
Author SHA1 Message Date
Pádraig Brady 8d13292a73 version 8.32
* NEWS: Record release date.
v8.32
2020-03-05 13:23:50 +00:00
Pádraig Brady c8539d36c7 tests: don't rely on system env(1) being present
* tests/misc/env-S.pl: `env -i env` will call the system env
due to the path being cleared, so pass the absolute path
of our env binary under test to avoid that.  This was seen
to be an issue on Guix where /usr/bin/env was not available.
2020-03-04 18:00:38 +00:00
Pádraig Brady 0ea65e29dc basenc: avoid undefined behaviour in z85 processing
* src/basenc.c (z85_decode_ctx_init): Ensure we're working
with unsigned, as otherwise ubsan triggers with:
  src/basenc.c:767:18: runtime error: signed integer overflow:
  43 * 52200625 cannot be represented in type 'int'
(z85_encode): Likewise to avoid the usban error:
  src/basenc.c:630:26: runtime error:
  left shift of 134 by 24 places cannot be represented in type 'int'
2020-03-04 12:38:27 +00:00
Pádraig Brady f2034aa9bb tests: avoid a false failure on OpenIndiana 11
* tests/misc/timeout-parameters.sh: Split the large timeout
handling to ...
* tests/misc/timeout-large-parameters.sh: ... here, so that
the 3 second delay is contained in its own test, and if
the test is skipped due invalid handling within timeout(1),
it will be more apparent.
Also adjust the check so we skip whenever the kernel timer
fires immediately, to handle the buggy OpenIndiana 11 kernel also.
Reported by Bruno Haible.
2020-03-01 23:36:05 +00:00
Pádraig Brady ab305f746e tests: avoid a hang on GNU/Hurd from 2019
* tests/du/8gb.sh: Add a timeout around:
`dd bs=1 seek=8G of=big < /dev/null`
2020-03-01 22:04:20 +00:00
Pádraig Brady 04e2b9b1a6 tests: use bash in some scripts to avoid false failures
* init.cfg (require_bash_as_SHELL_): A new function to replace
SHELL for the current test, with bash if available.
This is useful on OpenIndiana 11 where /bin/sh was seen
to have races in handling of SIGPIPE.
* tests/misc/seq-epipe.sh: Use the new function to enforce bash.
* tests/misc/env-signal-handler.sh: Likewise.
Reported by Bruno Haible
2020-03-01 22:02:39 +00:00
Pádraig Brady 5cdab9f26b tests: improve test coverage for ls stat checks
* tests/ls/stat-free-color.sh: Check for the availability
of various stat calls individually, and add statx() and fstatat64()
to the list to check.  Fix the stat counting logic to
ignore lines like "+++ exited with 0 +++".
* tests/ls/stat-free-symlinks.sh: Check syscalls other than stat().
2020-03-01 13:06:01 +00:00
Bruno Haible e840c78685 tests: enable 4 more tests to be executed on FreeBSD
* init.cfg (gcc_shared_libs_): New variable.
(gcc_shared_): Use it, instead of hardcoding -ldl.
(require_gcc_shared_): Determine the suitable value
for gcc_shared_libs_.
2020-03-01 10:56:12 +00:00
Pádraig Brady ea1c1b7515 tests: fix incorrect || fail pattern in tests
* tests/ls/stat-free-symlinks.sh: s/|| fail/|| fail=1/.
* tests/misc/tee.sh: Likewise.
* tests/touch/relative.sh: Likewise.
* cfg.mk (sc_prohibit_or_fail): A new syntax-check to avoid this.
2020-02-29 22:45:00 +00:00
Pádraig Brady de73a867ca tests: avoid false failures on darwin 19.2.0
With these adjustments, all tests pass on macOS Catalina.

* tests/dd/sparse.sh: Adjust so that systems like apfs that
don't create holes < 16 MiB do not fail erroneously.
* tests/touch/trailing-slash.sh: Darwin was seen to dereference
symlinks to files when given a trailing slash, so avoid
that particular case.
2020-02-29 22:45:00 +00:00
Bruno Haible 54bf2c2f1a tests: fix test failure on FreeBSD 12
* tests/misc/csplit-io-err.sh: Limit the effect of the fwrite
override to streams != stderr, as fwrite is in the error() path there.
2020-02-29 22:39:47 +00:00
Jan Nieuwenhuizen 516cdf38af build: once again distribute .tar.gz files
* configure.ac: Reenable distribution of gzip-compressed
tarballs, for Guix bootstrapping reasons as discussed at:
https://lists.gnu.org/r/coreutils/2020-02/msg00042.html
* THANKS.in: Remove me, as now a committer.
* NEWS (Build-related): Mention this.
2020-02-27 21:55:57 +00:00
Pádraig Brady cc4c55e4bc maint: ensure .deps/ in the project root is ignored by git
* .gitignore: s|*/.deps/|.deps|
2020-02-27 16:44:03 +00:00
Pádraig Brady 8de80feb51 doc: remove older ChangeLog items
* Makefile.am: Update the oldest documented version
to 8.23 which is now about 5 years old.
2020-02-27 16:11:42 +00:00
Colin Watson 05a99f7d7f ls: issue error message on removed directory
If the current directory has been removed, then "ls" confusingly
produced no output and no error message, indistinguishable from
running on an empty directory.

* src/ls.c (print_dir): Report ENOENT on GNU/Linux if readdir
finds no directory entries at all, not even "." or "..",
and a recheck with the getdents syscall returns ENOENT.
We recheck with getdents() as POSIX states that
"The directory entries for dot and dot-dot are optional".
* tests/ls/removed-directory.sh: New file.
* tests/local.mk (all_tests): Add new test.
* NEWS: Mention the change in behavior.
Reported by Owen Thomas.
2020-02-27 14:11:37 +00:00
Pádraig Brady 93db70867d build: update to latest gnulib
* bootstrap.conf: Adjust for changes to fchmodat and fchownat,
which are now separated from chmodat and chownat respectively.
2020-02-25 14:26:39 +00:00
Pádraig Brady ead8464e3a b2sum: sync better with upstream
* src/blake2/blake2-impl.h: Sync load16() implementation,
which doesn't change code generation.
Also leverage (builtin) memcpy to more efficiently
move data on little endian systems,
giving a 2% win with GCC 9.2.1 on an i3-2310M.
2020-02-25 13:51:21 +00:00
Pádraig Brady 9134993562 factor: sync longlong.h adjustments from upstream
* src/longlong.h: Sync changes from:
https://gmplib.org/repo/gmp/log/tip/longlong.h
mips64: Provide r6 asm code as default expression yields.
arm32: Define sub_ddmmss separately for non-thumb (no rsc instruction).
powerpc: Add "CLOBBER" descriptions for some registers.
x86: Fix criterion for when to use mulx in umul_ppmm.
2020-02-25 11:55:38 +00:00
Pádraig Brady 75f79010cd stat,tail: sync file system constants from the linux kernel
* src/stat.c: Add magic constants for "binderfs", "dma-buf-fs",
"erofs", "ppc-cmm-fs", and "z3fold".
* NEWS: Mention the improvement.
2020-02-25 00:03:40 +00:00
Pádraig Brady 8e81d44b52 uniq: avoid strcoll() to improve performance and consistency
strcoll() is only significant to uniq(1) if it returns 0,
and it generally only does so with buggy locales or mismatched
locales and data.  Some systems may have strcoll()
return 0 for equivalent normalized unicode forms,
but for consistency across platforms strcoll() is avoided.
The various cases are defined in the new test.
This is consistent with newer POSIX standards as discussed at:
https://www.austingroupbugs.net/view.php?id=963

* src/uniq.c: s/xstrcoll/memcmp/.
* tests/local.mk: Reference the new test.
* tests/misc/uniq-collate.sh: Add a new test.
* NEWS: Mention the change in behavior.
Fixes https://bugs.gnu.org/38627
2020-02-24 10:12:42 +00:00
Pádraig Brady 61c49d3693 doc: clarify that '%a' stat format outputs mode bits
* src/stat.c (usage): Mention permission bits rather than
"access" so there is no confusion with ACLs etc.
Also indicate we output the file type with '%A'.
* doc/coreutils.texi (stat invocation): Likewise.
Also indicate '%A' is similar to `ls -ld` output.
Addresses https://bugs.gnu.org/39613
2020-02-15 11:24:15 +00:00
Pádraig Brady 3150f4a82e tests: fix test for symlink
* tests/cp/preserve-gid.sh: s/-l/-L/.
Reported by Kamil Dudka
2020-02-10 15:06:21 +00:00
Kamil Dudka b96b1a4728 tests: ensure tests/cp/preserve-gid.sh works with single binary
* tests/cp/preserve-gid.sh: If configured with --enable-single-binary
copy the coreutils single binary, instead of the cp one-line launcher.

Discussed at https://bugzilla.redhat.com/1800597
Fixes https://bugs.gnu.org/39485
2020-02-09 14:19:35 +00:00
Pádraig Brady 304bc557af maint: avoid syntax-check failure in previous commit
* configure.ac: Restrict lines to 80 chars.
2020-02-09 14:19:35 +00:00
Jim Meyering d7da53c47e build: suppress new FP warning from gcc-10.0.1
* configure.ac (GNULIB_WARN_CFLAGS): Add -Wno-return-local-addr
to avoid FP warning about careadlinkat.c. Discussed starting in
https://lists.gnu.org/r/coreutils/2020-02/msg00006.html
2020-02-08 17:52:48 -08:00
Pádraig Brady fa6d154b7b build: update to latest gnulib
Pick up recent build fixes to avoid sysctl.h inclusion on glibc systems,
restrict the max file size supported by read-file to PTRDIFF_MAX,
and to avoid a -Werror=unused failure in test-canonicalize.
2020-02-04 19:12:33 +00:00
Pádraig Brady ab108667ba tests: avoid false failure due to varying /proc/kallsyms
* tests/cp/proc-short-read.sh: Switch to using /proc/cpuinfo,
rather than /proc/kallsyms which was seen to vary in some cases.
Fixes https://bugs.gnu.org/39357
2020-02-04 19:11:37 +00:00
Pádraig Brady 1f443fe572 rmdir: fix --ignore-fail-on-non-empty with permissions errors
Since v6.10-21-ged5c4e7 `rmdir --ignore-fail-on-non-empty`
had reversed the failure status for directories that failed
to be removed for permissions reasons.  I.E. it would have
returned a failure status for such non empty dirs, and vice versa.

* src/rmdir.c (errno_may_be_non_empty): Rename from the
more confusing errno_may_be_empty(), and remove the EEXIST
case (specific to Solaris), which is moot here since
handled in errno_rmdir_non_empty().
(ignorable_failure): Fix the logic error so that
_non_ empty dirs are deemed to have ignorable failures.
(main): Fix clobbering of errno by is_empty_dir().
(remove_parents): Likewise.
* tests/rmdir/ignore.sh: Add a test case.
* THANKS.in: Add reporter who fixed the errno handling.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/39364
2020-02-04 19:11:37 +00:00
Pádraig Brady 709b4d9d8d build: avoid vector performance warnings in randperm
* configure.ac: Add -Wno-vector-operation-performance to suppress the
following gcc-9.2 error in gl/lib/randperm.c:
  error: vector operation will be expanded piecewise
2020-02-03 19:38:02 +00:00
Pádraig Brady 18c9382801 build: avoid including sysctl.h on glibc
* src/uname.c: Avoid unneeded header with GLIBC,
which has been deprecated since glibc-2.30.
* src/uptime.c: Likewise.
2020-02-03 19:38:02 +00:00
Pádraig Brady 2cecc3cc99 ls: support --time=creation to show/sort birth time
* src/ls.c (usage): Reorganize help for --time,
and add description for --time=birth.
(do_statx): Store btime in mtime if available.
(get_stat_btime): A new function to read the creation time
from the appropriate stat structure member.
(cmp_btime): A new function to compare birth time.
(print_long_format): Output '?' when birth time unavailable.
* doc/coreutils.texi: Document --time={birth,creation}.
* tests/local.mk: Reference the new test.
* tests/ls/birthtime.sh: Add a new test.
* NEWS: Mention the new feature.
2020-02-03 19:37:56 +00:00
Chris Meyering dda53d75a8 build: rearrange yes(1) code to prevent GCC 10 warning
* src/yes.c (main): Convert for loop to do-while in order to indicate
that the loop will be run at least once.
This avoids the following warning after the second loop:
src/yes.c:110:20: error: writing 1 byte into a region of size 0
2020-01-30 18:38:18 +00:00
Emil Engler fbbf81ec00 maint: add lib/iconv_open-zos.h to .gitignore
* .gitignore: Add file newly generated by gnulib commit 49e78fc
2020-01-01 23:59:40 +00:00
Pádraig Brady 0d77e1b7ea build: auto enable use of openssl with >= version 3
* configure.ac: Set --with-openssl=auto-gpl-compat as the default,
so that openssl is used for md5sum etc., with openssl >= 3,
which is newly licensed under ASL v2.
* gnulib: Update to include "auto-gpl-compat" support.
2020-01-01 22:14:45 +00:00
Pádraig Brady ff2f85273a maint: adjust to split out xstrtol-error gnulib module
* bootstrap.conf: Depend on the new module split from xstrtol.
* src/df.c: Include "xstrtol-error.h" for xstrtol_fatal.
* src/du.c: Likewise.
* src/ls.c: Likewise.
* src/od.c: Likewise.
* src/pr.c: Likewise.
* src/sort.c: Likewise.
2020-01-01 14:36:12 +00:00
Pádraig Brady aaba82431c maint: update all copyright year number ranges
Run "make update-copyright" and then...

* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
2020-01-01 14:16:56 +00:00
Bernhard Voelker 7490e94aa7 doc: add example to demonstrate sub-second sleep times
* doc/coreutils.texi (sleep invocation): Add an example to demonstrate
how to use the floating-point and the scientific notation to sleep
for sub-second times, e.g. milli-, micro- and nanoseconds.

Inspired by Stephane Chazelas in:
  https://lists.gnu.org/r/coreutils/2019-12/msg00005.html
2019-12-08 17:44:37 +01:00
Pádraig Brady ec5ab42604 maint: fix syntax-check failure from recent adjustment
* cfg.mk (old_NEWS_hash): Regenerate after commit v8.31-56-gc1e1965.
2019-12-02 16:21:08 +00:00
Kamil Dudka 5118a2e392 chcon: do not validate security context if SELinux is disabled
* src/chcon.c (main): Skip call of security_check_context()
in case SELinux is disabled to avoid unnecessary failure.

Bug: https://bugzilla.redhat.com/1777831
2019-12-02 16:21:05 +00:00
Paul Eggert ad591a52b4 doc: remove colon from node name
* doc/sort-version.texi (Minus/Hyphen and Colon characters):
Rename from “Minus/Hyphen @samp{-} and Colon @samp{:} characters”,
as texi2any 6.6 complains about colons in node names.
2019-11-11 16:54:19 -08:00
Paul Eggert 0103fb0244 shred: modernize documentation
* doc/coreutils.texi (shred invocation):
Modernize discussion to today’s technology (Bug#38168).
* src/shred.c (usage): Omit lengthy duplication of the manual’s
discussion of file systems and storage devices, as that became out
of sync with the manual.  Instead, just cite the manual.
2019-11-11 16:54:19 -08:00
Paul Eggert 04b136e298 all: improve parsing of numeric arguments
This addresses a longstanding "update all callers" FIXME in
lib/xstrtol.c, by having programs check that numbers do not
have unknown suffixes.  The problem was also reported for
'shuf' by my student Maggie Huang while reimplementing a shuf
subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
This patch also improves the portability of the code to unusual
platforms where ULONG_MAX < SIZE_MAX.
* NEWS: Mention user-visible changes.
* src/chgrp.c (parse_group):
* src/chroot.c (parse_additional_groups):
* src/du.c (main):
* src/install.c (get_ids):
* src/join.c (string_to_join_field):
* src/ls.c (decode_switches):
* src/md5sum.c (split_3):
* src/shuf.c (main):
* src/sort.c (specify_nthreads):
* src/uniq.c (size_opt, main):
Use uintmax_t instead of unsigned long, for portability
to oddball platforms where unsigned long is not wide enough.
* src/du.c (main):
* src/expr.c (mpz_init_set_str) [!HAVE_GMP]:
* src/install.c (get_ids):
* src/ls.c (decode_switches):
* src/mknod.c (main):
* src/ptx.c (main):
* src/shuf.c (main):
* src/sort.c (specify_nmerge, specify_nthreads):
Reject numbers with suffixes.
* src/md5sum.c (split_3): Simplify.
2019-10-22 15:04:43 -07:00
Paul Eggert ed2db6886b stdbuf: improve size checking
* bootstrap.conf (gnulib_modules): Add minmax.
* src/libstdbuf.c: Include stdint.h, minmax.h.
(apply_mode): Don’t assume SIZE_MAX <= ULONG_MAX.
Improve checking for invalid sizes.
2019-10-22 15:04:43 -07:00
Paul Eggert 74163ea677 shuf: improve randperm overflow checking
* gl/lib/randperm.c: Include randperm.h first, since it’s the API.
Include stdint.h, count-leading-zeros.h, verify.h.
(floor_lg): Rename from ceil_log (which was not actually
implementing the ceiling!) and implement the floor using
count_leading_zeros.
(randperm_bound): Use floor_lg, not ceil_log.  Use uintmax_t
instead of size_t in case the size gets large on a 32-bit host.
* gl/modules/randperm (Depends-on): Add count-leading-zeros, stdint.
2019-10-22 15:04:43 -07:00
Paul Eggert 565dd395c3 build: don’t worry about logical-op checking
* configure.ac: Remove code tailoring --enable-gcc-warnings
to GCC 4.7 and earlier, as developers no longer need to worry
about GCCs that old.
2019-10-22 15:04:43 -07:00
Paul Eggert 6716a79773 build: re-enable type-limits checking
* configure.ac: When --enable-gcc-warnings is used, omit
-Wno-type-limits.  The need for -Wno-type-limits has passed, now
that intprops.h uses builtin primitives for GCC 5 and later, given
that recent GCCs issue type-limits warnings only for non-constant
expressions.  --enable-gcc-warnings is not intended for use with
old compilers, so we can drop -Wno-type-limits now.
2019-10-22 15:04:42 -07:00
Paul Eggert c1e19656c8 shuf: fix bug with ‘-r -n 0’
‘shuf -r -n 0 file’ would mistakenly read from standard input.
Problem reported by my student Jingnong Qu while reimplementing a
shuf subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
* NEWS: Mention the fix.  Also, ASCIIfy a previous item.
* src/shuf.c (main): Fix bug.
* tests/misc/shuf.sh: Add a test case for the bug.
2019-10-21 13:20:35 -07:00
Jeff Layton a99ab26611 ls: use statx instead of stat when available
statx allows ls to indicate interest in only certain inode metadata.
This is potentially a win on networked/clustered/distributed
file systems. In cases where we'd have to do a full, heavyweight stat()
call we can now do a much lighter statx() call.

As a real-world example, consider a file system like CephFS where one
client is actively writing to a file and another client does an
ls --color in the same directory. --color means that we need to fetch
the mode of the file.

Doing that with a stat() call means that we have to fetch the size and
mtime in addition to the mode. The MDS in that situation will have to
revoke caps in order to ensure that it has up-to-date values to report,
which disrupts the writer.

This has a measurable affect on performance. I ran a fio sequential
write test on one cephfs client and had a second client do "ls --color"
in a tight loop on the directory that held the file:

Baseline -- no activity on the second client:

WRITE: bw=76.7MiB/s (80.4MB/s), 76.7MiB/s-76.7MiB/s (80.4MB/s-80.4MB/s),
       io=4600MiB (4824MB), run=60016-60016msec

Without this patch series, we see a noticable performance hit:

WRITE: bw=70.4MiB/s (73.9MB/s), 70.4MiB/s-70.4MiB/s (73.9MB/s-73.9MB/s),
       io=4228MiB (4433MB), run=60012-60012msec

With this patch series, we gain most of that ground back:

WRITE: bw=75.9MiB/s (79.6MB/s), 75.9MiB/s-75.9MiB/s (79.6MB/s-79.6MB/s),
       io=4555MiB (4776MB), run=60019-60019msec

* src/stat.c: move statx to stat struct conversion to new header...
* src/statx.h: ...here.
* src/ls.c: Add wrapper functions for stat/lstat/fstat calls,
and add variants for when we are only interested in specific info.
Add statx-enabled functions and set the request mask based on the
output format and what values are needed.
* NEWS: Mention the Improvement.
2019-10-09 22:07:42 +01:00
Paul Eggert 86ee5548c1 truncate: avoid integer-overflow assumptions
* src/truncate.c (do_ftruncate): Simplify overflow checking,
and don’t rely on theoretically-nonportable assumptions
like assuming that OFF_MAX < UINTMAX_MAX.
2019-10-03 12:41:51 -07:00
Paul Eggert ba1ea8caee numfmt: avoid unlikely integer overflow
* src/numfmt.c (parse_format_string): Report overflow if
pad < -LONG_MAX, since that can’t be negated.
2019-10-03 12:41:51 -07:00