Commit Graph
8099 Commits
Author SHA1 Message Date
Pádraig Brady 75f79010cd stat,tail: sync file system constants from the linux kernel
* src/stat.c: Add magic constants for "binderfs", "dma-buf-fs",
"erofs", "ppc-cmm-fs", and "z3fold".
* NEWS: Mention the improvement.
2020-02-25 00:03:40 +00:00
Pádraig Brady 8e81d44b52 uniq: avoid strcoll() to improve performance and consistency
strcoll() is only significant to uniq(1) if it returns 0,
and it generally only does so with buggy locales or mismatched
locales and data.  Some systems may have strcoll()
return 0 for equivalent normalized unicode forms,
but for consistency across platforms strcoll() is avoided.
The various cases are defined in the new test.
This is consistent with newer POSIX standards as discussed at:
https://www.austingroupbugs.net/view.php?id=963

* src/uniq.c: s/xstrcoll/memcmp/.
* tests/local.mk: Reference the new test.
* tests/misc/uniq-collate.sh: Add a new test.
* NEWS: Mention the change in behavior.
Fixes https://bugs.gnu.org/38627
2020-02-24 10:12:42 +00:00
Pádraig Brady 61c49d3693 doc: clarify that '%a' stat format outputs mode bits
* src/stat.c (usage): Mention permission bits rather than
"access" so there is no confusion with ACLs etc.
Also indicate we output the file type with '%A'.
* doc/coreutils.texi (stat invocation): Likewise.
Also indicate '%A' is similar to `ls -ld` output.
Addresses https://bugs.gnu.org/39613
2020-02-15 11:24:15 +00:00
Pádraig Brady 1f443fe572 rmdir: fix --ignore-fail-on-non-empty with permissions errors
Since v6.10-21-ged5c4e7 `rmdir --ignore-fail-on-non-empty`
had reversed the failure status for directories that failed
to be removed for permissions reasons.  I.E. it would have
returned a failure status for such non empty dirs, and vice versa.

* src/rmdir.c (errno_may_be_non_empty): Rename from the
more confusing errno_may_be_empty(), and remove the EEXIST
case (specific to Solaris), which is moot here since
handled in errno_rmdir_non_empty().
(ignorable_failure): Fix the logic error so that
_non_ empty dirs are deemed to have ignorable failures.
(main): Fix clobbering of errno by is_empty_dir().
(remove_parents): Likewise.
* tests/rmdir/ignore.sh: Add a test case.
* THANKS.in: Add reporter who fixed the errno handling.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/39364
2020-02-04 19:11:37 +00:00
Pádraig Brady 18c9382801 build: avoid including sysctl.h on glibc
* src/uname.c: Avoid unneeded header with GLIBC,
which has been deprecated since glibc-2.30.
* src/uptime.c: Likewise.
2020-02-03 19:38:02 +00:00
Pádraig Brady 2cecc3cc99 ls: support --time=creation to show/sort birth time
* src/ls.c (usage): Reorganize help for --time,
and add description for --time=birth.
(do_statx): Store btime in mtime if available.
(get_stat_btime): A new function to read the creation time
from the appropriate stat structure member.
(cmp_btime): A new function to compare birth time.
(print_long_format): Output '?' when birth time unavailable.
* doc/coreutils.texi: Document --time={birth,creation}.
* tests/local.mk: Reference the new test.
* tests/ls/birthtime.sh: Add a new test.
* NEWS: Mention the new feature.
2020-02-03 19:37:56 +00:00
Chris Meyering dda53d75a8 build: rearrange yes(1) code to prevent GCC 10 warning
* src/yes.c (main): Convert for loop to do-while in order to indicate
that the loop will be run at least once.
This avoids the following warning after the second loop:
src/yes.c:110:20: error: writing 1 byte into a region of size 0
2020-01-30 18:38:18 +00:00
Pádraig Brady ff2f85273a maint: adjust to split out xstrtol-error gnulib module
* bootstrap.conf: Depend on the new module split from xstrtol.
* src/df.c: Include "xstrtol-error.h" for xstrtol_fatal.
* src/du.c: Likewise.
* src/ls.c: Likewise.
* src/od.c: Likewise.
* src/pr.c: Likewise.
* src/sort.c: Likewise.
2020-01-01 14:36:12 +00:00
Pádraig Brady aaba82431c maint: update all copyright year number ranges
Run "make update-copyright" and then...

* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
2020-01-01 14:16:56 +00:00
Kamil Dudka 5118a2e392 chcon: do not validate security context if SELinux is disabled
* src/chcon.c (main): Skip call of security_check_context()
in case SELinux is disabled to avoid unnecessary failure.

Bug: https://bugzilla.redhat.com/1777831
2019-12-02 16:21:05 +00:00
Paul Eggert 0103fb0244 shred: modernize documentation
* doc/coreutils.texi (shred invocation):
Modernize discussion to today’s technology (Bug#38168).
* src/shred.c (usage): Omit lengthy duplication of the manual’s
discussion of file systems and storage devices, as that became out
of sync with the manual.  Instead, just cite the manual.
2019-11-11 16:54:19 -08:00
Paul Eggert 04b136e298 all: improve parsing of numeric arguments
This addresses a longstanding "update all callers" FIXME in
lib/xstrtol.c, by having programs check that numbers do not
have unknown suffixes.  The problem was also reported for
'shuf' by my student Maggie Huang while reimplementing a shuf
subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
This patch also improves the portability of the code to unusual
platforms where ULONG_MAX < SIZE_MAX.
* NEWS: Mention user-visible changes.
* src/chgrp.c (parse_group):
* src/chroot.c (parse_additional_groups):
* src/du.c (main):
* src/install.c (get_ids):
* src/join.c (string_to_join_field):
* src/ls.c (decode_switches):
* src/md5sum.c (split_3):
* src/shuf.c (main):
* src/sort.c (specify_nthreads):
* src/uniq.c (size_opt, main):
Use uintmax_t instead of unsigned long, for portability
to oddball platforms where unsigned long is not wide enough.
* src/du.c (main):
* src/expr.c (mpz_init_set_str) [!HAVE_GMP]:
* src/install.c (get_ids):
* src/ls.c (decode_switches):
* src/mknod.c (main):
* src/ptx.c (main):
* src/shuf.c (main):
* src/sort.c (specify_nmerge, specify_nthreads):
Reject numbers with suffixes.
* src/md5sum.c (split_3): Simplify.
2019-10-22 15:04:43 -07:00
Paul Eggert ed2db6886b stdbuf: improve size checking
* bootstrap.conf (gnulib_modules): Add minmax.
* src/libstdbuf.c: Include stdint.h, minmax.h.
(apply_mode): Don’t assume SIZE_MAX <= ULONG_MAX.
Improve checking for invalid sizes.
2019-10-22 15:04:43 -07:00
Paul Eggert c1e19656c8 shuf: fix bug with ‘-r -n 0’
‘shuf -r -n 0 file’ would mistakenly read from standard input.
Problem reported by my student Jingnong Qu while reimplementing a
shuf subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
* NEWS: Mention the fix.  Also, ASCIIfy a previous item.
* src/shuf.c (main): Fix bug.
* tests/misc/shuf.sh: Add a test case for the bug.
2019-10-21 13:20:35 -07:00
Jeff Layton a99ab26611 ls: use statx instead of stat when available
statx allows ls to indicate interest in only certain inode metadata.
This is potentially a win on networked/clustered/distributed
file systems. In cases where we'd have to do a full, heavyweight stat()
call we can now do a much lighter statx() call.

As a real-world example, consider a file system like CephFS where one
client is actively writing to a file and another client does an
ls --color in the same directory. --color means that we need to fetch
the mode of the file.

Doing that with a stat() call means that we have to fetch the size and
mtime in addition to the mode. The MDS in that situation will have to
revoke caps in order to ensure that it has up-to-date values to report,
which disrupts the writer.

This has a measurable affect on performance. I ran a fio sequential
write test on one cephfs client and had a second client do "ls --color"
in a tight loop on the directory that held the file:

Baseline -- no activity on the second client:

WRITE: bw=76.7MiB/s (80.4MB/s), 76.7MiB/s-76.7MiB/s (80.4MB/s-80.4MB/s),
       io=4600MiB (4824MB), run=60016-60016msec

Without this patch series, we see a noticable performance hit:

WRITE: bw=70.4MiB/s (73.9MB/s), 70.4MiB/s-70.4MiB/s (73.9MB/s-73.9MB/s),
       io=4228MiB (4433MB), run=60012-60012msec

With this patch series, we gain most of that ground back:

WRITE: bw=75.9MiB/s (79.6MB/s), 75.9MiB/s-75.9MiB/s (79.6MB/s-79.6MB/s),
       io=4555MiB (4776MB), run=60019-60019msec

* src/stat.c: move statx to stat struct conversion to new header...
* src/statx.h: ...here.
* src/ls.c: Add wrapper functions for stat/lstat/fstat calls,
and add variants for when we are only interested in specific info.
Add statx-enabled functions and set the request mask based on the
output format and what values are needed.
* NEWS: Mention the Improvement.
2019-10-09 22:07:42 +01:00
Paul Eggert 86ee5548c1 truncate: avoid integer-overflow assumptions
* src/truncate.c (do_ftruncate): Simplify overflow checking,
and don’t rely on theoretically-nonportable assumptions
like assuming that OFF_MAX < UINTMAX_MAX.
2019-10-03 12:41:51 -07:00
Paul Eggert ba1ea8caee numfmt: avoid unlikely integer overflow
* src/numfmt.c (parse_format_string): Report overflow if
pad < -LONG_MAX, since that can’t be negated.
2019-10-03 12:41:51 -07:00
Paul Eggert ac2b4a43e6 nl: fix integer-overflow bug
Problem reported by Roland Illig (Bug#37585)
* src/nl.c (print_lineno): Don’t rely on undefined behavior when
checking for integer overflow.
2019-10-03 12:41:51 -07:00
Paul Eggert bb23a7563f cp: simplify integer overflow checking
* src/copy.c (sparse_copy): Use INT_ADD_WRAPV instead
of doing overflow checking by hand.
2019-10-03 12:41:51 -07:00
Pádraig Brady 1c8050cc4d seq: use faster processing for integer steps from 2 to 200
* src/seq.c: (seq_fast): Accept STEP as a parameter and use that
to skip the output of generated numbers.
(main): Relax to using seq_fast for integer steps between 1 and 200.
For larger steps the throughput was faster using the standard
incrementing procedure.
(cmp): Use the equivalent but faster memcmp for equal len strings.
* tests/misc/seq.pl: Update fast path cases.
Addresses https://bugs.gnu.org/37241
2019-09-08 18:59:24 +01:00
Bruno Haible a3d070fa32 build: adjust for recent gnulib pthread changes
Discussed in https://lists.gnu.org/r/coreutils/2019-08/msg00030.html .

* bootstrap.conf (gnulib_modules): Replace 'pthread' with
pthread-* modules.
* src/sort.c: Remove GNULIB_defined_pthread_functions conditional.
2019-08-12 15:10:46 -06:00
Pádraig Brady becf0db0da doc: clarify that truncate creates sparse files
* src/truncate.c (usage): Explicitly mention "sparse".
* doc/coreutils.texi (truncate invocation): Likewise.
Addresses https://bugs.gnu.org/36963
2019-08-08 13:34:54 +01:00
Mike Swanson eb5a9ebaf1 dircolors: recognize the WebP image format
* src/dircolors.hin: Add .webp for the WebP image format.
Fixes https://bugs.gnu.org/36899
2019-08-07 07:17:27 +02:00
Assaf Gordon 07f811a3c0 seq: fix superfluous output line
Under certain circumstances seq prints an extra line when the output
format has custom format with characters following the printed numbers:

    $ seq -f "%g " 1000000 1000000
     1e+06
     1e+06

This is due to the "print_extra_number" logic using strings to determine
whether a 'extra number' is needed, but only one string was trimmed
when using a custom printf format.

Prompted by https://lists.gnu.org/r/coreutils/2019-08/msg00001.html

* NEWS: Mention fix.
* src/seq.c (print_numbers): Trim the 'x0_str' string before comparing
it to the previous 'x_str' string.
* tests/misc/seq-extra-number.sh: Add this scenario.
* tests/local.mk (all_tests): Add new test.
2019-08-02 18:29:25 -06:00
Andreas Dilger a1a5e9a32e stat: don't explicitly request file size for filenames
When calling 'stat -c %N' to print the filename, don't explicitly
request the size of the file via statx(), as it may add overhead on
some filesystems.  The size is only needed to optimize an allocation
for the relatively rare case of reading a symlink name, and the worst
effect is a somewhat-too-large temporary buffer may be allocated for
areadlink_with_size(), or internal retries if buffer is too small.

The file size will be returned by statx() on most filesystems, even
if not requested, unless the filesystem considers this to be too
expensive for that file, in which case the tradeoff is worthwhile.

* src/stat.c: Don't explicitly request STATX_SIZE for filenames.
2019-07-12 19:51:09 +01:00
Paul Eggert 3ae38a5995 od: use fseek on non-regular files
Problem reported by Szőts Ákos (Bug#36291).
* NEWS: Mention this.
* src/od.c (skip): Try fseek even on files that do not have usable
sizes, falling back on fread if fseek fails.
2019-06-19 18:48:54 -07:00
Jeff Layton 0b9bac90d8 stat: fix enabling of statx logic
* src/stat.c: STATX_INO isn't defined until stat.h is included.
Move the test down so it works properly.
2019-06-14 22:43:46 +01:00
Jeff Layton 6cc35de16f stat: Use statx where available and support --cached
* src/stat.c: Drop statbuf argument from out_epoch_sec().
Use statx() rather than [lf]stat() where available,
so a separate call is not required to get birth time.
Set STATX_* mask bits only for things we want to print,
which can be more efficient on some file systems.
Add a new --cache= command-line option that sets the appropriate hint
flags in the statx call.  These are primarily used with network
file systems to indicate what level of cache coherency is desired.
The new option is available unconditionally for better portability,
and ignored where not implemented.
* doc/coreutils.texi: Add documention for --cached.
* man/stat.x (SEE ALSO): Mention statx().
* NEWS: Mention the new feature.
2019-06-10 22:59:28 +01:00
Pádraig Brady 738a746d82 split: fix failure for certain number of specified files
* src/split.c (set_suffix_length): Use a more standard
zero based logN calculation for the number of units.
* tests/split/suffix-auto-length.sh: Add a test case.
* THANKS.in: Mention the reporter.
* NEWS: Mention the fix.
Fixes https://bugs.gnu.org/35291
2019-06-08 22:58:01 +01:00
Paul Eggert 00d72e6122 dd: be more careful about signal handling
Problem reported by Hans Henrik Bergan (Bug#36007).
* NEWS: Mention this.
* src/dd.c (iclose, ifdatasync, ifstat, ifsync):
New functions, which are more careful about SIGINT.
(cleanup): Use iclose instead of close.
(finish_up): Process signals first.
(skip, dd_copy, main): Use ifstat instead of fstat.
(dd_copy): Use ifdatasync and ifsync instead of fdatasync and fsync.
2019-05-30 13:54:18 -07:00
Paul Eggert 3b1046f80f cp: fix /dev/stdin problem on Solaris
Problem reported by Jakub Kulik (Bug#35713).
* NEWS: Mention this.
* configure.ac (DEV_FD_MIGHT_BE_CHR): New macro.
* src/copy.c (DEV_FD_MIGHT_BE_CHR): Default to false.
(follow_fstatat): New function.
(copy_internal): Use it.
* src/copy.h (XSTAT): Remove; no longer used.
2019-05-28 13:44:40 -07:00
Kevin Locke 3a87ad5124 doc: clarify dd sparse detection is by *output* block
The wording of the dd --help text suggests that output will be skipped
for sparse *input* blocks (i.e. that NUL-checking is done on input
blocks) while the code actually checks/skips all-NUL *output* blocks.[1]

* src/dd.c (usage): Update the --help text to clarify the above.
* tests/dd/sparse.sh: Ensure sparseness is controlled with obs.

[1]: https://superuser.com/a/1136358
2019-05-26 12:53:26 +01:00
Paul Eggert a60af87f24 b2sum: port blake2b-ref.c to HP-UX aCC
Continue the fix for Bug#35650.
* src/blake2/blake2b-ref.c [HAVE_CONFIG_H]: Include <config.h>.
2019-05-17 07:58:32 -07:00
Paul Eggert de54e1a809 b2sum: sync better with upstream
* src/blake2/b2sum.c: Reorder source code to minimize diffs from:
https://github.com/BLAKE2/BLAKE2/blob/master/b2sum/b2sum.c
2019-05-15 12:58:13 -07:00
Paul Eggert 5f261e239f b2sum: port to HP-UX aCC
Its support for the -include option is flaky.  Problem reported by
Michael Osipov (Bug#35650).  Plus, we could run into other
compilers that don’t support any option like -include.  Change the
code so that -include is not needed.  Although this causes us to
depart from the upstream version, we’re already doing that for
other reasons.
* configure.ac (USE_XLC_INCLUDE): Remove, as there’s no
guarantee a compiler will support something like -include.
* src/blake2/b2sum.c [HAVE_CONFIG_H]: Include <config.h>.
* src/local.mk (src_b2sum_CPPFLAGS): Add -DHAVE_CONFIG_H.
Do not use -include or a substitute.
2019-05-15 12:42:58 -07:00
Paul Eggert 6e97d361b4 b2sum: port to HP-UX C
* src/blake2/blake2.h (BLAKE2_PACKED):
Don’t assume __attribute__ ((packed)) works on non-Microsoft
compilers.  Instead, assume it works only if we have good
reason to assume so, and fall back on Microsoft (or not packing)
otherwise.  In practice, not packing is good enough and the
BLAKE2_PACKED macro is mostly just for documentation.
2019-05-11 13:55:54 -07:00
Paul Eggert 6f4c2e06ae cp: port fiemap.h to C99
* src/extent-scan.c (extent_scan_read): Adjust to change in
struct fiemap.
* src/fiemap.h (struct fiemap): Use FLEXIBLE_ARRAY_MEMBER
to port to C99.
2019-05-11 13:55:54 -07:00
Paul Eggert 2fd0be3621 basenc: port to C99
* src/basenc.c: Various minor style cleanups.
(struct base_decode_context): Do not use anonymous unions, as
they’re not in C99.  Use a named union instead.  All uses changed.
2019-05-11 13:55:54 -07:00
Paul Eggert a970a97737 maint: adjust to recent verify_true removal
* src/system.h (X2NREALLOC, X2REALLOC, DECIMAL_DIGIT_ACCUMULATE):
Use verify_expr instead of verify_true, which has been removed.
(DECIMAL_DIGIT_ACCUMULATE): Remove unnecessary size check.
2019-05-11 13:55:54 -07:00
Shugo Maeda 6d78a28078 factor: output immediately if stdout is a tty but stdin is not
* src/factor.c (lbuf_putc): Use line buffered mode if the standard
output is a terminal in the same way as the stdio library.
User programs might use pty only for the standard out
like the example of Ruby's PTY module:
https://docs.ruby-lang.org/en/2.6.0/PTY.html#module-PTY-label-Example
* NEWS: Mention the fix.
Fixes https://bugs.gnu.orv/35046
2019-03-31 15:49:48 -07:00
Pádraig Brady ebf70f297c maint: fix syntax check failure
* src/ln.c: Remove leading TAB.
2019-03-30 15:07:14 -07:00
Martin Castillo ba36d5118f maint: tee: use STDIN_FILENO rather than 0
* src/tee.c (tee_files): Use the name rather than the value.
Addresses https://bugs.gnu.org/35041
2019-03-30 15:04:42 -07:00
Kamil Dudka ae61b10663 md5sum,b2sum,sha*sum: --help: add note about binary/text mode
* src/md5sum.c (usage): Make it clear that there is no difference
between binary mode and text mode on GNU systems.

Bug: https://bugzilla.redhat.com/406981
Bug: https://bugzilla.redhat.com/1688740
2019-03-18 08:48:57 +01:00
Paul Eggert 3e0dff3925 ln: port to symlink ("x", ".") failing with EINVAL
Problem reported by John Marino (Bug#34894).
* src/ln.c (main): Port ln -s to Solaris symlink function,
where symlink ("x", ".") fails with errno == EINVAL.
2019-03-17 13:22:32 -07:00
Paul Eggert 6a707feee8 ln: port to platforms lacking O_DIRECTORY
* src/ln.c (main): Port to older platforms lacking
support for POSIX.1-2008’s O_DIRECTORY flag (Bug#34876).
2019-03-16 11:24:58 -07:00
Pádraig Brady 4bb49f04a1 build: avoid statx related build failure on AIX
* src/stat.c (get_birthtime): Check also for STATX_BTIME define,
as a different statx is available on AIX 7.2.
2019-03-06 10:53:00 -08:00
Pádraig Brady 4bbbe49074 build: fix env build where SIGNUM_BOUND is not constant
* src/env.c (initialize_signals): A new function to initialize
the signals array on the heap, to avoid a build failure on
opensolaris, where SIGNUM_BOUND is not a constant.
2019-03-06 09:00:44 -08:00
Pádraig Brady 6bf6d7f461 env: add --list-signal-handling to output non default handling
* src/env.c (main): Output blocked or ignored signals
before a command is executed.
* doc/coreutils.texi (env invocation): Add the option.
* tests/misc/env-signal-handler.sh: Add a test case.
* NEWS: Mention the new feature.
2019-03-04 00:37:42 -08:00
Assaf Gordon 95adadd9a4 env: new options --{default,ignore,block}-signal[=SIG]
New options to set signal handlers for the command being executed.
--block-signal suggested by Paul Eggert in http://bugs.gnu.org/34488#71
--default-signal is useful to overcome the POSIX limitation that shell
must not override inherited signal state, e.g. the second 'trap' here is
a no-op:

   trap '' PIPE && sh -c 'trap - PIPE ; seq inf | head -n1'

Instead use:

   trap '' PIPE && sh -c 'env --default-signal=PIPE seq inf | head -n1'

Similarly, the following will prevent CTRL-C from terminating the
program:

   env --ignore-signal=INT seq inf > /dev/null

See https://bugs.gnu.org/34488#8

* NEWS: Mention new options.
* doc/coreutils.texi (env invocation): Document new options.
* man/env.x: Add example of --default-signal=SIG usage.
(SEE ALSO): Mention sigprocmask.
* src/env.c (signals): New global variable.
(longopts): Add new options.
(usage): Print new options.
(parse_signal_params): Parse comma-separated list of signals, store in
signals variable.
(reset_signal_handlers): Set each signal to SIG_DFL/SIG_IGN.
(parse_block_signal_params): Parse command-line options.
(set_signal_proc_mask): Call sigprocmask to block/unblock signals.
(main): Process new options.
* src/local.mk (src_env_SOURCES): Add operand2sig.c.
* tests/misc/env-signal-handler.sh: New test.
* tests/local.mk (all_tests): Add new test.
2019-03-04 00:37:07 -08:00
Martin Bukatovic 186896d65f stat: print birth time on systems supporting statx
* configure.ac: Check for statx(), available on glibc >= 2.28.
* src/stat.c (get_birthtime): Call statx() when available.
* NEWS: Mention the improvement.
2019-03-03 23:33:51 -08:00