Commit Graph
29214 Commits
Author SHA1 Message Date
Pádraig Brady d1316583b2 doc: add options summary list to tr texinfo
* doc/coreutils.texi (tr invocation): Provide a summary
list of the available options, which is useful to
provide a quick reminder for those already familiar
with the functionality of tr.
Fixes https://bugs.gnu.org/49764
2021-07-29 13:19:33 +01:00
Pádraig Brady dcd6e4172c tests: augment new ls --zero test cases
* tests/ls/zero-option.sh: Check for the disabled, disallowed,
and allowed option combinations.
2021-07-28 23:17:34 +01:00
Pádraig Brady eca33373ce maint: avoid syntax-check failures in recent ls changes
* src/ls.c: Fix ifdef indenting and long line.
2021-07-28 23:16:50 +01:00
Paul Eggert d60b2922ff doc: modernize usage of “disk” and “core”
In documentation and comments, don’t assume that secondary storage
devices are disk devices.  Similarly, don’t assume that main memory
uses magnetic cores, which became obsolete in the 1970s.
* src/du.c (usage):
* src/ls.c (usage):
* src/shred.c (usage): Reword to avoid “disk” in usage messages.
2021-07-28 12:30:12 -07:00
Paul Eggert 1625916a1b doc: improve ls documentation
* doc/coreutils.texi (ls invocation): Document implementation more
closely.  Be more consistent about style.  Omit some needless words.
* src/ls.c (usage): Don’t overdocument -f, as the details were wrong.
Omit -1 advice as it’s a bit obsolete now that we have --zero and
is a bit much for --usage output anyway.
2021-07-28 12:30:12 -07:00
Paul Eggert c75779cac3 ls: rename --null to --zero (Bug#49716)
* NEWS, doc/coreutils.texi (General output formatting):
* src/ls.c (usage):
Document this.
* src/ls.c (ZERO_OPTION): Rename from NULL_OPTION.
All uses changed.
(long_options): Rename --null to --zero.
(dired_dump_obstack, main, print_dir): Use '\n' instead of
eolbyte where eolbyte must equal '\n'.
(decode_switches): Decode --zero instead of --null.
--zero also implies -1, -N, --color=none, --show-control-chars.
Use easier-to-decipher code to set ‘format’ and ‘dired’.
Reject attempts to combine --dired and --zero.
* tests/local.mk: Adjust to test script renaming.
* tests/ls/zero-option.sh: Rename from tests/ls/null-option.sh,
and test --zero instead of --null.
2021-07-27 18:08:39 -07:00
Paul Eggert 9e4a6101f2 ls: compute defaults more lazily
* src/ls.c (enum time_type, enum sort_type, enum indicator_style)
(enum Dereference_symlink, ignore_mode):
Put ‘= 0’ after default values, since the code relies
on static storage defaulting to zero.
(enum sort_type): Reorder so that -1 can be used to represent unset.
(main): Test print_with_color after parse_ls_color may have reset it.
(decode_line_length): Return the line length instead of setting
static storage.  All uses changed.  Treat line lengths exceeding
PTRDIFF_MAX as infinite, to avoid pointer-subtraction glitches.
(stdout_isatty): New function, to avoid calling isatty twice.
(decode_switches): Calculate defaults more lazily, to avoid using
syscalls or getenv during startup unless the results are more
likely to be needed.  Use -1 to indicate options that haven’t been
set on the command line yet.  Move print_with_color test from
here to ‘main’.  Suppress bogus GCC warning.
(getenv_quoting_style): Return the quoting style instead of
setting static storage.
(init_column_info): New arg MAX_COLS, to avoid recalculating it.
Caller changed.
2021-07-27 18:08:38 -07:00
Pádraig Brady e094d2e3e1 maint: avoid recent syntax-check issues
* .gitignore: Cater for recently added poll module.
* src/stdbuf.c: Avoid false positive from sc_prohibit_readlink.
2021-07-26 15:42:29 +01:00
Paul Eggert 7d44ac224c ls: add --null option (Bug#49716)
* NEWS, doc/coreutils.texi (General output formatting):
* src/ls.c (usage): Document this.
* src/ls.c (NULL_OPTION): New constant.
(long_options): Add --null.
(eolbyte): New static var.
(dired_dump_obstack, main, print_dir, print_current_files)
(print_many_per_line, print_horizontal, print_with_separator):
Output eolbyte instead of '\n'.
(decode_switches): Decode --null.
* tests/ls/null-option.sh: New file.
* tests/local.mk (all_tests): Add it.
2021-07-26 00:59:37 -07:00
Paul Eggert 39673b6d4c ls: port to wider off_t, uid_t, gid_t
* src/ls.c (dired_pos): Now off_t, not size_t, since it counts
output file offsets.
(dired_dump_obstack): This obstack's file offsets are now
off_t, not size_t.
(format_user_or_group, format_user_or_group_width):
ID arg is now uintmax_t, not unsigned long, since uid_t and
gid_t values might exceed ULONG_MAX.
(format_user_or_group_width): Use snprintf with NULL instead of
sprintf with a discarded buffer.  This avoids a stack buffer,
and so should be safer.
2021-07-26 00:59:37 -07:00
Paul Eggert 0a41ad2cfb ls: demacroize
Prefer functions or constants to macros where either will do.
That’s cleaner, and nowadays there’s no performance reason to
prefer macros.  All uses changed.
* src/ls.c (INITIAL_TABLE_SIZE, MIN_COLUMN_WIDTH):
Now constants instead of macros.
(file_or_link_mode): New function, replacing the old macro
FILE_OR_LINK_MODE.
(dired_outbyte): New function, replacing the old macro DIRED_PUTCHAR.
(dired_outbuf): New function, replacing the old macro DIRED_FPUTS.
(dired_outstring): New function, replacing the old macro
DIRED_FPUTS_LITERAL.
(dired_indent): New function, replacing the old macro DIRED_INDENT.
(push_current_dired_pos): New function, replacing the old macro
PUSH_CURRENT_DIRED_POS.
(assert_matching_dev_ino): New function, replacing the old macro
ASSERT_MATCHING_DEV_INO.
(do_stat, do_lstat, stat_for_mode, stat_for_ino, fstat_for_ino)
(signal_init, signal_restore, cmp_ctime, cmp_mtime, cmp_atime)
(cmp_btime, cmp_size, cmp_name, cmp_extension)
(fileinfo_name_width, cmp_width, cmp_version):
No longer inline; compilers can deduce this well enough nowadays.
(main): Protect unused assert with ‘if (false)’ rather than
commenting it out, so that the compiler checks the code.
(print_dir): Output the space and newline in the same buffer
as the human-readable number they surround.
(dirfirst_check): New function, replacing the old macro
DIRFIRST_CHECK.  Simplify by using subtraction.
(off_cmp): New function, replacing the old macro longdiff.
(print_long_format): No need to null-terminate the string now.
(format_user_or_group): Let printf count the bytes.
2021-07-26 00:59:37 -07:00
Paul Eggert c09b25dd3f ls: simplify sprintf usage
* src/ls.c (format_user_or_group_width, print_long_format):
Use return value from sprintf instead of calling strlen on
the resulting buffer, or inferring the length some other way.
2021-07-26 00:59:36 -07:00
Paul Eggert 558959daa9 build: update gnulib submodule to latest 2021-07-26 00:59:36 -07:00
Paul Eggert 5622ab450e maint: fix white space 2021-07-26 00:59:36 -07:00
Paul Eggert 3753c706b7 env: fix usage typo
* src/env.c (usage): Fix pluralization typo.
2021-07-26 00:59:36 -07:00
Kamil Dudka d6125af095 df: fix duplicated remote entries due to bind mounts
As originally reported in <https://bugzilla.redhat.com/1962515>,
df invoked without -a printed duplicated entries for NFS mounts
of bind mounts.  This is a regression from commit v8.25-54-g1c17f61ef99,
which introduced the use of a hash table.

The proposed patch makes sure that the devlist entry seen the last time
is used for comparison when eliminating duplicated mount entries.  This
way it worked before introducing the hash table.

Patch co-authored by Roberto Bergantinos.

* src/ls.c (struct devlist): Introduce the seen_last pointer.
(devlist_for_dev): Return the devlist entry seen the last time if found.
(filter_mount_list): Remember the devlist entry seen the last time for
each hashed item.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/49298
2021-07-02 17:32:46 +01:00
Paul Eggert da0d448bca tail: use poll, not select
This fixes an unlikely stack out-of-bounds write reported by
Stepan Broz via Kamil Dudka (Bug#49209).
* bootstrap.conf (gnulib_modules): Replace select with poll.
* src/tail.c: Do not include <sys/select.h>.
[!_AIX]: Include poll.h.
(check_output_alive) [!_AIX]: Use poll instead of select.
(tail_forever_inotify): Likewise.  Simplify logic, as there is no
need for a ‘while (len <= evbuf_off)’ loop.
2021-06-26 18:40:26 -07:00
Paul Eggert 177bfdd7ca tail: fix abuse2 test race
* tests/tail-2/inotify-hash-abuse2.sh (fastpoll):
Fix race where tailed file ‘f’ temporarily did not exist.
2021-06-26 18:40:26 -07:00
Paul Eggert 53bb0235d5 maint: while (1) → while (true) 2021-06-26 18:40:26 -07:00
Nikolay Nechaev 18dbcae25a maint: remove redundant checks on buffer sizes in tail
* src/tail.c: remove redundant size checks before calls to
`xwrite_stdout`
2021-06-21 13:17:06 +01:00
Pádraig Brady 5f8e6fc202 stat: use decomposed decimal device numbers by default
* src/stat.c (default_format): Use decomposed decimal
representation (major,minor) in the default format.
This is least ambiguous for human interpretation,
and more consistent with ls for example.
Fixes https://bugs.gnu.org/48960
2021-06-21 12:03:42 +01:00
Pádraig Brady 4a8278229e stat: support more device number representations
In preparation for changing the default device number
representation (to decomposed decimal), provide more
formatting options for device numbers.

These new (FreeBSD compat) formatting options are added:

   %Hd  major device number in decimal (st_dev)
   %Ld  minor device number in decimal (st_dev)
   %Hr  major device type in decimal (st_rdev)
   %Lr  minor device type in decimal (st_rdev)
   %r   (composed) device type in decimal (st_rdev)
   %R   (composed) device type in hex (st_rdev)

* doc/coreutils.texi (stat invocation): Document new formats.
* src/stat.c (print_it): Handle the new %H and %L modifiers.
(print_statfs): Adjust to passing the format as two chars
rather than an int.  Using an int was introduced in commit db42ae78,
but using separate chars is cleaner and more extensible.
(print_stat): Likewise.  Handle any modifiers and the new 'r' format.
(usage): Document the new formats.
* tests/misc/stat-fmt.sh: Add a test case for new modifiers.
Addresses https://bugs.gnu.org/48960
2021-06-21 12:01:04 +01:00
Paul Eggert 4edad9e121 build: update gnulib submodule to latest
Coreutils mistakenly did not list xstrndup as a module
that it depends on directly.  When the latest Gnulib removed
the dirname module's dependency on xstrndup, this mistake
caused coreutils to not build.  Since all of Coreutils's
uses of xstrndup know the string length, xmemdup0 is a better
match for what's needed.  Since the size args are typically
signed or derived from subtracting pointers, the new Gnulib
ximemdup0 function is a better match yet.
So, use ximemdup0 instead of xstrndup.
* src/cut.c, src/dircolors.c, src/expand-common.c, src/expand.c:
* src/numfmt.c, src/set-fields.c, src/unexpand.c:
Do not include xstrndup.h; no longer needed.
* src/dircolors.c (parse_line):
* src/expand-common.c (parse_tab_stops):
* src/numfmt.c (parse_format_string):
* src/set-fields.c (set_fields):
Use ximemdup0 instead of xstrndup.
2021-06-11 18:05:57 -07:00
Jim Meyering 370c294018 maint: bootstrap: remove reference to unused hash-pjw module
* bootstrap.conf (gnulib_modules): Remove hash-pjw. No longer used.
2021-05-28 13:10:46 -07:00
Pádraig Brady d1b0257077 build: update gnulib submodule to latest
Fixes a false test failure with MALLOC_CHECK_ set.

* gnulib: Update to latest.
2021-05-15 20:33:26 +01:00
Pádraig Brady 26eccf6c98 copy: remove fiemap logic
This is now only used on 10 year old linux kernels,
and performs a sync before each copy.

* src/copy.c (extent_copy): Remove function and all callers.
* src/extent-scan.c: Remove.
* src/extent-scan.h: Remove.
* src/fiemap.h: Remove.
* src/local.mk: Adjust for removed files.
* NEWS: Adjust to say fiemap is removed.
2021-05-15 16:14:11 +01:00
Pádraig Brady ba5e6885d2 copy: disallow copy_file_range() on Linux kernels before 5.3
copy_file_range() before Linux kernel release 5.3 had many issues,
as described at https://lwn.net/Articles/789527/, which was
referenced from https://lwn.net/Articles/846403/; a more general
article discussing the generality of copy_file_range().
Linux kernel 5.3 was released in September 2019, which is new enough
that we need to actively avoid older kernels.

* src/copy.c (functional_copy_file_range): A new function
that returns false for Linux kernels before version 5.3.
(sparse_copy): Call this new function to gate use of
copy_file_range().
2021-05-13 00:14:22 +01:00
Pádraig Brady 6b499720fe tests: fix tests/cp/sparse-2.sh false failure on some systems
* tests/cp/sparse-2.sh: Double check cp --sparse=always,
with dd conv=sparse, in the case where the former didn't
create a sparse file.  Now that this test is being newly run
on macos, we're seeing a failure due to seek() not creating
holes on apfs unless the size is >= 16MiB.
2021-05-12 22:14:11 +01:00
Pádraig Brady 4eb2e9c11a tests: ensure we test SEEK_DATA where used
fiemap is no longer the default copy implementation,
so check for SEEK_DATA support instead as that's preferred.
This will ensure better test coverage on systems without fiemap.

* init.cfg: Replace fiemap_capable_ with seek_data_capable_.
This is best supported with python 3 so prefer that.
* tests/seek-data-capable: A new test script checking for
SEEK_DATA support on the passed file name,
called from seek_data_capable_.
* tests/fiemap-capable: Remove no longer used probing script.
* tests/cp/fiemap-perf.sh: Renamed to tests/cp/sparse-perf.sh
* tests/cp/fiemap-2.sh: Renamed to tests/cp/sparse-2.sh
* tests/cp/fiemap-extents.sh: Renamed to tests/cp/sparse-extents.sh
* tests/cp/sparse-fiemap.sh: Renamed to tests/cp/sparse-extents-2.sh
* tests/cp/fiemap-FMR.sh: Renamed to tests/cp/copy-FMR.sh
* tests/local.mk: Reference the renamed tests.
2021-05-12 22:14:11 +01:00
Pádraig Brady 2e66e1732f copy: handle system security config issues with copy_file_range()
* src/copy.c (sparse_copy): Upon EPERM from copy_file_range(),
fall back to a standard copy, which will give a more accurate
error as to whether the issue is with the source or destination.
Also this will avoid the issue where seccomp or apparmor are
not configured to handle copy_file_range(), in which case
the fall back standard copy would succeed without issue.
This specific issue with seccomp was noticed for example in:
https://github.com/golang/go/issues/40900
2021-05-12 22:14:11 +01:00
Pádraig Brady 017877bd08 copy: handle EOPNOTSUPP from SEEK_DATA
* src/copy.c (infer_scantype): Ensure we don't error out
if SEEK_DATA returns EOPNOTSUPP, on systems where this value
is distinct from ENOTSUP.  Generally both of these should be checked.
2021-05-12 22:14:11 +01:00
Pádraig Brady 8ec0d1799e copy: handle ENOTSUP from copy_file_range()
* src/copy.c (sparse_copy): Ensure we fall back to
a standard copy if copy_file_range() returns ENOTSUP.
This generally is best checked when checking ENOSYS,
but it also seems to be a practical concern on Centos 7,
as a quick search gave https://bugzilla.redhat.com/1840284
2021-05-12 22:14:11 +01:00
Pádraig Brady e35e4d1757 build: update gnulib submodule to latest
Fixes a bits/long-double.h include build issue on some systems.

* bootstrap: Sync new --version option from gnulib.
* gnulib: Update to lastest.
Reported by Carl Edquist
2021-05-10 17:44:57 +01:00
Carl Edquist bf483155e5 build: fix __get_cpuid_count check to catch link failure
The test program will compile successfully even if __get_cpuid_count
is not declared.  The error for the missing symbol will only show up
at link time.  Thus, use AC_LINK_IFELSE instead of AC_COMPILE_IFELSE.

* configure.ac (__get_cpuid_count check): Use C_LINK_IFELSE instead
of AC_COMPILE_IFELSE.
(__get_cpuid check): Likewise.
2021-05-10 13:35:20 +01:00
Pádraig Brady 09bbb2b703 maint: consistently free hash structures in dev mode
Ensure we call hash_free() to avoid valgrind and leak_sanitizer
"definitely lost" warnings.  These were not real leaks as
we terminate immediately after, but we should avoid these
"definitely lost" warnings where possible.

* src/copy.c: Add dest_info_free() and src_info_free().
* src/copy.h: Declare the above.
* src/cp-hash.c: Don't define unless "lint" is defined.
* src/install.c: Call dest_info_free() in dev mode.
* src/mv.c: Likewise.
* src/cp.c: Likewise. Also call src_info_free().
* src/ln.c: Call hash_free() in dev mode.
* src/tail.c: Call hash_free() even if about to exit, in dev mode.

Fixes https://bugs.gnu.org/48189
2021-05-08 14:57:33 +01:00
Bernhard Voelker 65e4d0f71e maint: fix sc_space_before_open_paren failure
* src/copy.c (dest_info_init): Add space before parens.
(src_info_init): Likewise.
Syntax-check failure introduced in the previous commit.
2021-05-06 23:42:57 +02:00
Pádraig Brady f1d2e8e289 copy: exit immediately upon failure to allocate hash memory
* src/copy.c (dest_info_init, src_info_init): Terminate immediately
upon memory exhaustion.
2021-05-03 18:53:35 +01:00
Pádraig Brady ea9af99234 copy: ensure we enforce --reflink=never
* src/copy.c (sparse_copy): Don't use copy_file_range()
with --reflink=never as copy_file_range() may implicitly
use acceleration techniques like reflinking.
(extent_copy): Pass through whether we allow reflinking.
(lseek_copy): Likewise.
Fixes https://bugs.gnu.org/48164
2021-05-02 21:27:17 +01:00
Pádraig Brady e7fff54c92 wc: add --debug to diagnose which implementation used
* src/wc.c: (main): Handle the new --debug option.
Only call avx2_supported if needed.
(avx2_supported): Diagnose various failures and attempts.
* NEWS: Mention the new wc improvement and --debug option.
2021-05-02 20:51:37 +01:00
Kristoffer Brånemyr 4064c57380 wc: use avx2 optimization when counting only lines
Use cpuid to detect CPU support for avx2 instructions.
Performance was seen to improve by 5x for a file with only newlines,
while the performance for a file with no such characters is unchanged.

* configure.ac [USE_AVX2_WC_LINECOUNT]: A new conditional,
set when __get_cpuid_count() and avx2 compiler intrinsics are supported.
* src/wc.c (avx2_supported): A new function using __get_cpuid_count()
to determine if avx2 instructions are supported.
(wc_lines): A new function refactored from wc(),
which implements the standard line counting logic,
and provides the fallback implementation for when avx2 is not supported.
* src/wc_avx2.c: A new module to implement using avx2 intrinsics.
* src/local.mk: Reference the new module.  Note we build as a separate
lib so that it can be portably built with separate -mavx2 etc. flags.
2021-05-02 20:51:37 +01:00
Paul Eggert d435cfc0bc touch: fix wrong diagnostic (Bug#48106)
Problem reported by Roland (Bug#48106).
* src/touch.c (touch): Take more care when deciding whether
to use open_errno or utime_errno in the diagnostic.
Stop worrying about SunOS 4 (which as part of the problem),
as it’s long obsolete.  For Solaris 10, verify that EINVAL
really means the file was a directory.
2021-05-01 15:47:13 -07:00
Paul Eggert 62a7ce5f50 maint: port to Autoconf 2.71
* configure.ac: Use AC_PROG_CC, not AC_PROG_CC_STDC.
* gl/modules/smack (configure.ac):
* m4/jm-macros.m4 (coreutils_MACROS):
* m4/xattr.m4 (gl_FUNC_XATTR):
Use AS_HELP_STRING, not AC_HELP_STRING.
* m4/check-decl.m4 (gl_CHECK_DECLS):
Do not require AC_HEADER_TIME; we no longer care about it directly.
* m4/jm-macros.m4 (coreutils_MACROS):
Do not require AC_ISC_POSIX, which became obsolete in 2006.
Use AC_LINK_IFELSE instead of AC_TRY_LINK.
2021-04-26 23:32:45 -07:00
Paul Eggert 8527942835 csplit: size_t overflow check
* src/csplit.c (get_new_buffer): Fix unlikely size_t overflow.
2021-04-26 23:32:45 -07:00
Paul Eggert 5fcf8c6487 build: update gnulib submodule to latest
* src/csplit.c (load_buffer):
* src/pinky.c (create_fullname):
Use intprops-based checks rather than xalloc_oversized,
since Gnulib xalloc.h no longer includes xalloc-oversized.h.
2021-04-26 23:32:45 -07:00
Zorro Lang 785478013b copy: do not refuse to copy a swap file
* src/copy.c (sparse_copy): Fallback to read() if copy_file_range()
fails with ETXTBSY.  Otherwise it would be impossible to copy files
that are being used as swap.  This used to work before introducing
the support for copy_file_range() in coreutils.  (Bug#48036)
2021-04-26 18:00:58 -07:00
Bernhard Voelker b7091093bb tests: fix FP in ls/stat-free-color.sh
On newer systems like Fedora 34 and openSUSE Tumbleweed, ls(1) calls
newfstatat(STDOUT_FILENO, ...), but only when there is something to
output.

* tests/ls/stat-free-color.sh: Add -a option to the reference invocation
of ls, thus enforcing something gets output.
2021-04-22 20:14:32 +02:00
Pádraig Brady 34a48bf0f0 doc: clarify that ln --relative requires --symbolic to be specified
* doc/coreutils.texi (ln invocation): State --symbolic is required.
* src/ln.c (usage): Explicitly state -s is not implied.
Fixes https://bugs.gnu.org/47703
2021-04-11 19:23:37 +01:00
Pádraig Brady 512fe04902 doc: clarify what's counted by wc
* src/wc.c (usage): State that only printable characters are considered
when counting words.  This also disambiguates wether we're talking
about bytes or characters in this context.
* doc/coreutils.texi (wc invocation): Likewise.  Also clarify
that --characters counts valid locale aware characters,
and that --lines does not count a trailing "line" unless
it ends with a newline character.
Fixes https://bugs.gnu.org/47702
2021-04-11 19:23:13 +01:00
Pádraig Brady ef772bf97f maint: use "char const *" rather than "const char *"
* cfg.mk (sc_prohibit-const-char): Add a new syntax-check to
enforce this style.
* *.[ch]: sed -i 's/const char \*/char const */g'
2021-04-11 18:33:45 +01:00
Pádraig Brady 34ed19eed8 ls: cache name width determination
This is especially important now for --sort=width,
as that can greatly increase how often this
expensive quote_name_width() function is called per file.

This also helps the default invocation of ls,
or specifically the --format={across,vertical} cases
(when --width is not set to 0),
to avoid two calls to this function per file.

Note the only case where we later compute the width,
is for --format=commas.  That's only done once though,
so we leave the computation close to use to
maximize hardware caching.

* src/ls.c (struct fileinfo): Add a WIDTH member to cache
the screen width of the file name.
(update_current_files_info): Set the WIDTH members for cases
they're needed multiple times.  Note we do this explicitly here,
rather than caching at use, so that the fileinfo
structures can remain const in the sorting and presentation functions.
(sort_files): Call the new update_current_files_info() in this
initialization function.
(fileinfo_name_width): Renamed from fileinfo_width,
and adjusted to return the cached value if available.
2021-04-11 14:58:45 +01:00