Commit Graph
29197 Commits
Author SHA1 Message Date
Paul Eggert 177bfdd7ca tail: fix abuse2 test race
* tests/tail-2/inotify-hash-abuse2.sh (fastpoll):
Fix race where tailed file ‘f’ temporarily did not exist.
2021-06-26 18:40:26 -07:00
Paul Eggert 53bb0235d5 maint: while (1) → while (true) 2021-06-26 18:40:26 -07:00
Nikolay Nechaev 18dbcae25a maint: remove redundant checks on buffer sizes in tail
* src/tail.c: remove redundant size checks before calls to
`xwrite_stdout`
2021-06-21 13:17:06 +01:00
Pádraig Brady 5f8e6fc202 stat: use decomposed decimal device numbers by default
* src/stat.c (default_format): Use decomposed decimal
representation (major,minor) in the default format.
This is least ambiguous for human interpretation,
and more consistent with ls for example.
Fixes https://bugs.gnu.org/48960
2021-06-21 12:03:42 +01:00
Pádraig Brady 4a8278229e stat: support more device number representations
In preparation for changing the default device number
representation (to decomposed decimal), provide more
formatting options for device numbers.

These new (FreeBSD compat) formatting options are added:

   %Hd  major device number in decimal (st_dev)
   %Ld  minor device number in decimal (st_dev)
   %Hr  major device type in decimal (st_rdev)
   %Lr  minor device type in decimal (st_rdev)
   %r   (composed) device type in decimal (st_rdev)
   %R   (composed) device type in hex (st_rdev)

* doc/coreutils.texi (stat invocation): Document new formats.
* src/stat.c (print_it): Handle the new %H and %L modifiers.
(print_statfs): Adjust to passing the format as two chars
rather than an int.  Using an int was introduced in commit db42ae78,
but using separate chars is cleaner and more extensible.
(print_stat): Likewise.  Handle any modifiers and the new 'r' format.
(usage): Document the new formats.
* tests/misc/stat-fmt.sh: Add a test case for new modifiers.
Addresses https://bugs.gnu.org/48960
2021-06-21 12:01:04 +01:00
Paul Eggert 4edad9e121 build: update gnulib submodule to latest
Coreutils mistakenly did not list xstrndup as a module
that it depends on directly.  When the latest Gnulib removed
the dirname module's dependency on xstrndup, this mistake
caused coreutils to not build.  Since all of Coreutils's
uses of xstrndup know the string length, xmemdup0 is a better
match for what's needed.  Since the size args are typically
signed or derived from subtracting pointers, the new Gnulib
ximemdup0 function is a better match yet.
So, use ximemdup0 instead of xstrndup.
* src/cut.c, src/dircolors.c, src/expand-common.c, src/expand.c:
* src/numfmt.c, src/set-fields.c, src/unexpand.c:
Do not include xstrndup.h; no longer needed.
* src/dircolors.c (parse_line):
* src/expand-common.c (parse_tab_stops):
* src/numfmt.c (parse_format_string):
* src/set-fields.c (set_fields):
Use ximemdup0 instead of xstrndup.
2021-06-11 18:05:57 -07:00
Jim Meyering 370c294018 maint: bootstrap: remove reference to unused hash-pjw module
* bootstrap.conf (gnulib_modules): Remove hash-pjw. No longer used.
2021-05-28 13:10:46 -07:00
Pádraig Brady d1b0257077 build: update gnulib submodule to latest
Fixes a false test failure with MALLOC_CHECK_ set.

* gnulib: Update to latest.
2021-05-15 20:33:26 +01:00
Pádraig Brady 26eccf6c98 copy: remove fiemap logic
This is now only used on 10 year old linux kernels,
and performs a sync before each copy.

* src/copy.c (extent_copy): Remove function and all callers.
* src/extent-scan.c: Remove.
* src/extent-scan.h: Remove.
* src/fiemap.h: Remove.
* src/local.mk: Adjust for removed files.
* NEWS: Adjust to say fiemap is removed.
2021-05-15 16:14:11 +01:00
Pádraig Brady ba5e6885d2 copy: disallow copy_file_range() on Linux kernels before 5.3
copy_file_range() before Linux kernel release 5.3 had many issues,
as described at https://lwn.net/Articles/789527/, which was
referenced from https://lwn.net/Articles/846403/; a more general
article discussing the generality of copy_file_range().
Linux kernel 5.3 was released in September 2019, which is new enough
that we need to actively avoid older kernels.

* src/copy.c (functional_copy_file_range): A new function
that returns false for Linux kernels before version 5.3.
(sparse_copy): Call this new function to gate use of
copy_file_range().
2021-05-13 00:14:22 +01:00
Pádraig Brady 6b499720fe tests: fix tests/cp/sparse-2.sh false failure on some systems
* tests/cp/sparse-2.sh: Double check cp --sparse=always,
with dd conv=sparse, in the case where the former didn't
create a sparse file.  Now that this test is being newly run
on macos, we're seeing a failure due to seek() not creating
holes on apfs unless the size is >= 16MiB.
2021-05-12 22:14:11 +01:00
Pádraig Brady 4eb2e9c11a tests: ensure we test SEEK_DATA where used
fiemap is no longer the default copy implementation,
so check for SEEK_DATA support instead as that's preferred.
This will ensure better test coverage on systems without fiemap.

* init.cfg: Replace fiemap_capable_ with seek_data_capable_.
This is best supported with python 3 so prefer that.
* tests/seek-data-capable: A new test script checking for
SEEK_DATA support on the passed file name,
called from seek_data_capable_.
* tests/fiemap-capable: Remove no longer used probing script.
* tests/cp/fiemap-perf.sh: Renamed to tests/cp/sparse-perf.sh
* tests/cp/fiemap-2.sh: Renamed to tests/cp/sparse-2.sh
* tests/cp/fiemap-extents.sh: Renamed to tests/cp/sparse-extents.sh
* tests/cp/sparse-fiemap.sh: Renamed to tests/cp/sparse-extents-2.sh
* tests/cp/fiemap-FMR.sh: Renamed to tests/cp/copy-FMR.sh
* tests/local.mk: Reference the renamed tests.
2021-05-12 22:14:11 +01:00
Pádraig Brady 2e66e1732f copy: handle system security config issues with copy_file_range()
* src/copy.c (sparse_copy): Upon EPERM from copy_file_range(),
fall back to a standard copy, which will give a more accurate
error as to whether the issue is with the source or destination.
Also this will avoid the issue where seccomp or apparmor are
not configured to handle copy_file_range(), in which case
the fall back standard copy would succeed without issue.
This specific issue with seccomp was noticed for example in:
https://github.com/golang/go/issues/40900
2021-05-12 22:14:11 +01:00
Pádraig Brady 017877bd08 copy: handle EOPNOTSUPP from SEEK_DATA
* src/copy.c (infer_scantype): Ensure we don't error out
if SEEK_DATA returns EOPNOTSUPP, on systems where this value
is distinct from ENOTSUP.  Generally both of these should be checked.
2021-05-12 22:14:11 +01:00
Pádraig Brady 8ec0d1799e copy: handle ENOTSUP from copy_file_range()
* src/copy.c (sparse_copy): Ensure we fall back to
a standard copy if copy_file_range() returns ENOTSUP.
This generally is best checked when checking ENOSYS,
but it also seems to be a practical concern on Centos 7,
as a quick search gave https://bugzilla.redhat.com/1840284
2021-05-12 22:14:11 +01:00
Pádraig Brady e35e4d1757 build: update gnulib submodule to latest
Fixes a bits/long-double.h include build issue on some systems.

* bootstrap: Sync new --version option from gnulib.
* gnulib: Update to lastest.
Reported by Carl Edquist
2021-05-10 17:44:57 +01:00
Carl Edquist bf483155e5 build: fix __get_cpuid_count check to catch link failure
The test program will compile successfully even if __get_cpuid_count
is not declared.  The error for the missing symbol will only show up
at link time.  Thus, use AC_LINK_IFELSE instead of AC_COMPILE_IFELSE.

* configure.ac (__get_cpuid_count check): Use C_LINK_IFELSE instead
of AC_COMPILE_IFELSE.
(__get_cpuid check): Likewise.
2021-05-10 13:35:20 +01:00
Pádraig Brady 09bbb2b703 maint: consistently free hash structures in dev mode
Ensure we call hash_free() to avoid valgrind and leak_sanitizer
"definitely lost" warnings.  These were not real leaks as
we terminate immediately after, but we should avoid these
"definitely lost" warnings where possible.

* src/copy.c: Add dest_info_free() and src_info_free().
* src/copy.h: Declare the above.
* src/cp-hash.c: Don't define unless "lint" is defined.
* src/install.c: Call dest_info_free() in dev mode.
* src/mv.c: Likewise.
* src/cp.c: Likewise. Also call src_info_free().
* src/ln.c: Call hash_free() in dev mode.
* src/tail.c: Call hash_free() even if about to exit, in dev mode.

Fixes https://bugs.gnu.org/48189
2021-05-08 14:57:33 +01:00
Bernhard Voelker 65e4d0f71e maint: fix sc_space_before_open_paren failure
* src/copy.c (dest_info_init): Add space before parens.
(src_info_init): Likewise.
Syntax-check failure introduced in the previous commit.
2021-05-06 23:42:57 +02:00
Pádraig Brady f1d2e8e289 copy: exit immediately upon failure to allocate hash memory
* src/copy.c (dest_info_init, src_info_init): Terminate immediately
upon memory exhaustion.
2021-05-03 18:53:35 +01:00
Pádraig Brady ea9af99234 copy: ensure we enforce --reflink=never
* src/copy.c (sparse_copy): Don't use copy_file_range()
with --reflink=never as copy_file_range() may implicitly
use acceleration techniques like reflinking.
(extent_copy): Pass through whether we allow reflinking.
(lseek_copy): Likewise.
Fixes https://bugs.gnu.org/48164
2021-05-02 21:27:17 +01:00
Pádraig Brady e7fff54c92 wc: add --debug to diagnose which implementation used
* src/wc.c: (main): Handle the new --debug option.
Only call avx2_supported if needed.
(avx2_supported): Diagnose various failures and attempts.
* NEWS: Mention the new wc improvement and --debug option.
2021-05-02 20:51:37 +01:00
Kristoffer Brånemyr 4064c57380 wc: use avx2 optimization when counting only lines
Use cpuid to detect CPU support for avx2 instructions.
Performance was seen to improve by 5x for a file with only newlines,
while the performance for a file with no such characters is unchanged.

* configure.ac [USE_AVX2_WC_LINECOUNT]: A new conditional,
set when __get_cpuid_count() and avx2 compiler intrinsics are supported.
* src/wc.c (avx2_supported): A new function using __get_cpuid_count()
to determine if avx2 instructions are supported.
(wc_lines): A new function refactored from wc(),
which implements the standard line counting logic,
and provides the fallback implementation for when avx2 is not supported.
* src/wc_avx2.c: A new module to implement using avx2 intrinsics.
* src/local.mk: Reference the new module.  Note we build as a separate
lib so that it can be portably built with separate -mavx2 etc. flags.
2021-05-02 20:51:37 +01:00
Paul Eggert d435cfc0bc touch: fix wrong diagnostic (Bug#48106)
Problem reported by Roland (Bug#48106).
* src/touch.c (touch): Take more care when deciding whether
to use open_errno or utime_errno in the diagnostic.
Stop worrying about SunOS 4 (which as part of the problem),
as it’s long obsolete.  For Solaris 10, verify that EINVAL
really means the file was a directory.
2021-05-01 15:47:13 -07:00
Paul Eggert 62a7ce5f50 maint: port to Autoconf 2.71
* configure.ac: Use AC_PROG_CC, not AC_PROG_CC_STDC.
* gl/modules/smack (configure.ac):
* m4/jm-macros.m4 (coreutils_MACROS):
* m4/xattr.m4 (gl_FUNC_XATTR):
Use AS_HELP_STRING, not AC_HELP_STRING.
* m4/check-decl.m4 (gl_CHECK_DECLS):
Do not require AC_HEADER_TIME; we no longer care about it directly.
* m4/jm-macros.m4 (coreutils_MACROS):
Do not require AC_ISC_POSIX, which became obsolete in 2006.
Use AC_LINK_IFELSE instead of AC_TRY_LINK.
2021-04-26 23:32:45 -07:00
Paul Eggert 8527942835 csplit: size_t overflow check
* src/csplit.c (get_new_buffer): Fix unlikely size_t overflow.
2021-04-26 23:32:45 -07:00
Paul Eggert 5fcf8c6487 build: update gnulib submodule to latest
* src/csplit.c (load_buffer):
* src/pinky.c (create_fullname):
Use intprops-based checks rather than xalloc_oversized,
since Gnulib xalloc.h no longer includes xalloc-oversized.h.
2021-04-26 23:32:45 -07:00
Zorro Lang 785478013b copy: do not refuse to copy a swap file
* src/copy.c (sparse_copy): Fallback to read() if copy_file_range()
fails with ETXTBSY.  Otherwise it would be impossible to copy files
that are being used as swap.  This used to work before introducing
the support for copy_file_range() in coreutils.  (Bug#48036)
2021-04-26 18:00:58 -07:00
Bernhard Voelker b7091093bb tests: fix FP in ls/stat-free-color.sh
On newer systems like Fedora 34 and openSUSE Tumbleweed, ls(1) calls
newfstatat(STDOUT_FILENO, ...), but only when there is something to
output.

* tests/ls/stat-free-color.sh: Add -a option to the reference invocation
of ls, thus enforcing something gets output.
2021-04-22 20:14:32 +02:00
Pádraig Brady 34a48bf0f0 doc: clarify that ln --relative requires --symbolic to be specified
* doc/coreutils.texi (ln invocation): State --symbolic is required.
* src/ln.c (usage): Explicitly state -s is not implied.
Fixes https://bugs.gnu.org/47703
2021-04-11 19:23:37 +01:00
Pádraig Brady 512fe04902 doc: clarify what's counted by wc
* src/wc.c (usage): State that only printable characters are considered
when counting words.  This also disambiguates wether we're talking
about bytes or characters in this context.
* doc/coreutils.texi (wc invocation): Likewise.  Also clarify
that --characters counts valid locale aware characters,
and that --lines does not count a trailing "line" unless
it ends with a newline character.
Fixes https://bugs.gnu.org/47702
2021-04-11 19:23:13 +01:00
Pádraig Brady ef772bf97f maint: use "char const *" rather than "const char *"
* cfg.mk (sc_prohibit-const-char): Add a new syntax-check to
enforce this style.
* *.[ch]: sed -i 's/const char \*/char const */g'
2021-04-11 18:33:45 +01:00
Pádraig Brady 34ed19eed8 ls: cache name width determination
This is especially important now for --sort=width,
as that can greatly increase how often this
expensive quote_name_width() function is called per file.

This also helps the default invocation of ls,
or specifically the --format={across,vertical} cases
(when --width is not set to 0),
to avoid two calls to this function per file.

Note the only case where we later compute the width,
is for --format=commas.  That's only done once though,
so we leave the computation close to use to
maximize hardware caching.

* src/ls.c (struct fileinfo): Add a WIDTH member to cache
the screen width of the file name.
(update_current_files_info): Set the WIDTH members for cases
they're needed multiple times.  Note we do this explicitly here,
rather than caching at use, so that the fileinfo
structures can remain const in the sorting and presentation functions.
(sort_files): Call the new update_current_files_info() in this
initialization function.
(fileinfo_name_width): Renamed from fileinfo_width,
and adjusted to return the cached value if available.
2021-04-11 14:58:45 +01:00
Carl Edquist 751ad58e00 ls: add --sort=width option to sort by file name width
This helps identify the outliers for long filenames, and also produces
a more compact display of columns when listing a directory with many
entries of various widths.

* src/ls.c (sort_type, sort_types, sort_width): New sort_width sort
type.
(sort_args): Add "width" sort arg.
(cmp_width, fileinfo_width): New sort function and helper for file name
width.
(quote_name_width): Add function prototype declaration.
(usage): Document --sort=width option.
* doc/coreutils.texi: Document --sort=width option.
* tests/ls/sort-width-option.sh: New test for --sort=width option.
* tests/local.mk: Reference new test.
* NEWS: Mention the new feature.
2021-04-11 14:58:38 +01:00
Paul Eggert e3766c5db1 env: simplify --split-string memory management
* bootstrap.conf (gnulib_modules): Add idx.
* src/env.c: Include idx.h, minmax.h.
Prefer idx_t to ptrdiff_t when values are nonnegative.
(valid_escape_sequence, escape_char, validate_split_str)
(CHECK_START_NEW_ARG):
Remove; no longer needed now that we validate as we go.
(struct splitbuf): New type.
(splitbuf_grow, splitbuf_append_byte, check_start_new_arg)
(splitbuf_finishup): New functions.
(build_argv): New arg ARGC.  Validate and process in one go, using
the new functions; this is simpler and more reliable than the old
approach (as witness the recent bug).  Avoid integer overflow in
the unlikely case where the string contains more than INT_MAX
arguments.
(parse_split_string): Simplify by exploiting the new build_argv.
2021-03-29 21:44:58 -07:00
Paul Eggert 5b5622f605 build: update gnulib submodule to latest 2021-03-29 21:44:58 -07:00
Pádraig Brady 9f1bda18f2 tests: add a test case for recent env fix
* tests/misc/env-S.pl: Add a test case for recent commit ec6904f0.
2021-03-29 15:48:39 +01:00
Pádraig Brady f20797f1d7 maint: ignore all .a files in .gitignore
* .gitignore: Rather than add the new src/libcksum_pclmul.a,
just ignore any such libs.
* src/.gitignore: Likewise.
Reported by Kristoffer Brånemyr.
2021-03-29 14:10:09 +01:00
Paul Eggert 12abdb757a env: prefer ptrdiff_t
* src/env.c (usvars_used, vnlen, unset_envvars, expansion)
(build_argv): Prefer ptrdiff_t to size_t when either will do.
2021-03-26 15:17:28 -07:00
Paul Eggert 5f99c7533d env: improve whitespace warning
* src/env.c (main): Issue -S warning for any whitespace,
not just space.
2021-03-26 15:17:28 -07:00
Paul Eggert 7fc887299e env: use GNU coding style
* src/env.c: Use GNU coding style for recentish changes.
2021-03-26 15:17:28 -07:00
Paul Eggert 52f11d8b7c env: remove asserts
The assertions didn’t help catch the most recent bug which
was in their area, and kind of get in the way.
* src/env.c: Do not include <assert.h>, and remove all assertions.
These seem to have been put in to pacify gcov, but surely there’s
a better way.
(escape_char): Pacify GCC with 'assume' instead.
2021-03-26 15:17:28 -07:00
Paul Eggert 6c4efdc0f5 doc: document env fix
* NEWS, doc/coreutils.texi (env invocation): Document recent change.
2021-03-26 15:17:28 -07:00
Paul Eggert ec6904f0ed env: fix address violation with '\v' in -S
Problem reported by Frank Busse (Bug#47412).
* src/env.c (C_ISSPACE_CHARS): New macro.
(shortopts, build_argv, main): Treate all C-locale space
characters like space and tab, for compatibility with FreeBSD.
(validate_split_str, build_argv, parse_split_string):
Use the C locale, not the current locale, to determine whether a
byte is a space character.
2021-03-26 13:50:16 -07:00
Paul Eggert 4698e284f3 hostname: pacify valgrind
* src/hostname.c (main) [IF_LINT]: Free hostname (Bug#47384).
2021-03-25 11:21:15 -07:00
Paul Eggert c7a588ac36 hostname: use puts
* src/hostname.c (main): Prefer puts to printf "%s\n".
2021-03-25 09:19:02 -07:00
Paul Eggert 8980b7c898 maint: indenting
* src/ln.c: Fix indenting.
2021-03-25 09:19:02 -07:00
Kamil Dudka 6e98f67758 ln: fix memory leaks in do_link
* src/ln.c (do_link): Free memory allocated by convert_abs_rel
on all code paths (Bug#47373).
2021-03-25 09:19:02 -07:00
Paul Eggert 238dd8eb76 stdbuf: port lib to macOS
* src/libstdbuf.c (fprintf, free, strtoumax): Undef these too,
since Gnulib might replace them.
2021-03-24 17:00:46 -07:00
Paul Eggert 1677294076 cksum: port recent changes to macOS
* src/cksum.c (cksum_slice8): Fix bug on little-endian
platforms lacking __bswap_32: the SWAP macro evaluates
its argument multiple times, but the macro has a side effect.
2021-03-24 16:47:50 -07:00