Commit Graph
29188 Commits
Author SHA1 Message Date
Pádraig Brady ba5e6885d2 copy: disallow copy_file_range() on Linux kernels before 5.3
copy_file_range() before Linux kernel release 5.3 had many issues,
as described at https://lwn.net/Articles/789527/, which was
referenced from https://lwn.net/Articles/846403/; a more general
article discussing the generality of copy_file_range().
Linux kernel 5.3 was released in September 2019, which is new enough
that we need to actively avoid older kernels.

* src/copy.c (functional_copy_file_range): A new function
that returns false for Linux kernels before version 5.3.
(sparse_copy): Call this new function to gate use of
copy_file_range().
2021-05-13 00:14:22 +01:00
Pádraig Brady 6b499720fe tests: fix tests/cp/sparse-2.sh false failure on some systems
* tests/cp/sparse-2.sh: Double check cp --sparse=always,
with dd conv=sparse, in the case where the former didn't
create a sparse file.  Now that this test is being newly run
on macos, we're seeing a failure due to seek() not creating
holes on apfs unless the size is >= 16MiB.
2021-05-12 22:14:11 +01:00
Pádraig Brady 4eb2e9c11a tests: ensure we test SEEK_DATA where used
fiemap is no longer the default copy implementation,
so check for SEEK_DATA support instead as that's preferred.
This will ensure better test coverage on systems without fiemap.

* init.cfg: Replace fiemap_capable_ with seek_data_capable_.
This is best supported with python 3 so prefer that.
* tests/seek-data-capable: A new test script checking for
SEEK_DATA support on the passed file name,
called from seek_data_capable_.
* tests/fiemap-capable: Remove no longer used probing script.
* tests/cp/fiemap-perf.sh: Renamed to tests/cp/sparse-perf.sh
* tests/cp/fiemap-2.sh: Renamed to tests/cp/sparse-2.sh
* tests/cp/fiemap-extents.sh: Renamed to tests/cp/sparse-extents.sh
* tests/cp/sparse-fiemap.sh: Renamed to tests/cp/sparse-extents-2.sh
* tests/cp/fiemap-FMR.sh: Renamed to tests/cp/copy-FMR.sh
* tests/local.mk: Reference the renamed tests.
2021-05-12 22:14:11 +01:00
Pádraig Brady 2e66e1732f copy: handle system security config issues with copy_file_range()
* src/copy.c (sparse_copy): Upon EPERM from copy_file_range(),
fall back to a standard copy, which will give a more accurate
error as to whether the issue is with the source or destination.
Also this will avoid the issue where seccomp or apparmor are
not configured to handle copy_file_range(), in which case
the fall back standard copy would succeed without issue.
This specific issue with seccomp was noticed for example in:
https://github.com/golang/go/issues/40900
2021-05-12 22:14:11 +01:00
Pádraig Brady 017877bd08 copy: handle EOPNOTSUPP from SEEK_DATA
* src/copy.c (infer_scantype): Ensure we don't error out
if SEEK_DATA returns EOPNOTSUPP, on systems where this value
is distinct from ENOTSUP.  Generally both of these should be checked.
2021-05-12 22:14:11 +01:00
Pádraig Brady 8ec0d1799e copy: handle ENOTSUP from copy_file_range()
* src/copy.c (sparse_copy): Ensure we fall back to
a standard copy if copy_file_range() returns ENOTSUP.
This generally is best checked when checking ENOSYS,
but it also seems to be a practical concern on Centos 7,
as a quick search gave https://bugzilla.redhat.com/1840284
2021-05-12 22:14:11 +01:00
Pádraig Brady e35e4d1757 build: update gnulib submodule to latest
Fixes a bits/long-double.h include build issue on some systems.

* bootstrap: Sync new --version option from gnulib.
* gnulib: Update to lastest.
Reported by Carl Edquist
2021-05-10 17:44:57 +01:00
Carl Edquist bf483155e5 build: fix __get_cpuid_count check to catch link failure
The test program will compile successfully even if __get_cpuid_count
is not declared.  The error for the missing symbol will only show up
at link time.  Thus, use AC_LINK_IFELSE instead of AC_COMPILE_IFELSE.

* configure.ac (__get_cpuid_count check): Use C_LINK_IFELSE instead
of AC_COMPILE_IFELSE.
(__get_cpuid check): Likewise.
2021-05-10 13:35:20 +01:00
Pádraig Brady 09bbb2b703 maint: consistently free hash structures in dev mode
Ensure we call hash_free() to avoid valgrind and leak_sanitizer
"definitely lost" warnings.  These were not real leaks as
we terminate immediately after, but we should avoid these
"definitely lost" warnings where possible.

* src/copy.c: Add dest_info_free() and src_info_free().
* src/copy.h: Declare the above.
* src/cp-hash.c: Don't define unless "lint" is defined.
* src/install.c: Call dest_info_free() in dev mode.
* src/mv.c: Likewise.
* src/cp.c: Likewise. Also call src_info_free().
* src/ln.c: Call hash_free() in dev mode.
* src/tail.c: Call hash_free() even if about to exit, in dev mode.

Fixes https://bugs.gnu.org/48189
2021-05-08 14:57:33 +01:00
Bernhard Voelker 65e4d0f71e maint: fix sc_space_before_open_paren failure
* src/copy.c (dest_info_init): Add space before parens.
(src_info_init): Likewise.
Syntax-check failure introduced in the previous commit.
2021-05-06 23:42:57 +02:00
Pádraig Brady f1d2e8e289 copy: exit immediately upon failure to allocate hash memory
* src/copy.c (dest_info_init, src_info_init): Terminate immediately
upon memory exhaustion.
2021-05-03 18:53:35 +01:00
Pádraig Brady ea9af99234 copy: ensure we enforce --reflink=never
* src/copy.c (sparse_copy): Don't use copy_file_range()
with --reflink=never as copy_file_range() may implicitly
use acceleration techniques like reflinking.
(extent_copy): Pass through whether we allow reflinking.
(lseek_copy): Likewise.
Fixes https://bugs.gnu.org/48164
2021-05-02 21:27:17 +01:00
Pádraig Brady e7fff54c92 wc: add --debug to diagnose which implementation used
* src/wc.c: (main): Handle the new --debug option.
Only call avx2_supported if needed.
(avx2_supported): Diagnose various failures and attempts.
* NEWS: Mention the new wc improvement and --debug option.
2021-05-02 20:51:37 +01:00
Kristoffer Brånemyr 4064c57380 wc: use avx2 optimization when counting only lines
Use cpuid to detect CPU support for avx2 instructions.
Performance was seen to improve by 5x for a file with only newlines,
while the performance for a file with no such characters is unchanged.

* configure.ac [USE_AVX2_WC_LINECOUNT]: A new conditional,
set when __get_cpuid_count() and avx2 compiler intrinsics are supported.
* src/wc.c (avx2_supported): A new function using __get_cpuid_count()
to determine if avx2 instructions are supported.
(wc_lines): A new function refactored from wc(),
which implements the standard line counting logic,
and provides the fallback implementation for when avx2 is not supported.
* src/wc_avx2.c: A new module to implement using avx2 intrinsics.
* src/local.mk: Reference the new module.  Note we build as a separate
lib so that it can be portably built with separate -mavx2 etc. flags.
2021-05-02 20:51:37 +01:00
Paul Eggert d435cfc0bc touch: fix wrong diagnostic (Bug#48106)
Problem reported by Roland (Bug#48106).
* src/touch.c (touch): Take more care when deciding whether
to use open_errno or utime_errno in the diagnostic.
Stop worrying about SunOS 4 (which as part of the problem),
as it’s long obsolete.  For Solaris 10, verify that EINVAL
really means the file was a directory.
2021-05-01 15:47:13 -07:00
Paul Eggert 62a7ce5f50 maint: port to Autoconf 2.71
* configure.ac: Use AC_PROG_CC, not AC_PROG_CC_STDC.
* gl/modules/smack (configure.ac):
* m4/jm-macros.m4 (coreutils_MACROS):
* m4/xattr.m4 (gl_FUNC_XATTR):
Use AS_HELP_STRING, not AC_HELP_STRING.
* m4/check-decl.m4 (gl_CHECK_DECLS):
Do not require AC_HEADER_TIME; we no longer care about it directly.
* m4/jm-macros.m4 (coreutils_MACROS):
Do not require AC_ISC_POSIX, which became obsolete in 2006.
Use AC_LINK_IFELSE instead of AC_TRY_LINK.
2021-04-26 23:32:45 -07:00
Paul Eggert 8527942835 csplit: size_t overflow check
* src/csplit.c (get_new_buffer): Fix unlikely size_t overflow.
2021-04-26 23:32:45 -07:00
Paul Eggert 5fcf8c6487 build: update gnulib submodule to latest
* src/csplit.c (load_buffer):
* src/pinky.c (create_fullname):
Use intprops-based checks rather than xalloc_oversized,
since Gnulib xalloc.h no longer includes xalloc-oversized.h.
2021-04-26 23:32:45 -07:00
Zorro Lang 785478013b copy: do not refuse to copy a swap file
* src/copy.c (sparse_copy): Fallback to read() if copy_file_range()
fails with ETXTBSY.  Otherwise it would be impossible to copy files
that are being used as swap.  This used to work before introducing
the support for copy_file_range() in coreutils.  (Bug#48036)
2021-04-26 18:00:58 -07:00
Bernhard Voelker b7091093bb tests: fix FP in ls/stat-free-color.sh
On newer systems like Fedora 34 and openSUSE Tumbleweed, ls(1) calls
newfstatat(STDOUT_FILENO, ...), but only when there is something to
output.

* tests/ls/stat-free-color.sh: Add -a option to the reference invocation
of ls, thus enforcing something gets output.
2021-04-22 20:14:32 +02:00
Pádraig Brady 34a48bf0f0 doc: clarify that ln --relative requires --symbolic to be specified
* doc/coreutils.texi (ln invocation): State --symbolic is required.
* src/ln.c (usage): Explicitly state -s is not implied.
Fixes https://bugs.gnu.org/47703
2021-04-11 19:23:37 +01:00
Pádraig Brady 512fe04902 doc: clarify what's counted by wc
* src/wc.c (usage): State that only printable characters are considered
when counting words.  This also disambiguates wether we're talking
about bytes or characters in this context.
* doc/coreutils.texi (wc invocation): Likewise.  Also clarify
that --characters counts valid locale aware characters,
and that --lines does not count a trailing "line" unless
it ends with a newline character.
Fixes https://bugs.gnu.org/47702
2021-04-11 19:23:13 +01:00
Pádraig Brady ef772bf97f maint: use "char const *" rather than "const char *"
* cfg.mk (sc_prohibit-const-char): Add a new syntax-check to
enforce this style.
* *.[ch]: sed -i 's/const char \*/char const */g'
2021-04-11 18:33:45 +01:00
Pádraig Brady 34ed19eed8 ls: cache name width determination
This is especially important now for --sort=width,
as that can greatly increase how often this
expensive quote_name_width() function is called per file.

This also helps the default invocation of ls,
or specifically the --format={across,vertical} cases
(when --width is not set to 0),
to avoid two calls to this function per file.

Note the only case where we later compute the width,
is for --format=commas.  That's only done once though,
so we leave the computation close to use to
maximize hardware caching.

* src/ls.c (struct fileinfo): Add a WIDTH member to cache
the screen width of the file name.
(update_current_files_info): Set the WIDTH members for cases
they're needed multiple times.  Note we do this explicitly here,
rather than caching at use, so that the fileinfo
structures can remain const in the sorting and presentation functions.
(sort_files): Call the new update_current_files_info() in this
initialization function.
(fileinfo_name_width): Renamed from fileinfo_width,
and adjusted to return the cached value if available.
2021-04-11 14:58:45 +01:00
Carl Edquist 751ad58e00 ls: add --sort=width option to sort by file name width
This helps identify the outliers for long filenames, and also produces
a more compact display of columns when listing a directory with many
entries of various widths.

* src/ls.c (sort_type, sort_types, sort_width): New sort_width sort
type.
(sort_args): Add "width" sort arg.
(cmp_width, fileinfo_width): New sort function and helper for file name
width.
(quote_name_width): Add function prototype declaration.
(usage): Document --sort=width option.
* doc/coreutils.texi: Document --sort=width option.
* tests/ls/sort-width-option.sh: New test for --sort=width option.
* tests/local.mk: Reference new test.
* NEWS: Mention the new feature.
2021-04-11 14:58:38 +01:00
Paul Eggert e3766c5db1 env: simplify --split-string memory management
* bootstrap.conf (gnulib_modules): Add idx.
* src/env.c: Include idx.h, minmax.h.
Prefer idx_t to ptrdiff_t when values are nonnegative.
(valid_escape_sequence, escape_char, validate_split_str)
(CHECK_START_NEW_ARG):
Remove; no longer needed now that we validate as we go.
(struct splitbuf): New type.
(splitbuf_grow, splitbuf_append_byte, check_start_new_arg)
(splitbuf_finishup): New functions.
(build_argv): New arg ARGC.  Validate and process in one go, using
the new functions; this is simpler and more reliable than the old
approach (as witness the recent bug).  Avoid integer overflow in
the unlikely case where the string contains more than INT_MAX
arguments.
(parse_split_string): Simplify by exploiting the new build_argv.
2021-03-29 21:44:58 -07:00
Paul Eggert 5b5622f605 build: update gnulib submodule to latest 2021-03-29 21:44:58 -07:00
Pádraig Brady 9f1bda18f2 tests: add a test case for recent env fix
* tests/misc/env-S.pl: Add a test case for recent commit ec6904f0.
2021-03-29 15:48:39 +01:00
Pádraig Brady f20797f1d7 maint: ignore all .a files in .gitignore
* .gitignore: Rather than add the new src/libcksum_pclmul.a,
just ignore any such libs.
* src/.gitignore: Likewise.
Reported by Kristoffer Brånemyr.
2021-03-29 14:10:09 +01:00
Paul Eggert 12abdb757a env: prefer ptrdiff_t
* src/env.c (usvars_used, vnlen, unset_envvars, expansion)
(build_argv): Prefer ptrdiff_t to size_t when either will do.
2021-03-26 15:17:28 -07:00
Paul Eggert 5f99c7533d env: improve whitespace warning
* src/env.c (main): Issue -S warning for any whitespace,
not just space.
2021-03-26 15:17:28 -07:00
Paul Eggert 7fc887299e env: use GNU coding style
* src/env.c: Use GNU coding style for recentish changes.
2021-03-26 15:17:28 -07:00
Paul Eggert 52f11d8b7c env: remove asserts
The assertions didn’t help catch the most recent bug which
was in their area, and kind of get in the way.
* src/env.c: Do not include <assert.h>, and remove all assertions.
These seem to have been put in to pacify gcov, but surely there’s
a better way.
(escape_char): Pacify GCC with 'assume' instead.
2021-03-26 15:17:28 -07:00
Paul Eggert 6c4efdc0f5 doc: document env fix
* NEWS, doc/coreutils.texi (env invocation): Document recent change.
2021-03-26 15:17:28 -07:00
Paul Eggert ec6904f0ed env: fix address violation with '\v' in -S
Problem reported by Frank Busse (Bug#47412).
* src/env.c (C_ISSPACE_CHARS): New macro.
(shortopts, build_argv, main): Treate all C-locale space
characters like space and tab, for compatibility with FreeBSD.
(validate_split_str, build_argv, parse_split_string):
Use the C locale, not the current locale, to determine whether a
byte is a space character.
2021-03-26 13:50:16 -07:00
Paul Eggert 4698e284f3 hostname: pacify valgrind
* src/hostname.c (main) [IF_LINT]: Free hostname (Bug#47384).
2021-03-25 11:21:15 -07:00
Paul Eggert c7a588ac36 hostname: use puts
* src/hostname.c (main): Prefer puts to printf "%s\n".
2021-03-25 09:19:02 -07:00
Paul Eggert 8980b7c898 maint: indenting
* src/ln.c: Fix indenting.
2021-03-25 09:19:02 -07:00
Kamil Dudka 6e98f67758 ln: fix memory leaks in do_link
* src/ln.c (do_link): Free memory allocated by convert_abs_rel
on all code paths (Bug#47373).
2021-03-25 09:19:02 -07:00
Paul Eggert 238dd8eb76 stdbuf: port lib to macOS
* src/libstdbuf.c (fprintf, free, strtoumax): Undef these too,
since Gnulib might replace them.
2021-03-24 17:00:46 -07:00
Paul Eggert 1677294076 cksum: port recent changes to macOS
* src/cksum.c (cksum_slice8): Fix bug on little-endian
platforms lacking __bswap_32: the SWAP macro evaluates
its argument multiple times, but the macro has a side effect.
2021-03-24 16:47:50 -07:00
Bernhard Voelker b5a6a56ffc maint: update bootstrap from gnulib
* bootstrap: Sync from gnulib/build-aux/bootstrap; the previous gnulib
update (commit 1a3eb6c30) missed to update that file.
2021-03-22 16:52:11 -07:00
Paul Eggert a71539e628 ptx: remove use of diacrit module
The diacrit module is obsolete, and ptx’s use of it is obsolete
too; it assumes an 8-bit locale (not that common these days) and
that TeX cannot process the 8-bit characters (nowadays, it can).
* NEWS, doc/coreutils.texi (Charset selection in ptx): Document this.
* bootstrap.conf (gnulib_modules): Remove diacrit.
* src/ptx.c: Do not include diacrit.h.
(print_field, fix_output_parameters): Remove obsolete support
for 8-bit diacritics.
2021-03-21 14:01:15 -07:00
Paul Eggert 1a3eb6c305 build: update gnulib submodule to latest 2021-03-21 14:01:15 -07:00
Pádraig Brady 86c8dc2e85 cksum: don't exit immediately if a single file overflows
This behavior was introduced in commit FILEUTILS-4_0_44-4-g519b707b4.

* src/cksum.c (cksum_slice8): Only report the overflow, and continue.
* src/cksum_pclmul.c (cksum_pclmul): Likewise.
2021-03-15 13:12:29 +00:00
Pádraig Brady b73b9fcb1d cksum: add --debug to diagnose which implementation used
* src/cksum.c: (main): Use getopt_long to parse options,
and handle the new --debug option.
(pclmul_supported): Diagnose various failures and attempts.
* NEWS: Mention the new option.
2021-03-15 12:44:27 +00:00
Kristoffer Brånemyr 4b9118cdb3 cksum: use pclmul hardware instruction for CRC32 calculation
Use cpuid to detect CPU support for hardware instruction.
Fall back to slice by 8 algorithm if not supported.
A 500MiB file improves from 1.40s to 0.67s on an i3-2310M

* configure.ac [USE_PCLMUL_CRC32]: A new conditional,
set when __get_cpuid() and clmul compiler intrinsics are supported.
* src/cksum.c (pclmul_supported): A new function using __get_cpuid()
to determine if pclmul instructions are supported.
(cksum): A new function refactored from cksum_slice8(),
which calls pclmul_supported() and then cksum_slice8()
or cksum_pclmul() as appropriate.
* src/cksum.h: Export the crctab array for use in the new module.
* src/cksum_pclmul.c: A new module to implement using pclmul intrinsics.
* po/POTFILES.in: Reference the new cksum_pclmul module.
* src/local.mk: Likewise.  Note we build it as a separate library
so that it can be portably built with separate -mavx etc. flags.
* tests/misc/cksum.sh: Add new test modes for pertinent buffer sizes.
2021-03-15 12:44:22 +00:00
Pádraig Brady 90c4ef1f7d maint: propagate DEPENDENCIES to libs in single binary mode
build-aux/gen-single-binary.sh (override_single): A new function
to refactor the existing mappings for dir, vdir, and arch.
This function now also sets the DEPENDENCIES variable so that these
dependencies can be maintained later in the script, where
we now propagate the automake generated $(src_$cmd_DEPENDENCIES)
to our equivalent src_libsinglebin_$cmd_a_DEPENDENCIES.
This will ensure that any required libs are built,
which we require in a following change to cksum that
builds part of it as a separate library.
2021-03-14 23:45:46 +00:00
Pádraig Brady 9de1d153f8 rmdir: diagnose non following of symlinks with trailing slash
GNU/Linux is unusual here in that rmdir("symlink/") returns ENOTDIR,
whereas Solaris and FreeBSD at least, will follow the symlink
and remove the target directory.  We don't make the behavior
on Linux kernels consistent, but at least clarify
the confusing error message.

* src/rmdir (main): Output a specific error message for the above case.
(remove_parents): In the error message, don't assume intermediate paths
are directories, as they could be symlinks.
* tests/rmdir/symlink-errors.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
2021-02-19 11:35:30 +00:00
Kamil Dudka a5e0d8f387 stat,tail: add support for the exfat file system
Bug: https://bugzilla.redhat.com/1921427

* src/stat.c (human_fstype): Add case for the 'exfat' file system type.
* NEWS: Mention the Improvement.
Fixes https://bugs.gnu.org/46613
2021-02-18 13:50:29 +00:00