Commit Graph
7880 Commits
Author SHA1 Message Date
Michael Heimpold 73c8452e83 split: add new --hex-suffixes option
* doc/coreutils.texi (split invocation): Document the new option.
* src/split.c (usage): Likewise.
(main): Process the new option much like --numeric-suffixes,
but with an adjusted alphabet.
* tests/split/numeric.sh: Refactor to support --hex mode.
* NEWS: Mention the new feature.
2017-03-27 20:32:58 -07:00
Pádraig Brady 264f5446cd md5sum,b2sum,sha*sum: don't erroneously trigger BSD reversed mode
* src/md5sum.c (split_3): Verify hex digits internally before
triggering the global bsd_reversed mode flag.
(bsd_split_3): Likewise.
* tests/misc/md5sum-bsd.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/26263
2017-03-27 20:32:58 -07:00
Philipp Thomas 7c228bc55e df: avoid querying excluded file systems
* src/df.c (filter_mount_list): Avoid stat() on
explicitly excluded file systems, which is especially
significant in cases like `-x nfs` which may hang.
* NEWS: Mention the bug fix.
2017-03-26 22:48:33 -07:00
Pádraig Brady 8e7bcdd806 maint: avoid a static analysis warning in expand-common
* src/expand-common.c (next_file): We're dependent on calling
this function with NULL to initialize things appropriately.
So enforce this with assert(), which avoids a warning from
clang-anaylzer.
2017-03-26 15:20:26 -07:00
Pádraig Brady 0839e6d8d9 split: process more efficiently when filters exit early
* src/split.c (bytes_split): Don't write to an existing filter
if it has exited.  When filters exit early, skip input data if
possible.  Refactor out 2 redundant variables.
* tests/split/filter.sh: Improve test coverage given the
new more efficient processing.  Also use a 10TB file to
expand the file systems tested on.
2017-03-26 15:20:26 -07:00
Pádraig Brady a79dbb97bf split: ensure input is processed when filters exit early
commit v8.25-4-g62e7af0 introduced the issue as it
broke out of the processing loop irrespective of
the value of new_file_flag which was used to indicate
a finite number of filters or not.

For example, this ran forever (as it should):
  $ yes | split --filter="head -c1 >/dev/null" -b 1000
However this exited immediately due to EPIPE being propagated
back through cwrite and the loop not considering new filters:
  $ yes | split --filter="head -c1 >/dev/null" -b 100000

Similarly processing would exit early for a bounded number of
output files, resulting in empty data sent to all but the first:
  $ truncate -s10T big.in
  $ split --filter='head -c1 >$FILE' -n 2 big.in
  $ echo $(stat -c%s x??)
  1 0

I was alerted to this code by clang-analyzer,
which indicated dead assigments, which is often
an indication of code that hasn't considered all cases.

* src/split.c (bytes_split): Change the last condition in
the processing loop to also consider the number of files
before breaking out of the processing loop.
* tests/split/filter.sh: Add a test case.
* NEWS: Mention the bug fix.
2017-03-26 15:20:23 -07:00
Pádraig Brady 51d7a16c12 build: for factor use C in more cases for arm64 and ppc64
* src/longlong.h: Sync from gmp repo incorporating:
Use asm-free umul_ppmm() on arm64 and ppc64.
2017-03-08 23:02:06 -08:00
Pádraig Brady 79a5cb161d stat,tail: support "RDT" Linux kernel control file system
* src/stat.c (human_fstype): This file system is the user interface
for resource allocation in Intel's Resource Director Technology.
2017-03-08 17:55:09 -08:00
Bernhard Voelker 6dc540ed7a build: avoid redundant build of tr with --enable-single-binary
* src/local.mk [check-duplicate-no-install]: Depend on the
single-binary tr, or the system tr, as the edge case
where these are not available only result in the sanity
check being effectively ignored.
2017-03-07 21:34:06 -08:00
Pádraig Brady 6555f41883 timeout: handle multiple children on solaris
* src/timeout.c (install_sigchld): A new function to
install the SIGCHLD handler using sigaction() rather
than signal(), because with the latter on solaris
the signal handler is reset to default and thus
sigsuspend() only returns for the first finished child.
Reported by Assaf Gordon.
2017-03-03 00:30:20 -08:00
Pádraig Brady b88e4dd3c6 build: fix libstdbuf build on AIX 7
* src/libstdbuf.c: undef malloc so as libstdbuf is
not linked with gnulib, and anyway the replacement is
never needed since we never malloc(0).
Reported by Assaf Gordon.
2017-03-03 00:23:56 -08:00
Pádraig Brady 3be3da3bec expand: avoid an extraneous warning on 32 bit
* src/expand-common (parse-tab-stops): Exit earlier upon overflow
so another warning isn't issued (on 32 bit) in add_tab_stop().
Flagged in https://hydra.nixos.org/build/49499970
2017-03-01 09:25:28 -08:00
Pádraig Brady 06c3123f6e maint: avoid a -Werror=null-dereference with GCC-6.3.1
* src/stty.c (sane_mode): Assert to inform the compiler
we know the pointer will be valid.
2017-02-28 19:26:04 -08:00
Pádraig Brady b1413b6011 expand,unexpand: support specifying a trailing tab size
* doc/coreutils.texi (expand invocation): Document the feature.
(unexpand invocation): Likewise.
* src/expand-common.c (extend_size): A new global to use
when the last tab stop is prefixed by '/'.
(set_extend_size): A new function to validate and set
the new extend_size global.
(parse_tab_stops): Call set_extend_size() for '/' prefixes.
(finalize_tab_stops): Ensure a single specified '/' is
treated like a standard tabsize, but also ensure that
when '/' is specified with a single other entry that
we process as a list rather than a tab size.
(get_next_tab_stop): Use the tab size if set,
for items after the user specified tab position list.
* tests/misc/expand.pl: Add test cases
* NEWS: Mention the new feature.
Fixes http://bugs.gnu.org/25540
2017-02-28 19:26:01 -08:00
Pádraig Brady db277d6077 cp: set SELinux context for --parents directories
* src/copy.c (set_process_security_ctx, set_file_security_ctx):
Export for use in cp.c.
* src/copy.h: Likewise.
* src/cp.c (make_dir_parents_private): Call the exported functions
to set the security context for new and updated directories.
* tests/cp/cp-a-selinux.sh: Add a test case.

Fixes http://bugs.gnu.org/25378
2017-02-22 20:23:31 -08:00
Paul Eggert 1c27b56095 maint: xsetmode renamed to xbinary-io
* bootstrap.conf, src/base64.c, src/cat.c, src/cksum.c:
* src/head.c, src/md5sum.c, src/od.c, src/split.c, src/sum.c:
* src/tac.c, src/tail.c, src/tee.c, src/tr.c, src/wc.c:
Adjust to renaming of the xsetmode module to xbinary-io,
and of the xsetmode function to xset_binary_mode.
2017-02-16 00:40:50 -08:00
Paul Eggert 75aababed4 maint: use xsetmode, not xfreopen
This fixes a bug noted by Eric Blake.  Code was using xfreopen to
change files to binary mode, but this fails for stdout when in
append mode.  Such code should use xsetmode instead.  This affects
only the port on platforms like MS-Windows which distiguish text
from binary I/O.
* bootstrap.conf (gnulib_modules):
Remove xfreopen and add xsetmode.  Sort.
* src/base64.c (main):
* src/cat.c (main):
* src/cksum.c (cksum):
* src/head.c (head_file, main):
* src/md5sum.c (digest_file):
* src/od.c (open_next_file):
* src/split.c (main):
* src/sum.c (bsd_sum_file, sysv_sum_file):
* src/tac.c (tac_file, main):
* src/tail.c (tail_file):
* src/tee.c (tee_files):
* src/tr.c (main):
* src/wc.c (wc_file): Use xsetmode, not xfreopen.
2017-02-15 15:59:16 -08:00
Pádraig Brady bd4bb42d65 maint: tweaks so syntax tests pass for previous commit
* src/force-link.h: Don't include headers already included by system.h
* src/force-link.c: Likewise.  Also include system.h and
explicitly mark extern functions as such.
2017-02-12 18:05:37 -08:00
Paul Eggert 376967889e ln: replace destination links more atomically
If the file B already exists, commands like 'ln -f A B' and
'cp -fl A B' no longer remove B before creating the new link.
Instead, they arrange for the new link to replace B atomically.
This should fix a race condition reported by Mike Crowe (Bug#25680).
* NEWS, doc/coreutils.texi (cp invocation, ln invocation):
Document this.
* bootstrap.conf (gnulib_modules): Add symlinkat.
* src/copy.c, src/ln.c: Include force-link.h.
* src/copy.c (same_file_ok): It's also OK to remove a destination
symlink when creating symbolic links, or when the source and
destination are on the same file system and when creating hard links.
* src/copy.c (create_hard_link, copy_internal):
* src/ln.c (do_link):
Rewrite using force_linkat and force_symlinkat, to close a window
where the destination temporarily does not exist.
* src/cp.c (main): Do not set x.unlink_dest_before_opening
merely because we are in link-creation mode.
* src/force-link.c, src/force-link.h: New files.
* src/local.mk (copy_sources, src_ln_SOURCES): Add them.
* tests/cp/same-file.sh: Adjust test case to match fixed behavior.
2017-02-11 23:14:02 -08:00
Tobias Stoeckmann 2f69dba5df timeout: fix race possibly terminating wrong process
The race is unlikely, as timeout(1) needs to receive a signal
in the few operations between waitpid() returning and exit().
Also the system needs to have reallocated the just released pid
in this time window.

Previously we never disabled the signal handler that sent
the termination signal to the "child" pid.  However once waitpid()
has reaped the child, the system is free to allocate that pid,
so we must ensure we don't process any further signals.

* build-aux/gen-lists-of-programs.sh: Build timeout(1) optionally...
* configure.ac: ...predicated on sigsuspend() being available.
* src/timeout.c (block_cleanup): A new function to ensure the
cleanup() handler is disabled after waitpid has returned.
(main): Use sigsuspend() to wait with cleanup() enabled but
disabled once it returns, and thus disabled for the waitpid() call.
(monitored_pid): Change to the more accurate pid_t.
* NEWS: Mention the fix.

Fixes http://bugs.gnu.org/25624
2017-02-09 22:15:43 -08:00
Janne Snabb 1ddcd24d52 tail: fix output of redundant headers when resuming
* src/tail.c (check_fspec): Only enable printing of the file header
if we've actually read some data and this is a new file.  Also
move printing of the file header to...
(dump_remainder): ...here, to allow printing only when data read.
* tests/tail-2/overlay-headers.sh: A new test for suspension
and resumption of tail.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/23539
2017-02-08 14:42:18 -08:00
Pádraig Brady a520942efa tail: fix erroneous status about 'giving up' on file
* src/tail.c (recheck): Set f->ignore before we
use it to show the appropriate error.
* tests/tail-2/retry.sh: Ensure the "giving up" message
is not presented.
2017-02-08 14:40:22 -08:00
Pádraig Brady 7a9c524f15 build: fix issue with HAVE_FALLOCATE on centos5
* src/copy.c (punch_hole): Work around an empty definition
of HAVE_FALLOCATE which leads to a build error of:
"error: #if with no expression"
That was triggered by the inclusion of <linux/fs.h> in
commit v8.25-68-g89e1fef with kernel-headers-2.6.18.
Reported by Nelson H. F. Beebe
2017-01-25 18:24:29 +00:00
Manolis Ragkousis c974f835a5 build: Properly expand cu_install_program when cross-compiling
* src/local.mk (cu_install_program): Replace @INSTALL_PROGRAM@
with @INSTALL@ when cross-compiling; missed in commit 477a1e8e.
Message-Id: <20170125163329.5690-1-manolis837@gmail.com>
Copyright-paperwork-exempt: Yes
2017-01-25 10:55:48 -06:00
Mike Swanson 9f888b75c0 dircolors: highlight windows archive format
* src/dircolors.hin: Match *.{wim,swm,dwn,esd}
2017-01-21 16:05:58 +00:00
Assaf Gordon bd9beb5102 maint: appease syntax-check failures due to recent updates
* bootstrap: s/time stamp/timestamp/.
* old/fileutils/NEWS: Likewise.
* src/tail.c: Avoided a long line.
2017-01-21 15:04:43 +00:00
Paul Eggert b14be5085c date: fix TZ= regression
Problem reported by Paul Wise for Debian, in:
https://bugs.debian.org/851934
This is fallout from the fix for GNU Bug#23035.
* src/date.c (batch_convert): New args TZ and TZSTRING.
All uses changed.
(batch_convert, main): Adjust to parse_datetime2 API change.
(main): Allocate time zone object.
* tests/misc/date-debug.sh: Fix incorrect test case,
caught by the fix.
* tests/misc/date.pl: Test the fix.
2017-01-20 18:24:54 -08:00
Paul Eggert 0b61395910 date: new option spelling --rfc-email
* NEWS:
* doc/coreutils.texi (Time conversion specifiers)
(Options for date, Examples of date): Document this.
* src/date.c (rfc_email_format): Rename from rfc_2822_format.
All uses changed.
(usage, long_options): Support --rfc-email.
2017-01-14 23:59:01 -08:00
Paul Eggert 103d2b92da maint: modernize URLs
A lot of this is converting http: to https:.
Also, gmane went away, so remove URLs that no longer work and
are not easy to figure out what they were.
Some of this stuff is so old that it no longer matters anyway.
2017-01-14 23:59:01 -08:00
Paul Eggert f4650aadbe maint: standardize on "timestamp" as per POSIX 2017-01-09 15:15:04 -08:00
Pádraig Brady 9c0a3a27f7 stty: ensure no side effects from invalid options
* src/stty.c (apply_settings): A new function refactored
from main() that is used to both check and apply options.
(main): Call apply_settings before we open the device,
so all validation is done before interacting with a device.
* NEWS: Mention the improvement.
* tests/misc/stty.sh: Add a test case.
2017-01-09 00:27:07 +00:00
Pádraig Brady e7a2580b96 maint: update all copyright year number ranges
Run "make update-copyright" and then...

* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
2017-01-01 22:38:02 +00:00
Pádraig Brady e17e5f40b8 wc: with only --bytes, determine size more efficiently
* src/wc.c (wc): Avoid reading the end of the file
when the size is not a multiple of PAGE_SIZE,
as the special case handling for files in /proc and /sys
is only required when st_size is 0 or a multiple of PAGE_SIZE.
* tests/misc/wc-proc.sh: Add a test case.
2016-12-26 12:57:16 +00:00
William R. Fraser 9944e4763b wc: fix wrong byte counts when using --files-from0
* src/wc.c (main): Reset fstatus[0].failed between files when reusing
the fstatus[0] entry in --files-from0 mode.  This ensures a stat() is
done for each file, avoiding incorrect counts and redundant reading.
* NEWS: Mention the bug fix.
* tests/misc/wc-files0.sh: Add a test case.
Fixes http://bugs.gnu.org/23073
2016-12-19 20:13:51 +00:00
Torbjörn Granlund ca52f3bf3f factor: retry properly if Pollard rho gives a trivial factorization
* src/factor.c (factor_using_pollard_rho): Handle trivial factor g = n.
(factor_using_pollard_rho2): Handle trivial factor g1 = n1, g0 = n0.
* tests/misc/factor.pl: Add a test case.
Fixes http://bugs.gnu.org/25135
2016-12-08 10:10:38 +00:00
Niels Möller c44da11506 factor: fix infinite loop in gcd2_odd
* src/factor.c (gcd2_odd): Fix the case a1 == 0, a0 == 0.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/25135
2016-12-08 10:10:34 +00:00
Pádraig Brady 11c1b73c1b doc: fix --help for: od -t f[SIZE]
* src/od.c (usage): SIZE is that of float, not integer.
2016-12-02 13:10:18 +00:00
Pádraig Brady 0655b86a6e head: fix processing of non-seekable input as seekable
* src/head.c (elide_tail_bytes_file): Ensure we don't use
st_size unless we've previously used seek() to determine
the CURRENT_POS in the seekable file.
This was seen to cause issue on FreeBSD 11 when the pipe
buffer was filled with `yes | head --lines=-0`, in which
case st_size was 64KiB while ST_BLKSIZE() was 4KiB.
Reported by Assaf Gordon.
2016-11-28 18:36:32 +00:00
Kamil Dudka d8104265f2 install,mkdir: fix handling of -DZ and -pZ, respectively
... in the case where two or more directories nested in each other are
created and each of them defaults to a different SELinux context.

* src/install.c (make_ancestor): When calling defaultcon(), give it the
same path that is given to mkdir().  The other path is not always valid
wrt. current working directory.
* src/mkdir.c (make_ancestor): Likewise.
* NEWS: Mention the bug fix.

Reported at https://bugzilla.redhat.com/1398913
2016-11-28 16:17:31 +00:00
Pádraig Brady a39641cbb8 tac: fix mem corruption when failing to read non seekable inputs
This was detected with ASAN, but can also be seen without ASAN with:
  $ tac - - <&-
  tac: standard input: read error: Bad file descriptor
  *** Error in `tac': malloc(): memory corruption: 0x...

* src/tac.c (copy_to_temp): Don't close our output stream on
(possibly transient) output error, or on input error.
(temp_stream): clearerr() on the stream about to be reused,
to ensure future stream use is not impacted by transient errors.
* tests/misc/tac-2-nonseekable.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/25041
2016-11-28 13:19:24 +00:00
Pádraig Brady 6f30a99fa5 tail: fix uninitialized memory read when failing to read file
Reproduced under UBSAN with `tail -f <&-` giving:
  tail.c:2220:18: runtime error: load of value 190,
  which is not a valid value for type ‘_Bool'

* src/tail.c (tail_file): Ensure f->ignore is initialized
in all cases where we can't tail the specified file.
* tests/tail-2/follow-stdin.sh: Add a test case which
checks stderr has no UBSAN warnings.
Fixes http://bugs.gnu.org/25041
2016-11-27 21:10:15 +00:00
Pádraig Brady 286a401bc5 stat,tail: sync with latest Linux file systems
Update with the results from:

  kgit='https://git.kernel.org/cgit/linux/kernel/git'
  wget -q $kgit/torvalds/linux.git/plain/include/uapi/linux/magic.h \
    -O src/fs-latest-magic.h
  make src/fs-magic-compare

* src/stat.c (human_fstype): Add entries for:
BALLOON_KVM, CGROUP2, DAXFS, ZSMALLOC.
2016-11-27 00:28:16 +00:00
Pádraig Brady 4535563318 build: fix potential factor build failure on arm and risc
* src/longlong.h: Sync from gmp repo incorporating:
Protect umul_ppmm with do ... while (0)
Replace obsolete ARC asm 'J' constraints with 'Cal'
Provide umul_ppmm for riscv64
2016-11-27 00:28:16 +00:00
Pádraig Brady 12f6214d07 shred,sort: ensure faster unaligned access to rand module
glibc has changed the public define
from _STRING_ARCH_unaligned to _STRING_INLINE_unaligned as per
https://sourceware.org/bugzilla/show_bug.cgi?id=19462

* gl/lib/rand-isaac.c: Cater for both defines.
* gl/lib/randread.c: Likewise.
* src/system.h: Update commented out code.
2016-11-27 00:27:45 +00:00
Paul Eggert e490178c39 numfmt: pacify Sun C 5.14
* src/numfmt.c (main): Don't implicitly coerce pointer in bool
initializer.  Although it's portable C99 code, it's confusing.
2016-11-25 18:54:32 -08:00
Pádraig Brady d91aeef052 pr: fix read from invalid memory with tabs in separator
This was detected with:
  echo a > a; pr "-S$(printf "\t\t\t")" a -m a > /dev/null
Resulting in ASAN triggering:
  ====================================================
  ERROR: AddressSanitizer: global-buffer-overflow
  READ of size 1 at 0x00000041b622 thread T0
    #0 0x40506a in print_sep_string ../src/pr.c:2241
    #1 0x407ec4 in read_line ../src/pr.c:2493
    #2 0x40985c in print_page ../src/pr.c:1802
    #3 0x40985c in print_files ../src/pr.c:1618
    #4 0x4036e0 in main ../src/pr.c:1136

* src/pr.c (init_parameters): Ensure we only override the
specified separator when it's a single tab, thus matching
the calculated separator length.
* tests/pr/pr-tests.pl: Add a test case.
* NEWS: Mention the fix.
2016-11-25 14:08:42 +00:00
Pádraig Brady ca99c524e8 ptx: fix an invalid heap reference with short --width
* src/ptx.c (fix_output_parameters): Ensure line_width doesn't
go negative, which can happen when the --width is less
than the --gap-size.
* tests/misc/ptx-overrun.sh: Add a test case that triggers
with ASAN.  (Note the longer filename is needed to trigger).
Fixes http://bugs.gnu.org/25011
2016-11-24 16:21:59 +00:00
Bernhard Voelker 19157e87fd doc: clarify that readlink's --quiet option is on by default
* src/readlink.c (usage): Mark the --quiet/--silent option as active
by default.
* doc/coreutils.texi (readlink invocation): Likewise.
2016-11-24 10:51:27 +01:00
Pádraig Brady 4954f79ad2 split: fix memory corruption during chunk extraction
ASAN reported this error for: split -n2/3 /dev/null
  ERROR: AddressSanitizer: negative-size-param: (size=-1)
  #0 0x7f0d4c36951d in __asan_memmove (/lib64/libasan.so.2+0x8d51d)
  #1 0x404e06 in memmove /usr/include/bits/string3.h:59
  #2 0x404e06 in bytes_chunk_extract src/split.c:988
  #3 0x404e06 in main src/split.c:1626

Specifically there would be invalid memory access
and subsequent processing if the chunk to be extracted
was beyond the initial amount read from file (which is
currently capped at 128KiB).  This issue is not in a
released version, only being introduced in commit v8.25-4-g62e7af0

* src/split.c (bytes_chunk_extract): The initial_read != SIZE_MAX
should have been combined with && rather than ||, but also this
condition is always true in this function so remove entirely.
* tests/split/b-chunk.sh: Add a test case.

Fixes http://bugs.gnu.org/25003
2016-11-24 00:24:58 +00:00
Pádraig Brady 68c5eeccd0 tail: fix checking of remoteness when not using inotify
In recent commit v8.25-93-g7fc7206 we used the f->remote flag
which wasn't set in all cases.  This was detected with
ASAN giving this error when reading f->remote;
  runtime error: load of value 190,
  which is not a valid value for type '_Bool'

* src/tail.c (fremote): Query the system even without inotify.
(recheck): Always set f->fremote for valid files.
2016-11-24 00:24:21 +00:00