Commit Graph
26507 Commits
Author SHA1 Message Date
Jim Meyering 33a5de5a0e doc: tail: semi-deprecate --sleep-interval and --max-unchanged-stats
Those options are useful only on systems that lack inotify support
and in the unusual event that a system with inotify support must
resort to polling.
* src/tail.c (usage): Note that the --max-unchanged-stats=N and
--sleep-interval=N options are rarely useful on systems with
inotify support.
* doc/coreutils.texi (tail invocation): Likewise.
2010-12-14 20:13:05 +01:00
Paul Eggert 14ad7a2550 sort: fix very-unlikely buffer overrun when merging to input file
* src/sort.c (avoid_trashing_input): Fix a typo that could cause a
buffer overrun in theory.  In practice this is extremely unlikely,
as it requires running out of file descriptors in a small merge,
presumably because some other process is hogging all the OS's file
descriptors.
2010-12-14 11:10:45 -08:00
Paul Eggert 8f40ed634c sort: document --compress reaper fixes
* NEWS: Document the --compress reaper fixes installed yesterday.
2010-12-14 10:22:04 -08:00
Paul Eggert dcb3f60d0a tests: default to /tmp as the temporary directory
* tests/check.mk (TESTS_ENVIRONMENT): Default TMPDIR to /tmp,
rather than to the working directory; this is more common in
practice, which makes the tests more real-worldish; and it is
often faster.  Also, it avoids some problems with NFS cleanups.
* tests/misc/sort-compress: Remove unnecessary code setting TMPDIR.
* tests/misc/sort-compress-proc: Likewise.  Do the final sleep
only if TMPDIR is relative, which should be rarely given the
change to TESTS_ENVIRONMENT.
2010-12-14 10:08:04 -08:00
Paul Eggert 0da4d84300 sort: fix some --compress reaper bugs
* src/sort.c (uintptr): New type.
(enum procstate, struct procnode, update_proc): Remove.
(proctab_hasher, proctab_comparator, register_proc, wait_proc):
(reap_some): The proctab is now simply a hash of process-IDs
rather than of pointers to objects with reference counts and
states; this is smaller and faster and easier to understand.
(nprocs): Now pid_t, not size_t, since one cannot have more than
PID_MAX children.
(reap): If the argument is -1, wait; if 0 (a new value), do not.
Delete pid from proctab as needed.  Ignore children that are not
in proctab, as they are from the program that exec'ed us and are
irrelevant to our success or failure.
(delete_proc, reap_all): New functions.
(open_temp): Register the child.
(sort): Clean up all children afterwards; without this patch,
'sort' sometimes missed failures in children due to race conditions.
* tests/Makefile.am (TESTS): Add misc/sort-compress-proc.
* tests/misc/sort-compress-proc: New file, to test for the
bugs fixed above.
2010-12-13 23:23:47 -08:00
Paul Eggert 6d36bd4c64 tests: typo fix
* tests/misc/sort-stale-thread-mem: Fix typo in comment.
2010-12-13 10:02:43 -08:00
Jim Meyering 0c70708db7 tests: tweak basic-1 to use warn_ rather than literal "exit 77"
* tests/install/basic-1 (just_built_dd): Use warn_, rather than
cat and exit 77.
2010-12-13 08:29:42 +01:00
Jim Meyering 8351407f87 tests: mark new test as very expensive
* tests/misc/sort-stale-thread-mem: Don't initialize fail=0 here;
that is done in init.sh.  This avoids a syntax-check failure.
Invoke "Exit $fail" at end, too.
Mark as a very expensive test.
2010-12-13 08:28:19 +01:00
Paul Eggert b212633c41 tests: test for access to stale thread memory
* tests/misc/sort-stale-thread-mem: New tests.
* tests/Makefile.am (TESTS): Add it.
2010-12-12 13:38:49 -08:00
Jim Meyering 45ff91af2e tests: avoid FP failure when run under valgrind
* tests/misc/printenv: Filter out LD_PRELOAD, as the comment
said, not LD_LIBRARY.
2010-12-11 18:44:57 +01:00
Jim Meyering ad61335bf8 sort: avoid segfault when using two or more threads
This change does not fix the actual bug.  That was done by commit
c9db0ac6, "sort: preallocate merge tree nodes to heap".  The fix
was to store each "node" structure on the heap, not on the stack.
Otherwise, a node from one thread's stack could be used in another
thread after the first thread had expired (via pthread_join).
This bug was very hard to trigger when using spinlocks, but
easier once we began using mutexes.
* NEWS (Bug fixes): Mention it.
For details, see http://debbugs.gnu.org/7597.
2010-12-11 11:52:24 +01:00
Jim Meyering 9a9d69e9e4 sort: syntax cleanup
* src/sort.c (xfopen, debug_key, sortlines, sort, main): Adjust
formatting: fix misplaced braces, use consistent spacing,
split a 2-stmt line.
2010-12-11 11:29:38 +01:00
Paul Eggert 27e997d0eb sort: integer overflow checks in thread counts, etc.
* src/sort.c (specify_nthreads, merge_tree_init, init_node):
(queue_init, sortlines, struct thread_args, sort, main):
Use size_t, not unsigned long int, for thread counts, since thread
counts are now used to compute sizes.
(specify_nthreads): Check for size_t overflow.
(merge_tree_init, sort): Shorten name of local variable, for
readability.
(merge_tree_init): Move constants next to each other in product,
so that the constant folding is easier to see.
(init_node): Now static.  Add 'restrict' only where it might
be helpful for compiler optimization.
(queue_init): 2nd arg is now nthreads, not "reserve", which is
a bit harder to follow.  All uses changed.
(struct thread_args): Rename lo_child to is_lo_child, so that
it's obvious to the reader when we're talking about this boolean
as opposed to the new lo_child member of the other structure.
All uses changed.
(sort): Remove unused local variable end_node.
(main): Don't allow large thread counts to cause undefined behavior
later, due to integer overflow.
2010-12-11 00:29:13 -08:00
Chen Guo c9db0ac6de sort: preallocate merge tree nodes to heap.
* src/sort.c: (merge_tree_init) New function. Allocates memory for
merge tree nodes.
(merge_tree_destory) New function.
(init_node) New function.
(sortlines) Refactor node creation code to init_node. Remove now
superfluous arguments. All callers changed.
(sort) Initialize/destory merge tree. Refactor root node creation
to merge_tree_init.
2010-12-11 00:29:13 -08:00
Paul Eggert d1f7003556 sort: comment fix
* src/sort.c: Comment fix re spin locks.
2010-12-11 00:29:13 -08:00
Chen Guo 621876ff44 sort: use mutexes, not spinlocks (avoid busy loop on blocked output)
Running a command like this on a multi-core system
  sort < big-file | less
would peg all processors at near 100% utilization.
* src/sort.c: (struct merge_node) Change member lock to mutex.
All uses changed.
* tests/Makefile.am (XFAIL_TESTS): Remove definition, now that
this test passes once again.  I.e., the sort-spinlock-abuse test
no longer fails.
* NEWS (Bug reports): Mention this.
Reported by DJ Lucas in http://debbugs.gnu.org/7489.
2010-12-11 00:29:13 -08:00
Pádraig Brady 195c455d0a split: fix a case where --elide-empty causes invalid chunking
When -n l/N is used and long lines are present that both
span partitions and multiple buffers, one would get
inconsistent chunk sizes.

* src/split.c (main): Add a new undocumented ---io-blksize option
to support full testing with varied buffer sizes.
(cwrite): Refactor most handling of --elide-empty to here.
(bytes_split): Remove handling of --elide-empty.
(lines_chunk_split): Likewise.  The specific issue here
was the first handling of elide_empty_files interfered
with the replenishing of the input buffer.
* test/misc/split-lchunk: Add -e and the new ---io-blksize
combinations to the test.
2010-12-08 23:36:17 +00:00
Jim Meyering a1629ba1ec tests: remove useless definition of $SORT in sort-compress
* tests/misc/sort-compress (SORT): Remove unused definition.
2010-12-05 17:07:35 +01:00
Jim Meyering 231d8719f1 tests: make it harder to lose a race in spinlock-abuse
* tests/misc/sort-spinlock-abuse: On a busy system, with only 12
pauses of length 0.1 seconds, the buggy (busy-spinlock blocked)
sort would fail to accumulate 1 second of CPU time, and hence
would mistakenly pass.  Increase from 12 to 50.
2010-12-04 11:52:21 +01:00
Paul Eggert cd00fa6ee1 sort: merge_queue -> queue
* src/sort.c (struct thread_args, sortlines_thread, sortlines, sort):
Rename "merge_queue" to "queue", for consistency with other functions
that just use the name "queue" for these things.
2010-12-03 23:43:31 -08:00
Paul Eggert fb282c7b30 sort: clarify queue_check_insert
* src/sort.c (queue_check_insert): Clarify body a bit, and remove
no-longer-needed comment.
2010-12-03 23:43:31 -08:00
Paul Eggert fb41e82c77 sort: fix problems with merge node dest pointer
* src/sort.c (mergelines_node): Return void, not size_t.  All
callers changed.  Change *node->dest here, not in caller.
Do not change node->dest: it's not needed and could cause problems
on (mostly theoretical) hosts that do not allow adding integers to
null pointers.
(queue_check_insert_parent): Omit MERGED parameter; no longer needed.
All callers changed.
2010-12-03 23:43:31 -08:00
Paul Eggert f2d977aff1 sort: simplify write_unique
* src/sort.c (write_unique): Simplify slightly so that there is
just one call to write_line, not two.
2010-12-03 23:43:31 -08:00
Paul Eggert 6f4279421d sort: put queue arg first
* src/sort.c (queue_check_insert, queue_check_insert_parent): Make
the queue arg first, for consistency with other functions such as
queue_insert that put the queue arg first.  Rename from
check_insert and update_parent, respectively.  All callers
changed.
2010-12-03 23:43:30 -08:00
Paul Eggert f35f4b3396 sort: tune struct_merge_node slightly
* src/sort.c (struct merge_node): 'lock' is now the actual lock,
not a pointer to the lock; there's no need for indirection here.
Make 'level' unsigned int instead of size_t, since it is a
bit-shift count; also, move it next to a bool so that it's more
likely to take less space.  All uses changed.
(sortlines, sort): Spell out initialization instead of using an
initializer.  This makes the initializer a bit easier to understand,
and avoids unnecessary stores into the spin lock.
2010-12-03 23:43:30 -08:00
Paul Eggert eb989f4b75 sort: Clarify comments
* src/sort.c: Improve comments a bit.
2010-12-03 23:43:30 -08:00
Paul Eggert b6ef652e50 tests: cleanup rm -rf fails under NFS
This problem was observed on RHEL 5.5 x86-64 when running as a
client of a NetApp FAS2050.
* tests/cp/cp-mv-backup: Don't leave a file descriptor open to
a file in a directory that will be cleaned up with "rm -rf".
Under NFS, when the rm unlinks that file, it is instead renamed
to .nfsXXXX and then rm cannot remove the parent directory,
and the test fails.
* tests/cp/same-file: Likewise.
2010-12-03 11:09:27 -08:00
Paul Eggert b5a25a996d cp: fix bug with fine-grained src to nearby coarse-grained dest
The actual fix is in gnulib's lib/utimecmp.c.
* NEWS: Document fix.
2010-12-03 09:56:32 -08:00
Paul Eggert 038fcfcf1b build: update gnulib submodule to latest 2010-12-03 09:53:02 -08:00
Paul Eggert 0ec869e8be sort: fix bug on 64-bit hosts with at least 32768 processors
* src/sort.c (MAX_MERGE): Avoid integer overflow when on a machine
with (say) 32-bit int and 64-bit size_t and when level == 15.
Without this fix, on such a machine with 32768 or more processors,
the level computation could overflow on large input, and this
would result in division by zero.
2010-12-01 21:51:15 -08:00
Jim Meyering a5207bb139 tests: add test for parallel sort -u segfault bug
* tests/misc/sort-unique-segv: New file.
* tests/Makefile.am (TESTS): Add it.
2010-12-01 08:15:42 +01:00
Paul Eggert 3afda5f007 sort -u: fix a thread-race pointer corruption bug
* src/sort.c (write_unique): Save the entire "struct line", not
just a pointer to one.  Otherwise, with a multi-thread run,
sometimes, with some inputs, fillbuf would would win a race
and clobber a "saved->text" pointer in one thread just before
it was dereferenced in a comparison in another thread.
* NEWS (Bug fixes): Mention it.
2010-12-01 07:13:06 +01:00
Jim Meyering 43d1112d01 tests: don't let the OMP_NUM_THREADS envvar affect sort tests
* tests/envvar-check (vars): Add OMP_NUM_THREADS.
2010-11-27 16:32:10 +01:00
Jim Meyering fbb47bf424 tests: test for parallel sort spinlock abuse
* tests/misc/sort-spinlock-abuse: New file.
* tests/Makefile.am (TESTS): Add it.
(XFAIL_TESTS): Mark this as an expected-to-fail (for now) test.
2010-11-27 16:32:06 +01:00
Pádraig Brady b8cd747cc8 tsort: suppress a valgrind memory leak warning
* src/tsort.c (tsort): Unconditionally invoking the free()
doesn't increase scalability, so do it only with -Dlint
2010-11-27 03:27:49 +00:00
Pádraig Brady 67f7ed3e3a maint: update the valgrind support script
* README-valgrind: Include the "noinst" programs in
those wrapped by valgrind.  Update $PATH in check.mk
rather than Makefile.am.  Make wrapper scripts
work when suppressions not setup.  Keep lines < 80 chars.
2010-11-24 13:39:47 +00:00
Paul Eggert 013d95af11 cp: give a better diagnostic for nonexistent dest/
This patch was written by Jim Meyering and myself.
* src/copy.c (copy_reg): Turn EISDIR to ENOTDIR to improve the
quality of diagnostics for commands like "cp a nosuch/".  Reported
by Марк Коренберг and Alan Curry in the thread starting at:
http://lists.gnu.org/archive/html/bug-coreutils/2010-11/msg00178.html
* THANKS: Update.
* tests/mv/trailing-slash: Add a test.
2010-11-22 00:04:50 -08:00
Chen Guo be107398e5 split: add --number to generate a particular number of files
* src/split.c (usage, long_options, main): New options --number,
--unbuffered, --elide-empty-files.
(set_suffix_length): New function to auto increase suffix length
to handle a specified number of files.
(create): New function.  Refactored from cwrite() and ofile_open().
(bytes_split): Add max_files argument to support byte chunking.
(lines_chunk_split): New function.  Split file into chunks of lines.
(bytes_chunk_extract): New function.  Extract a chunk of file.
(of_info): New struct.  Used by functions lines_rr and ofile_open
to keep track of file descriptors associated with output files.
(ofile_open): New function.  Shuffle file descriptors when there
are more output files than available file descriptors.
(lines_rr): New function to distribute lines round-robin to files.
(chunk_parse): New function.  Parses K/N syntax.
* tests/misc/split-bchunk: New test for byte chunking.
* tests/misc/split-lchunk: New test for line delimited chunking.
* tests/misc/split-rchunk: New test for round-robin chunking.
* tests/Makefile.am: Reference new tests.
* tests/misc/split-fail: Add failure scenarios for new options.
* tests/misc/split-l: Fix a typo. s/ln/split/.
* doc/coreutils.texi (split invocation): Document --number.
* NEWS: Mention the new feature.
* .mailmap: Map new email address for shortlog.

Signed-off-by: Pádraig Brady <P@draigBrady.com>
2010-11-22 01:45:15 +00:00
Jim Meyering dadca988af build: update gnulib to fix a syntax error in a test 2010-11-19 14:29:32 +01:00
Jim Meyering c2bdd17856 maint: avoid NEWS-related syntax-check failure
* NEWS: Add 2nd blank line to separate latest changes from
those of 8.7, to avoid syntax-check failure.
2010-11-18 21:16:27 +01:00
Paul Eggert a71c22fdf0 od: fix bugs in displaying floating-point values
* NEWS: Describe patch.
* bootstrap.conf (gnulib_modules): Add ftoastr.
* src/od.c: Include ftoastr.h, not float.h.
(FLT_DIG, DBL_DIG): Remove.  No need to verify LDBL_DIG.
(FMT_BYTES_ALLOCATED): No need to worry about floating point now,
since this format is no longer used for floating point.
(PRINT_FIELDS): New macro, with most of the guts of the old PRINT_TYPE.
(PRINT_TYPE): Rewrite to use PRINT_FIELDS.
(PRINT_FLOATTYPE): New macro.  This uses the new functions from
ftoastr.
(print_float, print_double, print_long_double): Reimplement
using PRINT_FLOATTYPE.
(decode_one_format): Calculate field widths based on ftoastr-supplied
macros.
* tests/Makefile.am (TESTS): Add misc/od-float.
* tests/misc/od-float: New file.
2010-11-18 09:51:04 -08:00
Paul Eggert cf90729e09 build: update gnulib submodule to latest 2010-11-18 09:33:49 -08:00
Jim Meyering c1b41c5c3b maint: syntax-check: prevent new $VERBOSE/--version tests
* cfg.mk (sc_prohibit_verbose_version): New rule.
2010-11-18 06:29:54 +01:00
Jim Meyering ae5cfc9f12 tests: convert tests/misc/selinux manually 2010-11-17 21:46:57 +01:00
Jim Meyering 8f9be76b0d tests: convert the multi-prog $VERBOSE/--version uses
E.g.,

-test "$VERBOSE" = yes && { env -- pwd --version; readlink --version; }
+print_ver_ pwd readlink

-test "$VERBOSE" = yes && { stdbuf --version; mv --version; }
+print_ver_ stdbuf mv

Use this command:
git grep -l 'VERBOSE.*--version'|xargs perl -ni \
  -e '/^test "\$VERBOSE" = yes && { .*--version/ or print,next;' \
  -e 's/env -- //g;' \
  -e 's/test "\$VERBOSE" = yes && { /print_ver_ /;' \
  -e ' s/(\w+) --version;/$1/g; s/ *}$//; print'
2010-11-17 21:46:25 +01:00
Jim Meyering a8c8484ee1 tests: convert "... env -- prog --version" uses
E.g.,

-test "$VERBOSE" = yes && env -- pwd --version
+print_ver_ pwd

git grep -l 'VERBOSE.*--version'|xargs perl -pi -e \
  's/test "\$VERBOSE" = yes && env -- (\w+) --version/print_ver_ $1/'
2010-11-17 21:46:22 +01:00
Jim Meyering 293c37b906 tests: substitute the single-program $VERBOSE/--version uses
Automatically make all of the changes like this:

  -test "$VERBOSE" = yes && chgrp --version
  +print_ver_ chgrp

git grep -l 'VERBOSE.*--version'|xargs perl -pi -e \
  's/test "\$VERBOSE" = yes && (\w+) --version/print_ver_ $1/'
2010-11-17 21:45:51 +01:00
Jim Meyering 5c0275e953 tests: factor out VERBOSE-only --version-printing code
* tests/init.cfg (print_ver_): New function.
2010-11-17 21:34:54 +01:00
Jim Meyering 11d3389f5a tests: remove test-lib.sh; now all tests use gnulib's init.sh
* tests/test-lib.sh: Remove file.  No longer used.
* tests/Makefile.am (EXTRA_DIST): Remove it here, too.
* tests/sample-test: Correct a comment.
2010-11-17 20:48:57 +01:00
Jim Meyering fbcc100f5b tests: convert remaining uses of test-lib.sh to init.sh
RHS='. "\${srcdir=.}/init.sh"; path_prepend_ ../src'
git grep -l test-lib.sh \
  | xargs perl -pi -e 's,^\. \$srcdir/test-lib\.sh$,'"$RHS",
2010-11-17 20:48:57 +01:00