Commit Graph
26498 Commits
Author SHA1 Message Date
Jim Meyering 45ff91af2e tests: avoid FP failure when run under valgrind
* tests/misc/printenv: Filter out LD_PRELOAD, as the comment
said, not LD_LIBRARY.
2010-12-11 18:44:57 +01:00
Jim Meyering ad61335bf8 sort: avoid segfault when using two or more threads
This change does not fix the actual bug.  That was done by commit
c9db0ac6, "sort: preallocate merge tree nodes to heap".  The fix
was to store each "node" structure on the heap, not on the stack.
Otherwise, a node from one thread's stack could be used in another
thread after the first thread had expired (via pthread_join).
This bug was very hard to trigger when using spinlocks, but
easier once we began using mutexes.
* NEWS (Bug fixes): Mention it.
For details, see http://debbugs.gnu.org/7597.
2010-12-11 11:52:24 +01:00
Jim Meyering 9a9d69e9e4 sort: syntax cleanup
* src/sort.c (xfopen, debug_key, sortlines, sort, main): Adjust
formatting: fix misplaced braces, use consistent spacing,
split a 2-stmt line.
2010-12-11 11:29:38 +01:00
Paul Eggert 27e997d0eb sort: integer overflow checks in thread counts, etc.
* src/sort.c (specify_nthreads, merge_tree_init, init_node):
(queue_init, sortlines, struct thread_args, sort, main):
Use size_t, not unsigned long int, for thread counts, since thread
counts are now used to compute sizes.
(specify_nthreads): Check for size_t overflow.
(merge_tree_init, sort): Shorten name of local variable, for
readability.
(merge_tree_init): Move constants next to each other in product,
so that the constant folding is easier to see.
(init_node): Now static.  Add 'restrict' only where it might
be helpful for compiler optimization.
(queue_init): 2nd arg is now nthreads, not "reserve", which is
a bit harder to follow.  All uses changed.
(struct thread_args): Rename lo_child to is_lo_child, so that
it's obvious to the reader when we're talking about this boolean
as opposed to the new lo_child member of the other structure.
All uses changed.
(sort): Remove unused local variable end_node.
(main): Don't allow large thread counts to cause undefined behavior
later, due to integer overflow.
2010-12-11 00:29:13 -08:00
Chen Guo c9db0ac6de sort: preallocate merge tree nodes to heap.
* src/sort.c: (merge_tree_init) New function. Allocates memory for
merge tree nodes.
(merge_tree_destory) New function.
(init_node) New function.
(sortlines) Refactor node creation code to init_node. Remove now
superfluous arguments. All callers changed.
(sort) Initialize/destory merge tree. Refactor root node creation
to merge_tree_init.
2010-12-11 00:29:13 -08:00
Paul Eggert d1f7003556 sort: comment fix
* src/sort.c: Comment fix re spin locks.
2010-12-11 00:29:13 -08:00
Chen Guo 621876ff44 sort: use mutexes, not spinlocks (avoid busy loop on blocked output)
Running a command like this on a multi-core system
  sort < big-file | less
would peg all processors at near 100% utilization.
* src/sort.c: (struct merge_node) Change member lock to mutex.
All uses changed.
* tests/Makefile.am (XFAIL_TESTS): Remove definition, now that
this test passes once again.  I.e., the sort-spinlock-abuse test
no longer fails.
* NEWS (Bug reports): Mention this.
Reported by DJ Lucas in http://debbugs.gnu.org/7489.
2010-12-11 00:29:13 -08:00
Pádraig Brady 195c455d0a split: fix a case where --elide-empty causes invalid chunking
When -n l/N is used and long lines are present that both
span partitions and multiple buffers, one would get
inconsistent chunk sizes.

* src/split.c (main): Add a new undocumented ---io-blksize option
to support full testing with varied buffer sizes.
(cwrite): Refactor most handling of --elide-empty to here.
(bytes_split): Remove handling of --elide-empty.
(lines_chunk_split): Likewise.  The specific issue here
was the first handling of elide_empty_files interfered
with the replenishing of the input buffer.
* test/misc/split-lchunk: Add -e and the new ---io-blksize
combinations to the test.
2010-12-08 23:36:17 +00:00
Jim Meyering a1629ba1ec tests: remove useless definition of $SORT in sort-compress
* tests/misc/sort-compress (SORT): Remove unused definition.
2010-12-05 17:07:35 +01:00
Jim Meyering 231d8719f1 tests: make it harder to lose a race in spinlock-abuse
* tests/misc/sort-spinlock-abuse: On a busy system, with only 12
pauses of length 0.1 seconds, the buggy (busy-spinlock blocked)
sort would fail to accumulate 1 second of CPU time, and hence
would mistakenly pass.  Increase from 12 to 50.
2010-12-04 11:52:21 +01:00
Paul Eggert cd00fa6ee1 sort: merge_queue -> queue
* src/sort.c (struct thread_args, sortlines_thread, sortlines, sort):
Rename "merge_queue" to "queue", for consistency with other functions
that just use the name "queue" for these things.
2010-12-03 23:43:31 -08:00
Paul Eggert fb282c7b30 sort: clarify queue_check_insert
* src/sort.c (queue_check_insert): Clarify body a bit, and remove
no-longer-needed comment.
2010-12-03 23:43:31 -08:00
Paul Eggert fb41e82c77 sort: fix problems with merge node dest pointer
* src/sort.c (mergelines_node): Return void, not size_t.  All
callers changed.  Change *node->dest here, not in caller.
Do not change node->dest: it's not needed and could cause problems
on (mostly theoretical) hosts that do not allow adding integers to
null pointers.
(queue_check_insert_parent): Omit MERGED parameter; no longer needed.
All callers changed.
2010-12-03 23:43:31 -08:00
Paul Eggert f2d977aff1 sort: simplify write_unique
* src/sort.c (write_unique): Simplify slightly so that there is
just one call to write_line, not two.
2010-12-03 23:43:31 -08:00
Paul Eggert 6f4279421d sort: put queue arg first
* src/sort.c (queue_check_insert, queue_check_insert_parent): Make
the queue arg first, for consistency with other functions such as
queue_insert that put the queue arg first.  Rename from
check_insert and update_parent, respectively.  All callers
changed.
2010-12-03 23:43:30 -08:00
Paul Eggert f35f4b3396 sort: tune struct_merge_node slightly
* src/sort.c (struct merge_node): 'lock' is now the actual lock,
not a pointer to the lock; there's no need for indirection here.
Make 'level' unsigned int instead of size_t, since it is a
bit-shift count; also, move it next to a bool so that it's more
likely to take less space.  All uses changed.
(sortlines, sort): Spell out initialization instead of using an
initializer.  This makes the initializer a bit easier to understand,
and avoids unnecessary stores into the spin lock.
2010-12-03 23:43:30 -08:00
Paul Eggert eb989f4b75 sort: Clarify comments
* src/sort.c: Improve comments a bit.
2010-12-03 23:43:30 -08:00
Paul Eggert b6ef652e50 tests: cleanup rm -rf fails under NFS
This problem was observed on RHEL 5.5 x86-64 when running as a
client of a NetApp FAS2050.
* tests/cp/cp-mv-backup: Don't leave a file descriptor open to
a file in a directory that will be cleaned up with "rm -rf".
Under NFS, when the rm unlinks that file, it is instead renamed
to .nfsXXXX and then rm cannot remove the parent directory,
and the test fails.
* tests/cp/same-file: Likewise.
2010-12-03 11:09:27 -08:00
Paul Eggert b5a25a996d cp: fix bug with fine-grained src to nearby coarse-grained dest
The actual fix is in gnulib's lib/utimecmp.c.
* NEWS: Document fix.
2010-12-03 09:56:32 -08:00
Paul Eggert 038fcfcf1b build: update gnulib submodule to latest 2010-12-03 09:53:02 -08:00
Paul Eggert 0ec869e8be sort: fix bug on 64-bit hosts with at least 32768 processors
* src/sort.c (MAX_MERGE): Avoid integer overflow when on a machine
with (say) 32-bit int and 64-bit size_t and when level == 15.
Without this fix, on such a machine with 32768 or more processors,
the level computation could overflow on large input, and this
would result in division by zero.
2010-12-01 21:51:15 -08:00
Jim Meyering a5207bb139 tests: add test for parallel sort -u segfault bug
* tests/misc/sort-unique-segv: New file.
* tests/Makefile.am (TESTS): Add it.
2010-12-01 08:15:42 +01:00
Paul Eggert 3afda5f007 sort -u: fix a thread-race pointer corruption bug
* src/sort.c (write_unique): Save the entire "struct line", not
just a pointer to one.  Otherwise, with a multi-thread run,
sometimes, with some inputs, fillbuf would would win a race
and clobber a "saved->text" pointer in one thread just before
it was dereferenced in a comparison in another thread.
* NEWS (Bug fixes): Mention it.
2010-12-01 07:13:06 +01:00
Jim Meyering 43d1112d01 tests: don't let the OMP_NUM_THREADS envvar affect sort tests
* tests/envvar-check (vars): Add OMP_NUM_THREADS.
2010-11-27 16:32:10 +01:00
Jim Meyering fbb47bf424 tests: test for parallel sort spinlock abuse
* tests/misc/sort-spinlock-abuse: New file.
* tests/Makefile.am (TESTS): Add it.
(XFAIL_TESTS): Mark this as an expected-to-fail (for now) test.
2010-11-27 16:32:06 +01:00
Pádraig Brady b8cd747cc8 tsort: suppress a valgrind memory leak warning
* src/tsort.c (tsort): Unconditionally invoking the free()
doesn't increase scalability, so do it only with -Dlint
2010-11-27 03:27:49 +00:00
Pádraig Brady 67f7ed3e3a maint: update the valgrind support script
* README-valgrind: Include the "noinst" programs in
those wrapped by valgrind.  Update $PATH in check.mk
rather than Makefile.am.  Make wrapper scripts
work when suppressions not setup.  Keep lines < 80 chars.
2010-11-24 13:39:47 +00:00
Paul Eggert 013d95af11 cp: give a better diagnostic for nonexistent dest/
This patch was written by Jim Meyering and myself.
* src/copy.c (copy_reg): Turn EISDIR to ENOTDIR to improve the
quality of diagnostics for commands like "cp a nosuch/".  Reported
by Марк Коренберг and Alan Curry in the thread starting at:
http://lists.gnu.org/archive/html/bug-coreutils/2010-11/msg00178.html
* THANKS: Update.
* tests/mv/trailing-slash: Add a test.
2010-11-22 00:04:50 -08:00
Chen Guo be107398e5 split: add --number to generate a particular number of files
* src/split.c (usage, long_options, main): New options --number,
--unbuffered, --elide-empty-files.
(set_suffix_length): New function to auto increase suffix length
to handle a specified number of files.
(create): New function.  Refactored from cwrite() and ofile_open().
(bytes_split): Add max_files argument to support byte chunking.
(lines_chunk_split): New function.  Split file into chunks of lines.
(bytes_chunk_extract): New function.  Extract a chunk of file.
(of_info): New struct.  Used by functions lines_rr and ofile_open
to keep track of file descriptors associated with output files.
(ofile_open): New function.  Shuffle file descriptors when there
are more output files than available file descriptors.
(lines_rr): New function to distribute lines round-robin to files.
(chunk_parse): New function.  Parses K/N syntax.
* tests/misc/split-bchunk: New test for byte chunking.
* tests/misc/split-lchunk: New test for line delimited chunking.
* tests/misc/split-rchunk: New test for round-robin chunking.
* tests/Makefile.am: Reference new tests.
* tests/misc/split-fail: Add failure scenarios for new options.
* tests/misc/split-l: Fix a typo. s/ln/split/.
* doc/coreutils.texi (split invocation): Document --number.
* NEWS: Mention the new feature.
* .mailmap: Map new email address for shortlog.

Signed-off-by: Pádraig Brady <P@draigBrady.com>
2010-11-22 01:45:15 +00:00
Jim Meyering dadca988af build: update gnulib to fix a syntax error in a test 2010-11-19 14:29:32 +01:00
Jim Meyering c2bdd17856 maint: avoid NEWS-related syntax-check failure
* NEWS: Add 2nd blank line to separate latest changes from
those of 8.7, to avoid syntax-check failure.
2010-11-18 21:16:27 +01:00
Paul Eggert a71c22fdf0 od: fix bugs in displaying floating-point values
* NEWS: Describe patch.
* bootstrap.conf (gnulib_modules): Add ftoastr.
* src/od.c: Include ftoastr.h, not float.h.
(FLT_DIG, DBL_DIG): Remove.  No need to verify LDBL_DIG.
(FMT_BYTES_ALLOCATED): No need to worry about floating point now,
since this format is no longer used for floating point.
(PRINT_FIELDS): New macro, with most of the guts of the old PRINT_TYPE.
(PRINT_TYPE): Rewrite to use PRINT_FIELDS.
(PRINT_FLOATTYPE): New macro.  This uses the new functions from
ftoastr.
(print_float, print_double, print_long_double): Reimplement
using PRINT_FLOATTYPE.
(decode_one_format): Calculate field widths based on ftoastr-supplied
macros.
* tests/Makefile.am (TESTS): Add misc/od-float.
* tests/misc/od-float: New file.
2010-11-18 09:51:04 -08:00
Paul Eggert cf90729e09 build: update gnulib submodule to latest 2010-11-18 09:33:49 -08:00
Jim Meyering c1b41c5c3b maint: syntax-check: prevent new $VERBOSE/--version tests
* cfg.mk (sc_prohibit_verbose_version): New rule.
2010-11-18 06:29:54 +01:00
Jim Meyering ae5cfc9f12 tests: convert tests/misc/selinux manually 2010-11-17 21:46:57 +01:00
Jim Meyering 8f9be76b0d tests: convert the multi-prog $VERBOSE/--version uses
E.g.,

-test "$VERBOSE" = yes && { env -- pwd --version; readlink --version; }
+print_ver_ pwd readlink

-test "$VERBOSE" = yes && { stdbuf --version; mv --version; }
+print_ver_ stdbuf mv

Use this command:
git grep -l 'VERBOSE.*--version'|xargs perl -ni \
  -e '/^test "\$VERBOSE" = yes && { .*--version/ or print,next;' \
  -e 's/env -- //g;' \
  -e 's/test "\$VERBOSE" = yes && { /print_ver_ /;' \
  -e ' s/(\w+) --version;/$1/g; s/ *}$//; print'
2010-11-17 21:46:25 +01:00
Jim Meyering a8c8484ee1 tests: convert "... env -- prog --version" uses
E.g.,

-test "$VERBOSE" = yes && env -- pwd --version
+print_ver_ pwd

git grep -l 'VERBOSE.*--version'|xargs perl -pi -e \
  's/test "\$VERBOSE" = yes && env -- (\w+) --version/print_ver_ $1/'
2010-11-17 21:46:22 +01:00
Jim Meyering 293c37b906 tests: substitute the single-program $VERBOSE/--version uses
Automatically make all of the changes like this:

  -test "$VERBOSE" = yes && chgrp --version
  +print_ver_ chgrp

git grep -l 'VERBOSE.*--version'|xargs perl -pi -e \
  's/test "\$VERBOSE" = yes && (\w+) --version/print_ver_ $1/'
2010-11-17 21:45:51 +01:00
Jim Meyering 5c0275e953 tests: factor out VERBOSE-only --version-printing code
* tests/init.cfg (print_ver_): New function.
2010-11-17 21:34:54 +01:00
Jim Meyering 11d3389f5a tests: remove test-lib.sh; now all tests use gnulib's init.sh
* tests/test-lib.sh: Remove file.  No longer used.
* tests/Makefile.am (EXTRA_DIST): Remove it here, too.
* tests/sample-test: Correct a comment.
2010-11-17 20:48:57 +01:00
Jim Meyering fbcc100f5b tests: convert remaining uses of test-lib.sh to init.sh
RHS='. "\${srcdir=.}/init.sh"; path_prepend_ ../src'
git grep -l test-lib.sh \
  | xargs perl -pi -e 's,^\. \$srcdir/test-lib\.sh$,'"$RHS",
2010-11-17 20:48:57 +01:00
Jim Meyering ff7f0ff85c tests: convert 'if test "$VERBOSE" = yes; then' to test ... && 2010-11-17 20:48:57 +01:00
Jim Meyering f38becab02 tests: convert first batch of tests from test-lib.sh to init.sh
Initially, I did this,
git grep -l srcdir/test-lib.sh|xargs perl -p0i -e '~180-byte script'
but that line would have been much longer than the maximum permitted
by coreutils' commit hook, and wasn't readable besides, so here's a
more readable version:

lhs=$(printf '%s\\n' \
'if test "$VERBOSE" = yes; then' \
'  set -x' \
'  touch --version' \
'fi' \
'' \
'. $srcdir/test-lib.sh' \
  | sed 's/\$/\\\$/g;s/touch/(\\w+)/')

rhs=$(printf '%s\\n' \
'. "${srcdir=.}/init.sh"; path_prepend_ ../src' \
'test "$VERBOSE" = yes && FIXME --version' \
  | sed 's/\$/\\\$/g;s/FIXME/\$1/')

git grep -l srcdir/test-lib.sh|xargs perl -p0i -e "s,$lhs,$rhs,"
2010-11-17 20:48:57 +01:00
Jim Meyering 30b74dfff1 tests: move a comment so automated conversion works 2010-11-17 20:48:57 +01:00
Jim Meyering 01a27b9054 maint: add a NEWS-checking syntax-check rule
* cfg.mk (sc_NEWS_two_empty_lines): New syntax-check rule.
2010-11-17 20:48:56 +01:00
Pádraig Brady 79adacc431 build: add patch as a bootstrap dependency
* bootstrap.conf (buildreq): require `patch` as it's used
by gnulib-tool to apply local diffs to gnulib modules
2010-11-16 07:39:06 +00:00
Pádraig Brady 9ad13ad14a split: fail immediately if impossible to create a large file
* src/split.c (main): Error if -[bC] value > OFF_T_MAX
* tests/misc/split-fail: Adjust for the new lower limits
2010-11-16 00:50:09 +00:00
Pádraig Brady c8d570c2b3 truncate: fix a very unlikely case for undiagnosed errors
src/truncate.c (main): Use a bool to store if an error occurred,
rather than an int, to protect against overflow.
(do_ftruncate): Likewise.   Also change 0/false to mean failure
rather than success.
2010-11-16 00:47:36 +00:00
Pádraig Brady 0d1ba34494 maint: fix a new -Wpointer-sign gcc warning
* src/csplit.c (max_out): Fix a new warning introduced with
commit 6568b173, 2010-11-10, "csplit: do not rely on..."
2010-11-16 00:38:12 +00:00
Pádraig Brady f9cfda6ad2 build: enable the -Wpointer-sign gcc warning
... when configured with the --enable-gcc-warnings option.
This follows on from commit 34ef0a01, 2010-10-14,
"sort: fix unportable cast of unsigned char * -> char *"

* configure.ac: -Wall implicitly enables this warning
so remove the explicit disabling.
2010-11-16 00:05:41 +00:00