* src/sort.c (avoid_trashing_input): Fix an off-by-one error and
guard the use of memmove.
* NEWS (Bug fixes): Mention it.
* tests/misc/sort: Add tests to exercise the offending code.
* THANKS: Update.
Reported by Otavio Salvador in http://bugs.debian.org/525048.
Suggestion from Steven Schveighoffer at:
http://savannah.gnu.org/patch/?6797
to greatly speed up the random passes done by shred.
* gl/lib/randread.c: Default to using the internal
pseudorandom generator, rather than reading /dev/urandom
* src/shred.c (usage): remove mention of /dev/urandom
* src/shuf.c (usage); ditto
* src/sort.c (usage): ditto
* doc/coreutils.text: Document the new behaviour
for aquiring random data.
This is an alternative to my 9 March patch labeled "Silently lower
nmerge; don't (sometimes incorrectly) range-check"
<http://lists.gnu.org/archive/html/bug-coreutils/2009-03/msg00070.html>.
It differs by not using 'dup' to probe for extra file descriptors;
instead, it simply calls 'open' (and 'pipe') to open files and pipes,
until one of these calls fails due to file descriptor exhaustion; it
then backs off by 1, does a merge with the files that it has opened,
and then retries with the (now-smaller) number of files.
This patch requires quite a few more changes to the source code than
the earlier patch, but it is in some sense "better" because it doesn't
need to call "dup" ahead of time in order to decide whether "open" or
"pipe" will fail. Also, it's more robust in the case where "open" or
"pipe" fails with errno==EMFILE because some system-wide limit is
exhausted.
* src/sort.c (create_temp_file): New arg SURVIVE_FD_EXHAUSTION.
(stream_open): New function, containing guts of xfopen.
(xfopen): Use it.
(pipe_fork): Set errno on failure.
(maybe_create_temp): New function, containing guts of create_temp.
(create_temp): Use it.
(open_temp): Distinguish failures due to file descriptor exhaustion
from other failures, and on fd exhaustion return a notice to caller
rather than dying. Don't test execlp's return value; when it returns,
it *always* returns -1.
(open_input_files): New function.
(mergefps): New arg FPS. It's now the caller's responsibility to open
the input and output files. All callers changed.
(mergefiles): New function.
(avoid_trashing_input, merge): Handle the case where a single merge
can't merge as much as we wanted due to file descriptor exhaustion, by
merging as much as we can and then retrying.
* tests/Makefile.am (TESTS): Add misc/sort-continue.
* tests/misc/sort-continue: New file.
* THANKS: Add Glen Lenker and Matt Pham who coauthored this patch.
* src/sort.c: When no specific number of chars to skip
is specified for the end field, always skip the whole field.
Also never include leading spaces from next field.
* tests/misc/sort: Add 2 new tests for these cases.
* NEWS: Mention this bug fix.
* THANKS: Add bug reporter.
Reported by Davide Canova.
* doc/coreutils.texi: Describe the most common
usage of --files0-from=- to read names from stdin.
* src/du.c: Likewise.
* src/sort.c: Likewise.
* src/wc.c: Likewise.
for all but the C/POSIX locale.
* src/system.h: Include "hard-locale.h".
(emit_bug_reporting_address): Tell where to report translation bugs.
don't include "hard-locale.h", now that system.h does
* src/comm.c: Likewise.
* src/join.c: Likewise.
* src/ls.c: Likewise.
* src/pinky.c: Likewise.
* src/pr.c: Likewise.
* src/sort.c: Likewise.
* src/uniq.c: Likewise.
* src/who.c: Likewise.
* src/ls.c (cmp_version): Use filevercmp instead of strverscmp.
* src/sort.c (usage): Remove mna reference to strverscmp(3).
(compare_version): Use filevercmp instead of strverscmp.
* bootstrap.conf: Add filevercmp to list of gnulib modules.
* tests/misc/sort-version: Remove conflicting string and enhance test.
* NEWS: Mention the change.
* src/sort.c (usage): Mention that -k defaults to end of line if
POS2 omitted.
* THANKS: Update.
Reported by Tim Ryan.
Signed-off-by: Eric Blake <ebb9@byu.net>
* src/sort.c [struct keyfield] (version): New member.
(usage): Describe --version-sort.
(sort_options): Add 'V'.
(long_options): Add "version-sort".
(CHECK_TABLE, _ct_, SORT_TABLE, _st_): Define new macros.
(check_args, sort_args, sort_types): Use these new macros in declarations.
(ARGMATCH_VERIFY): Remove use. No longer needed.
(compare_version): New function.
(key_compare): Add a case.
(check_ordering_compatibility): Handle new type.
(main): Likewise. Reformat two expressions for readability.
* tests/misc/sort-version: new test file
* tests/Makefile.am: add it to the list
* doc/coreutils.texi (sort invocation): Document it.
* NEWS: Mention the new feature.
When mkstemp fails, the template buffer may have undefined
contents, so we must not print it.
* src/sort.c (create_temp_file): Use temp_dir, not "file"
when diagnosing failed mkstemp, because "file" may be undefined.
* tests/misc/sort-merge: Adjust for new expected output.
Jeph Cowan and Ralf Wildenhues reported the test failure:
http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/14235/focus=14257
* src/tac.c (copy_to_temp): Don't use template buffer after
failed mkstemp call, since its contents may be undefined.
* tests/misc/tac (pipe-bad-tmpdir): New test for the above.
* src/mktemp.c (main): Save a copy of the template string,
solely for use in case mkstemp fails.
* tests/misc/mktemp (pipe-bad-tmpdir): New test for the above.
* src/sort.c (OPEN_MAX): Define if not already defined.
(MAX_NMERGE): Remove definition.
(specify_nmerge): Don't cast MAX_NMERGE (of type size_t) to unsigned int.
Instead, use OPEN_MAX as the fall-back value.
* src/sort.c (specify_nmerge): Do use uinttostr value.
Provoke with e.g., sort -m --batch-size=18446744073709551617
Omit quotes around known-numeric value in diagnostic.
* tests/misc/sort-merge [nmerge-big]: Tighten ERR_SUBST regexp
to require a numeric value in that diagnostic, so this particular
failure cannot reappear.
* src/sort.c: (static unsigned int nmerge) Replace constant NMERGE.
(specify_nmerge) Validate and apply new option.
(mergefps) Replace some arrays with pointers to xnmalloc'd storage.
* tests/misc/sort-merge: Test new option.
* doc/coreutils.texi: Describe new option.
* NEWS: Advertise new option.
* src/sort.c: Support new option.
* tests/misc/sort-files0-from: Test new option.
* tests/misc/Makefile.am: Indicate new test.
* docs/coreutils.texi: Explain new option.
* NEWS: Advertise new option.
Signed-off-by: Bo Borgerson <gigabo@gmail.com>
* src/sort.c (fillbuf): When enlarging the line buffer, ensure that
the new size is a multiple of "sizeof (struct line)". This avoids
alignment problems when indexing from the end of the buffer.
Problem reported by Andreas Schwab in
<http://lists.gnu.org/archive/html/bug-coreutils/2007-07/msg00158.html>.
* bootstrap.conf: Remove findprog.
* doc/coreutils.texi (sort invocation): The default is to not
compress. Don't treat "" specially.
* src/sort.c: Don't include findprog.h.
(create_temp): Compress only if the user specified --compress-program.
* tests/misc/sort-compress: Adjusts tests to match new behavior.
an environment variable.
* doc/coreutils.texi (sort invocation): Document this.
* src/sort.c (usage): Likewise.
(COMPRESS_PROGRAM_OPTION): New const.
(long_options, create_temp, main): Support new option.
* tests/misc/sort-compress: Test it.
* src/sort.c (usage): Split a diagnostic that had grown to be
longer than the C89 maximum of 509 bytes.
* .x-sc_cast_of_argument_to_free: New file. Allow a cast in sort.c.
FIXME: this is just temporary, while we wait to remove the offending
access-calling code.
* Makefile.am (EXTRA_DIST): Add .x-sc_cast_of_argument_to_free.
* Makefile.maint (sc_cast_of_argument_to_free): Use the
canonical, $$($(CVS_LIST_EXCEPT)).
* m4/.gitignore, m4/.cvsignore, lib/.gitignore, lib/.cvsignore: Update.
like it will be approved. Also add --check=quiet, --check=silent
as long aliases, and --check=diagnose-first as an alias for -c.
* doc/coreutils.texi (sort invocation): Document this.
Also, mention that sort -c can take at most one file.
* src/sort.c: Implement this.
Include argmatch.h.
(usage): Document the change.
(CHECK_OPTION): New constant.
(long_options): --check now takes an optional argument, and is now
treated differently from 'c'.
(check_args, check_types): New constant arrays.
(check): New arg CHECKONLY, which suppresses diagnostic if -C.
(main): Parse the new options.
* tests/sort/Test.pm (02d, 02d, incompat5, incompat6):
New tests for -C.
In pipe_fork callers, use these named constants, not "2" and "8".
(proctab, nprocs): Declare to be "static".
(pipe_fork) [lint]: Initialize local, pid,
to avoid unwarranted may-be-used-uninitialized warning.
(create_temp): Use the active voice. Describe parameters, too.
2007-01-21 James Youngman <jay@gnu.org>
Centralize all the uses of sigprocmask(). Don't restore an invalid
saved mask.
* src/sort.c (enter_cs, leave_cs): New functions for protecting
code sequences against signal delivery.
* (exit_cleanup): Use enter_cs and leave_cs instead of
calling sigprocmask directly.
(create_temp_file, pipe_fork, zaptemp): Likewise
2007-01-21 Dan Hipschman <dsh@linux.ucla.edu>
Add compression of temp files to sort.
* NEWS: Mention this.
* bootstrap.conf: Import findprog.
* configure.ac: Add AC_FUNC_FORK.
* doc/coreutils.texi: Document GNUSORT_COMPRESSOR environment
variable.
* src/sort.c (compress_program): New global, holds the name of the
external compression program.
(struct sortfile): New type used by mergepfs and friends instead
of filenames to hold PIDs of compressor processes.
(proctab): New global, holds compressor PIDs on which to wait.
(enum procstate, struct procnode): New types used by proctab.
(proctab_hasher, proctab_comparator): New functions for proctab.
(nprocs): New global, number of forked but unreaped children.
(reap, reap_some): New function, wait for/cleanup forked processes.
(register_proc, update_proc, wait_proc): New functions for adding,
modifying and removing proctab entries.
(create_temp_file): Change parameter type to pointer to file
descriptor, and return type to pointer to struct tempnode.
(dup2_or_die): New function used in create_temp and open_temp.
(pipe_fork): New function, creates a pipe and child process.
(create_temp): Creates a temp file and possibly a compression
program to which we filter output.
(open_temp): Opens a compressed temp file and creates a
decompression process through which to filter the input.
(mergefps): Change FILES parameter type to struct sortfile array
and update access accordingly. Use open_temp and reap_some.
(avoid_trashing_input, merge): Change FILES parameter like
mergefps and call create_temp instead of create_temp_file.
(sort): Call create_temp instead of create_temp_file.
Use reap_some.
(avoid_trashing_input, merge, sort, main): Adapt to mergefps.
* src/csplit.c (main): Also catch SIGALRM, SIGPIPE, SIGPOLL,
SIGPROF, SIGVTALRM, SIGXCPU, SIGXFSZ.
* src/ls.c (main): Likewise (except SIGPIPE was already caught).
Note that ls.c is special, as it also catches SIGTSTP.
* src/sort.c (main): Likewise. Also catch SIGQUIT. More details in
<http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/9510>.
(exit_cleanup): New function.
(main): Don't invoke atexit until we're ready.
Invoke it with exit_cleanup, not with cleanup and close_stdout,
to avoid a race condition with cleanup and signal handling. More
details: http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/9508
so that commands like "sort -k 18446744073709551616" no longer fail merely
because 18446744073709551616 doesn't fit in uintmax_t. The trick is that
these fields can all be treated as effectively infinity; their exact
values don't matter, since no internal buffer can be that long.
* src/join.c (string_to_join_field): Verify that SIZE_MAX <= ULONG_MAX
if the code assumes this. Silently truncate too-large values to SIZE_MAX,
as the remaining code will do the right thing in this case.
* src/sort.c (parse_field_count): Likewise.
* src/uniq.c (size_opt, main): Likewise.
* tests/join/Test.pm (bigfield): New test.
* tests/sort/Test.pm (bigfield): New test.
* tests/uniq/Test.pm (121): New test.
Signed-off-by: Jim Meyering <jim@meyering.net>
* src/sort.c (main): Don't allocate memory for each new key here.
(insertkey): Allocate memory for each key here, instead.
(key_init): Rename from new_key. Don't allocate.