Commit Graph
211 Commits
Author SHA1 Message Date
Jim Meyering 593d17ffe4 (merge): Move declarations of local variables into
the inner scope where they're used.
(sort): Likewise.
2001-03-13 08:34:38 +00:00
Jim Meyering 86c5a50e86 Do not include closeout.h.
(xfopen): Use stdout if *how != 'r'.
(mergefps): Remove FPS arg.
Open all input files, and close all files when done.
If OFP is null, open the output file (but after opening input files).
All callers changed.
(first_same_file): New function.
(sort, merge): Remove arg OFP; we now open the output file as needed.
All callers changed.
(merge): New arg MAX_MERGE.  All callers changed.
(sort):	For "sort F -o F", close the input before opening the output.
(main): Do not use close_stdout; 'sort' and 'merge' now close stdout.
(This also fixes a close-stdout-twice bug.)
Remove test for overlapping input and output files, as 'sort' no longer
needs to worry about overlap, and 'merge' checks for overlap itself.
Use first_same_file to inform 'merge' about how much to merge at
the top level, to avoid overlap.
2001-03-13 08:28:09 +00:00
Jim Meyering 6d99caa5ee (die): New function.
(create_temp_file, xfopen, xfclose, write_bytes, sort_buffer_size,
fillbuf, main): Use it to regularize error messages.  The only change
in behavior is that write_bytes and the final close used to say "write
error" but now give just the output file name, which should be enough.
2001-03-03 19:31:37 +00:00
Jim Meyering 410e28bd78 (xfclose): Add FILE arg, and report the file name
on error.  All callers changed.
2001-03-03 19:21:41 +00:00
Jim Meyering 91b9ff7935 (main): When fclose (stdin) fails, do not mention
the output file in the error message; mention "-" instead.
2001-03-03 19:19:06 +00:00
Jim Meyering 5d34d8c327 (xfopen): Set have_read_stdin to 1 only if file is "-".
Use fopen_safer, not fopen, to avoid subtle bugs when fopen returns
stdin, stdout, or stderr.
(xfclose): stdout is no longer a special case.
(main): Close output file, don't just flush it; there might be
an error on the close.
2001-03-03 19:12:23 +00:00
Jim Meyering cac9a29ad8 (initbuf): If the desired size cannot be
allocated, repeatedly halve it until allocation succeeds.
2001-03-03 18:53:44 +00:00
Jim Meyering f10bbe70a9 (parse_field_count): Comment fix. 2001-03-03 18:40:44 +00:00
Jim Meyering 25f95ca91e Tune allocation and comparison of nodes
representing temp files.  This improved CPU performance of
'sort -S 1 *.[ch]' by 17% on my host.

(struct tempnode): name member now uses struct hack.
(temphead): Now a pointer, not a structure.  All uses changed.
(create_temp_file): Allocate node using struct hack.
(zaptemp): Free node using struct hack.  Use pointer comparison, not
string comparison.
2001-03-03 18:23:29 +00:00
Jim Meyering cfd53be2c8 (eolchar, trim_trailing_blanks): Now static. 2001-03-02 03:44:49 +00:00
Jim Meyering 2679dfb639 'sort' race condition fixes.
Defend against a DoS attack where someone else creates a
temporary file with the same name as ours.  Use mkstemp to do
this, supplying our own mkstemp if the system doesn't have one.

Also, fix a race condition during cleanup on hosts without
sigaction.


(NAME_MAX_IN_DIR): Remove.
(sigprocmask, sigset_t): New macros, defined only on older hosts.
(caught_signals): New var.
(xtmpfopen, tempname): Removed.
(create_temp_file): New function, combining the functions of the old
xtmpfopen and tempname.  All callers changed.
Use mkstemp to create the file.
(sighandler): On hosts without sigaction, ignore signals while
cleaning up, instead of letting them interrupt cleanup.
(main): Initialize caught_signals.  On hosts with sigaction, block all
caught signals while handling one.  Remove duplicate code.
2001-03-02 02:58:23 +00:00
Jim Meyering 5f001b35ed (keycompare): Move declarations of locals, lena and lenb,
into the inner scope where they are used.
2001-02-22 15:11:08 +00:00
Jim Meyering d4142ec88e (AUTHORS): Add Paul Eggert. 2001-02-19 08:53:24 +00:00
Jim Meyering 5af9b0048f Fix a race condition: freed storage accessed during a signal handler.
(struct tempnode.next): Now volatile.
(zaptemp): Free the file name after removing it from the temp list,
not before, because a signal can arrive between the two actions
and cleanup () traverses the list.
2001-02-19 08:52:53 +00:00
Jim Meyering 55d155dbbe Check for input size, and do not overallocate memory.
Also check for memory quotas.

Revamp storage management so that line tables and character data are
taken from the same buffer.  Line tables are now in reverse order,
since they grow down while the character data grow up.

(<sys/resource.h>): Include if HAVE_SYS_RESOURCE_H.
(struct rlimit, getrlimit): Define a replacement if RLIMIT_DATA
is not defined.
(RLIMIT_AS): Define to RLIMIT_DATA if not defined.
(struct lines): Remove.
(struct buffer): New members nlines, line_bytes, eof.
Remove member newline_free; no longer needed, since the code no longer
runs out of line table space.
(SORTALLOC_MIN, SORTALLOC_DEFAULT_MIN): Remove.
(sort_size): Renamed from sortalloc; now applies to the sum of the
character data and the line table, not just the character data.
(MIN_SORT_SIZE, INPUT_FILE_SIZE_GUESS): New macros.
(linelength): remove.
(specify_sort_size): Don't worry about the distinction between the
character data and the line table; that is now the caller's
responsibility.
(default_sort_size): Return the value, instead of being executed for
side effect.  Return half of available memory, or 1/16 of total memory,
whichever is greater; except do not exceed 1/2 of quota.
(sort_buffer_size): New function.
(initbuf): New arg LINE_BYTES.  Ensure that the line array is properly
aligned.  Initialize the new set of struct buffer members.
(buffer_linelim): New function.
(fillbuf): Return int, not size_t, since the callers merely care
whether the result is nonzero.  New arg FILE so that error messages
can report the file name.  Keep track of eof.  Initialize the line
table too, taking its memory from the input buffer's memory; this
subsumes the old findlines function and removes the need for worrying
about running out of line table entries.
(checkfp, mergefps, sortlines, merge, sort): Adjust to the new storage
management regime, in particular the fact that line tables are now
filled in by fillbuf and are in reverse order.
(checkfp): Now takes char *, not const char *, since subroutines
require that now.  Rewrite to avoid lint and duplicate code.
If line length alloc calculation overflows,
simply allocate enough memory to hold the line.
(mergefps): New arg FILES, used for buffer size calculation and error
messages.  Rewrite to avoid lint.  Do not loop if savealloc*2
overflows.
(mergefps, merge): Zap temporary files eagerly rather than lazily;
this is needed because we now pass FILES to mergefps.
(sortlines): Args now point at end of arrays, not at beginnings.
(sort): Do not allocate temporary line array for sortlines;
instead, take the space from the same buffer.
(main): Adjust to sort_size and default_sort_size changes.
2001-02-19 08:50:12 +00:00
Jim Meyering 56c1cbf2e8 (main): Use PACKAGE, not GNU_PACKAGE. 2001-01-17 09:47:41 +00:00
Jim Meyering bd485671ff (usage): Convert each TAB in --help output to a sequence of 8 spaces. 2001-01-17 09:47:32 +00:00
Jim Meyering a0439c2405 (main): Remove embedded \n from diagnostic. 2001-01-03 11:36:10 +00:00
Jim Meyering 3c46adfe19 adjust indentation 2000-12-19 09:21:42 +00:00
Jim Meyering bf86c62a33 Include physmem.h.
(SORTALLOC, mergealloc, LINEALLOC): Remove.
(sortalloc): Default to zero at program startup.
(SORTALLOC_MIN, SORTALLOC_DEFAULT_MIN): New macros.
(usage, main): Add support for new -S SIZE option.
(specify_sort_size, default_sort_size): New functions.
(initlines): Do not let alloc exceed limit.
(findlines): Likewise.
(checkfp, mergefps, sort): Use sortalloc to size everything
else, instead of relying on precomputed sizes.
2000-12-19 09:16:28 +00:00
Jim Meyering 8973e2b38c whoops. rename the use, too -- not just the decl!! 2000-12-02 21:14:41 +00:00
Jim Meyering 5edc91fc05 (checkfp): Rename local `buf' to avoid shadowing previous declaration. 2000-12-02 21:14:02 +00:00
Jim Meyering 1c71127370 (NONZERO): Define and use it to make the code a tiny
bit more readable.
2000-12-02 14:43:28 +00:00
Jim Meyering 339738f351 s/SIZE_T_MAX/SIZE_MAX/. 2000-11-30 20:30:04 +00:00
Jim Meyering d92f4ac85e Port GNU "sort" to hosts where sizes don't fit in "int",
e.g. 64-bit Solaris (sparc).

("human.h", "xstrtol.h"): Include.
(struct line): length member is now size_t, not int.
(struct lines): Likewise for used, alloc, limit members.
(struct buffer): Likewise for used, alloc, left, newline_free members.
(struct keyfield): Likewise for sword, schar, eword, echar members.
(sortalloc, mergealloc, linelength): Now size_t, not int.

(initbuf, fillbuf, initlines, begfield, limfield, findlines,
numcompare, getmonth, keycompare, compare, checkfp, mergefps,
sortlines, sort): Accept, return, and use size_t for sizes, not int.

(fillbuf, initlines, findlines, checkfp, sort): Check for overflow
when computing buffer sizes.

(begfield, limfield): Do not index past end of array.

(checkfp): Return a boolean, not a line number, as the line
number may not fit in int.  All callers changed.  Use
uintmax_t for line numbers, not int.

(sort): Don't allocate tmp until we need it (and know the right size).

(parse_field_count): New function.

(main): Use it to check for overflow in field counts.
"outfile" is now a pointer to const.
2000-11-30 11:33:49 +00:00
Jim Meyering b84871a279 add missing backslash 2000-10-22 07:20:24 +00:00
Jim Meyering fb5e934c73 (SORT_OUT_OF_ORDER): Define.
(main): Use it instead of hard-coding the `1'.
2000-10-21 13:05:40 +00:00
Jim Meyering 09ad98374b (main): Use EXIT_SUCCESS rather than 0.
Fail when checking (-c) with more than one file argument,
rather than simply ignoring the extra arguments.
2000-10-21 13:03:40 +00:00
Jim Meyering 5018ccfb7e (usage): Describe -d and -i in a locale-independent way. 2000-08-11 09:12:47 +00:00
Jim Meyering 43fde4395a (usage): Warn more succintly about the effects of
the locale on sort order.
2000-08-07 07:07:00 +00:00
Jim Meyering 773a575cd5 (main): Rename local t' to tmp_dir' to avoid shadowing
a previous local by that name.
(usage): Warn that GNU sort is now locale-aware, and suggest
people put LC_ALL=POSIX in their environment.
2000-08-05 20:27:49 +00:00
Jim Meyering d0f57e1004 (temp_dir): Remove.
(temp_dirs, temp_dir_count, temp_dir_alloc): New vars.
(process_id): New var.
(usage): Describe new use of -T.
(add_temp_dir): New function.
(tempname): Use new temp_dirs array.
Do not discard information from the process-id or sequence
number, unless we have short file names.
(sighandle): Use process_id instead of getpid.
(main): Initialize process_id.
Add support for the new use of -T.
2000-07-29 05:41:55 +00:00
Jim Meyering 02f7878928 Arrange to call close_stdout upon exit. Don't close stdout explicitly.
(but set exit status and file name, too)
2000-05-20 22:03:12 +00:00
Jim Meyering 57bb2e656e (struct buffer.newline_free): New member.
(initbuf, findlines): Set it.
(fillbuf): Do not double the size of a full buffer to append a newline
unless the buffer is known to be newline free.
2000-03-06 06:20:59 +00:00
Jim Meyering 2c43f8c612 (fillbuf): Move declaration of local, cc, into scope of
`while' loop where it's used.
2000-03-03 08:24:16 +00:00
Jim Meyering 8e1e6d357a Big performance improvement when sorting many small files,
building on a suggestion by Charles Randall.

(fillbuf): Skip memmove if it would be a no-op,
as many memmove implementations are slow in that case.
Don't examine leftover bytes for eolchar, since they may be left
over from a previous file, and we want to read from this file.

(sort): At end of file, if there is more input and buffer room,
concatenate the next input file.
2000-03-03 08:18:48 +00:00
Jim Meyering 45d534215a (keycompare): Use global, hard_LC_COLLATE in place of
local that is sometimes undeclared.
2000-01-22 22:54:51 +00:00
Jim Meyering c4acbcc996 Tweak sort performance.
(hard_LC_CTYPE): Remove.
(keylist): Renamed from keyhead.  Now a pointer, not a
mostly-unused struct.  All uses changed.
(findlines, keycompare, CMP_WITH_IGNORE, compare, checkfp, mergefps,
sort): Tune and use a more consistent style for reallocation.
(keycompare, main): Don't worry about LC_CTYPE;
it's buggy with multibyte chars anyway.
(compare): Invoke alloca (0) after each call to keycompare,
not just the ones that return nonzero.  This avoids a memory
leak on architectures without builtin alloca that occurs
sometimes when a file contains all duplicate lines.
2000-01-19 22:43:33 +00:00
Jim Meyering 3e115cf2b5 (sighandler, main):
Don't use SA_INTERRUPT to decide whether to call sigaction, as
POSIX.1 doesn't require SA_INTERRUPT and some systems
(e.g. Solaris 7) don't define it.  Use SA_NOCLDSTOP instead;
it's been part of POSIX.1 since day 1 (in 1988).
2000-01-18 07:36:53 +00:00
Jim Meyering a69a2d6856 (fillbuf): Avoid quadratic behavior with long lines.
Also, stop worrying about ancient memchr bug (misbehavior when
size is zero), since other code doesn't worry either.
2000-01-13 07:34:44 +00:00
Jim Meyering 2d81adfab6 (SORTALLOC): New macro.
(sortalloc, mergealloc, LINEALLOC): Use it.
(sortalloc, mergealloc, linelength): Now const.
(sortalloc): Increase from 0.5 to 8 MB.
(mergealloc): Increase from 16 to 256 kB.
(LINEALLOC): Increase from 0.25 to 4 MB.
1999-11-05 07:59:40 +00:00
Jim Meyering cec217ed93 (begfield, limfield, findlines, keycompare, compare):
Do not consider newline to be part of a line when comparing lines
in `sort' and `comm'.  POSIX.2 requires that we consider newline,
but this is a bug in the spec and the bug will likely be fixed.
1999-11-04 23:11:34 +00:00
Jim Meyering d1b4719534 Remove xstrdup declaration. 1999-09-02 21:31:19 +00:00
Jim Meyering 005c36b178 (checkfp): Use IF_LINT macro instead of #ifdef lint...
(mergefps): Likewise.
1999-08-22 09:50:05 +00:00
Jim Meyering e4024a94f8 Include file name in `write error' diagnostics.
(write_bytes): Add output_file parameter and use it.  Update callers.
(mergefps): Likewise.
(merge): Likewise.
(sort): Likewise.
Reported by John Summerfield.
1999-08-06 00:07:59 +00:00
Jim Meyering d67adf2b8c Include hard-locale.h, memcoll.h.
(hard_LC_COLLATE, hard_LC_CTYPE, hard_LC_TIME): New variables,
replacing `need_locale'.
(memcoll): Move to lib/memcoll.c.
(keycompare): No need to alloc (0), since our caller now does it.
(compare): alloca (0) before returning.
(my_setlocale): Remove; hard_locale now dows this.
(main): Invoke setlocale, bindtextdomain, and textdomain before
invoking anything that might print an error.
Use hard_locale to determine which locales are hard.
1999-07-04 10:23:19 +00:00
Jim Meyering 9e48e223db (general_numcompare): Put exceptional cases
first, not last, to be consistent with -M.
1999-05-22 12:52:14 +00:00
Jim Meyering 7bac5533d7 (strtod): Declare if STDC_HEADERS is not defined.
(general_numcompare): Use strtod, not xstrtod.
Do not consider partial conversions to be errors.
Put -infinity at the start, and +infinity at the end;
follow +infinity with NaNs (sorted by bit pattern),
and finally by conversion errors.
1999-05-22 02:47:45 +00:00
Jim Meyering d6261d35c9 Treat the trailing newline as part of the line, as required by POSIX.2.
(struct line, findlines, compare, checkfp, mergefps, sort):
A line now includes its trailing newline.
(findlines): Do not replace newline with NUL.
(memcoll, keycompare): Work even if the data to be compared are
adjacent strings; this is possible now that lines contain the
trailing newline.
(fillbuf): Always have an unused byte at the end of the	buffer,
since memcoll and keycompare want to modify a byte after the last line.
(sortalloc, mergealloc): Increase by 1, for trailing byte.
1999-05-21 20:24:19 +00:00
Jim Meyering d30365509d (keycompare): Ignore any length difference if the
localized comparison says the strings are equal.
1999-05-20 16:29:38 +00:00