Commit Graph
356 Commits
Author SHA1 Message Date
Assaf Gordon 2ce2a9fcde maint: add copyright/license information to configure.ac
Starting year chosen as 1993 based on earliest 'configure.in' found in
sed-1.18.tar.gz from that year.

* configure.ac: Add standard copyright and license information.
2016-07-26 22:13:37 -04:00
Assaf Gordon bc182c966e maint: update .gitignore after bootstrap
Added by running ./bootstrap on a clean sed repository.

* build-aux/.gitignore: Add config.rpath.
* lib/.gitignore: Add additional gnulib C files.
* m4/.gitignore: Add additional gnulib m4 files.
* po/.gitignore: Add additional PO-related files.
2016-07-26 22:13:37 -04:00
Assaf Gordon 1733161792 maint: update .gitignore
* .gitignore: Add coverage files, .dirstamp files, testsuite files.
2016-07-26 22:13:37 -04:00
Jim Meyering 2a033d37ea maint: skip a check when en_US.UTF-8 collation rules are broken
* cfg.mk (sc_THANKS_in_sorted): This check would fail on systems
for which "." is not ignored.  Add a quick sort-based check for
that error, and skip the check on any broken system.
This also corrects the command to use $(srcdir)/THANKS.in,
rather than just THANKS.in.
2016-07-25 08:28:36 -07:00
Jim Meyering 7cd155bb77 maint: sort THANKS.in
* THANKS.in: Move J.T. Conklin's name "down" so it is in
en_US.UTF-8-sorted order.
2016-07-24 20:10:32 -07:00
Assaf Gordon 3b29bece3e build: switch to non-recursive makefile
* .gitignore: Ignore '.dirstamp' files
* bootstrap.conf: Add 'non-recursive-gnulib-prefix-hack' gnulib module.
  Add missing 'stdalign' module.
  (bootstrap_post_import_hook): Use module to adjust gnulib.mk file.
* configure.ac: Add 'subdir-objects' automake option, don't generate
  makefiles in subdirectories. Adjust path of XFAIL tests.
* Makefile.am: Include 'local.mk' files, adjust for non-recursive make.
* doc/Makefile.am: Rename to ...
* doc/local.mk: Adjust paths. Rename SED to
  SEDBIN variable, to avoid overriding the default system-wide SED value
  which is automatically set by autotools.
* lib/Makefile.am: Rename to ...
* lib/local.mk: Adjust paths.
* lib/.gitignore: Ignore 'stdalign.h' from gnulib module.
* sed/Makefile.am: Rename to ...
* sed/local.mk: Adjust paths.
* testsuite/Makefile.am: Rename to ...
* testsuite/local.mk: Adjust paths.
* testsuite/init.cfg: Move to ...
* init.cfg: ... here. Moved to avoid changing the path in init.sh, and to
  keep the same directory structure as coreutils.
  (require_en_utf8_locale_): Adjust path for get-mb-cur-max executable.
  (require_ja_shiftjis_locale_,require_valid_ja_shiftjis_locale_): Adjust
  path for test-mbrtowc executable.
* m4/.gitignore: Add gnulib's non-recusive-make module.
* testsuite/runtest: Adjust paths of Makefile.tests and dir variables,
  Skip utility program 'get-mb-cur-max' which is not a standalone test.
* testsuite/Makefile.tests: Adjust path of sed exeutable.
* testsuite/eval.sed, testsuite/eval.good: Adjust path of sed executable.
* testsuite/cmd-R.sh,
  testsuite/cmd-l.sh,
  testsuite/colon-with-no-label.sh,
  testsuite/comment-n.sh,
  testsuite/compile-errors.sh,
  testsuite/compile-tests.sh,
  testsuite/convert-number.sh,
  testsuite/execute-tests.sh,
  testsuite/follow-symlinks-stdin.sh,
  testsuite/follow-symlinks.sh,
  testsuite/help-version.sh,
  testsuite/in-place-hyphen.sh,
  testsuite/in-place-suffix-backup.sh,
  testsuite/invalid-mb-seq-UMR.sh,
  testsuite/mb-bad-delim.sh,
  testsuite/mb-charclass-non-utf8.sh,
  testsuite/mb-match-slash.sh,
  testsuite/mb-y-translate.sh,
  testsuite/normalize-text.sh,
  testsuite/nulldata.sh,
  testsuite/panic-tests.sh,
  testsuite/posix-char-class.sh,
  testsuite/posix-mode-N.sh,
  testsuite/posix-mode-addr.sh,
  testsuite/posix-mode-bad-ref.sh,
  testsuite/posix-mode-s.sh,
  testsuite/range-overlap.sh,
  testsuite/recursive-escape-c.sh,
  testsuite/regex-errors.sh,
  testsuite/stdin-prog.sh,
  testsuite/subst-mb-incomplete.sh,
  testsuite/subst-options.sh,
  testsuite/subst-replacement.sh,
  testsuite/temp-file-cleanup.sh,
  testsuite/unbuffered.sh: Adjust paths of init.sh, sed executable.
2016-07-24 22:22:53 -04:00
Assaf Gordon f9070781f1 tests: test --follow-symlinks option
* configure.ac: (TEST_SYMLINKS): New AM_CONDITIONAL.
* testsuite/follow-symlinks.sh: New tests.
* testsuite/Makefile.am: (T): Conditionally add new test (and existing test
'follow-symlink-stdin.sh') based on TEST_SYMLINKS.
2016-07-17 19:43:22 -04:00
Assaf Gordon 12bdd75627 tests: test y command in multibyte locales
* testsuite/mb-y-translate.sh: New test.
* testsuite/Makefile.am: Add new test.
2016-07-17 19:43:22 -04:00
Assaf Gordon 941a09c82d tests: test y,s non-slash multibyte delimiters
Test various scenarios of using a multibyte delimiters (instead of
slash) for s,y commands under UTF-8 and C locales.

* testsuite/mb-bad-delim.sh: New test.
* testsuite/Makefile.am: Add new test.
2016-07-16 20:02:48 -04:00
Assaf Gordon 77a7c6a731 tests: skip tests upon buggy mbrtowc implementations
Sed tests purposefully use invalid non-utf8 multibyte sequences to
test sed's handling of invalid input/program.
Some implementations wrongly accept invalid sequences, causing false
alarms. Add a test program to detect such buggy implementations and skip
the tests if needed.
See discussion in:
http://lists.gnu.org/archive/html/sed-devel/2016-07/msg00005.html
http://lists.gnu.org/archive/html/sed-devel/2016-07/msg00000.html
http://lists.gnu.org/archive/html/sed-devel/2016-06/msg00031.html
http://lists.gnu.org/archive/html/sed-devel/2016-06/msg00014.html

* bootstrap.conf: Add gnulib's closeout module.
* m4/.gitignore, lib/.gitignore, po/POTFILES.in: Adjust after adding
closeout module.
* testsuite/Makefile.am (check_PROGRAMS): Add new program.
* testsuite/test-mbrtowc.c: New program.
* testsuite/init.cfg (require_valid_ja_eucjp_locale_)
(require_valid_ja_shiftjis_locale_): New functions. Call test-mbrtowc
with known invalid multibyte sequences to ensure implementation is not
buggy and rejects them; Skip test otherwise.
* testsuite/invalid-mb-seq-UMR.sh: Use require_valid_ja_eucjp_locale_.
* testsuite/mb-charclass-non-utf8.sh: Use require_valid_ja_shiftjis_locale_.
2016-07-06 20:29:07 -04:00
Assaf Gordon 0052daf1e5 sed: fix minor multibyte parsing bug
Previously sed would parse multibyte characters incorrectly in two scenarios:

1. Slash following an incomplete-yet-valid multibyte sequence (match_slash):
    $ LC_ALL=en_US.UTF-8 sed $'s/\316/X/'
    sed: -e expression #1, char 6: unterminated `s' command

2. Open/close brackets as part of a valid mutilbyte string inside a character
class (snarf_char_class). In the example below, '\203]' is a valid
multibyte character in SHIFT-JIS locale:

    $ LC_ALL=ja_JP.shiftjis sed $'/[\203]/]/p'
    sed: -e expression #1, char #5: Unmatched [ or [^

Both cases stem from mbcs.c:brlen() being non-intuitive:
It returned 1 for valid single-byte character, invalid multibyte-character,
and a for the last byte of a valid multibyte sequence - making it
non-trivial to use correctly.

This commit replaces brlen() with a simpler is_mb_char() function:
returns non-zero for multibyte sequences, zero for single/invalid sequences.

* sed/sed.h: (BRLEN, brlen): Remove delaration.
(IS_MB_CHAR,is_mb_char): Add macro and function declaration.
* sed/mbcs.c: (brlen): Remove function. (is_mb_char): New function.
* sed/compile.c: (snarf_char_class, match_slash): Use IS_MB_CHAR instead of
BRLEN; Adjust local variables accordingly.
* testsuite/mb-match-slash.sh: New test for scenario 1.
* testsuite/mb-charclass-non-utf8.sh: New test for scenario 2,
requires SHIFT-JIS locale.
* testsuite/Makefile.am: Add new tests
* testsuite/init.cfg: (require_ja_shiftjis_locale_): New function.
* NEWS: Mention bug fix.
2016-07-04 14:26:09 -04:00
Tristan Verniquet 6ad4acfbd5 sed: use unlocked-io
Sed already imports gnulib's unlocked-io module, but it was not included
in 'utils.c'. Suggested in http://bugs.gnu.org/23848 .

* sed/utils.c: Include 'unlocked-io.h' header.
* NEWS: Mention improvement.
2016-06-27 01:05:34 -04:00
Assaf Gordon 891057bbba tests: test text normalization (\X sequences)
Sed understands escape-sequences such as \r \t \n \cX.
Test various scenarios of text normalization.

* testsuite/normalize-text.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-05 22:25:33 -04:00
Assaf Gordon 417b6d59e5 tests: test \dNNN \oNNN \xNN escape sequences
Test different variations of \dNNN \oNNN \xNN escape sequences,
exercising compile.c:convert_number().

* testsuite/convert-number.sh: new test.
* testsuite/Makefile.am: add new test
2016-06-02 12:57:57 -04:00
Assaf Gordon b206f21f58 tests: test less-common compilation cases
Test various odds-and-ends stemming from GNU sed implementation
of the program compilation module.

* testsuite/compile-tests.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 12:57:57 -04:00
Assaf Gordon 8ff0f1074a tests: test less-common execution cases
* testsuite/execute-tests.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:04:12 -04:00
Assaf Gordon d884bd156a tests: test 'R' command
* testsuite/cmd-R.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:03:48 -04:00
Assaf Gordon 1047b1944f tests: test program file from STDIN
* testsuite/stdin-prog.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:03:47 -04:00
Assaf Gordon 5c27158013 tests: test auto-silent mode with '#n' comments
* testsuite/comment-n.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:03:47 -04:00
Assaf Gordon 0d3cc0ba69 tests: test character-classes escaping in POSIX mode
No \X escaping inside character classes in POSIX mode.

Compare:
  $ printf "t\t\n" | sed 's/[\t]/X/' | od -a
  0000000  t   X   nl
  $ printf "t\t\n" | sed --posix 's/[\t]/X/' | od -a
  0000000  X   ht  nl

* testsuite/posix-char-class.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:03:47 -04:00
Assaf Gordon c5a2873f98 tests: test substitue replacements (s//\N/)
Test standard POSIX (e.g. \1 \2 \3) and gnu extension (\U \u \L \l \E)
replacements in s/// command.

* testsuite/subst-replacement.st: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:03:47 -04:00
Assaf Gordon fad666a4a3 tests: test s/// modifiers
Test standard posix and gnu extension modifiers to s/// command.

* testsuite/subst-options.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:03:47 -04:00
Assaf Gordon e5c00cd814 tests: test compilation-related errors
Cover code paths in copmile.c leading to errors.

* testsuite/compile-errors.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 02:03:40 -04:00
Assaf Gordon 2a52c02147 tests: test regex compilation errors
Test scenarios leading to regex compilation errors.

* testsuite/regex-errors.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 01:53:58 -04:00
Assaf Gordon 5516a2e1c3 tests: exercise 'panic' calls
Test scenarios in which GNU sed terminates with a panic error message
(and verify the error message text).

* testsuite/panic-tests.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 01:53:58 -04:00
Assaf Gordon 26d3cbc417 tests: test -i/--inplace with backup suffix option
GNU Sed accepts optional backup filename/suffix with -i option.
If the name contains '*', sed replaces it with the input filename.
Test various scenarios of suffix names.
Example:
    sed -i'*-*-*.txt' = a
will generate backup file 'a-a-a.txt'.

* testsuite/in-place-suffix-backup.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 01:53:58 -04:00
Assaf Gordon b878dbf5e5 tests: test -u/--unbuffered mode
Test GNU Sed behaviour in --unbuffered mode (reading only the minimum
needed characters from STDIN, leaving output for next program when sed
terminates). Example:
   seq 10 | (sed -u 1q ; sort -nr )

* testsuite/unbuffered.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-02 01:53:58 -04:00
Assaf Gordon e697f54981 tests: test -z/--null-data option
Test input/output line-terminators with -z option.

* testsuite/nulldata.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-01 16:59:51 -04:00
Assaf Gordon dde9e7f391 tests: test 'l' command
Test output width of 'l' command, based on COLS envvar,
lN (GNU Extension), '-l N' command-line parameter,
and default output width.

* testsuite/cmd-l.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-01 16:59:51 -04:00
Assaf Gordon 9f5278aee1 tests: test gnu address extension in gnu/posix modes
The following addresses are gnu extensions:
  0,/regexp/
  addr1,+N
  addr2,~N

* testsuite/posix-mode-addr.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-01 16:59:51 -04:00
Assaf Gordon c7b50b72f1 tests: test 's///' gnu extensions in gnu/posix modes
s///[iImMe] should be rejected in --posix mode.
s//\[lLuUN]// should have no special meaning in --posix mode.

* testsuite/posix-mode-s.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-01 16:59:51 -04:00
Assaf Gordon 3939f507a2 tests: test invalid backreferences in gnu/posix modes
's/foo/\1/' is rejected by default (gnu extension),
silently accepted in POSIX.

* testsuite/posix-mode-bad-ref.sh: new test.
* testsuite/Makefile.am: add new test.
2016-06-01 16:59:51 -04:00
Assaf Gordon 8eadf4de43 tests: test N command in gnu/posix modes
'N' with insufficient output prints nothing in POSIX mode, prints the
last read line in GNU Extension mode.

Compare:
  echo a | sed N
  echo a | sed --posix N
  echo a | POSIXLY_CORRECT=y sed N

* testsuite/posix-mode-N.sh: test output of N command under gnu and
posix modes.
* testsuite/Makefile.am: add new test.
2016-06-01 16:59:51 -04:00
Assaf Gordon 49a0f87d9b tests: invalid-mb-seq-UMR: avoid false-failure
* testsuite/Makefile.am (TESTS_ENVIRONMENT): Propagate the
LOCALE_JA envvar setting determined at configure time into
the test environment.  Without this, the test would fail on
some systems: at least CentOS6 and 7, but not Fedora 24.
2016-06-01 08:24:15 -07:00
Jim Meyering b250bd43ef tests: don't let any envvar setting perturb tests
* testsuite/envvar-check: New file. Identical to the one from
grep, but with the addition of sed-specific "COLS".
* testsuite/Makefile.am (EXTRA_DIST): Add it here.
(TESTS_ENVIRONMENT): New prologue, mostly copied from grep.
2016-06-01 08:21:05 -07:00
Assaf Gordon 156e099817 sed: reject recursive escaping after \c
previously, sed 's/./\c\\/' or 's/./\c\d/' would produce incorrect
results. Require two backslashes after \c to denote control sequence
^\ (ASCII 0x34), and reject recursive escaping (e.g. \c\x61).

* sed/compile.c: (RECURSIVE_ESCAPE_C): New error message.
(normalize_text): Check for \c-backslash, reject recursive escaping.
* testsuite/recursive-escape-c.sh: New file. Test new behaviour.
* testsuite/Makefile.am (T): Add new test.
* NEWS (Bug fixes): Mention it.
2016-05-29 18:55:58 -07:00
Jim Meyering 61c0a53ec9 revert v4.2.2-87-gc033bde, to make sed -i treat "-" as a file name
Stephane Chazelas made a fine case for why sed's --in-place (-i)
should treat "-" as a file name, and not as standard input in
http://debbugs.gnu.org/21249

There was no specific test for the prior behavior, so rather than
letting the revert remove the sole test, adapt and retain it:
* testsuite/in-place-hyphen.sh: However, retain and adapt this
file to test for the restored behavior.
* testsuite/Makefile.am: Retain this change, too.
2016-01-02 19:19:46 -08:00
Jim Meyering 57f405ad56 maint: update copyright year, bootstrap, init.sh
Run "make update-copyright" and then...

* gnulib: Update to latest.
* tests/init.sh: Update from gnulib.
* bootstrap: Likewise.
2016-01-01 15:02:20 -08:00
Jim Meyering 9dd563c897 sed: do not elide an invalid byte in a substitution RHS
Now, sed copies such bytes into the output.
* sed/execute.c (str_append_modified): Copy each invalid byte
into the result string, as the comment suggests.
* testsuite/invalid-mb-seq-UMR.sh: Adjust this test to conform.
Perl does the same thing.  Also, remove an unnecessary "-e" option.
* NEWS (Bug fixes): Mention it.
Bug introduced prior to the switch to git in 2004,
between the releases of sed-4.0.9 and sed-4.1.
This addresses http://debbugs.gnu.org/22254.
2015-12-27 22:36:47 -08:00
Jim Meyering 67b3fcc980 sed: fix a heap-clobbering buffer overrun
* sed/execute.c (str_append_modified): Upon encountering an
incomplete multi-byte sequence in a substitution replacement
string, do not treat mbrtowc's return value of -2 as a large,
positive number.
* testsuite/subst-mb-incomplete.sh: New file/test.
* testsuite/Makefile.am (T): Add it.
Reported by Hanno Böck in http://debbugs.gnu.org/22127.
Hanno used the AFL fuzzer to find the segfault-inducing input
from which I derived the test's input.
Introduced by v4.2.2-76-g78e8e58, this bug was never in a release.
2015-12-27 20:19:08 -08:00
Jim Meyering 80c5866cb4 tests: add helper prog, get-mb-cur-max and re_en_utf8_locale_ function 2015-12-12 17:43:25 -08:00
Jim Meyering 31c84cbcfd sed: reject a ":" command without a label
Before, sed would accept it and treat it as a label whose name
had length 0, and that could be referenced via a "b" or "t"
command with no label.
* sed/compile.c (errors): Add the new diagnostic.
(COLON_LACKS_LABEL): Define.
(compile_program): Reject ":" command with no label.
* testsuite/colon-with-no-label.sh: New file.
* testsuite/Makefile.am (T): Add it.
* NEWS (Bug fixes): Mention it.
Reported by Stephane Chazelas in http://bugs.gnu.org/21250
2015-08-23 21:48:34 -07:00
Assaf Gordon 854b64b79b build: comment out definitions of unused macros
* sed/compile.c (OPEN_BRACE): Comment out definition, to prevent gcc-5.2
from failing with -Werror -Wunused-macros.
(END_ERRORS): Likewise.
2015-08-01 09:15:47 -07:00
Jim Meyering 51dc57ab9c tests: avoid false-failure on system with no ja_JP.eucJP locale
* testsuite/invalid-mb-seq-UMR.sh: Use the configure-detected
locale name, $LOCALE_JA, rather than hard-coding ja_JP.eucJP.
If the required type of locale was not detected, then skip
this test. Reported by Assaf Gordon in http://bugs.gnu.org/21124
2015-07-30 09:55:11 -07:00
Jim Meyering c38910d9e7 maint: remove dead code
* sed/utils.c (ck_mkstemp): Remove $TMP/$TMPDIR-envvar-using code
that would have been run only if this function's second parameter
were to be NULL, but that could never happen, given the current
and sole caller.
Also, use mode_t as the type for the saved umask value, not "int".
Also, move each declaration down to point of definition.
* sed/utils.h (ck_mkstemp): Mark each of the four parameters as nonnull.
2015-07-24 21:48:03 -07:00
Stanislav Brabec e387009c78 sed: fix --follow-symlinks to work when reading stdin
When reading from stdin, use of --follow-symlinks would cause failure:

  $ echo abc | sed --follow-symlinks s/a/d/
  sed: cannot stat -: No such file or directory

* sed/execute.c (open_next_file): Set input->in_file_name earlier,
so we can rearrange logic to avoid calling follow_symlink("-").
* testsuite/follow-symlinks-stdin.sh: New file.
* testsuite/Makefile.am (T): Add it.
* NEWS (Bug fixes): Mention it.
Bug introduced by commit v4.2.1-13-g84066bf.
http://bugs.gnu.org/20795
2015-07-15 21:40:31 -07:00
Jim Meyering fe9c9b6cfa typo fix: s/singlebyte/single-byte/
* sed/compile.c (compile_program, normalize_text): Fix typo.
* sed/mbcs.c (brlen): Likewise.
2015-06-23 15:32:35 -07:00
Jim Meyering c033bdee41 sed -i: do not treat "-" as a file name
Most GNU utilities treat "-" as standard input, sed itself does that --
in most contexts.  However, since the addition of support for
--in-place (-i) in sed-4.0, sed -i has treated a "-" argument as a
file name, i.e., like ./-.  Now, that usage evokes a diagnostic:

  $ sed -i s/a/b/ -
  sed: couldn't edit -: is a terminal

If you require the old behavior, specify the file name as "./-".
Prompted by the report/patch from Stanislav Brabec in
http://bugs.gnu.org/20796 to document the strangely
inconsistent legacy behavior.

* sed/execute.c (open_next_file): Call panic before
even attempting to operate on the file descriptor.
* testsuite/in-place-hyphen.sh: New file.  Test for this.
* testsuite/Makefile.am (T): Add it.
* NEWS (Feature removal): Mention it. Admittedly, the old behavior
feels more like a misfeature.
2015-06-20 19:58:08 -07:00
Jim Meyering 2714d42345 tests: initialize $fail to 0
* testsuite/range-overlap.sh: Otherwise, with $fail set to
nonzero in the environment, this test would report failure,
even when it should pass.
2015-06-20 19:58:08 -07:00
Jim Meyering 68e47c0c62 build: avoid warning about implicit declaration of unlink
* sed/sed.c: Include <unistd.h>, for declaration of unlink.
Without this change, commit makes sed v4.2.2-84-g7689015 fail
to compile when configured with --enable-gcc-warnings.
2015-05-10 09:33:27 -07:00