* .gitignore: Ignore '.dirstamp' files
* bootstrap.conf: Add 'non-recursive-gnulib-prefix-hack' gnulib module.
Add missing 'stdalign' module.
(bootstrap_post_import_hook): Use module to adjust gnulib.mk file.
* configure.ac: Add 'subdir-objects' automake option, don't generate
makefiles in subdirectories. Adjust path of XFAIL tests.
* Makefile.am: Include 'local.mk' files, adjust for non-recursive make.
* doc/Makefile.am: Rename to ...
* doc/local.mk: Adjust paths. Rename SED to
SEDBIN variable, to avoid overriding the default system-wide SED value
which is automatically set by autotools.
* lib/Makefile.am: Rename to ...
* lib/local.mk: Adjust paths.
* lib/.gitignore: Ignore 'stdalign.h' from gnulib module.
* sed/Makefile.am: Rename to ...
* sed/local.mk: Adjust paths.
* testsuite/Makefile.am: Rename to ...
* testsuite/local.mk: Adjust paths.
* testsuite/init.cfg: Move to ...
* init.cfg: ... here. Moved to avoid changing the path in init.sh, and to
keep the same directory structure as coreutils.
(require_en_utf8_locale_): Adjust path for get-mb-cur-max executable.
(require_ja_shiftjis_locale_,require_valid_ja_shiftjis_locale_): Adjust
path for test-mbrtowc executable.
* m4/.gitignore: Add gnulib's non-recusive-make module.
* testsuite/runtest: Adjust paths of Makefile.tests and dir variables,
Skip utility program 'get-mb-cur-max' which is not a standalone test.
* testsuite/Makefile.tests: Adjust path of sed exeutable.
* testsuite/eval.sed, testsuite/eval.good: Adjust path of sed executable.
* testsuite/cmd-R.sh,
testsuite/cmd-l.sh,
testsuite/colon-with-no-label.sh,
testsuite/comment-n.sh,
testsuite/compile-errors.sh,
testsuite/compile-tests.sh,
testsuite/convert-number.sh,
testsuite/execute-tests.sh,
testsuite/follow-symlinks-stdin.sh,
testsuite/follow-symlinks.sh,
testsuite/help-version.sh,
testsuite/in-place-hyphen.sh,
testsuite/in-place-suffix-backup.sh,
testsuite/invalid-mb-seq-UMR.sh,
testsuite/mb-bad-delim.sh,
testsuite/mb-charclass-non-utf8.sh,
testsuite/mb-match-slash.sh,
testsuite/mb-y-translate.sh,
testsuite/normalize-text.sh,
testsuite/nulldata.sh,
testsuite/panic-tests.sh,
testsuite/posix-char-class.sh,
testsuite/posix-mode-N.sh,
testsuite/posix-mode-addr.sh,
testsuite/posix-mode-bad-ref.sh,
testsuite/posix-mode-s.sh,
testsuite/range-overlap.sh,
testsuite/recursive-escape-c.sh,
testsuite/regex-errors.sh,
testsuite/stdin-prog.sh,
testsuite/subst-mb-incomplete.sh,
testsuite/subst-options.sh,
testsuite/subst-replacement.sh,
testsuite/temp-file-cleanup.sh,
testsuite/unbuffered.sh: Adjust paths of init.sh, sed executable.
* configure.ac: (TEST_SYMLINKS): New AM_CONDITIONAL.
* testsuite/follow-symlinks.sh: New tests.
* testsuite/Makefile.am: (T): Conditionally add new test (and existing test
'follow-symlink-stdin.sh') based on TEST_SYMLINKS.
Test various scenarios of using a multibyte delimiters (instead of
slash) for s,y commands under UTF-8 and C locales.
* testsuite/mb-bad-delim.sh: New test.
* testsuite/Makefile.am: Add new test.
Sed tests purposefully use invalid non-utf8 multibyte sequences to
test sed's handling of invalid input/program.
Some implementations wrongly accept invalid sequences, causing false
alarms. Add a test program to detect such buggy implementations and skip
the tests if needed.
See discussion in:
http://lists.gnu.org/archive/html/sed-devel/2016-07/msg00005.htmlhttp://lists.gnu.org/archive/html/sed-devel/2016-07/msg00000.htmlhttp://lists.gnu.org/archive/html/sed-devel/2016-06/msg00031.htmlhttp://lists.gnu.org/archive/html/sed-devel/2016-06/msg00014.html
* bootstrap.conf: Add gnulib's closeout module.
* m4/.gitignore, lib/.gitignore, po/POTFILES.in: Adjust after adding
closeout module.
* testsuite/Makefile.am (check_PROGRAMS): Add new program.
* testsuite/test-mbrtowc.c: New program.
* testsuite/init.cfg (require_valid_ja_eucjp_locale_)
(require_valid_ja_shiftjis_locale_): New functions. Call test-mbrtowc
with known invalid multibyte sequences to ensure implementation is not
buggy and rejects them; Skip test otherwise.
* testsuite/invalid-mb-seq-UMR.sh: Use require_valid_ja_eucjp_locale_.
* testsuite/mb-charclass-non-utf8.sh: Use require_valid_ja_shiftjis_locale_.
Previously sed would parse multibyte characters incorrectly in two scenarios:
1. Slash following an incomplete-yet-valid multibyte sequence (match_slash):
$ LC_ALL=en_US.UTF-8 sed $'s/\316/X/'
sed: -e expression #1, char 6: unterminated `s' command
2. Open/close brackets as part of a valid mutilbyte string inside a character
class (snarf_char_class). In the example below, '\203]' is a valid
multibyte character in SHIFT-JIS locale:
$ LC_ALL=ja_JP.shiftjis sed $'/[\203]/]/p'
sed: -e expression #1, char #5: Unmatched [ or [^
Both cases stem from mbcs.c:brlen() being non-intuitive:
It returned 1 for valid single-byte character, invalid multibyte-character,
and a for the last byte of a valid multibyte sequence - making it
non-trivial to use correctly.
This commit replaces brlen() with a simpler is_mb_char() function:
returns non-zero for multibyte sequences, zero for single/invalid sequences.
* sed/sed.h: (BRLEN, brlen): Remove delaration.
(IS_MB_CHAR,is_mb_char): Add macro and function declaration.
* sed/mbcs.c: (brlen): Remove function. (is_mb_char): New function.
* sed/compile.c: (snarf_char_class, match_slash): Use IS_MB_CHAR instead of
BRLEN; Adjust local variables accordingly.
* testsuite/mb-match-slash.sh: New test for scenario 1.
* testsuite/mb-charclass-non-utf8.sh: New test for scenario 2,
requires SHIFT-JIS locale.
* testsuite/Makefile.am: Add new tests
* testsuite/init.cfg: (require_ja_shiftjis_locale_): New function.
* NEWS: Mention bug fix.
Sed already imports gnulib's unlocked-io module, but it was not included
in 'utils.c'. Suggested in http://bugs.gnu.org/23848 .
* sed/utils.c: Include 'unlocked-io.h' header.
* NEWS: Mention improvement.
Sed understands escape-sequences such as \r \t \n \cX.
Test various scenarios of text normalization.
* testsuite/normalize-text.sh: new test.
* testsuite/Makefile.am: add new test.
Test different variations of \dNNN \oNNN \xNN escape sequences,
exercising compile.c:convert_number().
* testsuite/convert-number.sh: new test.
* testsuite/Makefile.am: add new test
Test various odds-and-ends stemming from GNU sed implementation
of the program compilation module.
* testsuite/compile-tests.sh: new test.
* testsuite/Makefile.am: add new test.
No \X escaping inside character classes in POSIX mode.
Compare:
$ printf "t\t\n" | sed 's/[\t]/X/' | od -a
0000000 t X nl
$ printf "t\t\n" | sed --posix 's/[\t]/X/' | od -a
0000000 X ht nl
* testsuite/posix-char-class.sh: new test.
* testsuite/Makefile.am: add new test.
Test standard POSIX (e.g. \1 \2 \3) and gnu extension (\U \u \L \l \E)
replacements in s/// command.
* testsuite/subst-replacement.st: new test.
* testsuite/Makefile.am: add new test.
Test scenarios in which GNU sed terminates with a panic error message
(and verify the error message text).
* testsuite/panic-tests.sh: new test.
* testsuite/Makefile.am: add new test.
GNU Sed accepts optional backup filename/suffix with -i option.
If the name contains '*', sed replaces it with the input filename.
Test various scenarios of suffix names.
Example:
sed -i'*-*-*.txt' = a
will generate backup file 'a-a-a.txt'.
* testsuite/in-place-suffix-backup.sh: new test.
* testsuite/Makefile.am: add new test.
Test GNU Sed behaviour in --unbuffered mode (reading only the minimum
needed characters from STDIN, leaving output for next program when sed
terminates). Example:
seq 10 | (sed -u 1q ; sort -nr )
* testsuite/unbuffered.sh: new test.
* testsuite/Makefile.am: add new test.
Test output width of 'l' command, based on COLS envvar,
lN (GNU Extension), '-l N' command-line parameter,
and default output width.
* testsuite/cmd-l.sh: new test.
* testsuite/Makefile.am: add new test.
The following addresses are gnu extensions:
0,/regexp/
addr1,+N
addr2,~N
* testsuite/posix-mode-addr.sh: new test.
* testsuite/Makefile.am: add new test.
s///[iImMe] should be rejected in --posix mode.
s//\[lLuUN]// should have no special meaning in --posix mode.
* testsuite/posix-mode-s.sh: new test.
* testsuite/Makefile.am: add new test.
's/foo/\1/' is rejected by default (gnu extension),
silently accepted in POSIX.
* testsuite/posix-mode-bad-ref.sh: new test.
* testsuite/Makefile.am: add new test.
'N' with insufficient output prints nothing in POSIX mode, prints the
last read line in GNU Extension mode.
Compare:
echo a | sed N
echo a | sed --posix N
echo a | POSIXLY_CORRECT=y sed N
* testsuite/posix-mode-N.sh: test output of N command under gnu and
posix modes.
* testsuite/Makefile.am: add new test.
* testsuite/Makefile.am (TESTS_ENVIRONMENT): Propagate the
LOCALE_JA envvar setting determined at configure time into
the test environment. Without this, the test would fail on
some systems: at least CentOS6 and 7, but not Fedora 24.
* testsuite/envvar-check: New file. Identical to the one from
grep, but with the addition of sed-specific "COLS".
* testsuite/Makefile.am (EXTRA_DIST): Add it here.
(TESTS_ENVIRONMENT): New prologue, mostly copied from grep.
previously, sed 's/./\c\\/' or 's/./\c\d/' would produce incorrect
results. Require two backslashes after \c to denote control sequence
^\ (ASCII 0x34), and reject recursive escaping (e.g. \c\x61).
* sed/compile.c: (RECURSIVE_ESCAPE_C): New error message.
(normalize_text): Check for \c-backslash, reject recursive escaping.
* testsuite/recursive-escape-c.sh: New file. Test new behaviour.
* testsuite/Makefile.am (T): Add new test.
* NEWS (Bug fixes): Mention it.
Stephane Chazelas made a fine case for why sed's --in-place (-i)
should treat "-" as a file name, and not as standard input in
http://debbugs.gnu.org/21249
There was no specific test for the prior behavior, so rather than
letting the revert remove the sole test, adapt and retain it:
* testsuite/in-place-hyphen.sh: However, retain and adapt this
file to test for the restored behavior.
* testsuite/Makefile.am: Retain this change, too.
Now, sed copies such bytes into the output.
* sed/execute.c (str_append_modified): Copy each invalid byte
into the result string, as the comment suggests.
* testsuite/invalid-mb-seq-UMR.sh: Adjust this test to conform.
Perl does the same thing. Also, remove an unnecessary "-e" option.
* NEWS (Bug fixes): Mention it.
Bug introduced prior to the switch to git in 2004,
between the releases of sed-4.0.9 and sed-4.1.
This addresses http://debbugs.gnu.org/22254.
* sed/execute.c (str_append_modified): Upon encountering an
incomplete multi-byte sequence in a substitution replacement
string, do not treat mbrtowc's return value of -2 as a large,
positive number.
* testsuite/subst-mb-incomplete.sh: New file/test.
* testsuite/Makefile.am (T): Add it.
Reported by Hanno Böck in http://debbugs.gnu.org/22127.
Hanno used the AFL fuzzer to find the segfault-inducing input
from which I derived the test's input.
Introduced by v4.2.2-76-g78e8e58, this bug was never in a release.
Before, sed would accept it and treat it as a label whose name
had length 0, and that could be referenced via a "b" or "t"
command with no label.
* sed/compile.c (errors): Add the new diagnostic.
(COLON_LACKS_LABEL): Define.
(compile_program): Reject ":" command with no label.
* testsuite/colon-with-no-label.sh: New file.
* testsuite/Makefile.am (T): Add it.
* NEWS (Bug fixes): Mention it.
Reported by Stephane Chazelas in http://bugs.gnu.org/21250
* testsuite/invalid-mb-seq-UMR.sh: Use the configure-detected
locale name, $LOCALE_JA, rather than hard-coding ja_JP.eucJP.
If the required type of locale was not detected, then skip
this test. Reported by Assaf Gordon in http://bugs.gnu.org/21124
* sed/utils.c (ck_mkstemp): Remove $TMP/$TMPDIR-envvar-using code
that would have been run only if this function's second parameter
were to be NULL, but that could never happen, given the current
and sole caller.
Also, use mode_t as the type for the saved umask value, not "int".
Also, move each declaration down to point of definition.
* sed/utils.h (ck_mkstemp): Mark each of the four parameters as nonnull.
When reading from stdin, use of --follow-symlinks would cause failure:
$ echo abc | sed --follow-symlinks s/a/d/
sed: cannot stat -: No such file or directory
* sed/execute.c (open_next_file): Set input->in_file_name earlier,
so we can rearrange logic to avoid calling follow_symlink("-").
* testsuite/follow-symlinks-stdin.sh: New file.
* testsuite/Makefile.am (T): Add it.
* NEWS (Bug fixes): Mention it.
Bug introduced by commit v4.2.1-13-g84066bf.
http://bugs.gnu.org/20795
Most GNU utilities treat "-" as standard input, sed itself does that --
in most contexts. However, since the addition of support for
--in-place (-i) in sed-4.0, sed -i has treated a "-" argument as a
file name, i.e., like ./-. Now, that usage evokes a diagnostic:
$ sed -i s/a/b/ -
sed: couldn't edit -: is a terminal
If you require the old behavior, specify the file name as "./-".
Prompted by the report/patch from Stanislav Brabec in
http://bugs.gnu.org/20796 to document the strangely
inconsistent legacy behavior.
* sed/execute.c (open_next_file): Call panic before
even attempting to operate on the file descriptor.
* testsuite/in-place-hyphen.sh: New file. Test for this.
* testsuite/Makefile.am (T): Add it.
* NEWS (Feature removal): Mention it. Admittedly, the old behavior
feels more like a misfeature.
* sed/sed.c: Include <unistd.h>, for declaration of unlink.
Without this change, commit makes sed v4.2.2-84-g7689015 fail
to compile when configured with --enable-gcc-warnings.
For example, running a command like "sed -i s//b/ F" would fail
to remove a temporary file named sedXXXXXX (for random XXXXXX)
in the directory alongside F.
* sed/sed.c (G_file_to_unlink): New global.
(register_cleanup_file, cancel_cleanup, cleanup): New functions.
(main): Call atexit.
* sed/execute.c (open_next_file): Register for unlink.
(closedown): Call cancel_cleanup right after the rename.
* sed/sed.h: Declare two of the new functions.
* NEWS (Bug fixes): Mention it.
* testsuite/temp-file-cleanup.sh: New file. Test for this.
* testsuite/Makefile.am (T): Add it.
Reported by David Jones in http://bugs.gnu.org/20002.
* testsuite/invalid-mb-seq-UMR.sh: Upon failure due to a
valgrind-internal assertion, skip this test.
Reported by Norihiro Tanaka in http://debbugs.gnu.org/20490
When the line number ranges of two or more editing commands overlap,
sed applies all commands but the first to a line that is just beyond
the union of all ranges. E.g., with this command,
seq 9|sed '2,7d;3,6d', sed would mistakenly delete line 8.
* sed/execute.c (match_an_address_p) [ADDR_IS_NUM]: Make this
function work also in this case.
(match_address_p): Move the ADDR_IS_NUM +
...->line_number == ...->addr_number comparison "up" into
match_an_address_p, so we can hoist two similar if/return
blocks out of "if" and "else" branches.
Change so that this function returns false when the current
line number is outside the specified range.
* testsuite/range-overlap.sh: New file, to test for this.
* testsuite/Makefile.am (T): Add it to the list.
* NEWS (Bug fixes): Mention it.
Reported as http://bugs.gnu.org/19899