Reported in https://bugs.gnu.org/30479 .
doc/sed.x: Change a HYPHEN-MINUS (code 0x55, 2D) to a dash (\-, minus)
if it matches " -[:alpha:]" or \(aq-[:alpha:] (for options); Use a pronoun
instead of a repeated noun.
Copyright-paperwork-exempt: Yes
Perl-regexp syntax (PCRE) in GNU Sed is shelved indefinitely.
See https://bugs.gnu.org/22801 , https://bugs.gnu.org/22647 .
Remove all (unused) REG_PERL related code.
* sed/sed.c, sed/sed.h, sed/regexp.c, sed/compile.c: Remove REG_PERL code.
Use the locally built sed binary only during 'check*' targets,
but not during build/installation targets - it might be buggy, or contain
temporary debugging/ASAN features that can break the build.
See: https://lists.gnu.org/r/sed-devel/2018-08/msg00013.html .
* cfg.mk (PATH): Add the 'sed' directory to the PATH only if this is a
'check' target.
Under normal operation there is no need for explicit freeing,
as all memory will be released when sed terminates.
During development (and specifically, valgrind and address-sanitizing)
enabling lint prevents false-positive warnings about memory
leaks.
Lint mode can be enabled with CFLAGS="-Dlint". It is also automatically
enabled by default when building from git (as opposed to tarball).
For consistency all instances of "#ifdef DEBUG_LEAKS" are replaced
with "#ifdef lint".
* sed/sed.h (struct subst): Add member variable to keep the address of
allocated buffer in compile.c:setup_replacement().
(release_regex): Add declaration.
(finish_program): Function now takes an argument: the sed program
vector.
* sed/sed.c (main): Adjust call to finish_program.
* sed/compile.c (finish_program): Release program allocations.
(setup_replacement): Remember the allocated buffer address.
(compile_program): Free temporary array in 'y' command.
* sed/execute.c (execute_program): Free allocated but unused buffer in
'R' command when reaching EOF.
* sed/regexp.c (release_regex): Remove 'static', free the allocated
dfa struct.
* sed/utils.c (panic): Free open files linked-list elements.
sed would access invalid memory when matching EOF combined with
s///n flag:
$ yes 0 | fmt -w 40 | head -n2 | valgrind sed 'N;s/$//2m'
==13131== Conditional jump or move depends on uninitialised value(s)
==13131== at 0x4C3002B: memchr (vg_replace_strmem.c:883)
==13131== by 0x1120BD: match_regex (regexp.c:286)
==13131== by 0x110736: do_subst (execute.c:1101)
==13131== by 0x1115D3: execute_program (execute.c:1591)
==13131== by 0x111A4C: process_files (execute.c:1774)
==13131== by 0x112E1C: main (sed.c:405)
==13131==
==13131== Invalid read of size 1
==13131== at 0x4C30027: memchr (vg_replace_strmem.c:883)
==13131== by 0x1120BD: match_regex (regexp.c:286)
==13131== by 0x110736: do_subst (execute.c:1101)
==13131== by 0x1115D3: execute_program (execute.c:1591)
==13131== by 0x111A4C: process_files (execute.c:1774)
==13131== by 0x112E1C: main (sed.c:405)
==13131== Address 0x55ec765 is 0 bytes after a block of size 101 alloc'd
==13131== at 0x4C2DDCF: realloc (vg_replace_malloc.c:785)
==13131== by 0x113BA2: ck_realloc (utils.c:418)
==13131== by 0x10E682: resize_line (execute.c:154)
==13131== by 0x10E6F0: str_append (execute.c:165)
==13131== by 0x110779: do_subst (execute.c:1106)
==13131== by 0x1115D3: execute_program (execute.c:1591)
==13131== by 0x111A4C: process_files (execute.c:1774)
==13131== by 0x112E1C: main (sed.c:405)
==13131==
The ^/$ optimization code added in v4.2.2-161-g6dea75e called memchr()
using 'buflen', ignoring the value of 'buf_start_offset' (which, if not
zero, reduces the number of bytes available for the search).
Reported by bugs@feusi.co (bug#32271) in
https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .
* NEWS: Mention the fix.
* sed/regexp.c (match_regex): Use correct buffer length in memchr().
* testsuite/bug-32271-2.sh: Test using valgrind.
* testsuite/local.mk (T): Add new test.
Under certain conditions sed would add an extraneous NUL:
$ echo 0 | sed -e 's/$/a/2' | od -tx1 -An
30 00 0a
This would happen when the regex is an empty (zero-length) match at the
end of the line (e.g. '$' and 'a*$') and the substitute number flag
('n' in s///n) is higher than the number of actual matches (multiple
EOL matches are possible with multiline match, e.g. 's/$/a/3m').
Details:
The comment at the top of 'execute.c:do_subst()' says:
/* The first part of the loop optimizes s/xxx// when xxx is at the
start, and s/xxx$// */
Which refers to lines 1051-3:
1051 /* Copy stuff to the left of this match into the output string. */
1052 if (start < offset)
1053 str_append(&s_accum, line.active + start, offset - start);
The above code appends text to 's_accum' but does *not* update 'start'.
Later on, if the s/// command includes 'n' flag, and if 'matched == 0'
(an empty match), this comparison will be incorrect:
1081 if (start < line.length)
1082 matched = 1;
Will in turn will set 'matched' to 1, and the 'str_append' call that
follows (line 1087) will append an additional character.
Because the empty match is EOL, the appended character is NUL.
More examples that trigger the bug:
echo 0 | sed -e 's/a*$/X/3'
printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/4m'
Examples that do not trigger the bug:
# The 'a*' empty regex matches at the beginning of the line (in
# addition to the end of the line), and the optimization in line
# 1052 is skipped.
echo 0 | sed -e 's/a*/X/3'
# There are 3 EOLs in the pattern space, s///3 is not too large.
printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/3m'
This was discovered while investigating bug#32271 reported by bugs@feusi.co
in https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .
* NEWS: Mention the fix.
* sed/execute.c (do_subst): Update 'start' as needed.
* testsuite/bug-32271-1.sh: New test.
* testsuite/local.mk (T): Add test.
Previously sed would explicitly flush the output after
every output line, except if the output was stdout in unbuffered mode.
In practice this was equivalent to forcing line-buffering, and was
espcially was noticable with "sed -i" (where the output is a temporary
file).
With this change, explicit flushing only happens with "sed -u",
regardless of the type of output file, making "sed -i" much faster.
This change also affect other write commands such as 'w'/'W' and 's///w'.
Reported by Vidar Holen <vidar@vidarholen.net> in
https://lists.gnu.org/r/bug-sed/2018-07/msg00014.html .
* NEWS: Mention this.
* sed/execute.c (flush_output): Never flush output unless in unbuffered
mode, regardless of which file it is.
Under certain conditions sed would access invalid memory based on
the requested back-reference (e.g. "s//\9/" would access the 9th element
in the regex registers without checking it is at least 9 element in
size).
The following examples would trigger valgrind errors:
seq 2 | valgrind sed -e '/^/s///p ; 2s//\9/'
seq 2 | valgrind sed --posix -e '/2/p ; 2s//\9/'
Reported by bugs@feusi.co in
https://lists.gnu.org/r/bug-sed/2018-07/msg00004.html .
* NEWS: Mention the bugfix.
* sed/execute.c (append_replacement): Check number of allocated regex
replacement registers before accessing the array.
* sed/testsuite/bug32082.sh: Test sed for this behaviour under valgrind.
* sed/testsuite/local.mk (T): Add new test.
* testsuite/inplace-selinux.sh: Set the user (-u) portion of the security
context rather than the type (-t), and use system_u and user_u, which
appear to be reliably available, since some coreutils's tests use those.
Assaf Gordon reported that this test failed on those systems in
https://lists.gnu.org/r/sed-devel/2018-03/msg00018.html
Also, correct copyright date to include this year.
testsuite/runtest was not portable to Solaris 10 /bin/sh, but was
used only to keep automake test machinery from running two compiled
binaries stand-alone. We can inhibit that more cleanly by removing
those two files from the list of TESTS to run and then just remove
the offending script.
* testsuite/local.mk (LOG_COMPILER): Set to false.
(TESTS): Don't put $(check_PROGRAMS) on this list.
* testsuite/runtest: Remove file.
Nelson H.F. Beebe reported the Solaris 10 /bin/sh failure.
* testsuite/runtest (test): Accommodate Solaris 10 /bin/sh
by sourcing init.sh, as is already done in many other shell
scripts in this directory, to handle (or skip test upon)
use of this construct: ${1##*/}. Reported by Nelson H.F. Beebe.
Sed uses 'size_t' internally, but gnulib's re_search uses 'signed int'.
If the buffer is larger than INT_MAX, reject it (panic, exit code 4).
The test is marked 'very expensive' and skipped by default (due to
creating a 2GB input file). To run it, use:
make check SUBDIRS=. RUN_VERY_EXPENSIVE_TESTS=yes \
TESTS=testsuite/regex-max-int.sh
Reported by YushiOMOTE in https://bugs.gnu.org/30520 .
* NEWS: Mention this.
* Makefile.am (check-expensive, check-very-expensive): New targets.
* init.cfg (expensive_, very_expensive_): Copied from coreutils.
* sed/regexp.c (match_regex): Check buffer length and panic if too large.
* testsuite/regex-max-int.sh: New test.
* testsuite/local.mk (T): Add new test.
Sed incorrectly treated '\x5c' (ASCII 92, backslash) as an escape character.
Old behavior:
$ echo z | sed -E 's/(z)/\x5c1/' # identical to 's/(z)/\1/'
z
New behavior:
$ echo z | sed -E 's/(z)/\x5c1/'
\1
Reported in https://bugs.gnu.org/30794.
* NEWS: Mention bug fix.
* sed/compile.c (normalize_text): Pass backslash as literal character.
* testsuite/misc.pl: Add tests.
* doc/sed.texi: s/only accepts/accepts only/.
(BRE vs ERE): Tweak ERE and BRE definitions, and convert a
passive-voice sentence to active voice. Insert a comma.
* doc/config.texi, doc/sed-dummy.1, doc/sed.texi: Change http to https for
GNU URLs. External links (e.g. sed.sf.net, autsingroup.org) currently do
not support https and kept as-is.