sed would access invalid memory when matching EOF combined with
s///n flag:
$ yes 0 | fmt -w 40 | head -n2 | valgrind sed 'N;s/$//2m'
==13131== Conditional jump or move depends on uninitialised value(s)
==13131== at 0x4C3002B: memchr (vg_replace_strmem.c:883)
==13131== by 0x1120BD: match_regex (regexp.c:286)
==13131== by 0x110736: do_subst (execute.c:1101)
==13131== by 0x1115D3: execute_program (execute.c:1591)
==13131== by 0x111A4C: process_files (execute.c:1774)
==13131== by 0x112E1C: main (sed.c:405)
==13131==
==13131== Invalid read of size 1
==13131== at 0x4C30027: memchr (vg_replace_strmem.c:883)
==13131== by 0x1120BD: match_regex (regexp.c:286)
==13131== by 0x110736: do_subst (execute.c:1101)
==13131== by 0x1115D3: execute_program (execute.c:1591)
==13131== by 0x111A4C: process_files (execute.c:1774)
==13131== by 0x112E1C: main (sed.c:405)
==13131== Address 0x55ec765 is 0 bytes after a block of size 101 alloc'd
==13131== at 0x4C2DDCF: realloc (vg_replace_malloc.c:785)
==13131== by 0x113BA2: ck_realloc (utils.c:418)
==13131== by 0x10E682: resize_line (execute.c:154)
==13131== by 0x10E6F0: str_append (execute.c:165)
==13131== by 0x110779: do_subst (execute.c:1106)
==13131== by 0x1115D3: execute_program (execute.c:1591)
==13131== by 0x111A4C: process_files (execute.c:1774)
==13131== by 0x112E1C: main (sed.c:405)
==13131==
The ^/$ optimization code added in v4.2.2-161-g6dea75e called memchr()
using 'buflen', ignoring the value of 'buf_start_offset' (which, if not
zero, reduces the number of bytes available for the search).
Reported by bugs@feusi.co (bug#32271) in
https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .
* NEWS: Mention the fix.
* sed/regexp.c (match_regex): Use correct buffer length in memchr().
* testsuite/bug-32271-2.sh: Test using valgrind.
* testsuite/local.mk (T): Add new test.
Under certain conditions sed would add an extraneous NUL:
$ echo 0 | sed -e 's/$/a/2' | od -tx1 -An
30 00 0a
This would happen when the regex is an empty (zero-length) match at the
end of the line (e.g. '$' and 'a*$') and the substitute number flag
('n' in s///n) is higher than the number of actual matches (multiple
EOL matches are possible with multiline match, e.g. 's/$/a/3m').
Details:
The comment at the top of 'execute.c:do_subst()' says:
/* The first part of the loop optimizes s/xxx// when xxx is at the
start, and s/xxx$// */
Which refers to lines 1051-3:
1051 /* Copy stuff to the left of this match into the output string. */
1052 if (start < offset)
1053 str_append(&s_accum, line.active + start, offset - start);
The above code appends text to 's_accum' but does *not* update 'start'.
Later on, if the s/// command includes 'n' flag, and if 'matched == 0'
(an empty match), this comparison will be incorrect:
1081 if (start < line.length)
1082 matched = 1;
Will in turn will set 'matched' to 1, and the 'str_append' call that
follows (line 1087) will append an additional character.
Because the empty match is EOL, the appended character is NUL.
More examples that trigger the bug:
echo 0 | sed -e 's/a*$/X/3'
printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/4m'
Examples that do not trigger the bug:
# The 'a*' empty regex matches at the beginning of the line (in
# addition to the end of the line), and the optimization in line
# 1052 is skipped.
echo 0 | sed -e 's/a*/X/3'
# There are 3 EOLs in the pattern space, s///3 is not too large.
printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/3m'
This was discovered while investigating bug#32271 reported by bugs@feusi.co
in https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .
* NEWS: Mention the fix.
* sed/execute.c (do_subst): Update 'start' as needed.
* testsuite/bug-32271-1.sh: New test.
* testsuite/local.mk (T): Add test.
Previously sed would explicitly flush the output after
every output line, except if the output was stdout in unbuffered mode.
In practice this was equivalent to forcing line-buffering, and was
espcially was noticable with "sed -i" (where the output is a temporary
file).
With this change, explicit flushing only happens with "sed -u",
regardless of the type of output file, making "sed -i" much faster.
This change also affect other write commands such as 'w'/'W' and 's///w'.
Reported by Vidar Holen <vidar@vidarholen.net> in
https://lists.gnu.org/r/bug-sed/2018-07/msg00014.html .
* NEWS: Mention this.
* sed/execute.c (flush_output): Never flush output unless in unbuffered
mode, regardless of which file it is.
Under certain conditions sed would access invalid memory based on
the requested back-reference (e.g. "s//\9/" would access the 9th element
in the regex registers without checking it is at least 9 element in
size).
The following examples would trigger valgrind errors:
seq 2 | valgrind sed -e '/^/s///p ; 2s//\9/'
seq 2 | valgrind sed --posix -e '/2/p ; 2s//\9/'
Reported by bugs@feusi.co in
https://lists.gnu.org/r/bug-sed/2018-07/msg00004.html .
* NEWS: Mention the bugfix.
* sed/execute.c (append_replacement): Check number of allocated regex
replacement registers before accessing the array.
* sed/testsuite/bug32082.sh: Test sed for this behaviour under valgrind.
* sed/testsuite/local.mk (T): Add new test.
* testsuite/inplace-selinux.sh: Set the user (-u) portion of the security
context rather than the type (-t), and use system_u and user_u, which
appear to be reliably available, since some coreutils's tests use those.
Assaf Gordon reported that this test failed on those systems in
https://lists.gnu.org/r/sed-devel/2018-03/msg00018.html
Also, correct copyright date to include this year.
testsuite/runtest was not portable to Solaris 10 /bin/sh, but was
used only to keep automake test machinery from running two compiled
binaries stand-alone. We can inhibit that more cleanly by removing
those two files from the list of TESTS to run and then just remove
the offending script.
* testsuite/local.mk (LOG_COMPILER): Set to false.
(TESTS): Don't put $(check_PROGRAMS) on this list.
* testsuite/runtest: Remove file.
Nelson H.F. Beebe reported the Solaris 10 /bin/sh failure.
* testsuite/runtest (test): Accommodate Solaris 10 /bin/sh
by sourcing init.sh, as is already done in many other shell
scripts in this directory, to handle (or skip test upon)
use of this construct: ${1##*/}. Reported by Nelson H.F. Beebe.
Sed uses 'size_t' internally, but gnulib's re_search uses 'signed int'.
If the buffer is larger than INT_MAX, reject it (panic, exit code 4).
The test is marked 'very expensive' and skipped by default (due to
creating a 2GB input file). To run it, use:
make check SUBDIRS=. RUN_VERY_EXPENSIVE_TESTS=yes \
TESTS=testsuite/regex-max-int.sh
Reported by YushiOMOTE in https://bugs.gnu.org/30520 .
* NEWS: Mention this.
* Makefile.am (check-expensive, check-very-expensive): New targets.
* init.cfg (expensive_, very_expensive_): Copied from coreutils.
* sed/regexp.c (match_regex): Check buffer length and panic if too large.
* testsuite/regex-max-int.sh: New test.
* testsuite/local.mk (T): Add new test.
Sed incorrectly treated '\x5c' (ASCII 92, backslash) as an escape character.
Old behavior:
$ echo z | sed -E 's/(z)/\x5c1/' # identical to 's/(z)/\1/'
z
New behavior:
$ echo z | sed -E 's/(z)/\x5c1/'
\1
Reported in https://bugs.gnu.org/30794.
* NEWS: Mention bug fix.
* sed/compile.c (normalize_text): Pass backslash as literal character.
* testsuite/misc.pl: Add tests.
* doc/sed.texi: s/only accepts/accepts only/.
(BRE vs ERE): Tweak ERE and BRE definitions, and convert a
passive-voice sentence to active voice. Insert a comma.
* doc/config.texi, doc/sed-dummy.1, doc/sed.texi: Change http to https for
GNU URLs. External links (e.g. sed.sf.net, autsingroup.org) currently do
not support https and kept as-is.
When editing a file in place, the SELinux context (if exists)
should be based on the link instead of the target file itself.
--follow-symlinks option remains unchanged.
Bug reported by Jakub Jelen, fix proposed by Petr Lautrbach,
discussed in https://lists.gnu.org/r/sed-devel/2017-12/msg00000.html
* NEWS: Mention this.
* sed/execute.c (open_next_file): Use lgetfilecon (instead of getfilecon)
to get the context of the symlink instead of the target file.
* testsuite/inplace-selinux.sh: Test the above change.
* testsuite/local.mk (T): Add new test file.
* init.cfg (require_selinux_): Copied from coreutils, skip the test if
selinux is not available.
* sed/sed.h (FALLTHROUGH): Define.
* sed/execute.c (execute_program): Use new FALLTHROUGH macro in place
of each comment.
* sed/compile.c (mark_subst_opts, compile_program): Likewise.
Reported by Bamber Ward in https://bugs.gnu.org/28140 .
* doc/sed.texi (Line length adjustment): Fix sed script;
Rewrite example to long script with inlined comments;
Remove second example.
Reported by Dennis Clarke in https://bugs.gnu.org/28665#14 .
* init.cfg (require_ru_utf8_locale_): New function.
* testsuite/utf8-ru.sh: Skip test is required locale was not found.
For development builds (using ./bootstrap), require texinfo version 6.1
or later (Same as GNU coreutils).
Sed's manual uses texinfo's @U{} command, introduced in texinfo-6.0.
This should not affect building from released tarballs, as the info
manual is pre-built and pre-packaged in the tar archive.
Reported by Dennis Clarke in https://bugs.gnu.org/28665#14 .
* bootstrap.conf: Require makeinfo 6.1 (up from 4.13).
These C-based tests were not updated since 2004, and were not included
by default. Enabling them with "./configure --enable-regex-tests"
resulted in false positives:
https://bugs.gnu.org/28665 , https://bugs.gnu.org/25381 .
This patch removes them completely.
* configure.ac: Remove "--with-regex-tests" option.
* local.mk (check_PROGRAMS): Remove tests.
* bug-regex10.c,
bug-regex11.c,
bug-regex12.c,
bug-regex13.c,
bug-regex14.c,
bug-regex15.c,
bug-regex16.c,
bug-regex21.c,
bug-regex27.c,
bug-regex28.c,
bug-regex7.c,
bug-regex8.c,
bug-regex9.c,
runptests.c,
runtests.c,
tst-boost.c,
tst-pcre.c,
tst-regex.c,
tst-regex2.c,
tst-rxspencer.c: Remove files.
* doc/sed.texi, doc/config.texi: Remove all uses of @acronym{...},
per recommendation by Karl Berry.
* cfg.mk (local-checks-to-skip): Remove exemption, enabling
the @acronym{-prohibiting syntax-check rule.
All input files (mac-mf.{inp,good,sed}) are large and are kept as-is.
* testsuite/mac-mf.sh: New test.
* testsuite/local.mk (SEDTEST): Remove mac-mf. (T): Add mac-mf.sh
All input files (madding.{inp,good,sed}) contain very long lines and
are kept as-is (to avoid failing 'make syntax-check').
* testsuite/madding.sh: New test.
* testsuite/local.mk (SEDTEST): Remove madding. (T): Add madding.sh