* bootstrap.conf (gnulib_modules): Remove
non-recursive-gnulib-prefix-hack.
(gnulib_tool_option_extras): Add --automake-subdir.
(bootstrap_post_import_hook): No need to massage lib/gnulib.mk.
* configure.ac [AM_INIT_AUTOMAKE]: Move subdir-objects onto the same line
as the function name, so gnulib-tool sees it.
* doc/sed.texi: When declaring that an example replaces
all occurrences, use "g" and make the description more
precise. Prompted by http://bugs.gnu.org/50361
Stop using "the word" when describing a match, to avoid the
implication of a connection with "word"-delimited (\b) matching.
Prefer to match "on lines ...", not "in lines".
The 'r' command can be used with address zero, effectively prepending
a file to the beginning of the input file, e.g.:
sed '0rA.TXT' B.TXT > C.TXT
is equivalent to:
cat A.TXT B.TXT > C.TXT
With "sed -i", this allows safe in-place prepending of files.
A typical example would be adding a license header to multiple source
files:
sed -i '0rLICENSE' *.c *.h
find -iname '*.cpp' | xargs sed -i '0rLICENSE'
A current cumbersome alternative is:
sed -i -e 'x;${p;x};1rA.TXT' -e '1d' B.TXT
* NEWS: Mention new feature.
* sed/sed.h (struct readcmd): New struct. (struct sed_cmd): Use new
struct instead of a char* for the filename.
* sed/compile.c (compile_program): Expand conditional detecting invalid
usage of "0" address to allow "0r"; Adjust '0r' to '1r' with prepending
(instead of appending).
* sed/execute.c (execute_program): 'r' command: support prepending.
* sed/debug.c (debug_print_function): Use the new 'struct readcmd'.
* testsuite/cmd-0r.sh: New test.
* testsuite/local.mk (TESTS): Add new test.
* doc/sed.texi (Zero Address): New section. (Adding a header to multiple
files): New example section.
Upon a failed temp file fclose, do_ck_fclose would call panic,
which would then attempt to fclose and unlink that same pointer.
Caught by gcc's new -Wanalyzer-double-fclose.
* sed/utils.c (struct open_file) [fclose_failed]: New member.
(panic): Don't double-close.
(register_open_file): Clear new member.
(mark_as_fclose_failed): Use new member to avoid double fclose.
(do_ck_fclose): Call mark_as_fclose_failed upon fclose failure.
* configure.ac: Avoid autoconf warnings:
Switch from obsolete AM_CONFIG_HEADER to AC_CONFIG_HEADERS, and quote.
Use AC_PROG_CC_STDC, not AC_PROG_CC.
Convert from obsolete AC_TRY_RUN to AC_RUN_IFELSE.
GCC compilation fails with:
sed/compile.c:206:3: error: format not a string literal, argument types
not checked [-Werror=format-nonliteral]
sprintf (unknown_cmd, msg, ch);
^~~~~~~
While there is suppression of "-Wformat-nonliteral" in configure.ac,
it later '-Wformat=2' which turns "-Wformat-nonliteral" on.
* sed/compile.c (bad_command): Add explicit "#pragma GCC ... ignore".
The format string is fixed and known ahead of time.
* build-aux/help2man: Update from coreutils.
This updates from 1.28 to 1.47.3
* cfg.mk (exclude_file_name_regexp--sc_long_lines):
Exempt help2man from the 80-column-max limit.
This addresses https://bugs.gnu.org/30471
It is possible to trigger an out of bounds memory access when
using the sed command R with an input file containing very long
lines.
The problem is that the line length of parsed file is returned as
a size_t by ck_getdelim, but temporarily stored in an int and
then converted back into a size_t. On systems like amd64, on which
this problem can be triggered, size_t and int have different sizes.
If the input line is longer than 2 GB (which is parseable on amd64
or other 64 bit systems), this means that the temporarily stored
int turns negative. Converting the negative int back into a size_t
will lead to an excessively large size_t, as the conversion leads to
a lot of leading 1 bits.
Eventually ck_fwrite is called with this huge size_t which in turn
will lead to an out of bounds access on amd64 systems -- after all
the parsed text was just a bit above 2 GB, not near SIZE_MAX.
You can trigger this issue with GNU sed on OpenBSD like this:
$ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > long.txt
$ sed Rlong.txt /etc/fstab
Segmentation fault (core dumped)
I was unable to trigger the bug on a Linux system with glibc due to
a bug in glibc's fwrite implementation -- it leads to a short write
and sed treats that correctly as an error.
* sed/execute.c (execute_program) [case 'R']: Declare result
to be of type size_t, not int.
* NEWS (Bug fixes): Mention it.
This addresses https://bugs.gnu.org/39166
If sed is called with an excessively long execution line, then it is
prone to an out of bounds memory access.
The problem is that the length of the execution line, which is a
size_t, is temporarily stored in an int. This means that on systems
which have a 64 bit size_t and a 32 bit int (e.g. linux amd64) an
execution line which exceeds 2 GB will overflow int. If it is just
slightly larger than 2 GB, the negative int value is used as an
array index to finish the execution line string with '\0' which
therefore triggers the out of bounds access.
This problem is probably never triggered in reality, but can be
provoked like this (given that 'e' support is compiled in):
$ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > e-command.txt
$ sed -f e-command.txt /etc/fstab
Segmentation fault (core dumped)
Also adjust another int/size_t conversion, even though it is a
purely cosmetic change, because it can never be larger than 4096.
* sed/execute.c (execute_program) [case 'e']: Declare cmd_length
to be of type size_t, not int. Likewise for "n" just below.
* NEWS (Bug fixes): Mention it.
This addresses https://bugs.gnu.org/39165
* testsuite/bug32271-2.sh: Filter out lines starting with "==".
This avoids unwarranted failures on e.g., windows systems that
would emit this to stderr:
+==16281== error calling PR_SET_PTRACER, vgdb might block
* testsuite/newline-dfa-bug.sh: Likewise.
Reported by Vagelis Prokopiou on Windows subsystem for Linux
(Debian 10) in https://lists.gnu.org/r/sed-devel/2020-01/msg00018.html
and by Bruno Haible for mingw in
https://lists.gnu.org/r/sed-devel/2020-01/msg00005.html
* testsuite/title-case.sh: Correct printf usage not to trigger
unspecified behavior for '\(\)'. Instead, just use sed with its -r
option so those parentheses don't need any backslash at all.
Reported by Bruno Haible in
https://lists.gnu.org/r/sed-devel/2020-01/msg00008.html
Before this change, the two perl-requiring tests would fail
on systems without perl. Now, they are skipped, as intended.
* bootstrap.conf (gnulib_modules): Add perl.
* testsuite/no-perl: New file. From coreutils.
* testsuite/local.mk (EXTRA_DIST): Add it.
(TESTSUITE_PERL): Include testsuite/ prefix.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* bootstrap.conf (gnulib_modules): Remove explicit mention of threadlib.
It was added back in 2012 to avoid a bootstrap failure; that explicit
mention has long been unnecessary.
"sed -i" now creates temporary files with correct umask (limited to u=rwx).
Previously sed would incorrectly set umask, and combined with mkostemp
creating file with mode 0600, the result would be a file with
permission mode 0.
Reported by Dr N.W. Filardo <nwf20@cam.ac.uk>:
https://lists.gnu.org/r/sed-devel/2019-08/msg00000.html
"The net effect is that this patch does not do what it says on the tin:
it does not improve the security story at all. Things continue to
function because the subsequent operations are via f*() APIs, which
take the open file handle, and in particular fchmod() will put the
bits back to something sensible.
However, when running atop, for example, fuse-style filesystems which do
not keep open descriptors to underlying files, this is catastrophic:
the underlying file will have I_SRWXU of zero, and so the filesystem
server will be unable to open the file for the fchmod() and that's
the end of that."
"fuse-overlayfs" is an example of a filesystem with such issues.
This change was made in commit 5156c19b23
and was released in sed 4.2.1.
* NEWS: Mention change.
* sed/utils.c (ck_mkstemp): Set correct umask.
Avoid false-positive failure on system with SELinux enabled when SED
is built without SELinux suppport.
Reported by Jeffrey Walton <noloader@gmail.com> in
https://bugs.gnu.org/35997 .
* testsuite/inplace-selinux.sh: Call 'sed --version' to check if sed
was built with SELinux support before continuing with the test.
Report whether sed was built with SELinux support (determined at compile
time), and whether it is enabled on this system (determined at runtime).
Relates to https://bugs.gnu.org/35997 .
* sed/sed.c (selinux_support): New function, print SELinux status to
stdout. (main): Call selinux_support on --version.
Cause developer builds to fail for any use of a VLA.
VLAs (variable length arrays) limit portability.
* configure.ac (nw): Remove -Wvla from the list of disabled warnings,
thus enabling the warning when configured with --enable-gcc-warnings.
(GNULIB_NO_VLA) Define, disabling use of VLAs in gnulib. This commit
is functionally equivalent to coreutils' v8.30-44-gd26dece5d.
Under some conditions, \b would mistakenly fail to match. E.g.,
this would mistakenly print "123-x" instead of "123":
echo 123-x|LC_ALL=C sed 's/.\bx//'
* NEWS (Bug fixes): Mention it
* gnulib: Update to latest, for DFA regression fix.
* testsuite/word-delim.sh: New file, to test for the dfa.c regression.
* testsuite/local.mk (T): Add it.
Reported by Jan Palus in
https://lists.gnu.org/r/sed-devel/2018-12/msg00022.html