GCC compilation fails with:
sed/compile.c:206:3: error: format not a string literal, argument types
not checked [-Werror=format-nonliteral]
sprintf (unknown_cmd, msg, ch);
^~~~~~~
While there is suppression of "-Wformat-nonliteral" in configure.ac,
it later '-Wformat=2' which turns "-Wformat-nonliteral" on.
* sed/compile.c (bad_command): Add explicit "#pragma GCC ... ignore".
The format string is fixed and known ahead of time.
* build-aux/help2man: Update from coreutils.
This updates from 1.28 to 1.47.3
* cfg.mk (exclude_file_name_regexp--sc_long_lines):
Exempt help2man from the 80-column-max limit.
This addresses https://bugs.gnu.org/30471
It is possible to trigger an out of bounds memory access when
using the sed command R with an input file containing very long
lines.
The problem is that the line length of parsed file is returned as
a size_t by ck_getdelim, but temporarily stored in an int and
then converted back into a size_t. On systems like amd64, on which
this problem can be triggered, size_t and int have different sizes.
If the input line is longer than 2 GB (which is parseable on amd64
or other 64 bit systems), this means that the temporarily stored
int turns negative. Converting the negative int back into a size_t
will lead to an excessively large size_t, as the conversion leads to
a lot of leading 1 bits.
Eventually ck_fwrite is called with this huge size_t which in turn
will lead to an out of bounds access on amd64 systems -- after all
the parsed text was just a bit above 2 GB, not near SIZE_MAX.
You can trigger this issue with GNU sed on OpenBSD like this:
$ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > long.txt
$ sed Rlong.txt /etc/fstab
Segmentation fault (core dumped)
I was unable to trigger the bug on a Linux system with glibc due to
a bug in glibc's fwrite implementation -- it leads to a short write
and sed treats that correctly as an error.
* sed/execute.c (execute_program) [case 'R']: Declare result
to be of type size_t, not int.
* NEWS (Bug fixes): Mention it.
This addresses https://bugs.gnu.org/39166
If sed is called with an excessively long execution line, then it is
prone to an out of bounds memory access.
The problem is that the length of the execution line, which is a
size_t, is temporarily stored in an int. This means that on systems
which have a 64 bit size_t and a 32 bit int (e.g. linux amd64) an
execution line which exceeds 2 GB will overflow int. If it is just
slightly larger than 2 GB, the negative int value is used as an
array index to finish the execution line string with '\0' which
therefore triggers the out of bounds access.
This problem is probably never triggered in reality, but can be
provoked like this (given that 'e' support is compiled in):
$ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > e-command.txt
$ sed -f e-command.txt /etc/fstab
Segmentation fault (core dumped)
Also adjust another int/size_t conversion, even though it is a
purely cosmetic change, because it can never be larger than 4096.
* sed/execute.c (execute_program) [case 'e']: Declare cmd_length
to be of type size_t, not int. Likewise for "n" just below.
* NEWS (Bug fixes): Mention it.
This addresses https://bugs.gnu.org/39165
* testsuite/bug32271-2.sh: Filter out lines starting with "==".
This avoids unwarranted failures on e.g., windows systems that
would emit this to stderr:
+==16281== error calling PR_SET_PTRACER, vgdb might block
* testsuite/newline-dfa-bug.sh: Likewise.
Reported by Vagelis Prokopiou on Windows subsystem for Linux
(Debian 10) in https://lists.gnu.org/r/sed-devel/2020-01/msg00018.html
and by Bruno Haible for mingw in
https://lists.gnu.org/r/sed-devel/2020-01/msg00005.html
* testsuite/title-case.sh: Correct printf usage not to trigger
unspecified behavior for '\(\)'. Instead, just use sed with its -r
option so those parentheses don't need any backslash at all.
Reported by Bruno Haible in
https://lists.gnu.org/r/sed-devel/2020-01/msg00008.html
Before this change, the two perl-requiring tests would fail
on systems without perl. Now, they are skipped, as intended.
* bootstrap.conf (gnulib_modules): Add perl.
* testsuite/no-perl: New file. From coreutils.
* testsuite/local.mk (EXTRA_DIST): Add it.
(TESTSUITE_PERL): Include testsuite/ prefix.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* bootstrap.conf (gnulib_modules): Remove explicit mention of threadlib.
It was added back in 2012 to avoid a bootstrap failure; that explicit
mention has long been unnecessary.
"sed -i" now creates temporary files with correct umask (limited to u=rwx).
Previously sed would incorrectly set umask, and combined with mkostemp
creating file with mode 0600, the result would be a file with
permission mode 0.
Reported by Dr N.W. Filardo <nwf20@cam.ac.uk>:
https://lists.gnu.org/r/sed-devel/2019-08/msg00000.html
"The net effect is that this patch does not do what it says on the tin:
it does not improve the security story at all. Things continue to
function because the subsequent operations are via f*() APIs, which
take the open file handle, and in particular fchmod() will put the
bits back to something sensible.
However, when running atop, for example, fuse-style filesystems which do
not keep open descriptors to underlying files, this is catastrophic:
the underlying file will have I_SRWXU of zero, and so the filesystem
server will be unable to open the file for the fchmod() and that's
the end of that."
"fuse-overlayfs" is an example of a filesystem with such issues.
This change was made in commit 5156c19b23
and was released in sed 4.2.1.
* NEWS: Mention change.
* sed/utils.c (ck_mkstemp): Set correct umask.
Avoid false-positive failure on system with SELinux enabled when SED
is built without SELinux suppport.
Reported by Jeffrey Walton <noloader@gmail.com> in
https://bugs.gnu.org/35997 .
* testsuite/inplace-selinux.sh: Call 'sed --version' to check if sed
was built with SELinux support before continuing with the test.
Report whether sed was built with SELinux support (determined at compile
time), and whether it is enabled on this system (determined at runtime).
Relates to https://bugs.gnu.org/35997 .
* sed/sed.c (selinux_support): New function, print SELinux status to
stdout. (main): Call selinux_support on --version.
Cause developer builds to fail for any use of a VLA.
VLAs (variable length arrays) limit portability.
* configure.ac (nw): Remove -Wvla from the list of disabled warnings,
thus enabling the warning when configured with --enable-gcc-warnings.
(GNULIB_NO_VLA) Define, disabling use of VLAs in gnulib. This commit
is functionally equivalent to coreutils' v8.30-44-gd26dece5d.
Under some conditions, \b would mistakenly fail to match. E.g.,
this would mistakenly print "123-x" instead of "123":
echo 123-x|LC_ALL=C sed 's/.\bx//'
* NEWS (Bug fixes): Mention it
* gnulib: Update to latest, for DFA regression fix.
* testsuite/word-delim.sh: New file, to test for the dfa.c regression.
* testsuite/local.mk (T): Add it.
Reported by Jan Palus in
https://lists.gnu.org/r/sed-devel/2018-12/msg00022.html
This test is skipped on most platforms (which do not support O_BINARY
and do not differentiate between TEXT and BINARY files).
On cygwin, the test is skipped because default cygwin installations
do not use O_TEXT/O_BINARY modes (unless file systems are mounted with
special options).
In practice this test will only run on native Windows (e.g. when compiled
with mingw or msvc). Note that almost every other test will fail with
such a binary (due to excesssive "\r" in the output). Thus it is
recommended to explictly run just this test alone:
make check SUBDIRS=. TESTS=testsuite/obinary.sh
The script tries to cover all cases mentioned in https://bugs.gnu.org/25459 .
See also https://lists.gnu.org/r/sed-devel/2018-10/msg00001.html .
* testsuite/obinary.sh: New test.
* testsuite/local.mk (TESTS): Add new test.
Discussed in https://lists.gnu.org/r/sed-devel/2018-10/msg00001.html .
* NEWS: Mention change.
* bootstrap.conf: Add gnulib's binary-io module.
* sed/sed.c (main): Set stdin/stdout to binary mode if needed.
* sed/utils.c (ck_mkstemp): Explicitly set binary mode on file
descriptor. It seems that on (non-cygwin) Windows the fdopen(3) call
ignores the 'b' in the 'mode' argument - and the file was always opened
in O_TEXT mode. Thus "--binary" was not working with "--inplace".
Require version 2.64 to bootstrap. This is already required in practice
by bootstrap.conf, but was not updated in configure.ac.
After recent gnulib update, gnulib-tool would complain:
$ ./gnulib/gnulib-tool
./gnulib/gnulib-tool: *** minimum supported autoconf version is 2.63.\
Try adding AC_PREREQ([2.63]) to your configure.ac.
./gnulib/gnulib-tool: *** Stop.
Hence this update.
* configure.ac: Require autoconf version 2.64 or later.
The NULL-initialized char string in_place_extension is free'ed everytime
it is redefined (using xstrdup) and at program exit with any return code.
See: https://lists.gnu.org/r/sed-devel/2018-11/msg00005.html
* sed/sed.c (main, cleanup): Free 'in_place_extension' if running with
lint.
* testsuite/debug.pl: Don't use Data::Dump. Unnecessary.
That package, perl-Data-Dump, appear not to be default-installed
on at least a Fedora 29 beta system.
* sed/debug.c (debug_print_function) [b, t, T]: For a b, t or T
command with no LABEL, do not access uninitialized memory.
I.e., print the label name only when there is one.
$ seq 3 | sed --debug -e 's/./--&--/ ; 2d'
SED PROGRAM:
s/./--&--/
2 d
INPUT: 'STDIN' line 1
PATTERN: 1
COMMAND: s/./--&--/
MATCHED REGEX REGISTERS
regex[0] = 0-1 '1'
PATTERN: --1--
COMMAND: 2 d
END-OF-CYCLE:
--1--
INPUT: 'STDIN' line 2
PATTERN: 2
COMMAND: s/./--&--/
MATCHED REGEX REGISTERS
regex[0] = 0-1 '2'
PATTERN: --2--
COMMAND: 2 d
END-OF-CYCLE:
INPUT: 'STDIN' line 3
PATTERN: 3
COMMAND: s/./--&--/
MATCHED REGEX REGISTERS
regex[0] = 0-1 '3'
PATTERN: --3--
COMMAND: 2 d
END-OF-CYCLE:
--3--
Discussed in https://lists.gnu.org/r/sed-devel/2018-07/msg00006.html
and https://lists.gnu.org/r/sed-devel/2018-10/msg00007.html .
* NEWS: Mention new option.
* doc/sed.texi (Program options): Mention new option.
* sed/debug.c: New unit with debug printing functions.
* sed/sed.h (debug_print_command, debug_print_char, debug_print_program,
debug): Declare functions and global variable.
(struct sed_cmd): Add label_name member variable.
* sed/compile.c (compile_program): Save the label's name.
(cleanup_program_filenames): extracted function to free filenames.
(check_final_program) Don't delete the filenames, instead move it to ...
(finish_program) ... here.
* sed/execute.c (debug_print_end_of_cycle, debug_print_input,
debug_print_line): New debug functions (cannot be defined in debug.c as
execute's structures are private).
(execute_program, process_files): Call debug functions.
* sed/sed.c: (DEBUG_OPTION): New option for getoptlong.
(debug): New global variable.
(usage): Mention new option.
(main): Process new option and call debug functions if needed.
* testsuite/debug.pl: New tests.
* testsuite/local.mk (T): Add new tests.
For the 'R' command, keep a pointer to 'struct output' instead of just
to FILE*. No change in functionality. This will help future debug code
to know the filename of the associated file.
As a side effect, fix invalid code in W/w execution:
The previous code checked for 'cur_cmd->x.fp' (which was related to R
command, not W/w). Since 'x' is a union, 'x.fp' was not relevant to
the 'x.outf' (which is the struct associted with W/w commands).
* sed/sed.h (struct sed_cmd): Replace 'FILE*' with 'struct *output' for
R command.
* sed/compile.c (compile_program): Adjust as needed.
* sed/execute.c (execute_program): Adjust as needed.
w/W (and s///w) commands Without a filename would print a confusing error
message:
$ sed w
sed: couldn't open file : No such file or directory
While r/R commands with empty file name were a silent no-op.
With this change, sed programs with empty filename are rejected with a
clear error:
$ sed 's/1/2/w'
sed: -e expression #1, char 7: missing filename in r/R/w/W commands
$ sed r
sed: -e expression #1, char 1: missing filename in r/R/w/W commands
* NEWS: Mention change.
* sed/compile.c (get_openfile): Exit with an error message if filename
is missing. (compile_program): Same for 'r' command code.
* testsuite/missing-filename.sh: New test.
* testsuite/local.mk (T): Add new test.
Discussed in https://bugs.gnu.org/22636 .
* doc/sed.texi: Use '@kbd{@key{TAB}}' instead of '@kbd{tab}' to improve
rendering in HTML and info formats.
Copyright-paperwork-exempt: Yes
Reported in https://bugs.gnu.org/30479 .
doc/sed.x: Change a HYPHEN-MINUS (code 0x55, 2D) to a dash (\-, minus)
if it matches " -[:alpha:]" or \(aq-[:alpha:] (for options); Use a pronoun
instead of a repeated noun.
Copyright-paperwork-exempt: Yes