Commit Graph
633 Commits
Author SHA1 Message Date
Assaf Gordon fb0c57d55e sed: silence -Wformat-nonoliteral warning
GCC compilation fails with:
  sed/compile.c:206:3: error: format not a string literal, argument types
                       not checked [-Werror=format-nonliteral]
    sprintf (unknown_cmd, msg, ch);
             ^~~~~~~
While there is suppression of "-Wformat-nonliteral" in configure.ac,
it later '-Wformat=2' which turns "-Wformat-nonliteral" on.

* sed/compile.c (bad_command): Add explicit "#pragma GCC ... ignore".
The format string is fixed and known ahead of time.
2021-01-05 17:02:41 -07:00
Assaf Gordon 71de42fd34 maint: update all copyright year number ranges
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* bootstrap.conf (gnulib_modules): Remove getopt module, depracated in
https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=0abc38bd2a1398f0
* po/POTFILES.in: Remove getopt.c
* sed/utils.h (panic): Rename _GL_ATTRIBUTE_FORMAT_PRINTF to
_GL_ATTRIBUTE_FORMAT_PRINTF_STANDARD following gnulib's change:
https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=387d654cabd7bc15
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
2021-01-05 17:02:10 -07:00
Jim Meyering 36e24f199f build: update to newer help2man
* build-aux/help2man: Update from coreutils.
This updates from 1.28 to 1.47.3
* cfg.mk (exclude_file_name_regexp--sc_long_lines):
Exempt help2man from the 80-column-max limit.
This addresses https://bugs.gnu.org/30471
2020-03-23 22:35:39 -07:00
Tobias Stoeckmann acabfdb582 sed: handle very long input lines with R (tiny change)
It is possible to trigger an out of bounds memory access when
using the sed command R with an input file containing very long
lines.

The problem is that the line length of parsed file is returned as
a size_t by ck_getdelim, but temporarily stored in an int and
then converted back into a size_t. On systems like amd64, on which
this problem can be triggered, size_t and int have different sizes.

If the input line is longer than 2 GB (which is parseable on amd64
or other 64 bit systems), this means that the temporarily stored
int turns negative. Converting the negative int back into a size_t
will lead to an excessively large size_t, as the conversion leads to
a lot of leading 1 bits.

Eventually ck_fwrite is called with this huge size_t which in turn
will lead to an out of bounds access on amd64 systems -- after all
the parsed text was just a bit above 2 GB, not near SIZE_MAX.

You can trigger this issue with GNU sed on OpenBSD like this:

$ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > long.txt
$ sed Rlong.txt /etc/fstab
Segmentation fault (core dumped)

I was unable to trigger the bug on a Linux system with glibc due to
a bug in glibc's fwrite implementation -- it leads to a short write
and sed treats that correctly as an error.

* sed/execute.c (execute_program) [case 'R']: Declare result
to be of type size_t, not int.
* NEWS (Bug fixes): Mention it.
This addresses https://bugs.gnu.org/39166
2020-01-18 08:25:10 -08:00
Tobias Stoeckmann 61b5e58f18 sed: handle very long execution lines (tiny change)
If sed is called with an excessively long execution line, then it is
prone to an out of bounds memory access.

The problem is that the length of the execution line, which is a
size_t, is temporarily stored in an int. This means that on systems
which have a 64 bit size_t and a 32 bit int (e.g. linux amd64) an
execution line which exceeds 2 GB will overflow int. If it is just
slightly larger than 2 GB, the negative int value is used as an
array index to finish the execution line string with '\0' which
therefore triggers the out of bounds access.

This problem is probably never triggered in reality, but can be
provoked like this (given that 'e' support is compiled in):

$ dd if=/dev/zero bs=1M count=2049 | tr '\0' e > e-command.txt
$ sed -f e-command.txt /etc/fstab
Segmentation fault (core dumped)

Also adjust another int/size_t conversion, even though it is a
purely cosmetic change, because it can never be larger than 4096.

* sed/execute.c (execute_program) [case 'e']: Declare cmd_length
to be of type size_t, not int.  Likewise for "n" just below.
* NEWS (Bug fixes): Mention it.
This addresses https://bugs.gnu.org/39165
2020-01-18 08:25:10 -08:00
Jim Meyering 8827762c4a maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2020-01-14 20:10:42 -08:00
Jim Meyering 6205998563 version 4.8
* NEWS: Record release date.
v4.8
2020-01-14 20:08:55 -08:00
Jim Meyering ca9be2751e doc: mention changes inherited through gnulib
* NEWS (Improvements): Mention gnulib.
2020-01-14 20:08:45 -08:00
Jim Meyering 5d1590baaf build: update gnulib to latest (for updated perl.m4) 2020-01-11 21:09:23 -08:00
Jim Meyering 8f2f1b7182 tests: remove valgrind-added diagnostics from stderr
* testsuite/bug32271-2.sh: Filter out lines starting with "==".
This avoids unwarranted failures on e.g., windows systems that
would emit this to stderr:
+==16281== error calling PR_SET_PTRACER, vgdb might block
* testsuite/newline-dfa-bug.sh: Likewise.
Reported by Vagelis Prokopiou on Windows subsystem for Linux
(Debian 10) in https://lists.gnu.org/r/sed-devel/2020-01/msg00018.html
and by Bruno Haible for mingw in
https://lists.gnu.org/r/sed-devel/2020-01/msg00005.html
2020-01-11 20:49:51 -08:00
Jim Meyering def38eb00e sed: avoid MSVC build failure
* sed/sed.c (usage): Remove unnecessary forward declaration.
Reported by Bruno Haible in
https://lists.gnu.org/r/sed-devel/2020-01/msg00007.html
2020-01-11 19:24:08 -08:00
Jim Meyering 5766a98647 tests: avoid hpux11 test failure
* testsuite/title-case.sh: Correct printf usage not to trigger
unspecified behavior for '\(\)'. Instead, just use sed with its -r
option so those parentheses don't need any backslash at all.
Reported by Bruno Haible in
https://lists.gnu.org/r/sed-devel/2020-01/msg00008.html
2020-01-11 16:18:50 -08:00
Jim Meyering 598c3a0dd9 tests: handle missing perl
Before this change, the two perl-requiring tests would fail
on systems without perl. Now, they are skipped, as intended.
* bootstrap.conf (gnulib_modules): Add perl.
* testsuite/no-perl: New file. From coreutils.
* testsuite/local.mk (EXTRA_DIST): Add it.
(TESTSUITE_PERL): Include testsuite/ prefix.
2020-01-09 22:56:39 -08:00
Jim Meyering 6c175df230 maint: once again distribute .tar.gz files
* configure.ac: Reenable distribution of gzip-compressed
tarballs, on request from Jan Nieuwenhuizen in
https://lists.gnu.org/r/sed-devel/2020-01/msg00013.html
* NEWS (Release): Mention this.
2020-01-06 16:55:02 -08:00
Jim Meyering 3c1e83363b maint: space-before-open-paren: avoid syntax-check error
* sed/sed.c (selinux_support): Insert space before open-paren.
2020-01-05 13:47:31 -08:00
Jim Meyering 174718e3c1 build: update gnulib to latest (for thread-related test fixes)
* .gitignore: Add /lib/inttypes.h.
2020-01-05 13:44:17 -08:00
Jim Meyering 623f5cd2f9 maint: update all copyright year number ranges
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
2020-01-01 11:38:46 -08:00
Jim Meyering f8ab99468d maint: remove explicit mention of gnulib's threadlib module
* bootstrap.conf (gnulib_modules): Remove explicit mention of threadlib.
It was added back in 2012 to avoid a bootstrap failure; that explicit
mention has long been unnecessary.
2020-01-01 11:37:23 -08:00
Assaf Gordon f69b085d3e sed: set correct umask on temporary files
"sed -i" now creates temporary files with correct umask (limited to u=rwx).
Previously sed would incorrectly set umask, and combined with mkostemp
creating file with mode 0600, the result would be a file with
permission mode 0.

Reported by Dr N.W. Filardo <nwf20@cam.ac.uk>:
https://lists.gnu.org/r/sed-devel/2019-08/msg00000.html
  "The net effect is that this patch does not do what it says on the tin:
   it does not improve the security story at all.  Things continue to
   function because the subsequent operations are via f*() APIs, which
   take the open file handle, and in particular fchmod() will put the
   bits back to something sensible.

   However, when running atop, for example, fuse-style filesystems which do
   not keep open descriptors to underlying files, this is catastrophic:
   the underlying file will have I_SRWXU of zero, and so the filesystem
   server will be unable to open the file for the fchmod() and that's
   the end of that."

"fuse-overlayfs" is an example of a filesystem with such issues.
This change was made in commit 5156c19b23
and was released in sed 4.2.1.

* NEWS: Mention change.
* sed/utils.c (ck_mkstemp): Set correct umask.
2019-09-05 13:10:29 -06:00
Assaf Gordon a9cb52bcf3 tests: inplace-selinux.sh: skip if SELinux is not supported
Avoid false-positive failure on system with SELinux enabled when SED
is built without SELinux suppport.

Reported by Jeffrey Walton <noloader@gmail.com> in
https://bugs.gnu.org/35997 .

* testsuite/inplace-selinux.sh: Call 'sed --version' to check if sed
was built with SELinux support before continuing with the test.
2019-06-02 01:14:00 -06:00
Assaf Gordon 07ef8a3138 sed: report SELinux status in --version output
Report whether sed was built with SELinux support (determined at compile
time), and whether it is enabled on this system (determined at runtime).

Relates to https://bugs.gnu.org/35997 .

* sed/sed.c (selinux_support): New function, print SELinux status to
stdout. (main): Call selinux_support on --version.
2019-06-02 01:11:19 -06:00
Jim Meyering 2a7125a8b3 build: ensure no VLA is used
Cause developer builds to fail for any use of a VLA.
VLAs (variable length arrays) limit portability.
* configure.ac (nw): Remove -Wvla from the list of disabled warnings,
thus enabling the warning when configured with --enable-gcc-warnings.
(GNULIB_NO_VLA) Define, disabling use of VLAs in gnulib.  This commit
is functionally equivalent to coreutils' v8.30-44-gd26dece5d.
2019-01-20 22:22:45 -08:00
Jim Meyering c728a3fcb1 build: update gnulib to latest 2019-01-20 22:22:45 -08:00
Jim Meyering f850369d8c build: update gnulib to latest 2019-01-01 19:09:57 -08:00
Assaf Gordon 07df2c2e61 maint: update copyright dates for 2019
* all files: Run "make update-copyright".
2019-01-01 14:46:27 -07:00
Jim Meyering 8c60cebe0b maint: use https: in gnu mirror URL prefix, not http
This appears in the generated release announcement message.
* cfg.mk (url_dir_list): Use https: prefix, not http:.
2018-12-20 22:21:40 -08:00
Jim Meyering c0a4f0643f maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2018-12-20 22:04:03 -08:00
Jim Meyering 249cbda41c version 4.7
* NEWS: Record release date.
v4.7
2018-12-20 22:02:51 -08:00
Jim Meyering 799ce52011 sed: fix \b DFA-bug in C locale
Under some conditions, \b would mistakenly fail to match. E.g.,
this would mistakenly print "123-x" instead of "123":
  echo 123-x|LC_ALL=C sed 's/.\bx//'
* NEWS (Bug fixes): Mention it
* gnulib: Update to latest, for DFA regression fix.
* testsuite/word-delim.sh: New file, to test for the dfa.c regression.
* testsuite/local.mk (T): Add it.
Reported by Jan Palus in
https://lists.gnu.org/r/sed-devel/2018-12/msg00022.html
2018-12-20 21:23:11 -08:00
Jim Meyering 04eef69746 maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2018-12-19 23:25:32 -08:00
Jim Meyering 4383aeba93 version 4.6
* NEWS: Record release date.
v4.6
2018-12-19 23:23:46 -08:00
Jim Meyering 99f0df6c3d build: update gnulib to latest for linux/sparc c-stack fix 2018-12-17 06:57:28 -08:00
Jim Meyering 75b06ca114 build: update gnulib to latest for config.sub fix, etc.
* gnulib: Update for build- and test-related fixes,
including the config.sub fix for Solaris 10:
https://lists.gnu.org/r/grep-devel/2018-12/msg00013.html
* bootstrap: Update to latest.
2018-12-16 11:31:41 -08:00
Assaf Gordon 5155e56775 tests: fix obinary test error due to 'wc' spaces
On BSD systems "wc -c" prefixes the count with spaces,
leading to test error.
Discussed in https://lists.gnu.org/r/sed-devel/2018-12/msg00014.html .

* testsuite/obinary.sh: Remove leading spaces from 'wc -c' output.
2018-12-13 15:59:44 -07:00
Jim Meyering 58eb65b263 build: update gnulib to latest 2018-12-12 22:11:29 -08:00
Assaf Gordon 6187fc0e3a tests: add text/binary mode tests
This test is skipped on most platforms (which do not support O_BINARY
and do not differentiate between TEXT and BINARY files).

On cygwin, the test is skipped because default cygwin installations
do not use O_TEXT/O_BINARY modes (unless file systems are mounted with
special options).

In practice this test will only run on native Windows (e.g. when compiled
with mingw or msvc). Note that almost every other test will fail with
such a binary (due to excesssive "\r" in the output). Thus it is
recommended to explictly run just this test alone:

   make check SUBDIRS=. TESTS=testsuite/obinary.sh

The script tries to cover all cases mentioned in https://bugs.gnu.org/25459 .
See also https://lists.gnu.org/r/sed-devel/2018-10/msg00001.html .

* testsuite/obinary.sh: New test.
* testsuite/local.mk (TESTS): Add new test.
2018-12-10 00:15:34 -07:00
Assaf Gordon 46b9427132 sed: fix -b/--binary mode under windows/mingw
Discussed in https://lists.gnu.org/r/sed-devel/2018-10/msg00001.html .

* NEWS: Mention change.
* bootstrap.conf: Add gnulib's binary-io module.
* sed/sed.c (main): Set stdin/stdout to binary mode if needed.
* sed/utils.c (ck_mkstemp): Explicitly set binary mode on file
descriptor. It seems that on (non-cygwin) Windows the fdopen(3) call
ignores the 'b' in the 'mode' argument - and the file was always opened
in O_TEXT mode. Thus "--binary" was not working with "--inplace".
2018-12-05 21:07:57 -07:00
Assaf Gordon 159675c030 maint: update .gitignore
Following gnulib recent update.
* lib/.gitignore, m4/.gitignore: Update file list.
2018-12-05 21:07:57 -07:00
Assaf Gordon 51eb713854 build: update autoconf version requirements
Require version 2.64 to bootstrap. This is already required in practice
by bootstrap.conf, but was not updated in configure.ac.
After recent gnulib update, gnulib-tool would complain:

  $ ./gnulib/gnulib-tool
  ./gnulib/gnulib-tool: *** minimum supported autoconf version is 2.63.\
      Try adding AC_PREREQ([2.63]) to your configure.ac.
  ./gnulib/gnulib-tool: *** Stop.

Hence this update.

* configure.ac: Require autoconf version 2.64 or later.
2018-12-05 21:07:50 -07:00
Assaf Gordon b307652592 gnulib: update to latest (for dfa improvements) 2018-12-05 14:00:50 -07:00
Jannick dae3abf2af sed: fix memory leakage under lint
The NULL-initialized char string in_place_extension is free'ed everytime
it is redefined (using xstrdup) and at program exit with any return code.

See: https://lists.gnu.org/r/sed-devel/2018-11/msg00005.html

* sed/sed.c (main, cleanup): Free 'in_place_extension' if running with
lint.
2018-11-22 13:21:38 -07:00
Assaf Gordon 76ca1d2c53 build: rename stdin local variable
Under windows/msys2, defining 'bool stdin' variable results in
compilation errors:

  ../sed/execute.c:1252:8: error: expected declaration specifiers or '...'
  before numeric constant
   bool stdin = (input->fp && fileno (input->fp) == 0);
        ^~~~~

Reported by Jannick <thirdedition@gmx.net> in
https://lists.gnu.org/archive/html/sed-devel/2018-10/msg00019.html .

* sed/execute.c (debug_print_input): Rename 'stdin' to 'is_stdin'.
2018-11-01 10:06:47 -06:00
Jim Meyering 4708cf368a maint: fix a syntax-check nit
* sed/compile.c (MISSING_FILENAME): Insert a space before "(", and
in a following comment.
2018-10-28 04:21:20 -07:00
Jim Meyering 1c0866b1d3 tests: avoid unnecessary dependency on Data::Dump
* testsuite/debug.pl: Don't use Data::Dump.  Unnecessary.
That package, perl-Data-Dump, appear not to be default-installed
on at least a Fedora 29 beta system.
2018-10-27 15:53:03 -07:00
Jim Meyering 2bfa7984c8 sed: avoid UMR in --debug code path
* sed/debug.c (debug_print_function) [b, t, T]: For a b, t or T
command with no LABEL, do not access uninitialized memory.
I.e., print the label name only when there is one.
2018-10-27 15:47:41 -07:00
Assaf Gordon 3ac640ce3b sed: add --debug feature
$ seq 3 | sed --debug -e 's/./--&--/ ; 2d'
    SED PROGRAM:
      s/./--&--/
      2 d
    INPUT:   'STDIN' line 1
    PATTERN: 1
    COMMAND: s/./--&--/
    MATCHED REGEX REGISTERS
      regex[0] = 0-1 '1'
    PATTERN: --1--
    COMMAND: 2 d
    END-OF-CYCLE:
    --1--
    INPUT:   'STDIN' line 2
    PATTERN: 2
    COMMAND: s/./--&--/
    MATCHED REGEX REGISTERS
      regex[0] = 0-1 '2'
    PATTERN: --2--
    COMMAND: 2 d
    END-OF-CYCLE:
    INPUT:   'STDIN' line 3
    PATTERN: 3
    COMMAND: s/./--&--/
    MATCHED REGEX REGISTERS
      regex[0] = 0-1 '3'
    PATTERN: --3--
    COMMAND: 2 d
    END-OF-CYCLE:
    --3--

Discussed in https://lists.gnu.org/r/sed-devel/2018-07/msg00006.html
and https://lists.gnu.org/r/sed-devel/2018-10/msg00007.html .

* NEWS: Mention new option.
* doc/sed.texi (Program options): Mention new option.
* sed/debug.c: New unit with debug printing functions.
* sed/sed.h (debug_print_command, debug_print_char, debug_print_program,
debug): Declare functions and global variable.
(struct sed_cmd): Add label_name member variable.
* sed/compile.c (compile_program): Save the label's name.
(cleanup_program_filenames): extracted function to free filenames.
(check_final_program) Don't delete the filenames, instead move it to ...
(finish_program) ... here.
* sed/execute.c (debug_print_end_of_cycle, debug_print_input,
debug_print_line): New debug functions (cannot be defined in debug.c as
execute's structures are private).
(execute_program, process_files): Call debug functions.
* sed/sed.c: (DEBUG_OPTION): New option for getoptlong.
(debug): New global variable.
(usage): Mention new option.
(main): Process new option and call debug functions if needed.
* testsuite/debug.pl: New tests.
* testsuite/local.mk (T): Add new tests.
2018-10-24 14:00:23 -06:00
Assaf Gordon f649e21458 sed: change internal storage for 'R' command
For the 'R' command, keep a pointer to 'struct output' instead of just
to FILE*. No change in functionality. This will help future debug code
to know the filename of the associated file.

As a side effect, fix invalid code in W/w execution:
The previous code checked for 'cur_cmd->x.fp' (which was related to R
command, not W/w). Since 'x' is a union, 'x.fp' was not relevant to
the 'x.outf' (which is the struct associted with W/w commands).

* sed/sed.h (struct sed_cmd): Replace 'FILE*' with 'struct *output' for
R command.
* sed/compile.c (compile_program): Adjust as needed.
* sed/execute.c (execute_program): Adjust as needed.
2018-10-19 10:37:32 -06:00
Assaf Gordon 38e9a954ff sed: reject r/R/w/W commands without a filename
w/W (and s///w) commands Without a filename would print a confusing error
message:
  $ sed w
  sed: couldn't open file : No such file or directory
While r/R commands with empty file name were a silent no-op.

With this change, sed programs with empty filename are rejected with a
clear error:

  $ sed 's/1/2/w'
  sed: -e expression #1, char 7: missing filename in r/R/w/W commands
  $ sed r
  sed: -e expression #1, char 1: missing filename in r/R/w/W commands

* NEWS: Mention change.
* sed/compile.c (get_openfile): Exit with an error message if filename
is missing. (compile_program): Same for 'r' command code.
* testsuite/missing-filename.sh: New test.
* testsuite/local.mk (T): Add new test.
2018-10-18 04:44:19 -06:00
Clint Adams 8e52c0aff0 doc: use @key{TAB} in texinfo
Discussed in https://bugs.gnu.org/22636 .

* doc/sed.texi: Use '@kbd{@key{TAB}}' instead of '@kbd{tab}' to improve
rendering in HTML and info formats.
Copyright-paperwork-exempt: Yes
2018-10-12 11:45:56 -06:00
Bjarni Ingi Gislason 19089418c6 doc: sed.x: some formatting corrections
Reported in https://bugs.gnu.org/30479 .

doc/sed.x: Change a HYPHEN-MINUS (code 0x55, 2D) to a dash (\-, minus)
if it matches " -[:alpha:]" or \(aq-[:alpha:] (for options); Use a pronoun
instead of a repeated noun.
Copyright-paperwork-exempt: Yes
2018-10-12 11:45:56 -06:00