Commit Graph
565 Commits
Author SHA1 Message Date
Assaf Gordon 293d1fd372 maint: syntax-check: prohibit-form-feed
* cfg.mk (sc_prohibit-form-feed): Copied from coreutils' cfg.mk.
* sed/compile.c, sed/execute.c, sed/regexp.c, sed/sed.h, sed/utils.c:
Remove form feed characters.
2018-08-04 22:09:57 -06:00
Assaf Gordon 08b9f66a0c maint: syntax-check: space-before-open-paren
* cfg.mk (sc_space_before_open_paren): Copy target from coreutils' cfg.mk.
* basicdefs.h, sed/compile.c, sed/execute.c, sed/regexp.c,
sed/sed.c, sed/utils.c: Add space before parentheses.
2018-08-04 22:09:57 -06:00
Assaf Gordon 06fe083ed2 maint: syntax-check: fix tab indentation
* sed/execute.c (open_next_file): Replace tab with spaces.
2018-08-04 22:09:41 -06:00
Assaf Gordon 5491c2bf85 sed: replace myname with gnulib's progname
* compile.c (bad_prog): Use program_name instead of myname.
* execute.c (open_next_file): Same.
* utils.c (panic):  Same.
  (myname): Remove variable.
* sed.c (usage): Use program_name instead of myname.
  (main): Call set_program_name once, don't set variables directly.
2018-08-03 20:02:39 -06:00
Assaf Gordon cd549fb19a sed: replace ck_realloc with gnulib's xrealloc/xnrealloc
* basicdefs.h (REALLOC): Call xnrealloc instead of ck_realloc.
* sed/utils.c (ck_realloc): Remove.
2018-08-03 20:02:39 -06:00
Assaf Gordon e68a03d750 sed: replace MALLOC/ck_malloc with gnulib's XCALLOC
ck_malloc zero'd out the allocated buffer, thus replaced with
xcalloc/xzalloc (not xmalloc).

* basicdefs.h (MALLOC): Remove.
* utils.c (ck_malloc): Remove.
* sed/compile.c, sed/execute.c, sed/regexp.c, sed/sed.c: Replace
MALLOC with XCALLOC; ck_malloc with xzalloc.
2018-08-03 20:02:39 -06:00
Assaf Gordon 8dda6ea11a sed: replace ck_strdup with gnulib's xstrdup
* sed/utils.{c,h} (ck_strdup): Remove.
* sed/compile.c, sed/execute.c, sed/sed.c: Replace ck_strdup with
xstrdup.
2018-08-03 20:02:39 -06:00
Assaf Gordon f593b21980 sed: replace ck_memdup with gnulib's xmemdup
* bootstrap.conf: Add gnulib's xalloc module.
* basicdefs.h (MEMDUP): Remove.
* sed/utils.{c,h} (ck_memdup): Remove.
2018-08-03 20:02:35 -06:00
Assaf Gordon 007a417687 sed: fix heap buffer overflow from multiline EOL regex optimization
sed would access invalid memory when matching EOF combined with
s///n flag:

  $ yes 0 | fmt -w 40 | head -n2 | valgrind sed 'N;s/$//2m'
  ==13131== Conditional jump or move depends on uninitialised value(s)
  ==13131==    at 0x4C3002B: memchr (vg_replace_strmem.c:883)
  ==13131==    by 0x1120BD: match_regex (regexp.c:286)
  ==13131==    by 0x110736: do_subst (execute.c:1101)
  ==13131==    by 0x1115D3: execute_program (execute.c:1591)
  ==13131==    by 0x111A4C: process_files (execute.c:1774)
  ==13131==    by 0x112E1C: main (sed.c:405)
  ==13131==
  ==13131== Invalid read of size 1
  ==13131==    at 0x4C30027: memchr (vg_replace_strmem.c:883)
  ==13131==    by 0x1120BD: match_regex (regexp.c:286)
  ==13131==    by 0x110736: do_subst (execute.c:1101)
  ==13131==    by 0x1115D3: execute_program (execute.c:1591)
  ==13131==    by 0x111A4C: process_files (execute.c:1774)
  ==13131==    by 0x112E1C: main (sed.c:405)
  ==13131==  Address 0x55ec765 is 0 bytes after a block of size 101 alloc'd
  ==13131==    at 0x4C2DDCF: realloc (vg_replace_malloc.c:785)
  ==13131==    by 0x113BA2: ck_realloc (utils.c:418)
  ==13131==    by 0x10E682: resize_line (execute.c:154)
  ==13131==    by 0x10E6F0: str_append (execute.c:165)
  ==13131==    by 0x110779: do_subst (execute.c:1106)
  ==13131==    by 0x1115D3: execute_program (execute.c:1591)
  ==13131==    by 0x111A4C: process_files (execute.c:1774)
  ==13131==    by 0x112E1C: main (sed.c:405)
  ==13131==

The ^/$ optimization code added in v4.2.2-161-g6dea75e called memchr()
using 'buflen', ignoring the value of 'buf_start_offset' (which, if not
zero, reduces the number of bytes available for the search).

Reported by bugs@feusi.co (bug#32271) in
https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .

* NEWS: Mention the fix.
* sed/regexp.c (match_regex): Use correct buffer length in memchr().
* testsuite/bug-32271-2.sh: Test using valgrind.
* testsuite/local.mk (T): Add new test.
2018-08-03 19:50:09 -06:00
Assaf Gordon 2cb09e1463 sed: fix extraneous NUL in s///n command
Under certain conditions sed would add an extraneous NUL:

    $ echo 0 | sed -e 's/$/a/2' | od -tx1 -An
     30 00 0a

This would happen when the regex is an empty (zero-length) match at the
end of the line (e.g. '$' and 'a*$') and the substitute number flag
('n' in s///n) is higher than the number of actual matches (multiple
EOL matches are possible with multiline match, e.g. 's/$/a/3m').

Details:
The comment at the top of 'execute.c:do_subst()' says:

       /* The first part of the loop optimizes s/xxx// when xxx is at the
          start, and s/xxx$// */

Which refers to lines 1051-3:

  1051 /* Copy stuff to the left of this match into the output string. */
  1052 if (start < offset)
  1053   str_append(&s_accum, line.active + start, offset - start);

The above code appends text to 's_accum' but does *not* update 'start'.

Later on, if the s/// command includes 'n' flag, and if 'matched == 0'
(an empty match), this comparison will be incorrect:

  1081  if (start < line.length)
  1082     matched = 1;

Will in turn will set 'matched' to 1, and the 'str_append' call that
follows (line 1087) will append an additional character.
Because the empty match is EOL, the appended character is NUL.

More examples that trigger the bug:

  echo 0 | sed -e 's/a*$/X/3'
  printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/4m'

Examples that do not trigger the bug:

  # The 'a*' empty regex matches at the beginning of the line (in
  # addition to the end of the line), and the optimization in line
  # 1052 is skipped.
  echo 0 | sed -e 's/a*/X/3'

  # There are 3 EOLs in the pattern space, s///3 is not too large.
  printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/3m'

This was discovered while investigating bug#32271 reported by bugs@feusi.co
in https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .

* NEWS: Mention the fix.
* sed/execute.c (do_subst): Update 'start' as needed.
* testsuite/bug-32271-1.sh: New test.
* testsuite/local.mk (T): Add test.
2018-08-03 19:36:19 -06:00
Assaf Gordon 0144eeeb3d sed: do not flush output stream unless in unbuffered mode
Previously sed would explicitly flush the output after
every output line, except if the output was stdout in unbuffered mode.

In practice this was equivalent to forcing line-buffering, and was
espcially was noticable with "sed -i" (where the output is a temporary
file).

With this change, explicit flushing only happens with "sed -u",
regardless of the type of output file, making "sed -i" much faster.
This change also affect other write commands such as 'w'/'W' and 's///w'.

Reported by Vidar Holen <vidar@vidarholen.net> in
https://lists.gnu.org/r/bug-sed/2018-07/msg00014.html .

* NEWS: Mention this.
* sed/execute.c (flush_output): Never flush output unless in unbuffered
mode, regardless of which file it is.
2018-07-26 18:06:57 -06:00
Bernhard Voelker a847466684 doc: tiny fix
* doc/sed.texi (Regexp Addresses): Remove the word 'two': there are
3 examples following.
2018-07-25 10:45:16 -06:00
Assaf Gordon c52a676e5e sed: fix heap buffer overflow from invalid references
Under certain conditions sed would access invalid memory based on
the requested back-reference (e.g. "s//\9/" would access the 9th element
in the regex registers without checking it is at least 9 element in
size).

The following examples would trigger valgrind errors:
    seq 2 | valgrind sed         -e '/^/s///p ; 2s//\9/'
    seq 2 | valgrind sed --posix -e '/2/p ;     2s//\9/'

Reported by bugs@feusi.co in
https://lists.gnu.org/r/bug-sed/2018-07/msg00004.html .

* NEWS: Mention the bugfix.
* sed/execute.c (append_replacement): Check number of allocated regex
replacement registers before accessing the array.
* sed/testsuite/bug32082.sh: Test sed for this behaviour under valgrind.
* sed/testsuite/local.mk (T): Add new test.
2018-07-08 20:09:51 -06:00
Jim Meyering 831fe3a5f0 maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2018-03-31 18:41:03 -07:00
Jim Meyering 7c8b9abf15 version 4.5
* NEWS: Record release date.
v4.5
2018-03-31 18:39:15 -07:00
Jim Meyering 4330bf9334 tests: port inplace-selinux.sh to Centos7 and Fedora 25
* testsuite/inplace-selinux.sh: Set the user (-u) portion of the security
context rather than the type (-t), and use system_u and user_u, which
appear to be reliably available, since some coreutils's tests use those.
Assaf Gordon reported that this test failed on those systems in
https://lists.gnu.org/r/sed-devel/2018-03/msg00018.html
Also, correct copyright date to include this year.
2018-03-30 21:33:01 -07:00
Jim Meyering 20cce35c16 build: update gnulib to latest 2018-03-28 13:22:25 -07:00
Jim Meyering ac16c564d1 maint: fix preceding change so "make dist" works
* testsuite/local.mk (EXTRA_DIST): Also remove runtest, here.
2018-03-28 13:22:18 -07:00
Jim Meyering 7daa26d068 tests: don't run $(check_PROGRAMS) directly
testsuite/runtest was not portable to Solaris 10 /bin/sh, but was
used only to keep automake test machinery from running two compiled
binaries stand-alone.  We can inhibit that more cleanly by removing
those two files from the list of TESTS to run and then just remove
the offending script.
* testsuite/local.mk (LOG_COMPILER): Set to false.
(TESTS): Don't put $(check_PROGRAMS) on this list.
* testsuite/runtest: Remove file.
Nelson H.F. Beebe reported the Solaris 10 /bin/sh failure.
2018-03-27 22:11:08 -07:00
Jim Meyering 4fa982396f tests: port to solaris 10's /bin/sh
* testsuite/runtest (test): Accommodate Solaris 10 /bin/sh
by sourcing init.sh, as is already done in many other shell
scripts in this directory, to handle (or skip test upon)
use of this construct: ${1##*/}. Reported by Nelson H.F. Beebe.
2018-03-27 21:40:06 -07:00
Jim Meyering 290c59315b maint: remove old definition of UNUSED: use _GL_UNUSED instead
* sed/execute.c (UNUSED): Remove macro definition.
(read_always_fail): Use _GL_UNUSED instead.
2018-03-23 13:26:54 -07:00
Assaf Gordon 5433dc245b sed: reject RE searches on buffers larger than INT_MAX
Sed uses 'size_t' internally, but gnulib's re_search uses 'signed int'.
If the buffer is larger than INT_MAX, reject it (panic, exit code 4).

The test is marked 'very expensive' and skipped by default (due to
creating a 2GB input file). To run it, use:

   make check SUBDIRS=. RUN_VERY_EXPENSIVE_TESTS=yes \
              TESTS=testsuite/regex-max-int.sh

Reported by YushiOMOTE in https://bugs.gnu.org/30520 .

* NEWS: Mention this.
* Makefile.am (check-expensive, check-very-expensive): New targets.
* init.cfg (expensive_, very_expensive_): Copied from coreutils.
* sed/regexp.c (match_regex): Check buffer length and panic if too large.
* testsuite/regex-max-int.sh: New test.
* testsuite/local.mk (T): Add new test.
2018-03-23 08:58:01 -06:00
Jim Meyering c0375163a4 maint: ignore more artifacts
* .gitignore: Also ignore sed-*.tar.xz.sig files.
* lib/.gitignore: Ignore more gnulib-provided sources.
2018-03-22 16:12:26 -07:00
Jim Meyering 9b1e01da2e build: update gnulib to latest
* gnulib: Update submodule.
* bootstrap: Update from gnulib.
* testsuite/init.sh: Likewise.
2018-03-22 16:00:43 -07:00
0xddaa b5f5236a4b sed: treat '\x5c' as literal backslash
Sed incorrectly treated '\x5c' (ASCII 92, backslash) as an escape character.

Old behavior:
  $ echo z | sed -E 's/(z)/\x5c1/' # identical to 's/(z)/\1/'
  z

New behavior:
  $ echo z | sed -E 's/(z)/\x5c1/'
  \1

Reported in https://bugs.gnu.org/30794.

* NEWS: Mention bug fix.
* sed/compile.c (normalize_text): Pass backslash as literal character.
* testsuite/misc.pl: Add tests.
2018-03-14 18:41:09 -04:00
Mike Frysinger ef9fcbc4cd sed: improve sandbox description in --help output
* sed/sed.c (usage): Add "(disable e/r/w commands)" to --sandbox text.
Copyright-paperwork-exempt: Yes
2018-03-02 16:21:42 -07:00
Jim Meyering 41e506e12b doc: small improvements
* doc/sed.texi: s/only accepts/accepts only/.
(BRE vs ERE): Tweak ERE and BRE definitions, and convert a
passive-voice sentence to active voice.  Insert a comma.
2018-02-15 17:41:20 -07:00
Assaf Gordon cb6cb45fed doc: change URLs to https
* doc/config.texi, doc/sed-dummy.1, doc/sed.texi: Change http to https for
GNU URLs. External links (e.g. sed.sf.net, autsingroup.org) currently do
not support https and kept as-is.
2018-02-13 16:07:29 -07:00
Assaf Gordon a2daab7fb1 maint: change http to https in license headers
* all files: Change http to https for all licenses URL.
2018-02-13 16:02:12 -07:00
Assaf Gordon a328263a41 sed: use https in usage screen contact information
* sed/sed.c (contact): Change http to https.
2018-02-13 15:51:55 -07:00
Jim Meyering 7747982294 build: update gnulib to latest
* gnulib: This fixes a syntax-check failure that was due to the
new use of "/proc/filesystems" in init.cfg.
2018-02-04 02:36:36 -08:00
Jim Meyering 85d8340acd maint: make cfg.mk slightly more generic
* cfg.mk (announcement_Cc_): Don't hard-code "sed-devel@".
Use "$(PACKAGE)-devel@" instead.
2018-01-31 22:11:27 +13:00
Jakub Martisko 057ab92dfe sed: with --in-place and selinux, use the symlink's context
When editing a file in place, the SELinux context (if exists)
should be based on the link instead of the target file itself.
--follow-symlinks option remains unchanged.

Bug reported by Jakub Jelen, fix proposed by Petr Lautrbach,
discussed in https://lists.gnu.org/r/sed-devel/2017-12/msg00000.html

* NEWS: Mention this.
* sed/execute.c (open_next_file): Use lgetfilecon (instead of getfilecon)
to get the context of the symlink instead of the target file.
* testsuite/inplace-selinux.sh: Test the above change.
* testsuite/local.mk (T): Add new test file.
* init.cfg (require_selinux_): Copied from coreutils, skip the test if
selinux is not available.
2018-01-09 15:44:51 -07:00
Assaf Gordon 4d4c0d5ad0 maint: update gnulib and copyright dates for 2018
* gnulib: Update to latest.
* all files: Run "make update-copyright".
2018-01-02 13:56:16 -07:00
Jim Meyering 80ebb116b7 maint: update to work with GCC7's -Werror=implicit-fallthrough=
* sed/sed.h (FALLTHROUGH): Define.
* sed/execute.c (execute_program): Use new FALLTHROUGH macro in place
of each comment.
* sed/compile.c (mark_subst_opts, compile_program): Likewise.
2017-11-22 19:22:54 -08:00
Assaf Gordon 490b4997fe doc: fix incorrect line-wrapping example
Reported by Bamber Ward in https://bugs.gnu.org/28140 .

* doc/sed.texi (Line length adjustment): Fix sed script;
  Rewrite example to long script with inlined comments;
  Remove second example.
2017-10-28 21:24:42 -06:00
Assaf Gordon 9f99a4f51e doc: fix missing slash in example
Reported by Nick Chambers in https://bugs.gnu.org/28881 .

* doc/sed.texi (Invoking sed::overview): Add missing slash in
's/hello/world/' command.
2017-10-17 16:48:49 -06:00
Assaf Gordon 315c804175 tests: skip utf8-ru test if required locale is not found
Reported by Dennis Clarke in https://bugs.gnu.org/28665#14 .

* init.cfg (require_ru_utf8_locale_): New function.
* testsuite/utf8-ru.sh: Skip test is required locale was not found.
2017-10-05 23:38:30 -06:00
Assaf Gordon 96e56370a4 build: require texinfo v6.1 for development builds
For development builds (using ./bootstrap), require texinfo version 6.1
or later (Same as GNU coreutils).
Sed's manual uses texinfo's @U{} command, introduced in texinfo-6.0.
This should not affect building from released tarballs, as the info
manual is pre-built and pre-packaged in the tar archive.
Reported by Dennis Clarke in https://bugs.gnu.org/28665#14 .

* bootstrap.conf: Require makeinfo 6.1 (up from 4.13).
2017-10-05 23:38:16 -06:00
Assaf Gordon 97478b089c tests: remove non-functional regex C tests
These C-based tests were not updated since 2004, and were not included
by default. Enabling them with "./configure --enable-regex-tests"
resulted in false positives:
https://bugs.gnu.org/28665 , https://bugs.gnu.org/25381 .

This patch removes them completely.

* configure.ac: Remove "--with-regex-tests" option.
* local.mk (check_PROGRAMS): Remove tests.
* bug-regex10.c,
  bug-regex11.c,
  bug-regex12.c,
  bug-regex13.c,
  bug-regex14.c,
  bug-regex15.c,
  bug-regex16.c,
  bug-regex21.c,
  bug-regex27.c,
  bug-regex28.c,
  bug-regex7.c,
  bug-regex8.c,
  bug-regex9.c,
  runptests.c,
  runtests.c,
  tst-boost.c,
  tst-pcre.c,
  tst-regex.c,
  tst-regex2.c,
  tst-rxspencer.c: Remove files.
2017-10-05 23:07:19 -06:00
Jim Meyering a59fb8826a maint: stop using @acronym{...} in texinfo sources
* doc/sed.texi, doc/config.texi: Remove all uses of @acronym{...},
per recommendation by Karl Berry.
* cfg.mk (local-checks-to-skip): Remove exemption, enabling
the @acronym{-prohibiting syntax-check rule.
2017-05-13 18:42:21 -07:00
Jim Meyering bebaa6945d gnulib: update to latest 2017-05-13 18:36:43 -07:00
Assaf Gordon 115829fc1f tests: fix various misspellings
Reported by Jim Meyering in
https://lists.gnu.org/archive/html/sed-devel/2017-05/msg00001.html .

* testsuite/BOOST.tests,
  testsuite/posix-mode-bad-ref.sh,
  testsuite/runptests.c,
  testsuite/runtests.c,
  testsuite/tst-pcre.c: Fix misspellings.
2017-05-10 21:39:57 -04:00
Assaf Gordon b8377e4303 tests: remove the 'Makefile.tests' used for old-style tests
* testsuite/Makefile.tests: Removed.
* testsuite/runtest: Remove code to run old-style tests.
* testsuite/local.mk (EXTRA_DIST): Remove MAkefile.tests.
2017-05-10 21:39:57 -04:00
Assaf Gordon b85182196a tests: convert "noeolw" test to new framework
* testsuite/misc.pl (Tests): Add noeolw.
* testsuite/noeolw.good: Delete file.
* testsuite/noeolw.1good: Likewise.
* testsuite/noeolw.2good: Likewise
* testsuite/noeolw.inp: Likewise.
* testsuite/noeolw.sed: Likewise.
* testsuite/local.mk (EXTRA_DIST): Remove their names.
(SEDTESTS): Remove "noeolw".
2017-05-10 21:39:57 -04:00
Assaf Gordon 0ea46106d3 tests: convert "mac-mf" test to new framework
All input files (mac-mf.{inp,good,sed}) are large and are kept as-is.

* testsuite/mac-mf.sh: New test.
* testsuite/local.mk (SEDTEST): Remove mac-mf. (T): Add mac-mf.sh
2017-05-10 21:39:57 -04:00
Assaf Gordon 63931dcff4 tests: convert "newjis" test to new framework
* testsuite/newjis.sh: New test.
* testsuite/newjis.{good,inp,sed}: Removed.
* testsuite/local.mk (EXTRA_DIST): Remove unneeded files.
(SEDTEST): Remove newjis. (T): Add newjis.sh
2017-05-10 21:39:57 -04:00
Assaf Gordon e86c034722 tests: convert "8to7" test to new framework
* testsuite/8to7.sh: New test.
* testsuite/8to7.{good,inp,sed}: Removed.
* testsuite/local.mk (EXTRA_DIST): Remove unneeded files.
(SEDTEST): Remove 8to7. (T): Add 8to7.sh
2017-05-10 21:39:57 -04:00
Assaf Gordon e51a067bc7 tests: convert "madding" test to new framework
All input files (madding.{inp,good,sed}) contain very long lines and
are kept as-is (to avoid failing 'make syntax-check').

* testsuite/madding.sh: New test.
* testsuite/local.mk (SEDTEST): Remove madding. (T): Add madding.sh
2017-05-10 21:39:56 -04:00
Assaf Gordon 22d1b59157 tests: convert "badenc" test to new framework
* testsuite/badenc.sh: New test.
* testsuite/distrib.{good,inp,sed}: Removed.
* testsuite/local.mk (EXTRA_DIST): Remove unneeded files.
(SEDTEST) Remove badenc. (T): Add badenc.sh
2017-05-10 21:39:56 -04:00