Commit Graph
584 Commits
Author SHA1 Message Date
Bjarni Ingi Gislason 19089418c6 doc: sed.x: some formatting corrections
Reported in https://bugs.gnu.org/30479 .

doc/sed.x: Change a HYPHEN-MINUS (code 0x55, 2D) to a dash (\-, minus)
if it matches " -[:alpha:]" or \(aq-[:alpha:] (for options); Use a pronoun
instead of a repeated noun.
Copyright-paperwork-exempt: Yes
2018-10-12 11:45:56 -06:00
Assaf Gordon 50f598b6a4 sed: update authors
* sed/sed.c (AUTHORS): Update.
* doc/sed.texi (@author): Update.
2018-10-12 11:35:07 -06:00
Assaf Gordon 261c7f145d maint: remove REG_PERL code
Perl-regexp syntax (PCRE) in GNU Sed is shelved indefinitely.
See https://bugs.gnu.org/22801 , https://bugs.gnu.org/22647 .
Remove all (unused) REG_PERL related code.

* sed/sed.c, sed/sed.h, sed/regexp.c, sed/compile.c: Remove REG_PERL code.
2018-08-23 14:34:18 -06:00
Assaf Gordon 41a169a9a1 maint: remove EXPERIMENTAL_DASH_N_OPTIMIZATION code
The optimization was buggy and was never enabled in a released version.

* sed/execute.c: Remove EXPERIMENTAL_DASH_N_OPTIMIZATION code.
2018-08-23 14:34:17 -06:00
Assaf Gordon 30192e7587 maint: replace street address with URL in license text
* basicdefs.h, sed/compile.c, sed/execute.c, sed/mbcs.c, sed/regexp.c,
sed/sed.c, sed/sed.h, sed/utils.c, sed/utils.h, testsuite/get-mb-cur-max.c,
testsuite/test-mbrtowc.c (GPL License notice): Replace FSF's office
address with 'https://www.gnu.org/licenses/'.
2018-08-23 14:34:17 -06:00
Assaf Gordon ac159313c7 build: use system's native sed during build
Use the locally built sed binary only during 'check*' targets,
but not during build/installation targets - it might be buggy, or contain
temporary debugging/ASAN features that can break the build.
See: https://lists.gnu.org/r/sed-devel/2018-08/msg00013.html .

* cfg.mk (PATH): Add the 'sed' directory to the PATH only if this is a
'check' target.
2018-08-23 14:34:09 -06:00
Assaf Gordon 36e3485e7f sed: mark function as attribute(malloc)
Suggested by gcc-8.2.0.

* sed/compile.c (read_label): Mark as _GL_ATTRIBUTE_MALLOC.
2018-08-08 14:14:51 -06:00
Assaf Gordon 665f9a9dc1 maint: add undefined-behavior build target
Using gcc-specific options, a recent gcc is required.
build with:
   make build-ubsan CC=gcc-8.2

* cfg.mk (build-ubsan): New target.
2018-08-08 14:14:51 -06:00
Assaf Gordon 0cc0ade423 maint: add address-sanitizer build target
use 'make build-asan' to rebuild sed with gcc's address sanitizer.

* cfg.mk (build-asan): New target.
2018-08-08 14:14:51 -06:00
Assaf Gordon 9a9f9e34f3 gnulib: update to latest (for regex memory leaks)
Specifically for gnulib commits 66b99e5259,c5e76a9560.
see https://lists.gnu.org/r/bug-gnulib/2018-07/msg00127.html
2018-08-08 14:14:51 -06:00
Assaf Gordon ba8e66de6d sed: fix memory leak
* sed/regexp.c (match_regex): Free the previously allocated regex struct
before re-building the regex during program execution.
2018-08-08 14:14:51 -06:00
Assaf Gordon 35b30b124c sed: free allocated memory under lint, remove DEBUG_LEAKS
Under normal operation there is no need for explicit freeing,
as all memory will be released when sed terminates.
During development (and specifically, valgrind and address-sanitizing)
enabling lint prevents false-positive warnings about memory
leaks.
Lint mode can be enabled with CFLAGS="-Dlint". It is also automatically
enabled by default when building from git (as opposed to tarball).

For consistency all instances of "#ifdef DEBUG_LEAKS" are replaced
with "#ifdef lint".

* sed/sed.h (struct subst): Add member variable to keep the address of
allocated buffer in compile.c:setup_replacement().
(release_regex): Add declaration.
(finish_program): Function now takes an argument: the sed program
vector.
* sed/sed.c (main): Adjust call to finish_program.
* sed/compile.c (finish_program): Release program allocations.
(setup_replacement): Remember the allocated buffer address.
(compile_program): Free temporary array in 'y' command.
* sed/execute.c (execute_program): Free allocated but unused buffer in
'R' command when reaching EOF.
* sed/regexp.c (release_regex): Remove 'static', free the allocated
dfa struct.
* sed/utils.c (panic): Free open files linked-list elements.
2018-08-08 14:14:39 -06:00
Assaf Gordon 08675afe9a sed: do not close stderr on exit
Not needed, and prevents leak-sanitizing from working.

* sed/utils.c (ck_fclose): Do not close stderr.
2018-08-08 14:14:20 -06:00
Jim Meyering 7c6a618904 doc: insert two missing words
doc/sed.texi (BRE vs ERE): Insert "with a".
2018-08-07 03:32:28 -07:00
Assaf Gordon a2cda80ecd maint: syntax-check: prohibit-operator-at-eol
* cfg.mk (sc_prohibit_operator_at_end_of_line): Copied from coreutils'
cfg.mk.
* sed/execute.c (do_subst,execute_program): Adjust as needed.
2018-08-04 22:09:58 -06:00
Assaf Gordon 4766cf6717 maint: syntax-check: fail-0 and fail-1 in shell script tests
* cfg.mk (prohibit_fail_0,sc_prohibit_and_fail_1): Copied from
coreutils' cfg.mk
* testsuite/*.sh: Remove 'fail=0'.
2018-08-04 22:09:58 -06:00
Assaf Gordon 6b3d0d6d37 maint: syntax-check: add various test-related checks
* cfg.mk (sc_prohibit_env_returns, sc_prohibit_perl_hash_quotes,
sc_prohibit_verbose_version, sc_prohibit_framework_failure,
sc_prohibit_test_backticks, sc_prohibit_test_empty): Copied from
coreutils' cfg.mk.
2018-08-04 22:09:58 -06:00
Assaf Gordon f5364300a6 maint: syntax-check: add sc_prohibit_strncmp
* cfg.mk (sc_prohibit_strncmp): Copied from coreutils' cfg.mk.
* basicdefs.h (STREQ,STREQ_LEN,STRPREFIX): Copied from coreutils' system.h.
* sed/execute.c (translate_mb): Replace strncmp with STREQ_LEN.
2018-08-04 22:09:58 -06:00
Assaf Gordon 58c26e2e66 maint: syntax-check: add sc_gitignore_missing
* cfg.mk (sc_gitignore_missing): Copied from coreutils' cfg.mk.
* .gitignore: Add missing files.
2018-08-04 22:09:57 -06:00
Assaf Gordon 293d1fd372 maint: syntax-check: prohibit-form-feed
* cfg.mk (sc_prohibit-form-feed): Copied from coreutils' cfg.mk.
* sed/compile.c, sed/execute.c, sed/regexp.c, sed/sed.h, sed/utils.c:
Remove form feed characters.
2018-08-04 22:09:57 -06:00
Assaf Gordon 08b9f66a0c maint: syntax-check: space-before-open-paren
* cfg.mk (sc_space_before_open_paren): Copy target from coreutils' cfg.mk.
* basicdefs.h, sed/compile.c, sed/execute.c, sed/regexp.c,
sed/sed.c, sed/utils.c: Add space before parentheses.
2018-08-04 22:09:57 -06:00
Assaf Gordon 06fe083ed2 maint: syntax-check: fix tab indentation
* sed/execute.c (open_next_file): Replace tab with spaces.
2018-08-04 22:09:41 -06:00
Assaf Gordon 5491c2bf85 sed: replace myname with gnulib's progname
* compile.c (bad_prog): Use program_name instead of myname.
* execute.c (open_next_file): Same.
* utils.c (panic):  Same.
  (myname): Remove variable.
* sed.c (usage): Use program_name instead of myname.
  (main): Call set_program_name once, don't set variables directly.
2018-08-03 20:02:39 -06:00
Assaf Gordon cd549fb19a sed: replace ck_realloc with gnulib's xrealloc/xnrealloc
* basicdefs.h (REALLOC): Call xnrealloc instead of ck_realloc.
* sed/utils.c (ck_realloc): Remove.
2018-08-03 20:02:39 -06:00
Assaf Gordon e68a03d750 sed: replace MALLOC/ck_malloc with gnulib's XCALLOC
ck_malloc zero'd out the allocated buffer, thus replaced with
xcalloc/xzalloc (not xmalloc).

* basicdefs.h (MALLOC): Remove.
* utils.c (ck_malloc): Remove.
* sed/compile.c, sed/execute.c, sed/regexp.c, sed/sed.c: Replace
MALLOC with XCALLOC; ck_malloc with xzalloc.
2018-08-03 20:02:39 -06:00
Assaf Gordon 8dda6ea11a sed: replace ck_strdup with gnulib's xstrdup
* sed/utils.{c,h} (ck_strdup): Remove.
* sed/compile.c, sed/execute.c, sed/sed.c: Replace ck_strdup with
xstrdup.
2018-08-03 20:02:39 -06:00
Assaf Gordon f593b21980 sed: replace ck_memdup with gnulib's xmemdup
* bootstrap.conf: Add gnulib's xalloc module.
* basicdefs.h (MEMDUP): Remove.
* sed/utils.{c,h} (ck_memdup): Remove.
2018-08-03 20:02:35 -06:00
Assaf Gordon 007a417687 sed: fix heap buffer overflow from multiline EOL regex optimization
sed would access invalid memory when matching EOF combined with
s///n flag:

  $ yes 0 | fmt -w 40 | head -n2 | valgrind sed 'N;s/$//2m'
  ==13131== Conditional jump or move depends on uninitialised value(s)
  ==13131==    at 0x4C3002B: memchr (vg_replace_strmem.c:883)
  ==13131==    by 0x1120BD: match_regex (regexp.c:286)
  ==13131==    by 0x110736: do_subst (execute.c:1101)
  ==13131==    by 0x1115D3: execute_program (execute.c:1591)
  ==13131==    by 0x111A4C: process_files (execute.c:1774)
  ==13131==    by 0x112E1C: main (sed.c:405)
  ==13131==
  ==13131== Invalid read of size 1
  ==13131==    at 0x4C30027: memchr (vg_replace_strmem.c:883)
  ==13131==    by 0x1120BD: match_regex (regexp.c:286)
  ==13131==    by 0x110736: do_subst (execute.c:1101)
  ==13131==    by 0x1115D3: execute_program (execute.c:1591)
  ==13131==    by 0x111A4C: process_files (execute.c:1774)
  ==13131==    by 0x112E1C: main (sed.c:405)
  ==13131==  Address 0x55ec765 is 0 bytes after a block of size 101 alloc'd
  ==13131==    at 0x4C2DDCF: realloc (vg_replace_malloc.c:785)
  ==13131==    by 0x113BA2: ck_realloc (utils.c:418)
  ==13131==    by 0x10E682: resize_line (execute.c:154)
  ==13131==    by 0x10E6F0: str_append (execute.c:165)
  ==13131==    by 0x110779: do_subst (execute.c:1106)
  ==13131==    by 0x1115D3: execute_program (execute.c:1591)
  ==13131==    by 0x111A4C: process_files (execute.c:1774)
  ==13131==    by 0x112E1C: main (sed.c:405)
  ==13131==

The ^/$ optimization code added in v4.2.2-161-g6dea75e called memchr()
using 'buflen', ignoring the value of 'buf_start_offset' (which, if not
zero, reduces the number of bytes available for the search).

Reported by bugs@feusi.co (bug#32271) in
https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .

* NEWS: Mention the fix.
* sed/regexp.c (match_regex): Use correct buffer length in memchr().
* testsuite/bug-32271-2.sh: Test using valgrind.
* testsuite/local.mk (T): Add new test.
2018-08-03 19:50:09 -06:00
Assaf Gordon 2cb09e1463 sed: fix extraneous NUL in s///n command
Under certain conditions sed would add an extraneous NUL:

    $ echo 0 | sed -e 's/$/a/2' | od -tx1 -An
     30 00 0a

This would happen when the regex is an empty (zero-length) match at the
end of the line (e.g. '$' and 'a*$') and the substitute number flag
('n' in s///n) is higher than the number of actual matches (multiple
EOL matches are possible with multiline match, e.g. 's/$/a/3m').

Details:
The comment at the top of 'execute.c:do_subst()' says:

       /* The first part of the loop optimizes s/xxx// when xxx is at the
          start, and s/xxx$// */

Which refers to lines 1051-3:

  1051 /* Copy stuff to the left of this match into the output string. */
  1052 if (start < offset)
  1053   str_append(&s_accum, line.active + start, offset - start);

The above code appends text to 's_accum' but does *not* update 'start'.

Later on, if the s/// command includes 'n' flag, and if 'matched == 0'
(an empty match), this comparison will be incorrect:

  1081  if (start < line.length)
  1082     matched = 1;

Will in turn will set 'matched' to 1, and the 'str_append' call that
follows (line 1087) will append an additional character.
Because the empty match is EOL, the appended character is NUL.

More examples that trigger the bug:

  echo 0 | sed -e 's/a*$/X/3'
  printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/4m'

Examples that do not trigger the bug:

  # The 'a*' empty regex matches at the beginning of the line (in
  # addition to the end of the line), and the optimization in line
  # 1052 is skipped.
  echo 0 | sed -e 's/a*/X/3'

  # There are 3 EOLs in the pattern space, s///3 is not too large.
  printf "%s\n" 0 0 0 | sed -e 'N;N;s/a*$/X/3m'

This was discovered while investigating bug#32271 reported by bugs@feusi.co
in https://lists.gnu.org/r/bug-sed/2018-07/msg00018.html .

* NEWS: Mention the fix.
* sed/execute.c (do_subst): Update 'start' as needed.
* testsuite/bug-32271-1.sh: New test.
* testsuite/local.mk (T): Add test.
2018-08-03 19:36:19 -06:00
Assaf Gordon 0144eeeb3d sed: do not flush output stream unless in unbuffered mode
Previously sed would explicitly flush the output after
every output line, except if the output was stdout in unbuffered mode.

In practice this was equivalent to forcing line-buffering, and was
espcially was noticable with "sed -i" (where the output is a temporary
file).

With this change, explicit flushing only happens with "sed -u",
regardless of the type of output file, making "sed -i" much faster.
This change also affect other write commands such as 'w'/'W' and 's///w'.

Reported by Vidar Holen <vidar@vidarholen.net> in
https://lists.gnu.org/r/bug-sed/2018-07/msg00014.html .

* NEWS: Mention this.
* sed/execute.c (flush_output): Never flush output unless in unbuffered
mode, regardless of which file it is.
2018-07-26 18:06:57 -06:00
Bernhard Voelker a847466684 doc: tiny fix
* doc/sed.texi (Regexp Addresses): Remove the word 'two': there are
3 examples following.
2018-07-25 10:45:16 -06:00
Assaf Gordon c52a676e5e sed: fix heap buffer overflow from invalid references
Under certain conditions sed would access invalid memory based on
the requested back-reference (e.g. "s//\9/" would access the 9th element
in the regex registers without checking it is at least 9 element in
size).

The following examples would trigger valgrind errors:
    seq 2 | valgrind sed         -e '/^/s///p ; 2s//\9/'
    seq 2 | valgrind sed --posix -e '/2/p ;     2s//\9/'

Reported by bugs@feusi.co in
https://lists.gnu.org/r/bug-sed/2018-07/msg00004.html .

* NEWS: Mention the bugfix.
* sed/execute.c (append_replacement): Check number of allocated regex
replacement registers before accessing the array.
* sed/testsuite/bug32082.sh: Test sed for this behaviour under valgrind.
* sed/testsuite/local.mk (T): Add new test.
2018-07-08 20:09:51 -06:00
Jim Meyering 831fe3a5f0 maint: post-release administrivia
* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.
2018-03-31 18:41:03 -07:00
Jim Meyering 7c8b9abf15 version 4.5
* NEWS: Record release date.
v4.5
2018-03-31 18:39:15 -07:00
Jim Meyering 4330bf9334 tests: port inplace-selinux.sh to Centos7 and Fedora 25
* testsuite/inplace-selinux.sh: Set the user (-u) portion of the security
context rather than the type (-t), and use system_u and user_u, which
appear to be reliably available, since some coreutils's tests use those.
Assaf Gordon reported that this test failed on those systems in
https://lists.gnu.org/r/sed-devel/2018-03/msg00018.html
Also, correct copyright date to include this year.
2018-03-30 21:33:01 -07:00
Jim Meyering 20cce35c16 build: update gnulib to latest 2018-03-28 13:22:25 -07:00
Jim Meyering ac16c564d1 maint: fix preceding change so "make dist" works
* testsuite/local.mk (EXTRA_DIST): Also remove runtest, here.
2018-03-28 13:22:18 -07:00
Jim Meyering 7daa26d068 tests: don't run $(check_PROGRAMS) directly
testsuite/runtest was not portable to Solaris 10 /bin/sh, but was
used only to keep automake test machinery from running two compiled
binaries stand-alone.  We can inhibit that more cleanly by removing
those two files from the list of TESTS to run and then just remove
the offending script.
* testsuite/local.mk (LOG_COMPILER): Set to false.
(TESTS): Don't put $(check_PROGRAMS) on this list.
* testsuite/runtest: Remove file.
Nelson H.F. Beebe reported the Solaris 10 /bin/sh failure.
2018-03-27 22:11:08 -07:00
Jim Meyering 4fa982396f tests: port to solaris 10's /bin/sh
* testsuite/runtest (test): Accommodate Solaris 10 /bin/sh
by sourcing init.sh, as is already done in many other shell
scripts in this directory, to handle (or skip test upon)
use of this construct: ${1##*/}. Reported by Nelson H.F. Beebe.
2018-03-27 21:40:06 -07:00
Jim Meyering 290c59315b maint: remove old definition of UNUSED: use _GL_UNUSED instead
* sed/execute.c (UNUSED): Remove macro definition.
(read_always_fail): Use _GL_UNUSED instead.
2018-03-23 13:26:54 -07:00
Assaf Gordon 5433dc245b sed: reject RE searches on buffers larger than INT_MAX
Sed uses 'size_t' internally, but gnulib's re_search uses 'signed int'.
If the buffer is larger than INT_MAX, reject it (panic, exit code 4).

The test is marked 'very expensive' and skipped by default (due to
creating a 2GB input file). To run it, use:

   make check SUBDIRS=. RUN_VERY_EXPENSIVE_TESTS=yes \
              TESTS=testsuite/regex-max-int.sh

Reported by YushiOMOTE in https://bugs.gnu.org/30520 .

* NEWS: Mention this.
* Makefile.am (check-expensive, check-very-expensive): New targets.
* init.cfg (expensive_, very_expensive_): Copied from coreutils.
* sed/regexp.c (match_regex): Check buffer length and panic if too large.
* testsuite/regex-max-int.sh: New test.
* testsuite/local.mk (T): Add new test.
2018-03-23 08:58:01 -06:00
Jim Meyering c0375163a4 maint: ignore more artifacts
* .gitignore: Also ignore sed-*.tar.xz.sig files.
* lib/.gitignore: Ignore more gnulib-provided sources.
2018-03-22 16:12:26 -07:00
Jim Meyering 9b1e01da2e build: update gnulib to latest
* gnulib: Update submodule.
* bootstrap: Update from gnulib.
* testsuite/init.sh: Likewise.
2018-03-22 16:00:43 -07:00
0xddaa b5f5236a4b sed: treat '\x5c' as literal backslash
Sed incorrectly treated '\x5c' (ASCII 92, backslash) as an escape character.

Old behavior:
  $ echo z | sed -E 's/(z)/\x5c1/' # identical to 's/(z)/\1/'
  z

New behavior:
  $ echo z | sed -E 's/(z)/\x5c1/'
  \1

Reported in https://bugs.gnu.org/30794.

* NEWS: Mention bug fix.
* sed/compile.c (normalize_text): Pass backslash as literal character.
* testsuite/misc.pl: Add tests.
2018-03-14 18:41:09 -04:00
Mike Frysinger ef9fcbc4cd sed: improve sandbox description in --help output
* sed/sed.c (usage): Add "(disable e/r/w commands)" to --sandbox text.
Copyright-paperwork-exempt: Yes
2018-03-02 16:21:42 -07:00
Jim Meyering 41e506e12b doc: small improvements
* doc/sed.texi: s/only accepts/accepts only/.
(BRE vs ERE): Tweak ERE and BRE definitions, and convert a
passive-voice sentence to active voice.  Insert a comma.
2018-02-15 17:41:20 -07:00
Assaf Gordon cb6cb45fed doc: change URLs to https
* doc/config.texi, doc/sed-dummy.1, doc/sed.texi: Change http to https for
GNU URLs. External links (e.g. sed.sf.net, autsingroup.org) currently do
not support https and kept as-is.
2018-02-13 16:07:29 -07:00
Assaf Gordon a2daab7fb1 maint: change http to https in license headers
* all files: Change http to https for all licenses URL.
2018-02-13 16:02:12 -07:00
Assaf Gordon a328263a41 sed: use https in usage screen contact information
* sed/sed.c (contact): Change http to https.
2018-02-13 15:51:55 -07:00
Jim Meyering 7747982294 build: update gnulib to latest
* gnulib: This fixes a syntax-check failure that was due to the
new use of "/proc/filesystems" in init.cfg.
2018-02-04 02:36:36 -08:00