sed is single-threaded, so does not need multithreading support.
* bootstrap.conf (avoided_gnulib_modules): Avoid lock, not lock-tests.
(bootstrap_post_import_hook): Copy lib/glthread/lock.h by hand,
as we now avoid ‘lock’.
* configure.ac (GNULIB_EXCLUDE_SINGLE_THREAD)
(GNULIB_LOCALENAME_SINGLE_THREAD)
(GNULIB_MBRTOWC_SINGLE_THREAD, GNULIB_REGEX_SINGLE_THREAD)
(GNULIB_WCHAR_SINGLE_LOCALE, GNULIB_MBRTOC32_REGULAR):
New macros.
* lib/.gitignore, m4/.gitignore: Adjust accordingly.
sed: also avoid ‘lock’ module
* bootstrap.conf
* basicdefs.h: Do not include wchar.h, wctype.h.
(ISASCII, ISBLANK, ISSPACE): Remove. All uses replaced
by c32isblank, c32isspace.
* bootstrap.conf (gnulib_modules): Add c32isblank, c32isprint,
c32isspace, c32rtomb, c32toower, c32toupper, mcel. Remove btowc,
localcharset, mbrlen, mbrtowc, mbsinit, wcrtomb, wctob, wctype-h.
* configure.ac: Do not check for headers locale.h, errno.h,
wchar.h, wctype.h, mcheck.h.
Do not check for isascii. Remove ac_cv_func_wcscoll hack.
(XFAIL_TESTS): Remove check for whether to run UTF-8 Cyrillic
tests, as I hope the other changes in this patch should fix
problems in that area.
* lib/.gitignore, m4/.gitignore: Adjust
* sed/compile.c (struct prog_info.prev): New member.
Adjust all users of this struct to update and use it when needed.
(INCHAR_EOF): New constant.
(inchar): Return a char32_t value for each wide character, a
negative byte value for encoding errors, and INCHAR_EOF for end of
input. All callers changed.
(savchar): Now takes no arguments. All callers changed.
(in_nonblank): Use char32_t tests rather than ctype.h tests.
(add_prev_to_buffer): New function.
(add_then_next, match_slash, read_text):
Use add_prev_to_buffer instead of add1_buffer when the idea is to
copy a character from input to output, as the character might be
multibyte and add1_buffer does only single-byte.
(match_slash): Support multibyte delimiters. Test for matchers
individually rather than via strchr, to make it clearer that we
are not testing for '\0'.
* sed/compile.c (add_then_next, snarf_char_class):
* sed/debug.c (debug_print_mb):
* sed/execute.c (line_init, line_reset, line_copy, line_append)
(line_exchange):
Remove now-unnecessary arg. All callers changed.
* sed/compile.c (compile_program, normalize_text):
* sed/debug.c (debug_print_regex_pattern):
* sed/execute.c (str_append_modified, do_list, translate_mb)
(debug_print_line):
Use mcel_scan/c32rtomb1 not mbrtowc1/mbrlen1/wcrtomb,
as mcel_scan should work better on MS-Windows.
* sed/debug.c (debug_print_char): Replace args c and clen
with an mcel_t arg g. All callers changed.
(debug_print_mb): Adjust to new meaning of int “characters”.
* sed/execute.c (struct line.mbstate): Remove. All uses removed.
(str_append): Omit no-longer-needed code put in just to update a
no-longer-needed mbstate_t.
(str_append_modified): Don’t worry about c32rtomb1 returning
(size_t) -1, as it now returns idx_t and anyway that cannot happen.
* sed/mbcs.c: Don’t include string.h, localcharset.h.
(is_mb_char, is_utf8): Remove. All uses removed.
* sed/sed.h: Include mcel.h.
(mbrtowc1, mbrlen1): Remove. All callers changed to use mcel_scan.
(c32rtomb1): New function.
(IS_MB_CHAR): Remove; no longer used.
* sed/utils.c (add1_buffer): Now returns void, since no caller
used the return value. 2nd arg is now char, not int.
Remove special case for EOF. All callers changed.
* testsuite/mb-bad-delim.sh, testsuite/mb-y-translate.sh:
Multibyte delimiters are no longer prohibited, so adjust test
cases accordingly. Put newlines at end of text files,
as required by POSIX.
ISDIGIT is locale-dependent, and in theory could misfire if
any locale has a single-byte digit that is not '0'..'9'.
Fix this by using c_isdigit, which should be a bit faster anyway.
* basicdefs.h (ISDIGIT): Remove. All uses replaced by c_isdigit.
* bootstrap.conf (gnulib_modules): Add c-ctype.
* lib/.gitignore: Update.
* sed/compile.c: Include c-ctype.h.
This refactoring should help make future changes easier.
* bootstrap.conf (gnulib_modules): Add read-file.
* configure.ac (HAVE_FOPEN_RT): Remove; no longer used.
* lib/.gitignore: Adjust for read-file files.
* sed/compile.c: Include errno.h, read-file.h.
(struct prog_info.file): Remove. All uses removed.
(inchar, savchar, compile_program): Just read from the string.
Assume ! (NULL < NULL) && (NULL - NULL) == 0,
as per Gnulib guidelines. This should speed things up here.
(compile_file): Read the file into a string, and process the string.
* bootstrap.conf (gnulib_modules): Add mbrlen.
* lib/.gitignore, m4/.gitignore: Adjust
* sed/mbcs.c (is_mb_char), sed/sed.h (mbrlen1):
Prefer mbrlen to mbrtowc with NULL, as it can be a bit faster.
In addition to the usual ‘git submodule foreach git pull origin master’,
this also makes the following changes to adjust to current Gnulib:
* bootstrap: Sync from gnulib.
* lib/.gitignore, m4/.gitignore: Update to match current Gnulib.
* sed/utils.c: Include min-eloop-threshold.h, not eloop-threshold.h.
(follow_symlink): Use MIN_ELOOP_THRESHOLD, not __eloop_threshold ().
* basicdefs.h: Do not include alloca.h.
* bootstrap.conf (gnulib_modules): Remove alloca, stdalign,
c-ctype, closeout, mbrlen, verify.
Add version-etc.
* lib/.gitignore, m4/.gitignore: Omit now-unused files.
* po/POTFILES.in: Remove closeout.c.
* testsuite/test-mbrtowc.c: Do not include closeout.h.
(main): Close stdout by hand, to avoid a dependency on closeout.
Fix some some longstanding but unlikely integer overflows.
Internally, 'sed' now more often prefers signed integer arithmetic,
which can be checked automatically via 'gcc -fsanitize=undefined'.
* basicdefs.h (countT): Remove. All uses replaced
with a more-specific signed type, e.g., idx_t.
Similarly, change uses of types like size_t to
signed types like idx_t when appropriate.
(REALLOC): Remove; no longer used. We now use xpalloc
because that detects integer overflow in size calculations.
Also, we no longer use XCALLOC since the code never relies
on the storage being zero, and leaving it uninitialized is
more likely to catch errors when debugging implementations
are used. We use XNMALLOC instead, or xpalloc.
* bootstrap.conf (gnulib_modules): Add stdckdint, strtoimax.
* lib/.gitignore, m4/.gitignore: Update for new Gnulib modules.
* sed/compile.c: Include stdckdint.h.
(VECTOR_ALLOC_INCREMENT): Remove; no longer used.
(in_integer): Return maximal value if integer overflow.
All callers changed to expect this.
(next_cmd_entry): Use xpalloc instead of reallocating by hand,
which might suffer integer overflow.
(normalize_text): Don’t rely on system-defined conversion
of out-of-range size_t to int.
(next_cmd_entry): Arg is now pointer, not pointer-to-pointer.
All uses changed.
* sed/debug.c (debug_print_function): Don’t attempt to
fwrite a null pointer with a zero size.
* sed/execute.c: Include <stdckdint.h>, "minmax.h".
(resize_line): LEN arg is now increment, not total length,
to avoid overflow when calculating total length.
All uses changed. Do not assume lb->alloc * 2 cannot overflow.
(resize_line, line_copy): Use xpalloc instead of doing realloc by
hand, which might suffer integer overflow.
(str_append_modified): Do not add n to to->length until
after it's known this cannot overflow.
(read_file_line): Don’t assume ssize_t fits in long.
(get_backup_file_name): Don’t assume string length fits in int.
Do not assume PTR-1+1 works; behavior is undefined if PTR
is at buffer start. Check for integer overflow in buffer
size calculation.
(read_pattern_space): Check for line number overflow.
(match_address_p): Check for address overflow.
(debug_print_line): Omit unnecessary test for in->active being null.
(execute_program): Check for Q overflow.
* sed/regexp.c: Include <stdckdint.h>.
(match_regex): Don’t assume TYPE_MAXIMUM (regoff_t) == INT_MAX.
* sed/sed.c: Include inttypes.h, for strtoimax.
(main): Use strtoimax, not atoi.
* sed/utils.c (init_buffer): Use xmalloc and xpalloc
instead of guessing sizes ourselves, and unnecessarily
initializing.
(resize_buffer): Remove; all callers changed to use xpalloc.
(free_buffer): Don’t call free (NULL), since we already
test whether the pointer is null.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* sed/regexp.c (compile_regex_1): Do away with use of DFA_CASE_FOLD.
This gnulib update pulls in a dfa module in which that symbol no
longer exists. Instead, it uses RE_ICASE in syntax bits.
* m4/.gitignore: Add files created by running bootstrap.
* lib/.gitignore: Likewise.
The following definitions caused trouble for no gain. They caused
some .o file names to have a long additional prefix and even
resulted in some .Po files not being removed by "make distclean"
when building with recent automake. The only reason to use these
definitions was -- long ago -- to require different compiler options
for files in lib/ than elsewhere. That is no longer necessary.
* lib/local.mk (lib_libsed_a_CPPFLAGS): Remove definition.
(lib_libsed_a_CFLAGS, lib_libsed_a_LIBADD): Likewise.
(lib_libsed_a_DEPENDENCIES): Likewise.
Sed incorrectly matches the anchor '^' to a newline character
(ASCII 10) despite using NUL line terminators (-z).
See https://bugs.gnu.org/24615 .
Previous (incorrect) behavior:
$ printf "a\0b\0" | sed -z 'N;s/^/X/mg;' | od -An -a
X a nul b nul
With this fix:
$ printf "a\0b\0" | sed -z 'N;s/^/X/mg;' | od -An -a
X a nul X b nul
Current regex engine can not handle NUL multiline anchors.
Implement such search manually (by breaking the pattern into separate
NUL-terminated lines) and searching each one individually.
* NEWS: Mention it.
* sed/regexp.c (compile_regex_1): Don't use newline_anchor of regex, if
the buffer delimiter is not newline character.
(match_regex): Do above case line-by-line.
* testsuite/zero-anchor.good, testsuite/zero-anchor.inp,
testsuite/zero-anchor.sed: New test.
* testsuite/local.mk: Add the test.
* testsuite/Makefile.tests: Add the test.
* bootstrap.conf: Add memrchr.
* lib/.gitignore, m4/.gitignore: Ignore memrchr files.
* doc/local.mk (distclean-local): Remove rule. Moved to ...
* Makefile.am (distclean-local): ...here, and add a command to
remove all .deps directories. It could have stayed in that .mk
file, but since with this change, it is no longer specific to doc/,
the top level makes more sense.
* lib/local.mk (MOSTLYCLEANFILES): Remove this. Covered by the above.
Reported by Assaf Gordon in
https://lists.gnu.org/archive/html/sed-devel/2016-08/msg00019.html
* doc/local.mk (doc/sed.1): Depend not on sed.c, but on the binary
that help2man must run. Also depend on .version, to be sure we
pick up a version change. Also, be careful not to write directly
to $@: instead write to $@-t, and make read-only to mark as generated,
and then, only if all is well, rename atomically to $@.
Also, prefix commands with $(AM_V_GEN) and $(AM_V_at) and ensure
that doc/ exists via $(MKDIR_P).
(distclean-local): New target. Use this instead of ...
(dist-hook-man-page): ... this. Remove rule.
(SEDBIN): Remove unnecessary $(top_builddir)/ prefix.
(doc/sed.texi, doc/s-texi): Remove some uses of $(top_builddir).
Replace others with $(srcdir).
* Makefile (dist-hook): remove dependency on dist-hook-man-page.
* lib/local.mk (MOSTLYCLEANFILES): Arrange for lib/.deps/*.Po
to be removed as part of "make distclean", so that make distcheck
no longer fails when using latest automake.
* .gitignore: Ignore '.dirstamp' files
* bootstrap.conf: Add 'non-recursive-gnulib-prefix-hack' gnulib module.
Add missing 'stdalign' module.
(bootstrap_post_import_hook): Use module to adjust gnulib.mk file.
* configure.ac: Add 'subdir-objects' automake option, don't generate
makefiles in subdirectories. Adjust path of XFAIL tests.
* Makefile.am: Include 'local.mk' files, adjust for non-recursive make.
* doc/Makefile.am: Rename to ...
* doc/local.mk: Adjust paths. Rename SED to
SEDBIN variable, to avoid overriding the default system-wide SED value
which is automatically set by autotools.
* lib/Makefile.am: Rename to ...
* lib/local.mk: Adjust paths.
* lib/.gitignore: Ignore 'stdalign.h' from gnulib module.
* sed/Makefile.am: Rename to ...
* sed/local.mk: Adjust paths.
* testsuite/Makefile.am: Rename to ...
* testsuite/local.mk: Adjust paths.
* testsuite/init.cfg: Move to ...
* init.cfg: ... here. Moved to avoid changing the path in init.sh, and to
keep the same directory structure as coreutils.
(require_en_utf8_locale_): Adjust path for get-mb-cur-max executable.
(require_ja_shiftjis_locale_,require_valid_ja_shiftjis_locale_): Adjust
path for test-mbrtowc executable.
* m4/.gitignore: Add gnulib's non-recusive-make module.
* testsuite/runtest: Adjust paths of Makefile.tests and dir variables,
Skip utility program 'get-mb-cur-max' which is not a standalone test.
* testsuite/Makefile.tests: Adjust path of sed exeutable.
* testsuite/eval.sed, testsuite/eval.good: Adjust path of sed executable.
* testsuite/cmd-R.sh,
testsuite/cmd-l.sh,
testsuite/colon-with-no-label.sh,
testsuite/comment-n.sh,
testsuite/compile-errors.sh,
testsuite/compile-tests.sh,
testsuite/convert-number.sh,
testsuite/execute-tests.sh,
testsuite/follow-symlinks-stdin.sh,
testsuite/follow-symlinks.sh,
testsuite/help-version.sh,
testsuite/in-place-hyphen.sh,
testsuite/in-place-suffix-backup.sh,
testsuite/invalid-mb-seq-UMR.sh,
testsuite/mb-bad-delim.sh,
testsuite/mb-charclass-non-utf8.sh,
testsuite/mb-match-slash.sh,
testsuite/mb-y-translate.sh,
testsuite/normalize-text.sh,
testsuite/nulldata.sh,
testsuite/panic-tests.sh,
testsuite/posix-char-class.sh,
testsuite/posix-mode-N.sh,
testsuite/posix-mode-addr.sh,
testsuite/posix-mode-bad-ref.sh,
testsuite/posix-mode-s.sh,
testsuite/range-overlap.sh,
testsuite/recursive-escape-c.sh,
testsuite/regex-errors.sh,
testsuite/stdin-prog.sh,
testsuite/subst-mb-incomplete.sh,
testsuite/subst-options.sh,
testsuite/subst-replacement.sh,
testsuite/temp-file-cleanup.sh,
testsuite/unbuffered.sh: Adjust paths of init.sh, sed executable.
Sed tests purposefully use invalid non-utf8 multibyte sequences to
test sed's handling of invalid input/program.
Some implementations wrongly accept invalid sequences, causing false
alarms. Add a test program to detect such buggy implementations and skip
the tests if needed.
See discussion in:
http://lists.gnu.org/archive/html/sed-devel/2016-07/msg00005.htmlhttp://lists.gnu.org/archive/html/sed-devel/2016-07/msg00000.htmlhttp://lists.gnu.org/archive/html/sed-devel/2016-06/msg00031.htmlhttp://lists.gnu.org/archive/html/sed-devel/2016-06/msg00014.html
* bootstrap.conf: Add gnulib's closeout module.
* m4/.gitignore, lib/.gitignore, po/POTFILES.in: Adjust after adding
closeout module.
* testsuite/Makefile.am (check_PROGRAMS): Add new program.
* testsuite/test-mbrtowc.c: New program.
* testsuite/init.cfg (require_valid_ja_eucjp_locale_)
(require_valid_ja_shiftjis_locale_): New functions. Call test-mbrtowc
with known invalid multibyte sequences to ensure implementation is not
buggy and rejects them; Skip test otherwise.
* testsuite/invalid-mb-seq-UMR.sh: Use require_valid_ja_eucjp_locale_.
* testsuite/mb-charclass-non-utf8.sh: Use require_valid_ja_shiftjis_locale_.