* sed/sed.h (c32rtomb1): When mbrtoc32 acts like musl (because
Gnulib has replaced it) but c32rtomb does not (because Gnulib does
not replace it), convert back to the byte by hand.
* testsuite/subst-mb-incomplete.sh:
Use en_US.UTF-8, not en_US.utf8, as the latter does not work
in NetBSD and require_en_utf8_locale_ tests for the former.
* cfg.mk (sc_THANKS_in_sorted): Do so here, too, for completeness.
Problem reported by Bruno Haible in:
https://lists.gnu.org/r/sed-devel/2026-08/msg00005.html
* testsuite/mb-charclass-non-utf8.sh:
Skip this test on macOS through 15.1 (Darwin through 24.1),
DragonFly, FreeBSD, NetBSD, OpenBSD.
Problem reported by Bruno Haible in:
https://lists.gnu.org/r/sed-devel/2026-08/msg00004.html
* testsuite/cmd-l.sh: Don’t test U+00A0 NO-BREAK SPACE as some
Solaris 11 derivatives think it’s nonprinting and it’s not worth
the trouble to convince them otherwise.
In production there’s no need to free storage just before exiting.
The freeing code might even have a bug and this would make the
program less reliable. And it slows things down a bit, compared
to just exiting. So, separate out PACIFY_LSAN (needed for ‘make
build-asan’) from lint (needed when building with
--enable-gcc-warnings).
* cfg.mk (ASAN_CFLAGS): Define PACIFY_LSAN, not lint.
* sed/compile.c (setup_replacement, finish_program):
* sed/execute.c (process_files):
* sed/regexp.c (release_regex):
* sed/sed.c (cleanup, main):
* sed/sed.h (struct subst):
* sed/utils.c (panic):
Use PACIFY_LSAN, not lint, to decide whether we are
pacifying the leak sanitizer.
* sed/sed.h (release_regex): Declare unconditionally,
as there’s no harm declaring functions nobody calls or defines,
and it’s even helpful to avoid inadvertent redefinition.
* bootstrap.conf (gnulib_modules): Add strnlen.
* lib/.gitignore, m4/.gitignore: Adjust.
* sed/compile.c (compile_program):
* sed/debug.c (debug_print_translation):
* sed/execute.c (translate_mb):
Adjust to data structure change.
* sed/sed.h (struct sed_cmd.translate.a.pair):
Keep the destination in multibyte rather than character form,
as this avoids a c32rtomb1 call during execution.
This fixes what in theory is undefined behavior when we have more than
one byte in the regular expression, though the behavior is well-defined
on all but debugging platforms.
* bootstrap.conf (gnulib_modules): Add flexmember, which we now use directly.
* sed/regexp.c: Include flexmember.h.
(compile_regex): Adjust to the fact that new_regex->re is now a
flexible array member.
* sed/sed.h (struct regex.re): Now a flexible array member.
* sed/compile.c (compile_program):
* sed/debug.c (debug_print_translation):
* sed/execute.c (execute_program):
Adjust to data structure change.
* sed/sed.h (struct sed_cmd): Rejigger representation of ‘y’ command
so that execution (and debugging) need not consult MB_CUR_MAX.
This is a bit faster on platforms where MB_CUR_MAX is slow.
* configure.ac: Remove calls to obsolete macros AC_C_CONST,
AC_TYPE_SIZE_T. Stop checking for the functions isatty, memcpy,
strchr, strtoul, pathconf, fchmod, as we no longer call those
functions ourselves. Remove no-longer-relevant --disable-i18n
help string. Sync the gcc-warnings stuff from Coreutils, except
for its single-binary stuff.
* sed/execute.c (match_address_p): Pacify gcc -Wswitch.
* sed/compile.c: Include minmax.h. Do not include ctype.h.
(convert_number): Examine starting at BUF, not at BUF + 1; caller
changed. Use a clearer way to stop the loop, which involves
addition not multiplication. Always store the resulting integer,
not a byte taken possibly from after end of input.
(normalize_text): Do not go past end of input. Treat incomplete
\c, \d, \o, and \x escapes like other unrecognized escapes.
Use c_toupper, not toupper, to avoid glitches in Turkish locales.
* testsuite/normalize-text.sh:
Test the fixed behavior, removing a TODO.
* testsuite/recursive-escape-c.sh: Adjust to new behavior.
This mostly avoids including files no longer needed.
It also moves dfa.h down, and removes stat-macros.h.
* bootstrap.conf (gnulib_modules): Remove stat-macros.
* lib/.gitignore: Adjust.
* sed/compile.c: Don’t include stdio.h, sys/types.h.
stdio.h is guaranteed by sed/sed.h, and sys/types.h should
now be redundant.
* sed/debug.c: Don’t include ctype.h, stdio.h, string.h, sys/types.h.
* sed/execute.c: Don’t include stat-macros.h, ctype.h,
selinux/context.h, stdio.h, sys/types.h.
* sed/local.mk (sed/version.c): Include "version.h"
to check interface consistency.
* sed/regexp.c: Don’t include ctype.h, limits.h, stdio.h.
Do include dfa.h.
* sed/sed.c: Don’t include stdio.h, sys/types.h, unistd.h.
* sed/sed.h: Don’t include dfa.h. Don’t include stdio.h or
unlocked-io.h, as utils.h does that for us.
* sed/utils.c: Don’t include unlocked-io.h, stdio.h, sys/types.h.
* sed/utils.h: Include unlocked-io.h instead of stdio.h.
Be more consistent about including files in this order:
- <config.h>
- The "foo.h" file in this directory that defines this module’s interface.
- Any other of this directory’s include files, in string syntax.
- Gnulib-only include files, using angle-bracket syntax.
- Include files that are standard on GNU/Linux.
In each group, use alphabetical order.
This is more likely to catch include-file issues,
such as an include file assuming some other include file.
It is now so small that separating it is more trouble than it’s worth.
* Makefile.am (noinst_HEADERS): Remove basicdefs.h.
* basicdefs.h: Remove.
* sed/sed.h: Do not include basicdefs.h. Instead, include gettext.h
and define ‘_’.
* sed/utils.c: Include utils.h first, to check that it’s standalone.
Do not include idx.h, since utils.h does that for us.
Include gettext.h and define ‘_’.
* sed/utils.h: Include idx.h instead of basicdefs.h.
* basicdefs.h: Move inclusion of locale.h to sed/sed.c.
(obstack_chunk_alloc, obstack_chunk_free):
Move these macro definitions to sed/compile.c.
* sed/debug.c: Don’t include basicdefs.h, as sed.h does that for us.
These tests should no longer be needed, as we no longer restart
mbrtowc after an incomplete scan. This effectively reverts commit
77a7c6a731 dated 2016-07-04.
* init.cfg (require_valid_ja_shiftjis_locale_)
(require_valid_ja_eucjp_locale_): Remove. All uses removed.
* testsuite/local.mk (check_PROGRAMS): Remove testsuite/test-mbrtowc.
(testsuite_test_mbrtowc_LDADD): Remove.
* testsuite/test-mbrtowc.c: Remove.
* sed/compile.c (bad_prog, bad_prog_notranslate)
(mark_subst_opts): Omit some unreachable code (including an
unreachable ‘unreachable ()’!) to pacify Oracle Developer Studio 12.6.
sed is single-threaded, so does not need multithreading support.
* bootstrap.conf (avoided_gnulib_modules): Avoid lock, not lock-tests.
(bootstrap_post_import_hook): Copy lib/glthread/lock.h by hand,
as we now avoid ‘lock’.
* configure.ac (GNULIB_EXCLUDE_SINGLE_THREAD)
(GNULIB_LOCALENAME_SINGLE_THREAD)
(GNULIB_MBRTOWC_SINGLE_THREAD, GNULIB_REGEX_SINGLE_THREAD)
(GNULIB_WCHAR_SINGLE_LOCALE, GNULIB_MBRTOC32_REGULAR):
New macros.
* lib/.gitignore, m4/.gitignore: Adjust accordingly.
sed: also avoid ‘lock’ module
* bootstrap.conf
Also, remove EMX support, as EMX is no longer used.
* basicdefs.h: Don’t include stdbool.h;
the Gnulib bool module means it’s not needed any more.
(PATH_MAX): Remove; no longer used.
(initialize_main): Remove. No longer needed now that we
no longer support EMX. All uses removed.
* sed/execute.c (open_next_file):
* sed/utils.c (ck_fdopen): Remove EMX support.
These days MB_CUR_MAX is so fast on its own, and we so rarely use it,
that there’s little point to our caching it.
* sed/mbcs.c: Remove.
All uses of mb_cur_max replaced by MB_CUR_MAX.
All uses of initialize_mbcs removed.
* sed/local.mk (sed_sed_SOURCES): Remove sed/mbcs.c.
* basicdefs.h: Do not include wchar.h, wctype.h.
(ISASCII, ISBLANK, ISSPACE): Remove. All uses replaced
by c32isblank, c32isspace.
* bootstrap.conf (gnulib_modules): Add c32isblank, c32isprint,
c32isspace, c32rtomb, c32toower, c32toupper, mcel. Remove btowc,
localcharset, mbrlen, mbrtowc, mbsinit, wcrtomb, wctob, wctype-h.
* configure.ac: Do not check for headers locale.h, errno.h,
wchar.h, wctype.h, mcheck.h.
Do not check for isascii. Remove ac_cv_func_wcscoll hack.
(XFAIL_TESTS): Remove check for whether to run UTF-8 Cyrillic
tests, as I hope the other changes in this patch should fix
problems in that area.
* lib/.gitignore, m4/.gitignore: Adjust
* sed/compile.c (struct prog_info.prev): New member.
Adjust all users of this struct to update and use it when needed.
(INCHAR_EOF): New constant.
(inchar): Return a char32_t value for each wide character, a
negative byte value for encoding errors, and INCHAR_EOF for end of
input. All callers changed.
(savchar): Now takes no arguments. All callers changed.
(in_nonblank): Use char32_t tests rather than ctype.h tests.
(add_prev_to_buffer): New function.
(add_then_next, match_slash, read_text):
Use add_prev_to_buffer instead of add1_buffer when the idea is to
copy a character from input to output, as the character might be
multibyte and add1_buffer does only single-byte.
(match_slash): Support multibyte delimiters. Test for matchers
individually rather than via strchr, to make it clearer that we
are not testing for '\0'.
* sed/compile.c (add_then_next, snarf_char_class):
* sed/debug.c (debug_print_mb):
* sed/execute.c (line_init, line_reset, line_copy, line_append)
(line_exchange):
Remove now-unnecessary arg. All callers changed.
* sed/compile.c (compile_program, normalize_text):
* sed/debug.c (debug_print_regex_pattern):
* sed/execute.c (str_append_modified, do_list, translate_mb)
(debug_print_line):
Use mcel_scan/c32rtomb1 not mbrtowc1/mbrlen1/wcrtomb,
as mcel_scan should work better on MS-Windows.
* sed/debug.c (debug_print_char): Replace args c and clen
with an mcel_t arg g. All callers changed.
(debug_print_mb): Adjust to new meaning of int “characters”.
* sed/execute.c (struct line.mbstate): Remove. All uses removed.
(str_append): Omit no-longer-needed code put in just to update a
no-longer-needed mbstate_t.
(str_append_modified): Don’t worry about c32rtomb1 returning
(size_t) -1, as it now returns idx_t and anyway that cannot happen.
* sed/mbcs.c: Don’t include string.h, localcharset.h.
(is_mb_char, is_utf8): Remove. All uses removed.
* sed/sed.h: Include mcel.h.
(mbrtowc1, mbrlen1): Remove. All callers changed to use mcel_scan.
(c32rtomb1): New function.
(IS_MB_CHAR): Remove; no longer used.
* sed/utils.c (add1_buffer): Now returns void, since no caller
used the return value. 2nd arg is now char, not int.
Remove special case for EOF. All callers changed.
* testsuite/mb-bad-delim.sh, testsuite/mb-y-translate.sh:
Multibyte delimiters are no longer prohibited, so adjust test
cases accordingly. Put newlines at end of text files,
as required by POSIX.
Instead of converting pointers between char * and unsigned char *,
convert their members between char and unsigned char.
This is a little cleaner as pointer casts are too powerful in C.
* sed/compile.c (struct prog_info, compile_program):
* sed/execute.c (execute_program):
* sed/sed.h (struct sed_cmd):
Use plain char * rather than unsigned char *.
All uses changed.
Fix a longstanding bug where the 'l' command mistakenly
escapes bytes in a multibyte character.
* basicdefs.h (ISPRINT): Remove. All uses removed.
* bootstrap.conf (gnulib_modules): Add wctype-h, which is now needed
directly instead of indirectly, as we now use iswprint.
* sed/debug.c (debug_print_char):
* sed/execute.c (do_list):
Use iswprint instead of ISPRINT, so that things work correctly
with multibyte locales.
* sed/debug.c (debug_print_char):
New args P and CLEN. All uses changed.
* testsuite/cmd-l.sh: Test for the bug.
ISDIGIT is locale-dependent, and in theory could misfire if
any locale has a single-byte digit that is not '0'..'9'.
Fix this by using c_isdigit, which should be a bit faster anyway.
* basicdefs.h (ISDIGIT): Remove. All uses replaced by c_isdigit.
* bootstrap.conf (gnulib_modules): Add c-ctype.
* lib/.gitignore: Update.
* sed/compile.c: Include c-ctype.h.
This refactoring should help make future changes easier.
* bootstrap.conf (gnulib_modules): Add read-file.
* configure.ac (HAVE_FOPEN_RT): Remove; no longer used.
* lib/.gitignore: Adjust for read-file files.
* sed/compile.c: Include errno.h, read-file.h.
(struct prog_info.file): Remove. All uses removed.
(inchar, savchar, compile_program): Just read from the string.
Assume ! (NULL < NULL) && (NULL - NULL) == 0,
as per Gnulib guidelines. This should speed things up here.
(compile_file): Read the file into a string, and process the string.
When handling the y command, or the case-changing replacement,
stay a bit more in the wchar_t world, to simplify allocation
and help make future changes simpler.
* basicdefs.h (STREQ_LEN): Remove; no longer used.
* cfg.mk (sc_prohibit_strncmp): Don’t suggest STREQ_LEN.
* sed/compile.c (compile_program):
* sed/execute.c (str_append_modified, translate_mb):
Instead of preserving the bytes of a multibyte character, just
save its wchar_t counterpart, or negative of its single-byte
encoding error.
* sed/debug.c (debug_print_mb): New static function.
(debug_print_translation): Use it to adjust to new data structure.
* sed/execute.c (translate_mb): Args are now NPAIRS and PAIR, not
TRANS. Caller changed.
* sed/sed.h (struct sed_cmd.translate.mb): Now a struct with
an npairs and pair members, instead of a single char **.
All uses changed.
* bootstrap.conf (gnulib_modules): Add mbrlen.
* lib/.gitignore, m4/.gitignore: Adjust
* sed/mbcs.c (is_mb_char), sed/sed.h (mbrlen1):
Prefer mbrlen to mbrtowc with NULL, as it can be a bit faster.
* sed/compile.c (compile_program, normalize_text):
Prefer separate locals to reusing locals, to make things a bit clearer.
* sed/execute.c (str_append_modified):
Treat invalid sequences consistently with elsewhere.
* sed/sed.c (SED_INLINE): New macro.
* sed/sed.h: Use _GL_INLINE_HEADER_BEGIN, _GL_INLINE_HEADER_END.
(SED_INLINE): Define if not already defined.
(MBRLEN): Remove this macro, replacing all uses with ...
(mbrlen1): ... this new function. All callers changed.
This simplifies them, and fixes a few inconsistencies.