838 Commits
Author SHA1 Message Date
Paul Eggert a179e7a8ad sed: revert "deal with musl-like mbrtoc32 ..."
* sed/sed.h (c32rtomb1): Revert recent change, as recent
Gnulib changes should make it unnecessary.
2026-09-20 17:07:12 -07:00
Paul Eggert 64d80e20f3 build: update gnulib submodule to latest 2026-09-20 17:07:12 -07:00
Bruno Haible ae24cdd560 tests: Skip mb-charclass-non-utf8.sh on macOS 15.
* testsuite/mb-charclass-non-utf8.sh: Skip this test on all macOS 15.x
versions.
2026-09-20 16:56:00 -07:00
Paul Eggert 0227bd67d1 sed: deal with musl-like mbrtoc32 behavior
* sed/sed.h (c32rtomb1): When mbrtoc32 acts like musl (because
Gnulib has replaced it) but c32rtomb does not (because Gnulib does
not replace it), convert back to the byte by hand.
2026-09-13 23:57:27 -07:00
Paul Eggert 4b0c1d1f0c build: update gnulib submodule to latest 2026-09-13 23:57:27 -07:00
Paul Eggert 7d45214948 tests: fix subst-mb-incomplete test on NetBSD
* testsuite/subst-mb-incomplete.sh:
Use en_US.UTF-8, not en_US.utf8, as the latter does not work
in NetBSD and require_en_utf8_locale_ tests for the former.
* cfg.mk (sc_THANKS_in_sorted): Do so here, too, for completeness.
2026-09-13 23:57:27 -07:00
Bruno Haible 1457250706 tests: Declare an expected failure on FreeBSD.
* bootstrap.conf (gnulib_modules): Add test-xfail.
* testsuite/local.mk (XFAIL_TESTS): New variable.
2026-09-13 23:57:27 -07:00
Paul Eggert 1cfb077a60 tests: skip mb-charclass-non-utf8.sh on macOS
Problem reported by Bruno Haible in:
https://lists.gnu.org/r/sed-devel/2026-08/msg00005.html
* testsuite/mb-charclass-non-utf8.sh:
Skip this test on macOS through 15.1 (Darwin through 24.1),
DragonFly, FreeBSD, NetBSD, OpenBSD.
2026-08-19 16:35:11 -07:00
Paul Eggert b235cac60b tests: don’t test no-break space
Problem reported by Bruno Haible in:
https://lists.gnu.org/r/sed-devel/2026-08/msg00004.html
* testsuite/cmd-l.sh: Don’t test U+00A0 NO-BREAK SPACE as some
Solaris 11 derivatives think it’s nonprinting and it’s not worth
the trouble to convince them otherwise.
2026-08-19 11:32:48 -07:00
Paul Eggert 308b851e4e maint: separate PACIFY_LSAN from lint
In production there’s no need to free storage just before exiting.
The freeing code might even have a bug and this would make the
program less reliable.  And it slows things down a bit, compared
to just exiting.  So, separate out PACIFY_LSAN (needed for ‘make
build-asan’) from lint (needed when building with
--enable-gcc-warnings).
* cfg.mk (ASAN_CFLAGS): Define PACIFY_LSAN, not lint.
* sed/compile.c (setup_replacement, finish_program):
* sed/execute.c (process_files):
* sed/regexp.c (release_regex):
* sed/sed.c (cleanup, main):
* sed/sed.h (struct subst):
* sed/utils.c (panic):
Use PACIFY_LSAN, not lint, to decide whether we are
pacifying the leak sanitizer.
* sed/sed.h (release_regex): Declare unconditionally,
as there’s no harm declaring functions nobody calls or defines,
and it’s even helpful to avoid inadvertent redefinition.
2026-08-17 01:59:47 -07:00
Paul Eggert f25dea95fc maint: update m4/.gitignore 2026-08-17 01:17:23 -07:00
Paul Eggert 4082bad781 build: update gnulib submodule to latest 2026-08-17 01:17:23 -07:00
Paul Eggert 80b85fe3ca sed: impove asymptotic y performance on long lines
* sed/execute.c (translate_mb): Avoid O(N**2) behavior when
executing ‘y’ on a long line in a multibyte locale.
2026-08-17 01:17:23 -07:00
Paul Eggert f7cc306958 sed: skip a c32rtomb1 call in y/a/b/
* bootstrap.conf (gnulib_modules): Add strnlen.
* lib/.gitignore, m4/.gitignore: Adjust.
* sed/compile.c (compile_program):
* sed/debug.c (debug_print_translation):
* sed/execute.c (translate_mb):
Adjust to data structure change.
* sed/sed.h (struct sed_cmd.translate.a.pair):
Keep the destination in multibyte rather than character form,
as this avoids a c32rtomb1 call during execution.
2026-08-17 01:17:23 -07:00
Paul Eggert e236a2345e sed: MB_LEN_MAX→MCEL_CUR_MAX
* sed/debug.c (debug_print_mb):
* sed/execute.c (str_append_modified, do_list, translate_mb):
Prefer MCEL_LEN_MAX to MB_LEN_MAX where either will do.
2026-08-17 01:17:23 -07:00
Paul Eggert c39cfe4fa3 sed: remove compile_program use of MB_CUR_MAX
* sed/compile.c (compile_program): Use localeinfo.multibyte
as it should be a bit faster than using MB_CUR_MAX.
2026-08-17 01:17:23 -07:00
Paul Eggert e4e57d8c0d sed: struct regex.re is now flexible
This fixes what in theory is undefined behavior when we have more than
one byte in the regular expression, though the behavior is well-defined
on all but debugging platforms.
* bootstrap.conf (gnulib_modules): Add flexmember, which we now use directly.
* sed/regexp.c: Include flexmember.h.
(compile_regex): Adjust to the fact that new_regex->re is now a
flexible array member.
* sed/sed.h (struct regex.re): Now a flexible array member.
2026-08-17 01:17:23 -07:00
Paul Eggert 814ac6d6a1 sed: consult MB_CUR_MAX only when compiling
* sed/compile.c (compile_program):
* sed/debug.c (debug_print_translation):
* sed/execute.c (execute_program):
Adjust to data structure change.
* sed/sed.h (struct sed_cmd): Rejigger representation of ‘y’ command
so that execution (and debugging) need not consult MB_CUR_MAX.
This is a bit faster on platforms where MB_CUR_MAX is slow.
2026-08-17 01:17:23 -07:00
Paul Eggert d933ecbc05 build: modernize configure.ac a bit
* configure.ac: Remove calls to obsolete macros AC_C_CONST,
AC_TYPE_SIZE_T.  Stop checking for the functions isatty, memcpy,
strchr, strtoul, pathconf, fchmod, as we no longer call those
functions ourselves.  Remove no-longer-relevant --disable-i18n
help string.  Sync the gcc-warnings stuff from Coreutils, except
for its single-binary stuff.
* sed/execute.c (match_address_p): Pacify gcc -Wswitch.
2026-08-17 01:17:23 -07:00
Paul Eggert d80e1ba179 Distribute lib/glthread/lock.h too 2026-08-17 01:17:23 -07:00
Paul Eggert b833ee016b doc: improve ASCII/multibyte doc 2026-08-17 01:17:23 -07:00
Paul Eggert 1e096959ac sed: fix \[cdox] at end of input
* sed/compile.c: Include minmax.h.  Do not include ctype.h.
(convert_number): Examine starting at BUF, not at BUF + 1; caller
changed.  Use a clearer way to stop the loop, which involves
addition not multiplication.  Always store the resulting integer,
not a byte taken possibly from after end of input.
(normalize_text): Do not go past end of input.  Treat incomplete
\c, \d, \o, and \x escapes like other unrecognized escapes.
Use c_toupper, not toupper, to avoid glitches in Turkish locales.
* testsuite/normalize-text.sh:
Test the fixed behavior, removing a TODO.
* testsuite/recursive-escape-c.sh: Adjust to new behavior.
2026-08-17 01:17:23 -07:00
Paul Eggert 8b7aa3d6cd sed: ptrdiff_t→idx_t in get_backup_file_name
* sed/execute.c (get_backup_file_name): Fix a stray ptrdiff_t.
2026-08-17 01:17:23 -07:00
Paul Eggert 077b4d4099 sed: #include file cleanup
This mostly avoids including files no longer needed.
It also moves dfa.h down, and removes stat-macros.h.
* bootstrap.conf (gnulib_modules): Remove stat-macros.
* lib/.gitignore: Adjust.
* sed/compile.c: Don’t include stdio.h, sys/types.h.
stdio.h is guaranteed by sed/sed.h, and sys/types.h should
now be redundant.
* sed/debug.c: Don’t include ctype.h, stdio.h, string.h, sys/types.h.
* sed/execute.c: Don’t include stat-macros.h, ctype.h,
selinux/context.h, stdio.h, sys/types.h.
* sed/local.mk (sed/version.c): Include "version.h"
to check interface consistency.
* sed/regexp.c: Don’t include ctype.h, limits.h, stdio.h.
Do include dfa.h.
* sed/sed.c: Don’t include stdio.h, sys/types.h, unistd.h.
* sed/sed.h: Don’t include dfa.h.  Don’t include stdio.h or
unlocked-io.h, as utils.h does that for us.
* sed/utils.c: Don’t include unlocked-io.h, stdio.h, sys/types.h.
* sed/utils.h: Include unlocked-io.h instead of stdio.h.
2026-08-17 01:17:23 -07:00
Paul Eggert 7a1fda23a9 sed: more-consistent #include ordering
Be more consistent about including files in this order:
- <config.h>
- The "foo.h" file in this directory that defines this module’s interface.
- Any other of this directory’s include files, in string syntax.
- Gnulib-only include files, using angle-bracket syntax.
- Include files that are standard on GNU/Linux.
In each group, use alphabetical order.
This is more likely to catch include-file issues,
such as an include file assuming some other include file.
2026-08-17 01:17:23 -07:00
Paul Eggert c7221b9610 sed: remove basicdefs.h
It is now so small that separating it is more trouble than it’s worth.
* Makefile.am (noinst_HEADERS): Remove basicdefs.h.
* basicdefs.h: Remove.
* sed/sed.h: Do not include basicdefs.h.  Instead, include gettext.h
and define ‘_’.
* sed/utils.c: Include utils.h first, to check that it’s standalone.
Do not include idx.h, since utils.h does that for us.
Include gettext.h and define ‘_’.
* sed/utils.h: Include idx.h instead of basicdefs.h.
2026-08-17 01:17:23 -07:00
Paul Eggert 5ba9bc5cfb sed: basicdefs.h trimdown
* basicdefs.h: Move inclusion of locale.h to sed/sed.c.
(obstack_chunk_alloc, obstack_chunk_free):
Move these macro definitions to sed/compile.c.
* sed/debug.c: Don’t include basicdefs.h, as sed.h does that for us.
2026-08-17 01:17:23 -07:00
Paul Eggert f3b29e0d76 tests: remove test for buggy mbrtowc
These tests should no longer be needed, as we no longer restart
mbrtowc after an incomplete scan.  This effectively reverts commit
77a7c6a731 dated 2016-07-04.
* init.cfg (require_valid_ja_shiftjis_locale_)
(require_valid_ja_eucjp_locale_): Remove.  All uses removed.
* testsuite/local.mk (check_PROGRAMS): Remove testsuite/test-mbrtowc.
(testsuite_test_mbrtowc_LDADD): Remove.
* testsuite/test-mbrtowc.c: Remove.
2026-08-17 01:17:23 -07:00
Paul Eggert 228652f200 sed: pacify ODS 12.6
* sed/compile.c (bad_prog, bad_prog_notranslate)
(mark_subst_opts): Omit some unreachable code (including an
unreachable ‘unreachable ()’!) to pacify Oracle Developer Studio 12.6.
2026-08-17 01:17:23 -07:00
Paul Eggert 598c3d0b95 sed: omit some multithreading cruft
sed is single-threaded, so does not need multithreading support.
* bootstrap.conf (avoided_gnulib_modules): Avoid lock, not lock-tests.
(bootstrap_post_import_hook): Copy lib/glthread/lock.h by hand,
as we now avoid ‘lock’.
* configure.ac (GNULIB_EXCLUDE_SINGLE_THREAD)
(GNULIB_LOCALENAME_SINGLE_THREAD)
(GNULIB_MBRTOWC_SINGLE_THREAD, GNULIB_REGEX_SINGLE_THREAD)
(GNULIB_WCHAR_SINGLE_LOCALE, GNULIB_MBRTOC32_REGULAR):
New macros.
* lib/.gitignore, m4/.gitignore: Adjust accordingly.

sed: also avoid ‘lock’ module

* bootstrap.conf
2026-08-17 01:17:23 -07:00
Paul Eggert 6d22645e41 sed: basicdefs.h cleanup
Also, remove EMX support, as EMX is no longer used.
* basicdefs.h: Don’t include stdbool.h;
the Gnulib bool module means it’s not needed any more.
(PATH_MAX): Remove; no longer used.
(initialize_main): Remove.  No longer needed now that we
no longer support EMX.  All uses removed.
* sed/execute.c (open_next_file):
* sed/utils.c (ck_fdopen): Remove EMX support.
2026-08-17 01:17:23 -07:00
Paul Eggert f137d0a95f sed: MB_CUR_MAX→MB_LEN_MAX for speed
* sed/execute.c (str_append_modified): Use MB_LEN_MAX
not MB_CUR_MAX, as it’s a bit faster and there’s little
point to space optimization here.
2026-08-17 01:17:23 -07:00
Paul Eggert 0222c34e3a sed: don’t cache MB_CUR_MAX
These days MB_CUR_MAX is so fast on its own, and we so rarely use it,
that there’s little point to our caching it.
* sed/mbcs.c: Remove.
All uses of mb_cur_max replaced by MB_CUR_MAX.
All uses of initialize_mbcs removed.
* sed/local.mk (sed_sed_SOURCES): Remove sed/mbcs.c.
2026-08-17 01:17:23 -07:00
Paul Eggert 2b155a6c8b sed: support multibyte delimiters
* basicdefs.h: Do not include wchar.h, wctype.h.
(ISASCII, ISBLANK, ISSPACE): Remove.  All uses replaced
by c32isblank, c32isspace.
* bootstrap.conf (gnulib_modules): Add c32isblank, c32isprint,
c32isspace, c32rtomb, c32toower, c32toupper, mcel.  Remove btowc,
localcharset, mbrlen, mbrtowc, mbsinit, wcrtomb, wctob, wctype-h.
* configure.ac: Do not check for headers locale.h, errno.h,
wchar.h, wctype.h, mcheck.h.
Do not check for isascii.  Remove ac_cv_func_wcscoll hack.
(XFAIL_TESTS): Remove check for whether to run UTF-8 Cyrillic
tests, as I hope the other changes in this patch should fix
problems in that area.
* lib/.gitignore, m4/.gitignore: Adjust
* sed/compile.c (struct prog_info.prev): New member.
Adjust all users of this struct to update and use it when needed.
(INCHAR_EOF): New constant.
(inchar): Return a char32_t value for each wide character, a
negative byte value for encoding errors, and INCHAR_EOF for end of
input.  All callers changed.
(savchar): Now takes no arguments.  All callers changed.
(in_nonblank): Use char32_t tests rather than ctype.h tests.
(add_prev_to_buffer): New function.
(add_then_next, match_slash, read_text):
Use add_prev_to_buffer instead of add1_buffer when the idea is to
copy a character from input to output, as the character might be
multibyte and add1_buffer does only single-byte.
(match_slash): Support multibyte delimiters.  Test for matchers
individually rather than via strchr, to make it clearer that we
are not testing for '\0'.
* sed/compile.c (add_then_next, snarf_char_class):
* sed/debug.c (debug_print_mb):
* sed/execute.c (line_init, line_reset, line_copy, line_append)
(line_exchange):
Remove now-unnecessary arg.  All callers changed.
* sed/compile.c (compile_program, normalize_text):
* sed/debug.c (debug_print_regex_pattern):
* sed/execute.c (str_append_modified, do_list, translate_mb)
(debug_print_line):
Use mcel_scan/c32rtomb1 not mbrtowc1/mbrlen1/wcrtomb,
as mcel_scan should work better on MS-Windows.
* sed/debug.c (debug_print_char): Replace args c and clen
with an mcel_t arg g.  All callers changed.
(debug_print_mb): Adjust to new meaning of int “characters”.
* sed/execute.c (struct line.mbstate): Remove.  All uses removed.
(str_append): Omit no-longer-needed code put in just to update a
no-longer-needed mbstate_t.
(str_append_modified): Don’t worry about c32rtomb1 returning
(size_t) -1, as it now returns idx_t and anyway that cannot happen.
* sed/mbcs.c: Don’t include string.h, localcharset.h.
(is_mb_char, is_utf8): Remove.  All uses removed.
* sed/sed.h: Include mcel.h.
(mbrtowc1, mbrlen1): Remove.  All callers changed to use mcel_scan.
(c32rtomb1): New function.
(IS_MB_CHAR): Remove; no longer used.
* sed/utils.c (add1_buffer): Now returns void, since no caller
used the return value.  2nd arg is now char, not int.
Remove special case for EOF.  All callers changed.
* testsuite/mb-bad-delim.sh, testsuite/mb-y-translate.sh:
Multibyte delimiters are no longer prohibited, so adjust test
cases accordingly.  Put newlines at end of text files,
as required by POSIX.
2026-08-17 01:17:23 -07:00
Paul Eggert 11f413d9c0 doc: say a:i::r:?
Document that there is no command that is like r but
acts before the current line (Bug#36201).
2026-08-17 01:17:23 -07:00
Paul Eggert c3744164d4 sed: streamline utils.c a bit
* sed/utils.c (quotef): Now static.
(ck_fdopen): Define only if needed.
(add_buffer): Remove; unused.
2026-08-17 01:17:23 -07:00
Paul Eggert 49336f7e4b sed: fix EOF value assumption
* sed/sed.c (main): Don’t assume EOF == -1; it’s not required
by either POSIX or C.
2026-08-17 01:17:23 -07:00
Paul Eggert 156efe87ae sed: prefer plain char *
Instead of converting pointers between char * and unsigned char *,
convert their members between char and unsigned char.
This is a little cleaner as pointer casts are too powerful in C.
* sed/compile.c (struct prog_info, compile_program):
* sed/execute.c (execute_program):
* sed/sed.h (struct sed_cmd):
Use plain char * rather than unsigned char *.
All uses changed.
2026-08-17 01:17:23 -07:00
Paul Eggert da5cecf553 sed: 'l' and printable multibyte characters
Fix a longstanding bug where the 'l' command mistakenly
escapes bytes in a multibyte character.
* basicdefs.h (ISPRINT): Remove.  All uses removed.
* bootstrap.conf (gnulib_modules): Add wctype-h, which is now needed
directly instead of indirectly, as we now use iswprint.
* sed/debug.c (debug_print_char):
* sed/execute.c (do_list):
Use iswprint instead of ISPRINT, so that things work correctly
with multibyte locales.
* sed/debug.c (debug_print_char):
New args P and CLEN.  All uses changed.
* testsuite/cmd-l.sh: Test for the bug.
2026-08-17 01:17:23 -07:00
Paul Eggert 6c92794c2b sed: remove ISALNUM etc
* basicdefs.h (ISALNUM, ISALPHA, ISCNTRL, ISLOWER, ISPUNCT)
(ISUPPER, ISXDIGIT): Remove; unused.
2026-08-17 01:17:23 -07:00
Paul Eggert fe0a844c0a sed: ISDIGIT → c_isdigit
ISDIGIT is locale-dependent, and in theory could misfire if
any locale has a single-byte digit that is not '0'..'9'.
Fix this by using c_isdigit, which should be a bit faster anyway.
* basicdefs.h (ISDIGIT): Remove.  All uses replaced by c_isdigit.
* bootstrap.conf (gnulib_modules): Add c-ctype.
* lib/.gitignore: Update.
* sed/compile.c: Include c-ctype.h.
2026-08-17 01:17:23 -07:00
Paul Eggert d43a870aa2 sed: simplify reading commands from file
This refactoring should help make future changes easier.
* bootstrap.conf (gnulib_modules): Add read-file.
* configure.ac (HAVE_FOPEN_RT): Remove; no longer used.
* lib/.gitignore: Adjust for read-file files.
* sed/compile.c: Include errno.h, read-file.h.
(struct prog_info.file): Remove.  All uses removed.
(inchar, savchar, compile_program): Just read from the string.
Assume ! (NULL < NULL) && (NULL - NULL) == 0,
as per Gnulib guidelines.  This should speed things up here.
(compile_file): Read the file into a string, and process the string.
2026-08-17 01:17:23 -07:00
Paul Eggert 4e2995cd01 sed: simplify multibyte substitution
When handling the y command, or the case-changing replacement,
stay a bit more in the wchar_t world, to simplify allocation
and help make future changes simpler.
* basicdefs.h (STREQ_LEN): Remove; no longer used.
* cfg.mk (sc_prohibit_strncmp): Don’t suggest STREQ_LEN.
* sed/compile.c (compile_program):
* sed/execute.c (str_append_modified, translate_mb):
Instead of preserving the bytes of a multibyte character, just
save its wchar_t counterpart, or negative of its single-byte
encoding error.
* sed/debug.c (debug_print_mb): New static function.
(debug_print_translation): Use it to adjust to new data structure.
* sed/execute.c (translate_mb): Args are now NPAIRS and PAIR, not
TRANS.  Caller changed.
* sed/sed.h (struct sed_cmd.translate.mb): Now a struct with
an npairs and pair members, instead of a single char **.
All uses changed.
2026-08-17 01:17:23 -07:00
Paul Eggert bd3b429b80 sed: shrink struct sed_cmd translate slots
* sed/sed.h (struct sed_cmd): Rename translate to translate.sb,
and translatemb to translate.mb, as they are never both used.
All uses changed.
2026-08-17 01:17:23 -07:00
Paul Eggert ff5bec34b1 sed: remove MEMDUP, OB_MALLOC
* basicdefs.h (MEMDUP, OB_MALLOC):
Remove.  All uses of these macros replaced by function calls,
which are simpler nowadays.
2026-08-17 01:17:23 -07:00
Paul Eggert f0e2d07924 sed: remove YMAP_LENGTH
* sed/compile.c (YMAP_LENGTH): Remove.
All uses replaced by UCHAR_MAX + 1.
2026-08-17 01:17:23 -07:00
Paul Eggert 0e9c603e82 sed: prefer mbrlen to mbrtowc+NULL
* bootstrap.conf (gnulib_modules): Add mbrlen.
* lib/.gitignore, m4/.gitignore: Adjust
* sed/mbcs.c (is_mb_char), sed/sed.h (mbrlen1):
Prefer mbrlen to mbrtowc with NULL, as it can be a bit faster.
2026-08-17 01:17:23 -07:00
Paul Eggert 2d2a87d3b6 sed: more consistency about encoding errors
* sed/compile.c (compile_program, normalize_text):
Prefer separate locals to reusing locals, to make things a bit clearer.
* sed/execute.c (str_append_modified):
Treat invalid sequences consistently with elsewhere.
* sed/sed.c (SED_INLINE): New macro.
* sed/sed.h: Use _GL_INLINE_HEADER_BEGIN, _GL_INLINE_HEADER_END.
(SED_INLINE): Define if not already defined.
(MBRLEN): Remove this macro, replacing all uses with ...
(mbrlen1): ... this new function.  All callers changed.
This simplifies them, and fixes a few inconsistencies.
2026-08-17 01:17:23 -07:00
Paul Eggert 818d3a5675 sed: don’t squash size_t to int
* sed/mbcs.c (is_mb_char): Don’t rely on implementation-defined
behavior when converting values greater than INT_MAX to int.
2026-08-17 01:17:23 -07:00
Paul Eggert de39cf77c0 sed: fix panic printf format
* sed/utils.c (ck_fwrite): %jd → %td for idx_t.
This matters for i686.
2026-08-17 01:17:23 -07:00