From bc7f50307ac7ce54f381cfe1b42b7142c28ced49 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Sun, 26 Dec 2004 15:44:47 +0000 Subject: [PATCH] warn against localization issues 2004-12-26 Paolo Bonzini * BUGS: Add section about [a-z] matching uppercase characters, and other locale issues. * doc/sed-in.texi [!PERL]: Likewise. git-archimport-id: bonzini@gnu.org--2004b/sed--stable--4.1--patch-31 --- BUGS | 18 ++++++++++++++++++ ChangeLog | 6 ++++++ doc/sed-in.texi | 28 +++++++++++++++++++++++++--- 3 files changed, 49 insertions(+), 3 deletions(-) diff --git a/BUGS b/BUGS index 023fa5d..c939031 100644 --- a/BUGS +++ b/BUGS @@ -101,3 +101,21 @@ regex syntax clashes (problems with backslashes) to delete every line up to the first one including `abc', and this is obtained with `0,/abc/d'. + +`[a-z]' is case insensitive + + You are encountering problems with locales. POSIX mandates that `[a-z]' + uses the current locale's collation order -- in C parlance, that means + strcoll(3) instead of strcmp(3). Some locales have a case insensitive + strcoll, others don't: one of those that have problems is Estonian. + + Another problem is that [a-z] tries to use collation symbols. This + only happens if you are on the GNU system, using GNU libc's regular + expression matcher instead of compiling the one supplied with GNU sed. + In a Danish locale, for example, the regular expression `^[a-z]$' + matches the string `aa', because aa is a single collating symbol that + comes after `a' and before `b'; `ll' behaves similarly in Spanish + locales, or `ij' in Dutch locales. + + To work around these problems, which may cause bugs in shell scripts, + set the LC_COLLATE and LC_CTYPE environment variables to `C'. diff --git a/ChangeLog b/ChangeLog index 13c8660..bbc6370 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,9 @@ +2004-12-26 Paolo Bonzini + + * BUGS: Add section about [a-z] matching uppercase characters, + and other locale issues. + * doc/sed-in.texi [!PERL]: Likewise. + 2004-11-15 Paolo Bonzini * sed/execute.c (str_append_modified): Copy the first character diff --git a/doc/sed-in.texi b/doc/sed-in.texi index 6cd4140..7d35999 100644 --- a/doc/sed-in.texi +++ b/doc/sed-in.texi @@ -2707,14 +2707,36 @@ bug@dots{}). @item @code{0a} does not work (gives an error) There is no line 0. 0 is a special address that is only used to treat -addresses like @samp{0,/@var{RE}/} as active when the script starts: if -you write @samp{1,/abc/d} and the first line includes the word @samp{abc}, +addresses like @code{0,/@var{RE}/} as active when the script starts: if +you write @code{1,/abc/d} and the first line includes the word @samp{abc}, then that match would be ignored because address ranges must span at least two lines (barring the end of the file); but what you probably wanted is to delete every line up to the first one including @samp{abc}, and this -is obtained with @samp{0,/abc/d}. +is obtained with @code{0,/abc/d}. + +@ifclear PERL +@item @code{[a-z]} is case insensitive +You are encountering problems with locales. POSIX mandates that @code{[a-z]} +uses the current locale's collation order -- in C parlance, that means using +@code{strcoll(3)} instead of @code{strcmp(3)}. Some locales have a +case-insensitive collation order, others don't: one of those that have +problems is Estonian. + +Another problem is that @code{[a-z]} tries to use collation symbols. +This only happens if you are on the @acronym{GNU} system, using +@acronym{GNU} libc's regular expression matcher instead of compiling the +one supplied with @acronym{GNU} sed. In a Danish locale, for example, +the regular expression @code{^[a-z]$} matches the string @samp{aa}, +because this is a single collating symbol that comes after @samp{a} +and before @samp{b}; @samp{ll} behaves similarly in Spanish +locales, or @samp{ij} in Dutch locales. + +To work around these problems, which may cause bugs in shell scripts, set +the @env{LC_COLLATE} and @env{LC_CTYPE} environment variables to @samp{C}. +@end ifclear @end table + @node Extended regexps @appendix Extended regular expressions @cindex Extended regular expressions, syntax