2012-02-05 Paolo Bonzini <bonzini@gnu.org> Jim Hill <gjthill@gmail.com> * autoboot.conf: Change getline to getdelim. * doc/sed-in.texi: Document -z/--null-data. * doc/sed.texi: Regenerate. * sed/execute.c: Change '\n' to buffer_delimiter. * sed/sed.c: Add support for -z/--null-data. * sed/sed.h: Add buffer_delimiter. * sed/utils.c: Change ck_getline to ck_getdelim. * sed/utils.h: Change ck_getline to ck_getdelim. * NEWS: Document new option.
4355 lines
141 KiB
Plaintext
4355 lines
141 KiB
Plaintext
\input texinfo @c -*-texinfo-*-
|
|
@c Do not edit this file!! It is automatically generated from sed-in.texi.
|
|
@c
|
|
@c -- Stuff that needs adding: ----------------------------------------------
|
|
@c (nothing!)
|
|
@c --------------------------------------------------------------------------
|
|
@c Check for consistency: regexps in @code, text that they match in @samp.
|
|
@c
|
|
@c Tips:
|
|
@c @command for command
|
|
@c @samp for command fragments: @samp{cat -s}
|
|
@c @code for sed commands and flags
|
|
@c Use ``quote'' not `quote' or "quote".
|
|
@c
|
|
@c %**start of header
|
|
@setfilename sed.info
|
|
@settitle sed, a stream editor
|
|
@c %**end of header
|
|
|
|
@c @smallbook
|
|
|
|
@include version.texi
|
|
|
|
@c Combine indices.
|
|
@syncodeindex ky cp
|
|
@syncodeindex pg cp
|
|
@syncodeindex tp cp
|
|
|
|
@defcodeindex op
|
|
@syncodeindex op fn
|
|
|
|
@include config.texi
|
|
|
|
@copying
|
|
This file documents version @value{VERSION} of
|
|
@value{SSED}, a stream editor.
|
|
|
|
Copyright @copyright{} 1998, 1999, 2001, 2002, 2003, 2004 Free
|
|
Software Foundation, Inc.
|
|
|
|
This document is released under the terms of the @acronym{GNU} Free
|
|
Documentation License as published by the Free Software Foundation;
|
|
either version 1.1, or (at your option) any later version.
|
|
|
|
You should have received a copy of the @acronym{GNU} Free Documentation
|
|
License along with @value{SSED}; see the file @file{COPYING.DOC}.
|
|
If not, write to the Free Software Foundation, 59 Temple Place - Suite
|
|
330, Boston, MA 02110-1301, USA.
|
|
|
|
There are no Cover Texts and no Invariant Sections; this text, along
|
|
with its equivalent in the printed manual, constitutes the Title Page.
|
|
@end copying
|
|
|
|
@setchapternewpage off
|
|
|
|
@titlepage
|
|
@title @command{sed}, a stream editor
|
|
@subtitle version @value{VERSION}, @value{UPDATED}
|
|
@author by Ken Pizzini, Paolo Bonzini
|
|
|
|
@page
|
|
@vskip 0pt plus 1filll
|
|
Copyright @copyright{} 1998, 1999 Free Software Foundation, Inc.
|
|
|
|
@insertcopying
|
|
|
|
Published by the Free Software Foundation, @*
|
|
51 Franklin Street, Fifth Floor @*
|
|
Boston, MA 02110-1301, USA
|
|
@end titlepage
|
|
|
|
|
|
@node Top
|
|
@top
|
|
|
|
@ifnottex
|
|
@insertcopying
|
|
@end ifnottex
|
|
|
|
@menu
|
|
* Introduction:: Introduction
|
|
* Invoking sed:: Invocation
|
|
* sed Programs:: @command{sed} programs
|
|
* Examples:: Some sample scripts
|
|
* Limitations:: Limitations and (non-)limitations of @value{SSED}
|
|
* Other Resources:: Other resources for learning about @command{sed}
|
|
* Reporting Bugs:: Reporting bugs
|
|
|
|
* Extended regexps:: @command{egrep}-style regular expressions
|
|
@ifset PERL
|
|
* Perl regexps:: Perl-style regular expressions
|
|
@end ifset
|
|
|
|
* Concept Index:: A menu with all the topics in this manual.
|
|
* Command and Option Index:: A menu with all @command{sed} commands and
|
|
command-line options.
|
|
|
|
@detailmenu
|
|
--- The detailed node listing ---
|
|
|
|
sed Programs:
|
|
* Execution Cycle:: How @command{sed} works
|
|
* Addresses:: Selecting lines with @command{sed}
|
|
* Regular Expressions:: Overview of regular expression syntax
|
|
* Common Commands:: Often used commands
|
|
* The "s" Command:: @command{sed}'s Swiss Army Knife
|
|
* Other Commands:: Less frequently used commands
|
|
* Programming Commands:: Commands for @command{sed} gurus
|
|
* Extended Commands:: Commands specific of @value{SSED}
|
|
* Escapes:: Specifying special characters
|
|
|
|
Examples:
|
|
* Centering lines::
|
|
* Increment a number::
|
|
* Rename files to lower case::
|
|
* Print bash environment::
|
|
* Reverse chars of lines::
|
|
* tac:: Reverse lines of files
|
|
* cat -n:: Numbering lines
|
|
* cat -b:: Numbering non-blank lines
|
|
* wc -c:: Counting chars
|
|
* wc -w:: Counting words
|
|
* wc -l:: Counting lines
|
|
* head:: Printing the first lines
|
|
* tail:: Printing the last lines
|
|
* uniq:: Make duplicate lines unique
|
|
* uniq -d:: Print duplicated lines of input
|
|
* uniq -u:: Remove all duplicated lines
|
|
* cat -s:: Squeezing blank lines
|
|
|
|
@ifset PERL
|
|
Perl regexps:: Perl-style regular expressions
|
|
* Backslash:: Introduces special sequences
|
|
* Circumflex/dollar sign/period:: Behave specially with regard to new lines
|
|
* Square brackets:: Are a bit different in strange cases
|
|
* Options setting:: Toggle modifiers in the middle of a regexp
|
|
* Non-capturing subpatterns:: Are not counted when backreferencing
|
|
* Repetition:: Allows for non-greedy matching
|
|
* Backreferences:: Allows for more than 10 back references
|
|
* Assertions:: Allows for complex look ahead matches
|
|
* Non-backtracking subpatterns:: Often gives more performance
|
|
* Conditional subpatterns:: Allows if/then/else branches
|
|
* Recursive patterns:: For example to match parentheses
|
|
* Comments:: Because things can get complex...
|
|
@end ifset
|
|
|
|
@end detailmenu
|
|
@end menu
|
|
|
|
|
|
@node Introduction
|
|
@chapter Introduction
|
|
|
|
@cindex Stream editor
|
|
@command{sed} is a stream editor.
|
|
A stream editor is used to perform basic text
|
|
transformations on an input stream
|
|
(a file or input from a pipeline).
|
|
While in some ways similar to an editor which
|
|
permits scripted edits (such as @command{ed}),
|
|
@command{sed} works by making only one pass over the
|
|
input(s), and is consequently more efficient.
|
|
But it is @command{sed}'s ability to filter text in a pipeline
|
|
which particularly distinguishes it from other types of
|
|
editors.
|
|
|
|
|
|
@node Invoking sed
|
|
@chapter Invocation
|
|
|
|
Normally @command{sed} is invoked like this:
|
|
|
|
@example
|
|
sed SCRIPT INPUTFILE...
|
|
@end example
|
|
|
|
The full format for invoking @command{sed} is:
|
|
|
|
@example
|
|
sed OPTIONS... [SCRIPT] [INPUTFILE...]
|
|
@end example
|
|
|
|
If you do not specify @var{INPUTFILE}, or if @var{INPUTFILE} is @file{-},
|
|
@command{sed} filters the contents of the standard input. The @var{script}
|
|
is actually the first non-option parameter, which @command{sed} specially
|
|
considers a script and not an input file if (and only if) none of the
|
|
other @var{options} specifies a script to be executed, that is if neither
|
|
of the @option{-e} and @option{-f} options is specified.
|
|
|
|
@command{sed} may be invoked with the following command-line options:
|
|
|
|
@table @code
|
|
@item --version
|
|
@opindex --version
|
|
@cindex Version, printing
|
|
Print out the version of @command{sed} that is being run and a copyright notice,
|
|
then exit.
|
|
|
|
@item --help
|
|
@opindex --help
|
|
@cindex Usage summary, printing
|
|
Print a usage message briefly summarizing these command-line options
|
|
and the bug-reporting address,
|
|
then exit.
|
|
|
|
@item -n
|
|
@itemx --quiet
|
|
@itemx --silent
|
|
@opindex -n
|
|
@opindex --quiet
|
|
@opindex --silent
|
|
@cindex Disabling autoprint, from command line
|
|
By default, @command{sed} prints out the pattern space
|
|
at the end of each cycle through the script (@pxref{Execution Cycle, ,
|
|
How @code{sed} works}).
|
|
These options disable this automatic printing,
|
|
and @command{sed} only produces output when explicitly told to
|
|
via the @code{p} command.
|
|
|
|
@item -e @var{script}
|
|
@itemx --expression=@var{script}
|
|
@opindex -e
|
|
@opindex --expression
|
|
@cindex Script, from command line
|
|
Add the commands in @var{script} to the set of commands to be
|
|
run while processing the input.
|
|
|
|
@item -f @var{script-file}
|
|
@itemx --file=@var{script-file}
|
|
@opindex -f
|
|
@opindex --file
|
|
@cindex Script, from a file
|
|
Add the commands contained in the file @var{script-file}
|
|
to the set of commands to be run while processing the input.
|
|
|
|
@item -i[@var{SUFFIX}]
|
|
@itemx --in-place[=@var{SUFFIX}]
|
|
@opindex -i
|
|
@opindex --in-place
|
|
@cindex In-place editing, activating
|
|
@cindex @value{SSEDEXT}, in-place editing
|
|
This option specifies that files are to be edited in-place.
|
|
@value{SSED} does this by creating a temporary file and
|
|
sending output to this file rather than to the standard
|
|
output.@footnote{This applies to commands such as @code{=},
|
|
@code{a}, @code{c}, @code{i}, @code{l}, @code{p}. You can
|
|
still write to the standard output by using the @code{w}
|
|
@cindex @value{SSEDEXT}, @file{/dev/stdout} file
|
|
or @code{W} commands together with the @file{/dev/stdout}
|
|
special file}.
|
|
|
|
This option implies @option{-s}.
|
|
|
|
When the end of the file is reached, the temporary file is
|
|
renamed to the output file's original name. The extension,
|
|
if supplied, is used to modify the name of the old file
|
|
before renaming the temporary file, thereby making a backup
|
|
copy@footnote{Note that @value{SSED} creates the backup
|
|
file whether or not any output is actually changed.}).
|
|
|
|
@cindex In-place editing, Perl-style backup file names
|
|
This rule is followed: if the extension doesn't contain a @code{*},
|
|
then it is appended to the end of the current filename as a
|
|
suffix; if the extension does contain one or more @code{*}
|
|
characters, then @emph{each} asterisk is replaced with the
|
|
current filename. This allows you to add a prefix to the
|
|
backup file, instead of (or in addition to) a suffix, or
|
|
even to place backup copies of the original files into another
|
|
directory (provided the directory already exists).
|
|
|
|
If no extension is supplied, the original file is
|
|
overwritten without making a backup.
|
|
|
|
@item -l @var{N}
|
|
@itemx --line-length=@var{N}
|
|
@opindex -l
|
|
@opindex --line-length
|
|
@cindex Line length, setting
|
|
Specify the default line-wrap length for the @code{l} command.
|
|
A length of 0 (zero) means to never wrap long lines. If
|
|
not specified, it is taken to be 70.
|
|
|
|
@item --posix
|
|
@opindex --posix
|
|
@cindex @value{SSEDEXT}, disabling
|
|
@value{SSED} includes several extensions to @acronym{POSIX}
|
|
sed. In order to simplify writing portable scripts, this
|
|
option disables all the extensions that this manual documents,
|
|
including additional commands.
|
|
@cindex @code{POSIXLY_CORRECT} behavior, enabling
|
|
Most of the extensions accept @command{sed} programs that
|
|
are outside the syntax mandated by @acronym{POSIX}, but some
|
|
of them (such as the behavior of the @command{N} command
|
|
described in @pxref{Reporting Bugs}) actually violate the
|
|
standard. If you want to disable only the latter kind of
|
|
extension, you can set the @code{POSIXLY_CORRECT} variable
|
|
to a non-empty value.
|
|
|
|
@item -b
|
|
@itemx --binary
|
|
@opindex -b
|
|
@opindex --binary
|
|
This option is available on every platform, but is only effective where the
|
|
operating system makes a distinction between text files and binary files.
|
|
When such a distinction is made---as is the case for MS-DOS, Windows,
|
|
Cygwin---text files are composed of lines separated by a carriage return
|
|
@emph{and} a line feed character, and @command{sed} does not see the
|
|
ending CR. When this option is specified, @command{sed} will open
|
|
input files in binary mode, thus not requesting this special processing
|
|
and considering lines to end at a line feed.
|
|
|
|
@item --follow-symlinks
|
|
@opindex --follow-symlinks
|
|
This option is available only on platforms that support
|
|
symbolic links and has an effect only if option @option{-i}
|
|
is specified. In this case, if the file that is specified
|
|
on the command line is a symbolic link, @command{sed} will
|
|
follow the link and edit the ultimate destination of the
|
|
link. The default behavior is to break the symbolic link,
|
|
so that the link destination will not be modified.
|
|
|
|
@item -r
|
|
@itemx --regexp-extended
|
|
@opindex -r
|
|
@opindex --regexp-extended
|
|
@cindex Extended regular expressions, choosing
|
|
@cindex @acronym{GNU} extensions, extended regular expressions
|
|
Use extended regular expressions rather than basic
|
|
regular expressions. Extended regexps are those that
|
|
@command{egrep} accepts; they can be clearer because they
|
|
usually have less backslashes, but are a @acronym{GNU} extension
|
|
and hence scripts that use them are not portable.
|
|
@xref{Extended regexps, , Extended regular expressions}.
|
|
|
|
@ifset PERL
|
|
@item -R
|
|
@itemx --regexp-perl
|
|
@opindex -R
|
|
@opindex --regexp-perl
|
|
@cindex Perl-style regular expressions, choosing
|
|
@cindex @value{SSEDEXT}, Perl-style regular expressions
|
|
Use Perl-style regular expressions rather than basic
|
|
regular expressions. Perl-style regexps are extremely
|
|
powerful but are a @value{SSED} extension and hence scripts that
|
|
use it are not portable. @xref{Perl regexps, ,
|
|
Perl-style regular expressions}.
|
|
@end ifset
|
|
|
|
@item -s
|
|
@itemx --separate
|
|
@opindex -s
|
|
@opindex --separate
|
|
@cindex Working on separate files
|
|
By default, @command{sed} will consider the files specified on the
|
|
command line as a single continuous long stream. This @value{SSED}
|
|
extension allows the user to consider them as separate files:
|
|
range addresses (such as @samp{/abc/,/def/}) are not allowed
|
|
to span several files, line numbers are relative to the start
|
|
of each file, @code{$} refers to the last line of each file,
|
|
and files invoked from the @code{R} commands are rewound at the
|
|
start of each file.
|
|
|
|
@item -u
|
|
@itemx --unbuffered
|
|
@opindex -u
|
|
@opindex --unbuffered
|
|
@cindex Unbuffered I/O, choosing
|
|
Buffer both input and output as minimally as practical.
|
|
(This is particularly useful if the input is coming from
|
|
the likes of @samp{tail -f}, and you wish to see the transformed
|
|
output as soon as possible.)
|
|
|
|
@item -z
|
|
@itemx --null-data
|
|
@itemx --zero-terminated
|
|
@opindex -z
|
|
@opindex --null-data
|
|
@opindex --zero-terminated
|
|
Treat the input as a set of lines, each terminated by a zero byte
|
|
(the ASCII @samp{NUL} character) instead of a newline. This option can
|
|
be used with commands like @samp{sort -z} and @samp{find -print0}
|
|
to process arbitrary file names.
|
|
@end table
|
|
|
|
If no @option{-e}, @option{-f}, @option{--expression}, or @option{--file}
|
|
options are given on the command-line,
|
|
then the first non-option argument on the command line is
|
|
taken to be the @var{script} to be executed.
|
|
|
|
@cindex Files to be processed as input
|
|
If any command-line parameters remain after processing the above,
|
|
these parameters are interpreted as the names of input files to
|
|
be processed.
|
|
@cindex Standard input, processing as input
|
|
A file name of @samp{-} refers to the standard input stream.
|
|
The standard input will be processed if no file names are specified.
|
|
|
|
|
|
@node sed Programs
|
|
@chapter @command{sed} Programs
|
|
|
|
@cindex @command{sed} program structure
|
|
@cindex Script structure
|
|
A @command{sed} program consists of one or more @command{sed} commands,
|
|
passed in by one or more of the
|
|
@option{-e}, @option{-f}, @option{--expression}, and @option{--file}
|
|
options, or the first non-option argument if zero of these
|
|
options are used.
|
|
This document will refer to ``the'' @command{sed} script;
|
|
this is understood to mean the in-order catenation
|
|
of all of the @var{script}s and @var{script-file}s passed in.
|
|
|
|
Commands within a @var{script} or @var{script-file} can be
|
|
separated by semicolons (@code{;}) or newlines (ASCII 10).
|
|
Some commands, due to their syntax, cannot be followed by semicolons
|
|
working as command separators and thus should be terminated
|
|
with newlines or be placed at the end of a @var{script} or @var{script-file}.
|
|
Commands can also be preceded with optional non-significant
|
|
whitespace characters.
|
|
|
|
Each @code{sed} command consists of an optional address or
|
|
address range, followed by a one-character command name
|
|
and any additional command-specific code.
|
|
|
|
@menu
|
|
* Execution Cycle:: How @command{sed} works
|
|
* Addresses:: Selecting lines with @command{sed}
|
|
* Regular Expressions:: Overview of regular expression syntax
|
|
* Common Commands:: Often used commands
|
|
* The "s" Command:: @command{sed}'s Swiss Army Knife
|
|
* Other Commands:: Less frequently used commands
|
|
* Programming Commands:: Commands for @command{sed} gurus
|
|
* Extended Commands:: Commands specific of @value{SSED}
|
|
* Escapes:: Specifying special characters
|
|
@end menu
|
|
|
|
|
|
@node Execution Cycle
|
|
@section How @command{sed} Works
|
|
|
|
@cindex Buffer spaces, pattern and hold
|
|
@cindex Spaces, pattern and hold
|
|
@cindex Pattern space, definition
|
|
@cindex Hold space, definition
|
|
@command{sed} maintains two data buffers: the active @emph{pattern} space,
|
|
and the auxiliary @emph{hold} space. Both are initially empty.
|
|
|
|
@command{sed} operates by performing the following cycle on each
|
|
line of input: first, @command{sed} reads one line from the input
|
|
stream, removes any trailing newline, and places it in the pattern space.
|
|
Then commands are executed; each command can have an address associated
|
|
to it: addresses are a kind of condition code, and a command is only
|
|
executed if the condition is verified before the command is to be
|
|
executed.
|
|
|
|
When the end of the script is reached, unless the @option{-n} option
|
|
is in use, the contents of pattern space are printed out to the output
|
|
stream, adding back the trailing newline if it was removed.@footnote{Actually,
|
|
if @command{sed} prints a line without the terminating newline, it will
|
|
nevertheless print the missing newline as soon as more text is sent to
|
|
the same output stream, which gives the ``least expected surprise''
|
|
even though it does not make commands like @samp{sed -n p} exactly
|
|
identical to @command{cat}.} Then the next cycle starts for the next
|
|
input line.
|
|
|
|
Unless special commands (like @samp{D}) are used, the pattern space is
|
|
deleted between two cycles. The hold space, on the other hand, keeps
|
|
its data between cycles (see commands @samp{h}, @samp{H}, @samp{x},
|
|
@samp{g}, @samp{G} to move data between both buffers).
|
|
|
|
|
|
@node Addresses
|
|
@section Selecting lines with @command{sed}
|
|
@cindex Addresses, in @command{sed} scripts
|
|
@cindex Line selection
|
|
@cindex Selecting lines to process
|
|
|
|
Addresses in a @command{sed} script can be in any of the following forms:
|
|
@table @code
|
|
@item @var{number}
|
|
@cindex Address, numeric
|
|
@cindex Line, selecting by number
|
|
Specifying a line number will match only that line in the input.
|
|
(Note that @command{sed} counts lines continuously across all input files
|
|
unless @option{-i} or @option{-s} options are specified.)
|
|
|
|
@item @var{first}~@var{step}
|
|
@cindex @acronym{GNU} extensions, @samp{@var{n}~@var{m}} addresses
|
|
This @acronym{GNU} extension matches every @var{step}th line
|
|
starting with line @var{first}.
|
|
In particular, lines will be selected when there exists
|
|
a non-negative @var{n} such that the current line-number equals
|
|
@var{first} + (@var{n} * @var{step}).
|
|
Thus, to select the odd-numbered lines,
|
|
one would use @code{1~2};
|
|
to pick every third line starting with the second, @samp{2~3} would be used;
|
|
to pick every fifth line starting with the tenth, use @samp{10~5};
|
|
and @samp{50~0} is just an obscure way of saying @code{50}.
|
|
|
|
@item $
|
|
@cindex Address, last line
|
|
@cindex Last line, selecting
|
|
@cindex Line, selecting last
|
|
This address matches the last line of the last file of input, or
|
|
the last line of each file when the @option{-i} or @option{-s} options
|
|
are specified.
|
|
|
|
@item /@var{regexp}/
|
|
@cindex Address, as a regular expression
|
|
@cindex Line, selecting by regular expression match
|
|
This will select any line which matches the regular expression @var{regexp}.
|
|
If @var{regexp} itself includes any @code{/} characters,
|
|
each must be escaped by a backslash (@code{\}).
|
|
|
|
@cindex empty regular expression
|
|
@cindex @value{SSEDEXT}, modifiers and the empty regular expression
|
|
The empty regular expression @samp{//} repeats the last regular
|
|
expression match (the same holds if the empty regular expression is
|
|
passed to the @code{s} command). Note that modifiers to regular expressions
|
|
are evaluated when the regular expression is compiled, thus it is invalid to
|
|
specify them together with the empty regular expression.
|
|
|
|
@item \%@var{regexp}%
|
|
(The @code{%} may be replaced by any other single character.)
|
|
|
|
@cindex Slash character, in regular expressions
|
|
This also matches the regular expression @var{regexp},
|
|
but allows one to use a different delimiter than @code{/}.
|
|
This is particularly useful if the @var{regexp} itself contains
|
|
a lot of slashes, since it avoids the tedious escaping of every @code{/}.
|
|
If @var{regexp} itself includes any delimiter characters,
|
|
each must be escaped by a backslash (@code{\}).
|
|
|
|
@item /@var{regexp}/I
|
|
@itemx \%@var{regexp}%I
|
|
@cindex @acronym{GNU} extensions, @code{I} modifier
|
|
@ifset PERL
|
|
@cindex Perl-style regular expressions, case-insensitive
|
|
@end ifset
|
|
The @code{I} modifier to regular-expression matching is a @acronym{GNU}
|
|
extension which causes the @var{regexp} to be matched in
|
|
a case-insensitive manner.
|
|
|
|
@item /@var{regexp}/M
|
|
@itemx \%@var{regexp}%M
|
|
@cindex @value{SSEDEXT}, @code{M} modifier
|
|
@ifset PERL
|
|
@cindex Perl-style regular expressions, multiline
|
|
@end ifset
|
|
The @code{M} modifier to regular-expression matching is a @value{SSED}
|
|
extension which directs @value{SSED} to match the regular expression
|
|
in @cite{multi-line} mode. The modifier causes @code{^} and @code{$} to
|
|
match respectively (in addition to the normal behavior) the empty string
|
|
after a newline, and the empty string before a newline. There are
|
|
special character sequences
|
|
@ifset PERL
|
|
(@code{\A} and @code{\Z} in Perl mode, @code{\`} and @code{\'}
|
|
in basic or extended regular expression modes)
|
|
@end ifset
|
|
@ifclear PERL
|
|
(@code{\`} and @code{\'})
|
|
@end ifclear
|
|
which always match the beginning or the end of the buffer.
|
|
In addition,
|
|
@ifset PERL
|
|
just like in Perl mode without the @code{S} modifier,
|
|
@end ifset
|
|
the period character does not match a new-line character in
|
|
multi-line mode.
|
|
|
|
@ifset PERL
|
|
@item /@var{regexp}/S
|
|
@itemx \%@var{regexp}%S
|
|
@cindex @value{SSEDEXT}, @code{S} modifier
|
|
@cindex Perl-style regular expressions, single line
|
|
The @code{S} modifier to regular-expression matching is only valid
|
|
in Perl mode and specifies that the dot character (@code{.}) will
|
|
match the newline character too. @code{S} stands for @cite{single-line}.
|
|
@end ifset
|
|
|
|
@ifset PERL
|
|
@item /@var{regexp}/X
|
|
@itemx \%@var{regexp}%X
|
|
@cindex @value{SSEDEXT}, @code{X} modifier
|
|
@cindex Perl-style regular expressions, extended
|
|
The @code{X} modifier to regular-expression matching is also
|
|
valid in Perl mode only. If it is used, whitespace in the
|
|
pattern (other than in a character class) and
|
|
characters between a @kbd{#} outside a character class and the
|
|
next newline character are ignored. An escaping backslash
|
|
can be used to include a whitespace or @kbd{#} character as part
|
|
of the pattern.
|
|
@end ifset
|
|
@end table
|
|
|
|
If no addresses are given, then all lines are matched;
|
|
if one address is given, then only lines matching that
|
|
address are matched.
|
|
|
|
@cindex Range of lines
|
|
@cindex Several lines, selecting
|
|
An address range can be specified by specifying two addresses
|
|
separated by a comma (@code{,}). An address range matches lines
|
|
starting from where the first address matches, and continues
|
|
until the second address matches (inclusively).
|
|
|
|
If the second address is a @var{regexp}, then checking for the
|
|
ending match will start with the line @emph{following} the
|
|
line which matched the first address: a range will always
|
|
span at least two lines (except of course if the input stream
|
|
ends).
|
|
|
|
If the second address is a @var{number} less than (or equal to)
|
|
the line matching the first address, then only the one line is
|
|
matched.
|
|
|
|
@cindex Special addressing forms
|
|
@cindex Range with start address of zero
|
|
@cindex Zero, as range start address
|
|
@cindex @var{addr1},+N
|
|
@cindex @var{addr1},~N
|
|
@cindex @acronym{GNU} extensions, special two-address forms
|
|
@cindex @acronym{GNU} extensions, @code{0} address
|
|
@cindex @acronym{GNU} extensions, 0,@var{addr2} addressing
|
|
@cindex @acronym{GNU} extensions, @var{addr1},+@var{N} addressing
|
|
@cindex @acronym{GNU} extensions, @var{addr1},~@var{N} addressing
|
|
@value{SSED} also supports some special two-address forms; all these
|
|
are @acronym{GNU} extensions:
|
|
@table @code
|
|
@item 0,/@var{regexp}/
|
|
A line number of @code{0} can be used in an address specification like
|
|
@code{0,/@var{regexp}/} so that @command{sed} will try to match
|
|
@var{regexp} in the first input line too. In other words,
|
|
@code{0,/@var{regexp}/} is similar to @code{1,/@var{regexp}/},
|
|
except that if @var{addr2} matches the very first line of input the
|
|
@code{0,/@var{regexp}/} form will consider it to end the range, whereas
|
|
the @code{1,/@var{regexp}/} form will match the beginning of its range and
|
|
hence make the range span up to the @emph{second} occurrence of the
|
|
regular expression.
|
|
|
|
Note that this is the only place where the @code{0} address makes
|
|
sense; there is no 0-th line and commands which are given the @code{0}
|
|
address in any other way will give an error.
|
|
|
|
@item @var{addr1},+@var{N}
|
|
Matches @var{addr1} and the @var{N} lines following @var{addr1}.
|
|
|
|
@item @var{addr1},~@var{N}
|
|
Matches @var{addr1} and the lines following @var{addr1}
|
|
until the next line whose input line number is a multiple of @var{N}.
|
|
@end table
|
|
|
|
@cindex Excluding lines
|
|
@cindex Selecting non-matching lines
|
|
Appending the @code{!} character to the end of an address
|
|
specification negates the sense of the match.
|
|
That is, if the @code{!} character follows an address range,
|
|
then only lines which do @emph{not} match the address range
|
|
will be selected.
|
|
This also works for singleton addresses,
|
|
and, perhaps perversely, for the null address.
|
|
|
|
|
|
@node Regular Expressions
|
|
@section Overview of Regular Expression Syntax
|
|
|
|
To know how to use @command{sed}, people should understand regular
|
|
expressions (@dfn{regexp} for short). A regular expression
|
|
is a pattern that is matched against a
|
|
subject string from left to right. Most characters are
|
|
@dfn{ordinary}: they stand for
|
|
themselves in a pattern, and match the corresponding characters
|
|
in the subject. As a trivial example, the pattern
|
|
|
|
@example
|
|
The quick brown fox
|
|
@end example
|
|
|
|
@noindent
|
|
matches a portion of a subject string that is identical to
|
|
itself. The power of regular expressions comes from the
|
|
ability to include alternatives and repetitions in the pattern.
|
|
These are encoded in the pattern by the use of @dfn{special characters},
|
|
which do not stand for themselves but instead
|
|
are interpreted in some special way. Here is a brief description
|
|
of regular expression syntax as used in @command{sed}.
|
|
|
|
@table @code
|
|
@item @var{char}
|
|
A single ordinary character matches itself.
|
|
|
|
@item *
|
|
@cindex @acronym{GNU} extensions, to basic regular expressions
|
|
Matches a sequence of zero or more instances of matches for the
|
|
preceding regular expression, which must be an ordinary character, a
|
|
special character preceded by @code{\}, a @code{.}, a grouped regexp
|
|
(see below), or a bracket expression. As a @acronym{GNU} extension, a
|
|
postfixed regular expression can also be followed by @code{*}; for
|
|
example, @code{a**} is equivalent to @code{a*}. @acronym{POSIX}
|
|
1003.1-2001 says that @code{*} stands for itself when it appears at
|
|
the start of a regular expression or subexpression, but many
|
|
non@acronym{GNU} implementations do not support this and portable
|
|
scripts should instead use @code{\*} in these contexts.
|
|
|
|
@item \+
|
|
@cindex @acronym{GNU} extensions, to basic regular expressions
|
|
As @code{*}, but matches one or more. It is a @acronym{GNU} extension.
|
|
|
|
@item \?
|
|
@cindex @acronym{GNU} extensions, to basic regular expressions
|
|
As @code{*}, but only matches zero or one. It is a @acronym{GNU} extension.
|
|
|
|
@item \@{@var{i}\@}
|
|
As @code{*}, but matches exactly @var{i} sequences (@var{i} is a
|
|
decimal integer; for portability, keep it between 0 and 255
|
|
inclusive).
|
|
|
|
@item \@{@var{i},@var{j}\@}
|
|
Matches between @var{i} and @var{j}, inclusive, sequences.
|
|
|
|
@item \@{@var{i},\@}
|
|
Matches more than or equal to @var{i} sequences.
|
|
|
|
@item \(@var{regexp}\)
|
|
Groups the inner @var{regexp} as a whole, this is used to:
|
|
|
|
@itemize @bullet
|
|
@item
|
|
@cindex @acronym{GNU} extensions, to basic regular expressions
|
|
Apply postfix operators, like @code{\(abcd\)*}:
|
|
this will search for zero or more whole sequences
|
|
of @samp{abcd}, while @code{abcd*} would search
|
|
for @samp{abc} followed by zero or more occurrences
|
|
of @samp{d}. Note that support for @code{\(abcd\)*} is
|
|
required by @acronym{POSIX} 1003.1-2001, but many non-@acronym{GNU}
|
|
implementations do not support it and hence it is not universally
|
|
portable.
|
|
|
|
@item
|
|
Use back references (see below).
|
|
@end itemize
|
|
|
|
@item .
|
|
Matches any character, including newline.
|
|
|
|
@item ^
|
|
Matches the null string at beginning of the pattern space, i.e. what
|
|
appears after the circumflex must appear at the beginning of the
|
|
pattern space.
|
|
|
|
In most scripts, pattern space is initialized to the content of each
|
|
line (@pxref{Execution Cycle, , How @code{sed} works}). So, it is a
|
|
useful simplification to think of @code{^#include} as matching only
|
|
lines where @samp{#include} is the first thing on line---if there are
|
|
spaces before, for example, the match fails. This simplification is
|
|
valid as long as the original content of pattern space is not modified,
|
|
for example with an @code{s} command.
|
|
|
|
@code{^} acts as a special character only at the beginning of the
|
|
regular expression or subexpression (that is, after @code{\(} or
|
|
@code{\|}). Portable scripts should avoid @code{^} at the beginning of
|
|
a subexpression, though, as @acronym{POSIX} allows implementations that
|
|
treat @code{^} as an ordinary character in that context.
|
|
|
|
@item $
|
|
It is the same as @code{^}, but refers to end of pattern space.
|
|
@code{$} also acts as a special character only at the end
|
|
of the regular expression or subexpression (that is, before @code{\)}
|
|
or @code{\|}), and its use at the end of a subexpression is not
|
|
portable.
|
|
|
|
|
|
@item [@var{list}]
|
|
@itemx [^@var{list}]
|
|
Matches any single character in @var{list}: for example,
|
|
@code{[aeiou]} matches all vowels. A list may include
|
|
sequences like @code{@var{char1}-@var{char2}}, which
|
|
matches any character between (inclusive) @var{char1}
|
|
and @var{char2}.
|
|
|
|
A leading @code{^} reverses the meaning of @var{list}, so that
|
|
it matches any single character @emph{not} in @var{list}. To include
|
|
@code{]} in the list, make it the first character (after
|
|
the @code{^} if needed), to include @code{-} in the list,
|
|
make it the first or last; to include @code{^} put
|
|
it after the first character.
|
|
|
|
@cindex @code{POSIXLY_CORRECT} behavior, bracket expressions
|
|
The characters @code{$}, @code{*}, @code{.}, @code{[}, and @code{\}
|
|
are normally not special within @var{list}. For example, @code{[\*]}
|
|
matches either @samp{\} or @samp{*}, because the @code{\} is not
|
|
special here. However, strings like @code{[.ch.]}, @code{[=a=]}, and
|
|
@code{[:space:]} are special within @var{list} and represent collating
|
|
symbols, equivalence classes, and character classes, respectively, and
|
|
@code{[} is therefore special within @var{list} when it is followed by
|
|
@code{.}, @code{=}, or @code{:}. Also, when not in
|
|
@env{POSIXLY_CORRECT} mode, special escapes like @code{\n} and
|
|
@code{\t} are recognized within @var{list}. @xref{Escapes}.
|
|
|
|
@item @var{regexp1}\|@var{regexp2}
|
|
@cindex @acronym{GNU} extensions, to basic regular expressions
|
|
Matches either @var{regexp1} or @var{regexp2}. Use
|
|
parentheses to use complex alternative regular expressions.
|
|
The matching process tries each alternative in turn, from
|
|
left to right, and the first one that succeeds is used.
|
|
It is a @acronym{GNU} extension.
|
|
|
|
@item @var{regexp1}@var{regexp2}
|
|
Matches the concatenation of @var{regexp1} and @var{regexp2}.
|
|
Concatenation binds more tightly than @code{\|}, @code{^}, and
|
|
@code{$}, but less tightly than the other regular expression
|
|
operators.
|
|
|
|
@item \@var{digit}
|
|
Matches the @var{digit}-th @code{\(@dots{}\)} parenthesized
|
|
subexpression in the regular expression. This is called a @dfn{back
|
|
reference}. Subexpressions are implicity numbered by counting
|
|
occurrences of @code{\(} left-to-right.
|
|
|
|
@item \n
|
|
Matches the newline character.
|
|
|
|
@item \@var{char}
|
|
Matches @var{char}, where @var{char} is one of @code{$},
|
|
@code{*}, @code{.}, @code{[}, @code{\}, or @code{^}.
|
|
Note that the only C-like
|
|
backslash sequences that you can portably assume to be
|
|
interpreted are @code{\n} and @code{\\}; in particular
|
|
@code{\t} is not portable, and matches a @samp{t} under most
|
|
implementations of @command{sed}, rather than a tab character.
|
|
|
|
@end table
|
|
|
|
@cindex Greedy regular expression matching
|
|
Note that the regular expression matcher is greedy, i.e., matches
|
|
are attempted from left to right and, if two or more matches are
|
|
possible starting at the same character, it selects the longest.
|
|
|
|
@noindent
|
|
Examples:
|
|
@table @samp
|
|
@item abcdef
|
|
Matches @samp{abcdef}.
|
|
|
|
@item a*b
|
|
Matches zero or more @samp{a}s followed by a single
|
|
@samp{b}. For example, @samp{b} or @samp{aaaaab}.
|
|
|
|
@item a\?b
|
|
Matches @samp{b} or @samp{ab}.
|
|
|
|
@item a\+b\+
|
|
Matches one or more @samp{a}s followed by one or more
|
|
@samp{b}s: @samp{ab} is the shortest possible match, but
|
|
other examples are @samp{aaaab} or @samp{abbbbb} or
|
|
@samp{aaaaaabbbbbbb}.
|
|
|
|
@item .*
|
|
@itemx .\+
|
|
These two both match all the characters in a string;
|
|
however, the first matches every string (including the empty
|
|
string), while the second matches only strings containing
|
|
at least one character.
|
|
|
|
@item ^main.*(.*)
|
|
This matches a string starting with @samp{main},
|
|
followed by an opening and closing
|
|
parenthesis. The @samp{n}, @samp{(} and @samp{)} need not
|
|
be adjacent.
|
|
|
|
@item ^#
|
|
This matches a string beginning with @samp{#}.
|
|
|
|
@item \\$
|
|
This matches a string ending with a single backslash. The
|
|
regexp contains two backslashes for escaping.
|
|
|
|
@item \$
|
|
Instead, this matches a string consisting of a single dollar sign,
|
|
because it is escaped.
|
|
|
|
@item [a-zA-Z0-9]
|
|
In the C locale, this matches any @acronym{ASCII} letters or digits.
|
|
|
|
@item [^ @kbd{tab}]\+
|
|
(Here @kbd{tab} stands for a single tab character.)
|
|
This matches a string of one or more
|
|
characters, none of which is a space or a tab.
|
|
Usually this means a word.
|
|
|
|
@item ^\(.*\)\n\1$
|
|
This matches a string consisting of two equal substrings separated by
|
|
a newline.
|
|
|
|
@item .\@{9\@}A$
|
|
This matches nine characters followed by an @samp{A}.
|
|
|
|
@item ^.\@{15\@}A
|
|
This matches the start of a string that contains 16 characters,
|
|
the last of which is an @samp{A}.
|
|
|
|
@end table
|
|
|
|
|
|
|
|
@node Common Commands
|
|
@section Often-Used Commands
|
|
|
|
If you use @command{sed} at all, you will quite likely want to know
|
|
these commands.
|
|
|
|
@table @code
|
|
@item #
|
|
[No addresses allowed.]
|
|
|
|
@findex # (comments)
|
|
@cindex Comments, in scripts
|
|
The @code{#} character begins a comment;
|
|
the comment continues until the next newline.
|
|
|
|
@cindex Portability, comments
|
|
If you are concerned about portability, be aware that
|
|
some implementations of @command{sed} (which are not @sc{posix}
|
|
conformant) may only support a single one-line comment,
|
|
and then only when the very first character of the script is a @code{#}.
|
|
|
|
@findex -n, forcing from within a script
|
|
@cindex Caveat --- #n on first line
|
|
Warning: if the first two characters of the @command{sed} script
|
|
are @code{#n}, then the @option{-n} (no-autoprint) option is forced.
|
|
If you want to put a comment in the first line of your script
|
|
and that comment begins with the letter @samp{n}
|
|
and you do not want this behavior,
|
|
then be sure to either use a capital @samp{N},
|
|
or place at least one space before the @samp{n}.
|
|
|
|
@item q [@var{exit-code}]
|
|
This command only accepts a single address.
|
|
|
|
@findex q (quit) command
|
|
@cindex @value{SSEDEXT}, returning an exit code
|
|
@cindex Quitting
|
|
Exit @command{sed} without processing any more commands or input.
|
|
Note that the current pattern space is printed if auto-print is
|
|
not disabled with the @option{-n} options. The ability to return
|
|
an exit code from the @command{sed} script is a @value{SSED} extension.
|
|
|
|
@item d
|
|
@findex d (delete) command
|
|
@cindex Text, deleting
|
|
Delete the pattern space;
|
|
immediately start next cycle.
|
|
|
|
@item p
|
|
@findex p (print) command
|
|
@cindex Text, printing
|
|
Print out the pattern space (to the standard output).
|
|
This command is usually only used in conjunction with the @option{-n}
|
|
command-line option.
|
|
|
|
@item n
|
|
@findex n (next-line) command
|
|
@cindex Next input line, replace pattern space with
|
|
@cindex Read next input line
|
|
If auto-print is not disabled, print the pattern space,
|
|
then, regardless, replace the pattern space with the next line of input.
|
|
If there is no more input then @command{sed} exits without processing
|
|
any more commands.
|
|
|
|
@item @{ @var{commands} @}
|
|
@findex @{@} command grouping
|
|
@cindex Grouping commands
|
|
@cindex Command groups
|
|
A group of commands may be enclosed between
|
|
@code{@{} and @code{@}} characters.
|
|
This is particularly useful when you want a group of commands
|
|
to be triggered by a single address (or address-range) match.
|
|
|
|
@end table
|
|
|
|
@node The "s" Command
|
|
@section The @code{s} Command
|
|
|
|
The syntax of the @code{s} (as in substitute) command is
|
|
@samp{s/@var{regexp}/@var{replacement}/@var{flags}}. The @code{/}
|
|
characters may be uniformly replaced by any other single
|
|
character within any given @code{s} command. The @code{/}
|
|
character (or whatever other character is used in its stead)
|
|
can appear in the @var{regexp} or @var{replacement}
|
|
only if it is preceded by a @code{\} character.
|
|
|
|
The @code{s} command is probably the most important in @command{sed}
|
|
and has a lot of different options. Its basic concept is simple:
|
|
the @code{s} command attempts to match the pattern
|
|
space against the supplied @var{regexp}; if the match is
|
|
successful, then that portion of the pattern
|
|
space which was matched is replaced with @var{replacement}.
|
|
|
|
@cindex Backreferences, in regular expressions
|
|
@cindex Parenthesized substrings
|
|
The @var{replacement} can contain @code{\@var{n}} (@var{n} being
|
|
a number from 1 to 9, inclusive) references, which refer to
|
|
the portion of the match which is contained between the @var{n}th
|
|
@code{\(} and its matching @code{\)}.
|
|
Also, the @var{replacement} can contain unescaped @code{&}
|
|
characters which reference the whole matched portion
|
|
of the pattern space.
|
|
@cindex @value{SSEDEXT}, case modifiers in @code{s} commands
|
|
Finally, as a @value{SSED} extension, you can include a
|
|
special sequence made of a backslash and one of the letters
|
|
@code{L}, @code{l}, @code{U}, @code{u}, or @code{E}.
|
|
The meaning is as follows:
|
|
|
|
@table @code
|
|
@item \L
|
|
Turn the replacement
|
|
to lowercase until a @code{\U} or @code{\E} is found,
|
|
|
|
@item \l
|
|
Turn the
|
|
next character to lowercase,
|
|
|
|
@item \U
|
|
Turn the replacement to uppercase
|
|
until a @code{\L} or @code{\E} is found,
|
|
|
|
@item \u
|
|
Turn the next character
|
|
to uppercase,
|
|
|
|
@item \E
|
|
Stop case conversion started by @code{\L} or @code{\U}.
|
|
@end table
|
|
|
|
When the @code{g} flag is being used, case conversion does not
|
|
propagate from one occurrence of the regular expression to
|
|
another. For example, when the following command is executed
|
|
with @samp{a-b-} in pattern space:
|
|
@example
|
|
s/\(b\?\)-/x\u\1/g
|
|
@end example
|
|
|
|
@noindent
|
|
the output is @samp{axxB}. When replacing the first @samp{-},
|
|
the @samp{\u} sequence only affects the empty replacement of
|
|
@samp{\1}. It does not affect the @code{x} character that is
|
|
added to pattern space when replacing @code{b-} with @code{xB}.
|
|
|
|
On the other hand, @code{\l} and @code{\u} do affect the remainder
|
|
of the replacement text if they are followed by an empty substitution.
|
|
With @samp{a-b-} in pattern space, the following command:
|
|
@example
|
|
s/\(b\?\)-/\u\1x/g
|
|
@end example
|
|
|
|
@noindent
|
|
will replace @samp{-} with @samp{X} (uppercase) and @samp{b-} with
|
|
@samp{Bx}. If this behavior is undesirable, you can prevent it by
|
|
adding a @samp{\E} sequence---after @samp{\1} in this case.
|
|
|
|
To include a literal @code{\}, @code{&}, or newline in the final
|
|
replacement, be sure to precede the desired @code{\}, @code{&},
|
|
or newline in the @var{replacement} with a @code{\}.
|
|
|
|
@findex s command, option flags
|
|
@cindex Substitution of text, options
|
|
The @code{s} command can be followed by zero or more of the
|
|
following @var{flags}:
|
|
|
|
@table @code
|
|
@item g
|
|
@cindex Global substitution
|
|
@cindex Replacing all text matching regexp in a line
|
|
Apply the replacement to @emph{all} matches to the @var{regexp},
|
|
not just the first.
|
|
|
|
@item @var{number}
|
|
@cindex Replacing only @var{n}th match of regexp in a line
|
|
Only replace the @var{number}th match of the @var{regexp}.
|
|
|
|
@cindex @acronym{GNU} extensions, @code{g} and @var{number} modifier interaction in @code{s} command
|
|
@cindex Mixing @code{g} and @var{number} modifiers in the @code{s} command
|
|
Note: the @sc{posix} standard does not specify what should happen
|
|
when you mix the @code{g} and @var{number} modifiers,
|
|
and currently there is no widely agreed upon meaning
|
|
across @command{sed} implementations.
|
|
For @value{SSED}, the interaction is defined to be:
|
|
ignore matches before the @var{number}th,
|
|
and then match and replace all matches from
|
|
the @var{number}th on.
|
|
|
|
@item p
|
|
@cindex Text, printing after substitution
|
|
If the substitution was made, then print the new pattern space.
|
|
|
|
Note: when both the @code{p} and @code{e} options are specified,
|
|
the relative ordering of the two produces very different results.
|
|
In general, @code{ep} (evaluate then print) is what you want,
|
|
but operating the other way round can be useful for debugging.
|
|
For this reason, the current version of @value{SSED} interprets
|
|
specially the presence of @code{p} options both before and after
|
|
@code{e}, printing the pattern space before and after evaluation,
|
|
while in general flags for the @code{s} command show their
|
|
effect just once. This behavior, although documented, might
|
|
change in future versions.
|
|
|
|
@item w @var{file-name}
|
|
@cindex Text, writing to a file after substitution
|
|
@cindex @value{SSEDEXT}, @file{/dev/stdout} file
|
|
@cindex @value{SSEDEXT}, @file{/dev/stderr} file
|
|
If the substitution was made, then write out the result to the named file.
|
|
As a @value{SSED} extension, two special values of @var{file-name} are
|
|
supported: @file{/dev/stderr}, which writes the result to the standard
|
|
error, and @file{/dev/stdout}, which writes to the standard
|
|
output.@footnote{This is equivalent to @code{p} unless the @option{-i}
|
|
option is being used.}
|
|
|
|
@item e
|
|
@cindex Evaluate Bourne-shell commands, after substitution
|
|
@cindex Subprocesses
|
|
@cindex @value{SSEDEXT}, evaluating Bourne-shell commands
|
|
@cindex @value{SSEDEXT}, subprocesses
|
|
This command allows one to pipe input from a shell command
|
|
into pattern space. If a substitution was made, the command
|
|
that is found in pattern space is executed and pattern space
|
|
is replaced with its output. A trailing newline is suppressed;
|
|
results are undefined if the command to be executed contains
|
|
a @sc{nul} character. This is a @value{SSED} extension.
|
|
|
|
@item I
|
|
@itemx i
|
|
@cindex @acronym{GNU} extensions, @code{I} modifier
|
|
@cindex Case-insensitive matching
|
|
@ifset PERL
|
|
@cindex Perl-style regular expressions, case-insensitive
|
|
@end ifset
|
|
The @code{I} modifier to regular-expression matching is a @acronym{GNU}
|
|
extension which makes @command{sed} match @var{regexp} in a
|
|
case-insensitive manner.
|
|
|
|
@item M
|
|
@itemx m
|
|
@cindex @value{SSEDEXT}, @code{M} modifier
|
|
@ifset PERL
|
|
@cindex Perl-style regular expressions, multiline
|
|
@end ifset
|
|
The @code{M} modifier to regular-expression matching is a @value{SSED}
|
|
extension which directs @value{SSED} to match the regular expression
|
|
in @cite{multi-line} mode. The modifier causes @code{^} and @code{$} to
|
|
match respectively (in addition to the normal behavior) the empty string
|
|
after a newline, and the empty string before a newline. There are
|
|
special character sequences
|
|
@ifset PERL
|
|
(@code{\A} and @code{\Z} in Perl mode, @code{\`} and @code{\'}
|
|
in basic or extended regular expression modes)
|
|
@end ifset
|
|
@ifclear PERL
|
|
(@code{\`} and @code{\'})
|
|
@end ifclear
|
|
which always match the beginning or the end of the buffer.
|
|
In addition,
|
|
@ifset PERL
|
|
just like in Perl mode without the @code{S} modifier,
|
|
@end ifset
|
|
the period character does not match a new-line character in
|
|
multi-line mode.
|
|
|
|
@ifset PERL
|
|
@item S
|
|
@itemx s
|
|
@cindex @value{SSEDEXT}, @code{S} modifier
|
|
@cindex Perl-style regular expressions, single line
|
|
The @code{S} modifier to regular-expression matching is only valid
|
|
in Perl mode and specifies that the dot character (@code{.}) will
|
|
match the newline character too. @code{S} stands for @cite{single-line}.
|
|
@end ifset
|
|
|
|
@ifset PERL
|
|
@item X
|
|
@itemx x
|
|
@cindex @value{SSEDEXT}, @code{X} modifier
|
|
@cindex Perl-style regular expressions, extended
|
|
The @code{X} modifier to regular-expression matching is also
|
|
valid in Perl mode only. If it is used, whitespace in the
|
|
pattern (other than in a character class) and
|
|
characters between a @kbd{#} outside a character class and the
|
|
next newline character are ignored. An escaping backslash
|
|
can be used to include a whitespace or @kbd{#} character as part
|
|
of the pattern.
|
|
@end ifset
|
|
@end table
|
|
|
|
|
|
@node Other Commands
|
|
@section Less Frequently-Used Commands
|
|
|
|
Though perhaps less frequently used than those in the previous
|
|
section, some very small yet useful @command{sed} scripts can be built with
|
|
these commands.
|
|
|
|
@table @code
|
|
@item y/@var{source-chars}/@var{dest-chars}/
|
|
(The @code{/} characters may be uniformly replaced by
|
|
any other single character within any given @code{y} command.)
|
|
|
|
@findex y (transliterate) command
|
|
@cindex Transliteration
|
|
Transliterate any characters in the pattern space which match
|
|
any of the @var{source-chars} with the corresponding character
|
|
in @var{dest-chars}.
|
|
|
|
Instances of the @code{/} (or whatever other character is used in its stead),
|
|
@code{\}, or newlines can appear in the @var{source-chars} or @var{dest-chars}
|
|
lists, provide that each instance is escaped by a @code{\}.
|
|
The @var{source-chars} and @var{dest-chars} lists @emph{must}
|
|
contain the same number of characters (after de-escaping).
|
|
|
|
@item a\
|
|
@itemx @var{text}
|
|
@cindex @value{SSEDEXT}, two addresses supported by most commands
|
|
As a @acronym{GNU} extension, this command accepts two addresses.
|
|
|
|
@findex a (append text lines) command
|
|
@cindex Appending text after a line
|
|
@cindex Text, appending
|
|
Queue the lines of text which follow this command
|
|
(each but the last ending with a @code{\},
|
|
which are removed from the output)
|
|
to be output at the end of the current cycle,
|
|
or when the next input line is read.
|
|
|
|
Escape sequences in @var{text} are processed, so you should
|
|
use @code{\\} in @var{text} to print a single backslash.
|
|
|
|
As a @acronym{GNU} extension, if between the @code{a} and the newline there is
|
|
other than a whitespace-@code{\} sequence, then the text of this line,
|
|
starting at the first non-whitespace character after the @code{a},
|
|
is taken as the first line of the @var{text} block.
|
|
(This enables a simplification in scripting a one-line add.)
|
|
This extension also works with the @code{i} and @code{c} commands.
|
|
|
|
@item i\
|
|
@itemx @var{text}
|
|
@cindex @value{SSEDEXT}, two addresses supported by most commands
|
|
As a @acronym{GNU} extension, this command accepts two addresses.
|
|
|
|
@findex i (insert text lines) command
|
|
@cindex Inserting text before a line
|
|
@cindex Text, insertion
|
|
Immediately output the lines of text which follow this command
|
|
(each but the last ending with a @code{\},
|
|
which are removed from the output).
|
|
|
|
@item c\
|
|
@itemx @var{text}
|
|
@findex c (change to text lines) command
|
|
@cindex Replacing selected lines with other text
|
|
Delete the lines matching the address or address-range,
|
|
and output the lines of text which follow this command
|
|
(each but the last ending with a @code{\},
|
|
which are removed from the output)
|
|
in place of the last line
|
|
(or in place of each line, if no addresses were specified).
|
|
A new cycle is started after this command is done,
|
|
since the pattern space will have been deleted.
|
|
|
|
@item =
|
|
@cindex @value{SSEDEXT}, two addresses supported by most commands
|
|
As a @acronym{GNU} extension, this command accepts two addresses.
|
|
|
|
@findex = (print line number) command
|
|
@cindex Printing line number
|
|
@cindex Line number, printing
|
|
Print out the current input line number (with a trailing newline).
|
|
|
|
@item l @var{n}
|
|
@findex l (list unambiguously) command
|
|
@cindex List pattern space
|
|
@cindex Printing text unambiguously
|
|
@cindex Line length, setting
|
|
@cindex @value{SSEDEXT}, setting line length
|
|
Print the pattern space in an unambiguous form:
|
|
non-printable characters (and the @code{\} character)
|
|
are printed in C-style escaped form; long lines are split,
|
|
with a trailing @code{\} character to indicate the split;
|
|
the end of each line is marked with a @code{$}.
|
|
|
|
@var{n} specifies the desired line-wrap length;
|
|
a length of 0 (zero) means to never wrap long lines. If omitted,
|
|
the default as specified on the command line is used. The @var{n}
|
|
parameter is a @value{SSED} extension.
|
|
|
|
@item r @var{filename}
|
|
@cindex @value{SSEDEXT}, two addresses supported by most commands
|
|
As a @acronym{GNU} extension, this command accepts two addresses.
|
|
|
|
@findex r (read file) command
|
|
@cindex Read text from a file
|
|
@cindex @value{SSEDEXT}, @file{/dev/stdin} file
|
|
Queue the contents of @var{filename} to be read and
|
|
inserted into the output stream at the end of the current cycle,
|
|
or when the next input line is read.
|
|
Note that if @var{filename} cannot be read, it is treated as
|
|
if it were an empty file, without any error indication.
|
|
|
|
As a @value{SSED} extension, the special value @file{/dev/stdin}
|
|
is supported for the file name, which reads the contents of the
|
|
standard input.
|
|
|
|
@item w @var{filename}
|
|
@findex w (write file) command
|
|
@cindex Write to a file
|
|
@cindex @value{SSEDEXT}, @file{/dev/stdout} file
|
|
@cindex @value{SSEDEXT}, @file{/dev/stderr} file
|
|
Write the pattern space to @var{filename}.
|
|
As a @value{SSED} extension, two special values of @var{file-name} are
|
|
supported: @file{/dev/stderr}, which writes the result to the standard
|
|
error, and @file{/dev/stdout}, which writes to the standard
|
|
output.@footnote{This is equivalent to @code{p} unless the @option{-i}
|
|
option is being used.}
|
|
|
|
The file will be created (or truncated) before the first input line is
|
|
read; all @code{w} commands (including instances of the @code{w} flag
|
|
on successful @code{s} commands) which refer to the same @var{filename}
|
|
are output without closing and reopening the file.
|
|
|
|
@item D
|
|
@findex D (delete first line) command
|
|
@cindex Delete first line from pattern space
|
|
If pattern space contains no newline, start a normal new cycle as if
|
|
the @code{d} command was issued. Otherwise, delete text in the pattern
|
|
space up to the first newline, and restart cycle with the resultant
|
|
pattern space, without reading a new line of input.
|
|
|
|
@item N
|
|
@findex N (append Next line) command
|
|
@cindex Next input line, append to pattern space
|
|
@cindex Append next input line to pattern space
|
|
Add a newline to the pattern space,
|
|
then append the next line of input to the pattern space.
|
|
If there is no more input then @command{sed} exits without processing
|
|
any more commands.
|
|
|
|
@item P
|
|
@findex P (print first line) command
|
|
@cindex Print first line from pattern space
|
|
Print out the portion of the pattern space up to the first newline.
|
|
|
|
@item h
|
|
@findex h (hold) command
|
|
@cindex Copy pattern space into hold space
|
|
@cindex Replace hold space with copy of pattern space
|
|
@cindex Hold space, copying pattern space into
|
|
Replace the contents of the hold space with the contents of the pattern space.
|
|
|
|
@item H
|
|
@findex H (append Hold) command
|
|
@cindex Append pattern space to hold space
|
|
@cindex Hold space, appending from pattern space
|
|
Append a newline to the contents of the hold space,
|
|
and then append the contents of the pattern space to that of the hold space.
|
|
|
|
@item g
|
|
@findex g (get) command
|
|
@cindex Copy hold space into pattern space
|
|
@cindex Replace pattern space with copy of hold space
|
|
@cindex Hold space, copy into pattern space
|
|
Replace the contents of the pattern space with the contents of the hold space.
|
|
|
|
@item G
|
|
@findex G (appending Get) command
|
|
@cindex Append hold space to pattern space
|
|
@cindex Hold space, appending to pattern space
|
|
Append a newline to the contents of the pattern space,
|
|
and then append the contents of the hold space to that of the pattern space.
|
|
|
|
@item x
|
|
@findex x (eXchange) command
|
|
@cindex Exchange hold space with pattern space
|
|
@cindex Hold space, exchange with pattern space
|
|
Exchange the contents of the hold and pattern spaces.
|
|
|
|
@end table
|
|
|
|
|
|
@node Programming Commands
|
|
@section Commands for @command{sed} gurus
|
|
|
|
In most cases, use of these commands indicates that you are
|
|
probably better off programming in something like @command{awk}
|
|
or Perl. But occasionally one is committed to sticking
|
|
with @command{sed}, and these commands can enable one to write
|
|
quite convoluted scripts.
|
|
|
|
@cindex Flow of control in scripts
|
|
@table @code
|
|
@item : @var{label}
|
|
[No addresses allowed.]
|
|
|
|
@findex : (label) command
|
|
@cindex Labels, in scripts
|
|
Specify the location of @var{label} for branch commands.
|
|
In all other respects, a no-op.
|
|
|
|
@item b @var{label}
|
|
@findex b (branch) command
|
|
@cindex Branch to a label, unconditionally
|
|
@cindex Goto, in scripts
|
|
Unconditionally branch to @var{label}.
|
|
The @var{label} may be omitted, in which case the next cycle is started.
|
|
|
|
@item t @var{label}
|
|
@findex t (test and branch if successful) command
|
|
@cindex Branch to a label, if @code{s///} succeeded
|
|
@cindex Conditional branch
|
|
Branch to @var{label} only if there has been a successful @code{s}ubstitution
|
|
since the last input line was read or conditional branch was taken.
|
|
The @var{label} may be omitted, in which case the next cycle is started.
|
|
|
|
@end table
|
|
|
|
@node Extended Commands
|
|
@section Commands Specific to @value{SSED}
|
|
|
|
These commands are specific to @value{SSED}, so you
|
|
must use them with care and only when you are sure that
|
|
hindering portability is not evil. They allow you to check
|
|
for @value{SSED} extensions or to do tasks that are required
|
|
quite often, yet are unsupported by standard @command{sed}s.
|
|
|
|
@table @code
|
|
@item e [@var{command}]
|
|
@findex e (evaluate) command
|
|
@cindex Evaluate Bourne-shell commands
|
|
@cindex Subprocesses
|
|
@cindex @value{SSEDEXT}, evaluating Bourne-shell commands
|
|
@cindex @value{SSEDEXT}, subprocesses
|
|
This command allows one to pipe input from a shell command
|
|
into pattern space. Without parameters, the @code{e} command
|
|
executes the command that is found in pattern space and
|
|
replaces the pattern space with the output; a trailing newline
|
|
is suppressed.
|
|
|
|
If a parameter is specified, instead, the @code{e} command
|
|
interprets it as a command and sends its output to the output stream
|
|
(like @code{r} does). The command can run across multiple
|
|
lines, all but the last ending with a back-slash.
|
|
|
|
In both cases, the results are undefined if the command to be
|
|
executed contains a @sc{nul} character.
|
|
|
|
@item F
|
|
@findex F (File name) command
|
|
@cindex Printing file name
|
|
@cindex File name, printing
|
|
Print out the file name of the current input file (with a trailing
|
|
newline).
|
|
|
|
@item L @var{n}
|
|
@findex L (fLow paragraphs) command
|
|
@cindex Reformat pattern space
|
|
@cindex Reformatting paragraphs
|
|
@cindex @value{SSEDEXT}, reformatting paragraphs
|
|
@cindex @value{SSEDEXT}, @code{L} command
|
|
This @value{SSED} extension fills and joins lines in pattern space
|
|
to produce output lines of (at most) @var{n} characters, like
|
|
@code{fmt} does; if @var{n} is omitted, the default as specified
|
|
on the command line is used. This command is considered a failed
|
|
experiment and unless there is enough request (which seems unlikely)
|
|
will be removed in future versions.
|
|
|
|
@ignore
|
|
Blank lines, spaces between words, and indentation are
|
|
preserved in the output; successive input lines with different
|
|
indentation are not joined; tabs are expanded to 8 columns.
|
|
|
|
If the pattern space contains multiple lines, they are joined, but
|
|
since the pattern space usually contains a single line, the behavior
|
|
of a simple @code{L;d} script is the same as @samp{fmt -s} (i.e.,
|
|
it does not join short lines to form longer ones).
|
|
|
|
@var{n} specifies the desired line-wrap length; if omitted,
|
|
the default as specified on the command line is used.
|
|
@end ignore
|
|
|
|
@item Q [@var{exit-code}]
|
|
This command only accepts a single address.
|
|
|
|
@findex Q (silent Quit) command
|
|
@cindex @value{SSEDEXT}, quitting silently
|
|
@cindex @value{SSEDEXT}, returning an exit code
|
|
@cindex Quitting
|
|
This command is the same as @code{q}, but will not print the
|
|
contents of pattern space. Like @code{q}, it provides the
|
|
ability to return an exit code to the caller.
|
|
|
|
This command can be useful because the only alternative ways
|
|
to accomplish this apparently trivial function are to use
|
|
the @option{-n} option (which can unnecessarily complicate
|
|
your script) or resorting to the following snippet, which
|
|
wastes time by reading the whole file without any visible effect:
|
|
|
|
@example
|
|
:eat
|
|
$d @i{@r{Quit silently on the last line}}
|
|
N @i{@r{Read another line, silently}}
|
|
g @i{@r{Overwrite pattern space each time to save memory}}
|
|
b eat
|
|
@end example
|
|
|
|
@item R @var{filename}
|
|
@findex R (read line) command
|
|
@cindex Read text from a file
|
|
@cindex @value{SSEDEXT}, reading a file a line at a time
|
|
@cindex @value{SSEDEXT}, @code{R} command
|
|
@cindex @value{SSEDEXT}, @file{/dev/stdin} file
|
|
Queue a line of @var{filename} to be read and
|
|
inserted into the output stream at the end of the current cycle,
|
|
or when the next input line is read.
|
|
Note that if @var{filename} cannot be read, or if its end is
|
|
reached, no line is appended, without any error indication.
|
|
|
|
As with the @code{r} command, the special value @file{/dev/stdin}
|
|
is supported for the file name, which reads a line from the
|
|
standard input.
|
|
|
|
@item T @var{label}
|
|
@findex T (test and branch if failed) command
|
|
@cindex @value{SSEDEXT}, branch if @code{s///} failed
|
|
@cindex Branch to a label, if @code{s///} failed
|
|
@cindex Conditional branch
|
|
Branch to @var{label} only if there have been no successful
|
|
@code{s}ubstitutions since the last input line was read or
|
|
conditional branch was taken. The @var{label} may be omitted,
|
|
in which case the next cycle is started.
|
|
|
|
@item v @var{version}
|
|
@findex v (version) command
|
|
@cindex @value{SSEDEXT}, checking for their presence
|
|
@cindex Requiring @value{SSED}
|
|
This command does nothing, but makes @command{sed} fail if
|
|
@value{SSED} extensions are not supported, simply because other
|
|
versions of @command{sed} do not implement it. In addition, you
|
|
can specify the version of @command{sed} that your script
|
|
requires, such as @code{4.0.5}. The default is @code{4.0}
|
|
because that is the first version that implemented this command.
|
|
|
|
This command enables all @value{SSEDEXT} even if
|
|
@env{POSIXLY_CORRECT} is set in the environment.
|
|
|
|
@item W @var{filename}
|
|
@findex W (write first line) command
|
|
@cindex Write first line to a file
|
|
@cindex @value{SSEDEXT}, writing first line to a file
|
|
Write to the given filename the portion of the pattern space up to
|
|
the first newline. Everything said under the @code{w} command about
|
|
file handling holds here too.
|
|
|
|
@item z
|
|
@findex z (Zap) command
|
|
@cindex @value{SSEDEXT}, emptying pattern space
|
|
@cindex Emptying pattern space
|
|
This command empties the content of pattern space. It is
|
|
usually the same as @samp{s/.*//}, but is more efficient
|
|
and works in the presence of invalid multibyte sequences
|
|
in the input stream. @sc{posix} mandates that such sequences
|
|
are @emph{not} matched by @samp{.}, so that there is no portable
|
|
way to clear @command{sed}'s buffers in the middle of the
|
|
script in most multibyte locales (including UTF-8 locales).
|
|
@end table
|
|
|
|
@node Escapes
|
|
@section @acronym{GNU} Extensions for Escapes in Regular Expressions
|
|
|
|
@cindex @acronym{GNU} extensions, special escapes
|
|
Until this chapter, we have only encountered escapes of the form
|
|
@samp{\^}, which tell @command{sed} not to interpret the circumflex
|
|
as a special character, but rather to take it literally. For
|
|
example, @samp{\*} matches a single asterisk rather than zero
|
|
or more backslashes.
|
|
|
|
@cindex @code{POSIXLY_CORRECT} behavior, escapes
|
|
This chapter introduces another kind of escape@footnote{All
|
|
the escapes introduced here are @acronym{GNU}
|
|
extensions, with the exception of @code{\n}. In basic regular
|
|
expression mode, setting @code{POSIXLY_CORRECT} disables them inside
|
|
bracket expressions.}---that
|
|
is, escapes that are applied to a character or sequence of characters
|
|
that ordinarily are taken literally, and that @command{sed} replaces
|
|
with a special character. This provides a way
|
|
of encoding non-printable characters in patterns in a visible manner.
|
|
There is no restriction on the appearance of non-printing characters
|
|
in a @command{sed} script but when a script is being prepared in the
|
|
shell or by text editing, it is usually easier to use one of
|
|
the following escape sequences than the binary character it
|
|
represents:
|
|
|
|
The list of these escapes is:
|
|
|
|
@table @code
|
|
@item \a
|
|
Produces or matches a @sc{bel} character, that is an ``alert'' (@sc{ascii} 7).
|
|
|
|
@item \f
|
|
Produces or matches a form feed (@sc{ascii} 12).
|
|
|
|
@item \n
|
|
Produces or matches a newline (@sc{ascii} 10).
|
|
|
|
@item \r
|
|
Produces or matches a carriage return (@sc{ascii} 13).
|
|
|
|
@item \t
|
|
Produces or matches a horizontal tab (@sc{ascii} 9).
|
|
|
|
@item \v
|
|
Produces or matches a so called ``vertical tab'' (@sc{ascii} 11).
|
|
|
|
@item \c@var{x}
|
|
Produces or matches @kbd{@sc{Control}-@var{x}}, where @var{x} is
|
|
any character. The precise effect of @samp{\c@var{x}} is as follows:
|
|
if @var{x} is a lower case letter, it is converted to upper case.
|
|
Then bit 6 of the character (hex 40) is inverted. Thus @samp{\cz} becomes
|
|
hex 1A, but @samp{\c@{} becomes hex 3B, while @samp{\c;} becomes hex 7B.
|
|
|
|
@item \d@var{xxx}
|
|
Produces or matches a character whose decimal @sc{ascii} value is @var{xxx}.
|
|
|
|
@item \o@var{xxx}
|
|
@ifset PERL
|
|
@item \@var{xxx}
|
|
@end ifset
|
|
Produces or matches a character whose octal @sc{ascii} value is @var{xxx}.
|
|
@ifset PERL
|
|
The syntax without the @code{o} is active in Perl mode, while the one
|
|
with the @code{o} is active in the normal or extended @sc{posix} regular
|
|
expression modes.
|
|
@end ifset
|
|
|
|
@item \x@var{xx}
|
|
Produces or matches a character whose hexadecimal @sc{ascii} value is @var{xx}.
|
|
@end table
|
|
|
|
@samp{\b} (backspace) was omitted because of the conflict with
|
|
the existing ``word boundary'' meaning.
|
|
|
|
Other escapes match a particular character class and are valid only in
|
|
regular expressions:
|
|
|
|
@table @code
|
|
@item \w
|
|
Matches any ``word'' character. A ``word'' character is any
|
|
letter or digit or the underscore character.
|
|
|
|
@item \W
|
|
Matches any ``non-word'' character.
|
|
|
|
@item \b
|
|
Matches a word boundary; that is it matches if the character
|
|
to the left is a ``word'' character and the character to the
|
|
right is a ``non-word'' character, or vice-versa.
|
|
|
|
@item \B
|
|
Matches everywhere but on a word boundary; that is it matches
|
|
if the character to the left and the character to the right
|
|
are either both ``word'' characters or both ``non-word''
|
|
characters.
|
|
|
|
@item \`
|
|
Matches only at the start of pattern space. This is different
|
|
from @code{^} in multi-line mode.
|
|
|
|
@item \'
|
|
Matches only at the end of pattern space. This is different
|
|
from @code{$} in multi-line mode.
|
|
|
|
@ifset PERL
|
|
@item \G
|
|
Match only at the start of pattern space or, when doing a global
|
|
substitution using the @code{s///g} command and option, at
|
|
the end-of-match position of the prior match. For example,
|
|
@samp{s/\Ga/Z/g} will change an initial run of @code{a}s to
|
|
a run of @code{Z}s
|
|
@end ifset
|
|
@end table
|
|
|
|
@node Examples
|
|
@chapter Some Sample Scripts
|
|
|
|
Here are some @command{sed} scripts to guide you in the art of mastering
|
|
@command{sed}.
|
|
|
|
@menu
|
|
Some exotic examples:
|
|
* Centering lines::
|
|
* Increment a number::
|
|
* Rename files to lower case::
|
|
* Print bash environment::
|
|
* Reverse chars of lines::
|
|
|
|
Emulating standard utilities:
|
|
* tac:: Reverse lines of files
|
|
* cat -n:: Numbering lines
|
|
* cat -b:: Numbering non-blank lines
|
|
* wc -c:: Counting chars
|
|
* wc -w:: Counting words
|
|
* wc -l:: Counting lines
|
|
* head:: Printing the first lines
|
|
* tail:: Printing the last lines
|
|
* uniq:: Make duplicate lines unique
|
|
* uniq -d:: Print duplicated lines of input
|
|
* uniq -u:: Remove all duplicated lines
|
|
* cat -s:: Squeezing blank lines
|
|
@end menu
|
|
|
|
@node Centering lines
|
|
@section Centering Lines
|
|
|
|
This script centers all lines of a file on a 80 columns width.
|
|
To change that width, the number in @code{\@{@dots{}\@}} must be
|
|
replaced, and the number of added spaces also must be changed.
|
|
|
|
Note how the buffer commands are used to separate parts in
|
|
the regular expressions to be matched---this is a common
|
|
technique.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -f
|
|
|
|
@group
|
|
# Put 80 spaces in the buffer
|
|
1 @{
|
|
x
|
|
s/^$/ /
|
|
s/^.*$/&&&&&&&&/
|
|
x
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
# del leading and trailing spaces
|
|
y/@kbd{tab}/ /
|
|
s/^ *//
|
|
s/ *$//
|
|
@end group
|
|
|
|
@group
|
|
# add a newline and 80 spaces to end of line
|
|
G
|
|
@end group
|
|
|
|
@group
|
|
# keep first 81 chars (80 + a newline)
|
|
s/^\(.\@{81\@}\).*$/\1/
|
|
@end group
|
|
|
|
@group
|
|
# \2 matches half of the spaces, which are moved to the beginning
|
|
s/^\(.*\)\n\(.*\)\2/\2\1/
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node Increment a number
|
|
@section Increment a Number
|
|
|
|
This script is one of a few that demonstrate how to do arithmetic
|
|
in @command{sed}. This is indeed possible,@footnote{@command{sed} guru Greg
|
|
Ubben wrote an implementation of the @command{dc} @sc{rpn} calculator!
|
|
It is distributed together with sed.} but must be done manually.
|
|
|
|
To increment one number you just add 1 to last digit, replacing
|
|
it by the following digit. There is one exception: when the digit
|
|
is a nine the previous digits must be also incremented until you
|
|
don't have a nine.
|
|
|
|
This solution by Bruno Haible is very clever and smart because
|
|
it uses a single buffer; if you don't have this limitation, the
|
|
algorithm used in @ref{cat -n, Numbering lines}, is faster.
|
|
It works by replacing trailing nines with an underscore, then
|
|
using multiple @code{s} commands to increment the last digit,
|
|
and then again substituting underscores with zeros.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -f
|
|
|
|
/[^0-9]/ d
|
|
|
|
@group
|
|
# replace all trailing 9s by _ (any other character except digits, could
|
|
# be used)
|
|
:d
|
|
s/9\(_*\)$/_\1/
|
|
td
|
|
@end group
|
|
|
|
@group
|
|
# incr last digit only. The first line adds a most-significant
|
|
# digit of 1 if we have to add a digit.
|
|
@end group
|
|
|
|
@group
|
|
s/^\(_*\)$/1\1/; tn
|
|
s/8\(_*\)$/9\1/; tn
|
|
s/7\(_*\)$/8\1/; tn
|
|
s/6\(_*\)$/7\1/; tn
|
|
s/5\(_*\)$/6\1/; tn
|
|
s/4\(_*\)$/5\1/; tn
|
|
s/3\(_*\)$/4\1/; tn
|
|
s/2\(_*\)$/3\1/; tn
|
|
s/1\(_*\)$/2\1/; tn
|
|
s/0\(_*\)$/1\1/; tn
|
|
@end group
|
|
|
|
@group
|
|
:n
|
|
y/_/0/
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node Rename files to lower case
|
|
@section Rename Files to Lower Case
|
|
|
|
This is a pretty strange use of @command{sed}. We transform text, and
|
|
transform it to be shell commands, then just feed them to shell.
|
|
Don't worry, even worse hacks are done when using @command{sed}; I have
|
|
seen a script converting the output of @command{date} into a @command{bc}
|
|
program!
|
|
|
|
The main body of this is the @command{sed} script, which remaps the name
|
|
from lower to upper (or vice-versa) and even checks out
|
|
if the remapped name is the same as the original name.
|
|
Note how the script is parameterized using shell
|
|
variables and proper quoting.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
@group
|
|
#! /bin/sh
|
|
# rename files to lower/upper case...
|
|
#
|
|
# usage:
|
|
# move-to-lower *
|
|
# move-to-upper *
|
|
# or
|
|
# move-to-lower -R .
|
|
# move-to-upper -R .
|
|
#
|
|
@end group
|
|
|
|
@group
|
|
help()
|
|
@{
|
|
cat << eof
|
|
Usage: $0 [-n] [-r] [-h] files...
|
|
@end group
|
|
|
|
@group
|
|
-n do nothing, only see what would be done
|
|
-R recursive (use find)
|
|
-h this message
|
|
files files to remap to lower case
|
|
@end group
|
|
|
|
@group
|
|
Examples:
|
|
$0 -n * (see if everything is ok, then...)
|
|
$0 *
|
|
@end group
|
|
|
|
$0 -R .
|
|
|
|
@group
|
|
eof
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
apply_cmd='sh'
|
|
finder='echo "$@@" | tr " " "\n"'
|
|
files_only=
|
|
@end group
|
|
|
|
@group
|
|
while :
|
|
do
|
|
case "$1" in
|
|
-n) apply_cmd='cat' ;;
|
|
-R) finder='find "$@@" -type f';;
|
|
-h) help ; exit 1 ;;
|
|
*) break ;;
|
|
esac
|
|
shift
|
|
done
|
|
@end group
|
|
|
|
@group
|
|
if [ -z "$1" ]; then
|
|
echo Usage: $0 [-h] [-n] [-r] files...
|
|
exit 1
|
|
fi
|
|
@end group
|
|
|
|
@group
|
|
LOWER='abcdefghijklmnopqrstuvwxyz'
|
|
UPPER='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
|
|
@end group
|
|
|
|
@group
|
|
case `basename $0` in
|
|
*upper*) TO=$UPPER; FROM=$LOWER ;;
|
|
*) FROM=$UPPER; TO=$LOWER ;;
|
|
esac
|
|
@end group
|
|
|
|
eval $finder | sed -n '
|
|
|
|
@group
|
|
# remove all trailing slashes
|
|
s/\/*$//
|
|
@end group
|
|
|
|
@group
|
|
# add ./ if there is no path, only a filename
|
|
/\//! s/^/.\//
|
|
@end group
|
|
|
|
@group
|
|
# save path+filename
|
|
h
|
|
@end group
|
|
|
|
@group
|
|
# remove path
|
|
s/.*\///
|
|
@end group
|
|
|
|
@group
|
|
# do conversion only on filename
|
|
y/'$FROM'/'$TO'/
|
|
@end group
|
|
|
|
@group
|
|
# now line contains original path+file, while
|
|
# hold space contains the new filename
|
|
x
|
|
@end group
|
|
|
|
@group
|
|
# add converted file name to line, which now contains
|
|
# path/file-name\nconverted-file-name
|
|
G
|
|
@end group
|
|
|
|
@group
|
|
# check if converted file name is equal to original file name,
|
|
# if it is, do not print anything
|
|
/^.*\/\(.*\)\n\1/b
|
|
@end group
|
|
|
|
@group
|
|
# escape special characters for the shell
|
|
s/["$`\\]/\\&/g
|
|
@end group
|
|
|
|
@group
|
|
# now, transform path/fromfile\n, into
|
|
# mv path/fromfile path/tofile and print it
|
|
s/^\(.*\/\)\(.*\)\n\(.*\)$/mv "\1\2" "\1\3"/p
|
|
@end group
|
|
|
|
' | $apply_cmd
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node Print bash environment
|
|
@section Print @command{bash} Environment
|
|
|
|
This script strips the definition of the shell functions
|
|
from the output of the @command{set} Bourne-shell command.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/bin/sh
|
|
|
|
@group
|
|
set | sed -n '
|
|
:x
|
|
@end group
|
|
|
|
@group
|
|
@ifinfo
|
|
# if no occurrence of "=()" print and load next line
|
|
@end ifinfo
|
|
@ifnotinfo
|
|
# if no occurrence of @samp{=()} print and load next line
|
|
@end ifnotinfo
|
|
/=()/! @{ p; b; @}
|
|
/ () $/! @{ p; b; @}
|
|
@end group
|
|
|
|
@group
|
|
# possible start of functions section
|
|
# save the line in case this is a var like FOO="() "
|
|
h
|
|
@end group
|
|
|
|
@group
|
|
# if the next line has a brace, we quit because
|
|
# nothing comes after functions
|
|
n
|
|
/^@{/ q
|
|
@end group
|
|
|
|
@group
|
|
# print the old line
|
|
x; p
|
|
@end group
|
|
|
|
@group
|
|
# work on the new line now
|
|
x; bx
|
|
'
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node Reverse chars of lines
|
|
@section Reverse Characters of Lines
|
|
|
|
This script can be used to reverse the position of characters
|
|
in lines. The technique moves two characters at a time, hence
|
|
it is faster than more intuitive implementations.
|
|
|
|
Note the @code{tx} command before the definition of the label.
|
|
This is often needed to reset the flag that is tested by
|
|
the @code{t} command.
|
|
|
|
Imaginative readers will find uses for this script. An example
|
|
is reversing the output of @command{banner}.@footnote{This requires
|
|
another script to pad the output of banner; for example
|
|
|
|
@example
|
|
#! /bin/sh
|
|
|
|
banner -w $1 $2 $3 $4 |
|
|
sed -e :a -e '/^.\@{0,'$1'\@}$/ @{ s/$/ /; ba; @}' |
|
|
~/sedscripts/reverseline.sed
|
|
@end example
|
|
}
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -f
|
|
|
|
/../! b
|
|
|
|
@group
|
|
# Reverse a line. Begin embedding the line between two newlines
|
|
s/^.*$/\
|
|
&\
|
|
/
|
|
@end group
|
|
|
|
@group
|
|
# Move first character at the end. The regexp matches until
|
|
# there are zero or one characters between the markers
|
|
tx
|
|
:x
|
|
s/\(\n.\)\(.*\)\(.\n\)/\3\2\1/
|
|
tx
|
|
@end group
|
|
|
|
@group
|
|
# Remove the newline markers
|
|
s/\n//g
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node tac
|
|
@section Reverse Lines of Files
|
|
|
|
This one begins a series of totally useless (yet interesting)
|
|
scripts emulating various Unix commands. This, in particular,
|
|
is a @command{tac} workalike.
|
|
|
|
Note that on implementations other than @acronym{GNU} @command{sed}
|
|
@ifset PERL
|
|
and @value{SSED}
|
|
@end ifset
|
|
this script might easily overflow internal buffers.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
# reverse all lines of input, i.e. first line became last, ...
|
|
|
|
@group
|
|
# from the second line, the buffer (which contains all previous lines)
|
|
# is *appended* to current line, so, the order will be reversed
|
|
1! G
|
|
@end group
|
|
|
|
@group
|
|
# on the last line we're done -- print everything
|
|
$ p
|
|
@end group
|
|
|
|
@group
|
|
# store everything on the buffer again
|
|
h
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node cat -n
|
|
@section Numbering Lines
|
|
|
|
This script replaces @samp{cat -n}; in fact it formats its output
|
|
exactly like @acronym{GNU} @command{cat} does.
|
|
|
|
Of course this is completely useless and for two reasons: first,
|
|
because somebody else did it in C, second, because the following
|
|
Bourne-shell script could be used for the same purpose and would
|
|
be much faster:
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
@group
|
|
#! /bin/sh
|
|
sed -e "=" $@@ | sed -e '
|
|
s/^/ /
|
|
N
|
|
s/^ *\(......\)\n/\1 /
|
|
'
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
It uses @command{sed} to print the line number, then groups lines two
|
|
by two using @code{N}. Of course, this script does not teach as much as
|
|
the one presented below.
|
|
|
|
The algorithm used for incrementing uses both buffers, so the line
|
|
is printed as soon as possible and then discarded. The number
|
|
is split so that changing digits go in a buffer and unchanged ones go
|
|
in the other; the changed digits are modified in a single step
|
|
(using a @code{y} command). The line number for the next line
|
|
is then composed and stored in the hold space, to be used in the
|
|
next iteration.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
@group
|
|
# Prime the pump on the first line
|
|
x
|
|
/^$/ s/^.*$/1/
|
|
@end group
|
|
|
|
@group
|
|
# Add the correct line number before the pattern
|
|
G
|
|
h
|
|
@end group
|
|
|
|
@group
|
|
# Format it and print it
|
|
s/^/ /
|
|
s/^ *\(......\)\n/\1 /p
|
|
@end group
|
|
|
|
@group
|
|
# Get the line number from hold space; add a zero
|
|
# if we're going to add a digit on the next line
|
|
g
|
|
s/\n.*$//
|
|
/^9*$/ s/^/0/
|
|
@end group
|
|
|
|
@group
|
|
# separate changing/unchanged digits with an x
|
|
s/.9*$/x&/
|
|
@end group
|
|
|
|
@group
|
|
# keep changing digits in hold space
|
|
h
|
|
s/^.*x//
|
|
y/0123456789/1234567890/
|
|
x
|
|
@end group
|
|
|
|
@group
|
|
# keep unchanged digits in pattern space
|
|
s/x.*$//
|
|
@end group
|
|
|
|
@group
|
|
# compose the new number, remove the newline implicitly added by G
|
|
G
|
|
s/\n//
|
|
h
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node cat -b
|
|
@section Numbering Non-blank Lines
|
|
|
|
Emulating @samp{cat -b} is almost the same as @samp{cat -n}---we only
|
|
have to select which lines are to be numbered and which are not.
|
|
|
|
The part that is common to this script and the previous one is
|
|
not commented to show how important it is to comment @command{sed}
|
|
scripts properly...
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
@group
|
|
/^$/ @{
|
|
p
|
|
b
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
# Same as cat -n from now
|
|
x
|
|
/^$/ s/^.*$/1/
|
|
G
|
|
h
|
|
s/^/ /
|
|
s/^ *\(......\)\n/\1 /p
|
|
x
|
|
s/\n.*$//
|
|
/^9*$/ s/^/0/
|
|
s/.9*$/x&/
|
|
h
|
|
s/^.*x//
|
|
y/0123456789/1234567890/
|
|
x
|
|
s/x.*$//
|
|
G
|
|
s/\n//
|
|
h
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node wc -c
|
|
@section Counting Characters
|
|
|
|
This script shows another way to do arithmetic with @command{sed}.
|
|
In this case we have to add possibly large numbers, so implementing
|
|
this by successive increments would not be feasible (and possibly
|
|
even more complicated to contrive than this script).
|
|
|
|
The approach is to map numbers to letters, kind of an abacus
|
|
implemented with @command{sed}. @samp{a}s are units, @samp{b}s are
|
|
tens and so on: we simply add the number of characters
|
|
on the current line as units, and then propagate the carry
|
|
to tens, hundreds, and so on.
|
|
|
|
As usual, running totals are kept in hold space.
|
|
|
|
On the last line, we convert the abacus form back to decimal.
|
|
For the sake of variety, this is done with a loop rather than
|
|
with some 80 @code{s} commands@footnote{Some implementations
|
|
have a limit of 199 commands per script}: first we
|
|
convert units, removing @samp{a}s from the number; then we
|
|
rotate letters so that tens become @samp{a}s, and so on
|
|
until no more letters remain.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
@group
|
|
# Add n+1 a's to hold space (+1 is for the newline)
|
|
s/./a/g
|
|
H
|
|
x
|
|
s/\n/a/
|
|
@end group
|
|
|
|
@group
|
|
# Do the carry. The t's and b's are not necessary,
|
|
# but they do speed up the thing
|
|
t a
|
|
: a; s/aaaaaaaaaa/b/g; t b; b done
|
|
: b; s/bbbbbbbbbb/c/g; t c; b done
|
|
: c; s/cccccccccc/d/g; t d; b done
|
|
: d; s/dddddddddd/e/g; t e; b done
|
|
: e; s/eeeeeeeeee/f/g; t f; b done
|
|
: f; s/ffffffffff/g/g; t g; b done
|
|
: g; s/gggggggggg/h/g; t h; b done
|
|
: h; s/hhhhhhhhhh//g
|
|
@end group
|
|
|
|
@group
|
|
: done
|
|
$! @{
|
|
h
|
|
b
|
|
@}
|
|
@end group
|
|
|
|
# On the last line, convert back to decimal
|
|
|
|
@group
|
|
: loop
|
|
/a/! s/[b-h]*/&0/
|
|
s/aaaaaaaaa/9/
|
|
s/aaaaaaaa/8/
|
|
s/aaaaaaa/7/
|
|
s/aaaaaa/6/
|
|
s/aaaaa/5/
|
|
s/aaaa/4/
|
|
s/aaa/3/
|
|
s/aa/2/
|
|
s/a/1/
|
|
@end group
|
|
|
|
@group
|
|
: next
|
|
y/bcdefgh/abcdefg/
|
|
/[a-h]/ b loop
|
|
p
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node wc -w
|
|
@section Counting Words
|
|
|
|
This script is almost the same as the previous one, once each
|
|
of the words on the line is converted to a single @samp{a}
|
|
(in the previous script each letter was changed to an @samp{a}).
|
|
|
|
It is interesting that real @command{wc} programs have optimized
|
|
loops for @samp{wc -c}, so they are much slower at counting
|
|
words rather than characters. This script's bottleneck,
|
|
instead, is arithmetic, and hence the word-counting one
|
|
is faster (it has to manage smaller numbers).
|
|
|
|
Again, the common parts are not commented to show the importance
|
|
of commenting @command{sed} scripts.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
@group
|
|
# Convert words to a's
|
|
s/[ @kbd{tab}][ @kbd{tab}]*/ /g
|
|
s/^/ /
|
|
s/ [^ ][^ ]*/a /g
|
|
s/ //g
|
|
@end group
|
|
|
|
@group
|
|
# Append them to hold space
|
|
H
|
|
x
|
|
s/\n//
|
|
@end group
|
|
|
|
@group
|
|
# From here on it is the same as in wc -c.
|
|
/aaaaaaaaaa/! bx; s/aaaaaaaaaa/b/g
|
|
/bbbbbbbbbb/! bx; s/bbbbbbbbbb/c/g
|
|
/cccccccccc/! bx; s/cccccccccc/d/g
|
|
/dddddddddd/! bx; s/dddddddddd/e/g
|
|
/eeeeeeeeee/! bx; s/eeeeeeeeee/f/g
|
|
/ffffffffff/! bx; s/ffffffffff/g/g
|
|
/gggggggggg/! bx; s/gggggggggg/h/g
|
|
s/hhhhhhhhhh//g
|
|
:x
|
|
$! @{ h; b; @}
|
|
:y
|
|
/a/! s/[b-h]*/&0/
|
|
s/aaaaaaaaa/9/
|
|
s/aaaaaaaa/8/
|
|
s/aaaaaaa/7/
|
|
s/aaaaaa/6/
|
|
s/aaaaa/5/
|
|
s/aaaa/4/
|
|
s/aaa/3/
|
|
s/aa/2/
|
|
s/a/1/
|
|
y/bcdefgh/abcdefg/
|
|
/[a-h]/ by
|
|
p
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node wc -l
|
|
@section Counting Lines
|
|
|
|
No strange things are done now, because @command{sed} gives us
|
|
@samp{wc -l} functionality for free!!! Look:
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
@group
|
|
#!/usr/bin/sed -nf
|
|
$=
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node head
|
|
@section Printing the First Lines
|
|
|
|
This script is probably the simplest useful @command{sed} script.
|
|
It displays the first 10 lines of input; the number of displayed
|
|
lines is right before the @code{q} command.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
@group
|
|
#!/usr/bin/sed -f
|
|
10q
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node tail
|
|
@section Printing the Last Lines
|
|
|
|
Printing the last @var{n} lines rather than the first is more complex
|
|
but indeed possible. @var{n} is encoded in the second line, before
|
|
the bang character.
|
|
|
|
This script is similar to the @command{tac} script in that it keeps the
|
|
final output in the hold space and prints it at the end:
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
@group
|
|
1! @{; H; g; @}
|
|
1,10 !s/[^\n]*\n//
|
|
$p
|
|
h
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
Mainly, the scripts keeps a window of 10 lines and slides it
|
|
by adding a line and deleting the oldest (the substitution command
|
|
on the second line works like a @code{D} command but does not
|
|
restart the loop).
|
|
|
|
The ``sliding window'' technique is a very powerful way to write
|
|
efficient and complex @command{sed} scripts, because commands like
|
|
@code{P} would require a lot of work if implemented manually.
|
|
|
|
To introduce the technique, which is fully demonstrated in the
|
|
rest of this chapter and is based on the @code{N}, @code{P}
|
|
and @code{D} commands, here is an implementation of @command{tail}
|
|
using a simple ``sliding window.''
|
|
|
|
This looks complicated but in fact the working is the same as
|
|
the last script: after we have kicked in the appropriate number
|
|
of lines, however, we stop using the hold space to keep inter-line
|
|
state, and instead use @code{N} and @code{D} to slide pattern
|
|
space by one line:
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -f
|
|
|
|
@group
|
|
1h
|
|
2,10 @{; H; g; @}
|
|
$q
|
|
1,9d
|
|
N
|
|
D
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
Note how the first, second and fourth line are inactive after
|
|
the first ten lines of input. After that, all the script does
|
|
is: exiting on the last line of input, appending the next input
|
|
line to pattern space, and removing the first line.
|
|
|
|
@node uniq
|
|
@section Make Duplicate Lines Unique
|
|
|
|
This is an example of the art of using the @code{N}, @code{P}
|
|
and @code{D} commands, probably the most difficult to master.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
@group
|
|
#!/usr/bin/sed -f
|
|
h
|
|
@end group
|
|
|
|
@group
|
|
:b
|
|
# On the last line, print and exit
|
|
$b
|
|
N
|
|
/^\(.*\)\n\1$/ @{
|
|
# The two lines are identical. Undo the effect of
|
|
# the n command.
|
|
g
|
|
bb
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
# If the @code{N} command had added the last line, print and exit
|
|
$b
|
|
@end group
|
|
|
|
@group
|
|
# The lines are different; print the first and go
|
|
# back working on the second.
|
|
P
|
|
D
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
As you can see, we mantain a 2-line window using @code{P} and @code{D}.
|
|
This technique is often used in advanced @command{sed} scripts.
|
|
|
|
@node uniq -d
|
|
@section Print Duplicated Lines of Input
|
|
|
|
This script prints only duplicated lines, like @samp{uniq -d}.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
@group
|
|
$b
|
|
N
|
|
/^\(.*\)\n\1$/ @{
|
|
# Print the first of the duplicated lines
|
|
s/.*\n//
|
|
p
|
|
@end group
|
|
|
|
@group
|
|
# Loop until we get a different line
|
|
:b
|
|
$b
|
|
N
|
|
/^\(.*\)\n\1$/ @{
|
|
s/.*\n//
|
|
bb
|
|
@}
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
# The last line cannot be followed by duplicates
|
|
$b
|
|
@end group
|
|
|
|
@group
|
|
# Found a different one. Leave it alone in the pattern space
|
|
# and go back to the top, hunting its duplicates
|
|
D
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node uniq -u
|
|
@section Remove All Duplicated Lines
|
|
|
|
This script prints only unique lines, like @samp{uniq -u}.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -f
|
|
|
|
@group
|
|
# Search for a duplicate line --- until that, print what you find.
|
|
$b
|
|
N
|
|
/^\(.*\)\n\1$/ ! @{
|
|
P
|
|
D
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
:c
|
|
# Got two equal lines in pattern space. At the
|
|
# end of the file we simply exit
|
|
$d
|
|
@end group
|
|
|
|
@group
|
|
# Else, we keep reading lines with @code{N} until we
|
|
# find a different one
|
|
s/.*\n//
|
|
N
|
|
/^\(.*\)\n\1$/ @{
|
|
bc
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
# Remove the last instance of the duplicate line
|
|
# and go back to the top
|
|
D
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node cat -s
|
|
@section Squeezing Blank Lines
|
|
|
|
As a final example, here are three scripts, of increasing complexity
|
|
and speed, that implement the same function as @samp{cat -s}, that is
|
|
squeezing blank lines.
|
|
|
|
The first leaves a blank line at the beginning and end if there are
|
|
some already.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -f
|
|
|
|
@group
|
|
# on empty lines, join with next
|
|
# Note there is a star in the regexp
|
|
:x
|
|
/^\n*$/ @{
|
|
N
|
|
bx
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
# now, squeeze all '\n', this can be also done by:
|
|
# s/^\(\n\)*/\1/
|
|
s/\n*/\
|
|
/
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
This one is a bit more complex and removes all empty lines
|
|
at the beginning. It does leave a single blank line at end
|
|
if one was there.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -f
|
|
|
|
@group
|
|
# delete all leading empty lines
|
|
1,/^./@{
|
|
/./!d
|
|
@}
|
|
@end group
|
|
|
|
@group
|
|
# on an empty line we remove it and all the following
|
|
# empty lines, but one
|
|
:x
|
|
/./!@{
|
|
N
|
|
s/^\n$//
|
|
tx
|
|
@}
|
|
@end group
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
This removes leading and trailing blank lines. It is also the
|
|
fastest. Note that loops are completely done with @code{n} and
|
|
@code{b}, without relying on @command{sed} to restart the
|
|
the script automatically at the end of a line.
|
|
|
|
@c start-------------------------------------------
|
|
@example
|
|
#!/usr/bin/sed -nf
|
|
|
|
@group
|
|
# delete all (leading) blanks
|
|
/./!d
|
|
@end group
|
|
|
|
@group
|
|
# get here: so there is a non empty
|
|
:x
|
|
# print it
|
|
p
|
|
# get next
|
|
n
|
|
# got chars? print it again, etc...
|
|
/./bx
|
|
@end group
|
|
|
|
@group
|
|
# no, don't have chars: got an empty line
|
|
:z
|
|
# get next, if last line we finish here so no trailing
|
|
# empty lines are written
|
|
n
|
|
# also empty? then ignore it, and get next... this will
|
|
# remove ALL empty lines
|
|
/./!bz
|
|
@end group
|
|
|
|
@group
|
|
# all empty lines were deleted/ignored, but we have a non empty. As
|
|
# what we want to do is to squeeze, insert a blank line artificially
|
|
i\
|
|
@end group
|
|
|
|
bx
|
|
@end example
|
|
@c end---------------------------------------------
|
|
|
|
@node Limitations
|
|
@chapter @value{SSED}'s Limitations and Non-limitations
|
|
|
|
@cindex @acronym{GNU} extensions, unlimited line length
|
|
@cindex Portability, line length limitations
|
|
For those who want to write portable @command{sed} scripts,
|
|
be aware that some implementations have been known to
|
|
limit line lengths (for the pattern and hold spaces)
|
|
to be no more than 4000 bytes.
|
|
The @sc{posix} standard specifies that conforming @command{sed}
|
|
implementations shall support at least 8192 byte line lengths.
|
|
@value{SSED} has no built-in limit on line length;
|
|
as long as it can @code{malloc()} more (virtual) memory,
|
|
you can feed or construct lines as long as you like.
|
|
|
|
However, recursion is used to handle subpatterns and indefinite
|
|
repetition. This means that the available stack space may limit
|
|
the size of the buffer that can be processed by certain patterns.
|
|
|
|
@ifset PERL
|
|
There are some size limitations in the regular expression
|
|
matcher but it is hoped that they will never in practice
|
|
be relevant. The maximum length of a compiled pattern
|
|
is 65539 (sic) bytes. All values in repeating quantifiers
|
|
must be less than 65536. The maximum nesting depth of
|
|
all parenthesized subpatterns, including capturing and
|
|
non-capturing subpatterns@footnote{The
|
|
distinction is meaningful when referring to Perl-style
|
|
regular expressions.}, assertions, and other types of
|
|
subpattern, is 200.
|
|
|
|
Also, @value{SSED} recognizes the @sc{posix} syntax
|
|
@code{[.@var{ch}.]} and @code{[=@var{ch}=]}
|
|
where @var{ch} is a ``collating element'', but these
|
|
are not supported, and an error is given if they are
|
|
encountered.
|
|
|
|
Here are a few distinctions between the real Perl-style
|
|
regular expressions and those that @option{-R} recognizes.
|
|
|
|
@enumerate
|
|
@item
|
|
Lookahead assertions do not allow repeat quantifiers after them
|
|
Perl permits them, but they do not mean what you
|
|
might think. For example, @samp{(?!a)@{3@}} does not assert that the
|
|
next three characters are not @samp{a}. It just asserts three times that the
|
|
next character is not @samp{a} --- a waste of time and nothing else.
|
|
|
|
@item
|
|
Capturing subpatterns that occur inside negative lookahead
|
|
head assertions are counted, but their entries are counted
|
|
as empty in the second half of an @code{s} command.
|
|
Perl sets its numerical variables from any such patterns
|
|
that are matched before the assertion fails to match
|
|
something (thereby succeeding), but only if the negative
|
|
lookahead assertion contains just one branch.
|
|
|
|
@item
|
|
The following Perl escape sequences are not supported:
|
|
@samp{\l}, @samp{\u}, @samp{\L}, @samp{\U}, @samp{\E},
|
|
@samp{\Q}. In fact these are implemented by Perl's general
|
|
string-handling and are not part of its pattern matching engine.
|
|
|
|
@item
|
|
The Perl @samp{\G} assertion is not supported as it is not
|
|
relevant to single pattern matches.
|
|
|
|
@item
|
|
Fairly obviously, @value{SSED} does not support the @samp{(?@{code@})}
|
|
and @samp{(?p@{code@})} constructions. However, there is some experimental
|
|
support for recursive patterns using the non-Perl item @samp{(?R)}.
|
|
|
|
@item
|
|
There are at the time of writing some oddities in Perl
|
|
5.005_02 concerned with the settings of captured strings
|
|
when part of a pattern is repeated. For example, matching
|
|
@samp{aba} against the pattern @samp{/^(a(b)?)+$/} sets
|
|
@samp{$2}@footnote{@samp{$2} would be @samp{\2} in @value{SSED}.}
|
|
to the value @samp{b}, but matching @samp{aabbaa}
|
|
against @samp{/^(aa(bb)?)+$/} leaves @samp{$2}
|
|
unset. However, if the pattern is changed to
|
|
@samp{/^(aa(b(b))?)+$/} then @samp{$2} (and @samp{$3}) are set.
|
|
In Perl 5.004 @samp{$2} is set in both cases, and that is also
|
|
true of @value{SSED}.
|
|
|
|
@item
|
|
Another as yet unresolved discrepancy is that in Perl
|
|
5.005_02 the pattern @samp{/^(a)?(?(1)a|b)+$/} matches
|
|
the string @samp{a}, whereas in @value{SSED} it does not.
|
|
However, in both Perl and @value{SSED} @samp{/^(a)?a/} matched
|
|
against @samp{a} leaves $1 unset.
|
|
@end enumerate
|
|
@end ifset
|
|
|
|
@node Other Resources
|
|
@chapter Other Resources for Learning About @command{sed}
|
|
|
|
@cindex Additional reading about @command{sed}
|
|
In addition to several books that have been written about @command{sed}
|
|
(either specifically or as chapters in books which discuss
|
|
shell programming), one can find out more about @command{sed}
|
|
(including suggestions of a few books) from the FAQ
|
|
for the @code{sed-users} mailing list, available from:
|
|
@display
|
|
@uref{http://sed.sourceforge.net/sedfaq.html}
|
|
@end display
|
|
|
|
Also of interest are
|
|
@uref{http://www.student.northpark.edu/pemente/sed/index.htm}
|
|
and @uref{http://sed.sf.net/grabbag},
|
|
which include @command{sed} tutorials and other @command{sed}-related goodies.
|
|
|
|
The @code{sed-users} mailing list itself maintained by Sven Guckes.
|
|
To subscribe, visit @uref{http://groups.yahoo.com} and search
|
|
for the @code{sed-users} mailing list.
|
|
|
|
@node Reporting Bugs
|
|
@chapter Reporting Bugs
|
|
|
|
@cindex Bugs, reporting
|
|
Email bug reports to @email{bonzini@@gnu.org}.
|
|
Be sure to include the word ``sed'' somewhere in the @code{Subject:} field.
|
|
Also, please include the output of @samp{sed --version} in the body
|
|
of your report if at all possible.
|
|
|
|
Please do not send a bug report like this:
|
|
|
|
@example
|
|
@i{@i{@r{while building frobme-1.3.4}}}
|
|
$ configure
|
|
@error{} sed: file sedscr line 1: Unknown option to 's'
|
|
@end example
|
|
|
|
If @value{SSED} doesn't configure your favorite package, take a
|
|
few extra minutes to identify the specific problem and make a stand-alone
|
|
test case. Unlike other programs such as C compilers, making such test
|
|
cases for @command{sed} is quite simple.
|
|
|
|
A stand-alone test case includes all the data necessary to perform the
|
|
test, and the specific invocation of @command{sed} that causes the problem.
|
|
The smaller a stand-alone test case is, the better. A test case should
|
|
not involve something as far removed from @command{sed} as ``try to configure
|
|
frobme-1.3.4''. Yes, that is in principle enough information to look
|
|
for the bug, but that is not a very practical prospect.
|
|
|
|
Here are a few commonly reported bugs that are not bugs.
|
|
|
|
@table @asis
|
|
@item @code{N} command on the last line
|
|
@cindex Portability, @code{N} command on the last line
|
|
@cindex Non-bugs, @code{N} command on the last line
|
|
|
|
Most versions of @command{sed} exit without printing anything when
|
|
the @command{N} command is issued on the last line of a file.
|
|
@value{SSED} prints pattern space before exiting unless of course
|
|
the @command{-n} command switch has been specified. This choice is
|
|
by design.
|
|
|
|
For example, the behavior of
|
|
@example
|
|
sed N foo bar
|
|
@end example
|
|
@noindent
|
|
would depend on whether foo has an even or an odd number of
|
|
lines@footnote{which is the actual ``bug'' that prompted the
|
|
change in behavior}. Or, when writing a script to read the
|
|
next few lines following a pattern match, traditional
|
|
implementations of @code{sed} would force you to write
|
|
something like
|
|
@example
|
|
/foo/@{ $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N @}
|
|
@end example
|
|
@noindent
|
|
instead of just
|
|
@example
|
|
/foo/@{ N;N;N;N;N;N;N;N;N; @}
|
|
@end example
|
|
|
|
@cindex @code{POSIXLY_CORRECT} behavior, @code{N} command
|
|
In any case, the simplest workaround is to use @code{$d;N} in
|
|
scripts that rely on the traditional behavior, or to set
|
|
the @code{POSIXLY_CORRECT} variable to a non-empty value.
|
|
|
|
@item Regex syntax clashes (problems with backslashes)
|
|
@cindex @acronym{GNU} extensions, to basic regular expressions
|
|
@cindex Non-bugs, regex syntax clashes
|
|
@command{sed} uses the @sc{posix} basic regular expression syntax. According to
|
|
the standard, the meaning of some escape sequences is undefined in
|
|
this syntax; notable in the case of @command{sed} are @code{\|},
|
|
@code{\+}, @code{\?}, @code{\`}, @code{\'}, @code{\<},
|
|
@code{\>}, @code{\b}, @code{\B}, @code{\w}, and @code{\W}.
|
|
|
|
As in all @acronym{GNU} programs that use @sc{posix} basic regular
|
|
expressions, @command{sed} interprets these escape sequences as special
|
|
characters. So, @code{x\+} matches one or more occurrences of @samp{x}.
|
|
@code{abc\|def} matches either @samp{abc} or @samp{def}.
|
|
|
|
This syntax may cause problems when running scripts written for other
|
|
@command{sed}s. Some @command{sed} programs have been written with the
|
|
assumption that @code{\|} and @code{\+} match the literal characters
|
|
@code{|} and @code{+}. Such scripts must be modified by removing the
|
|
spurious backslashes if they are to be used with modern implementations
|
|
of @command{sed}, like
|
|
@ifset PERL
|
|
@value{SSED} or
|
|
@end ifset
|
|
@acronym{GNU} @command{sed}.
|
|
|
|
On the other hand, some scripts use s|abc\|def||g to remove occurrences
|
|
of @emph{either} @code{abc} or @code{def}. While this worked until
|
|
@command{sed} 4.0.x, newer versions interpret this as removing the
|
|
string @code{abc|def}. This is again undefined behavior according to
|
|
@acronym{POSIX}, and this interpretation is arguably more robust: older
|
|
@command{sed}s, for example, required that the regex matcher parsed
|
|
@code{\/} as @code{/} in the common case of escaping a slash, which is
|
|
again undefined behavior; the new behavior avoids this, and this is good
|
|
because the regex matcher is only partially under our control.
|
|
|
|
@cindex @acronym{GNU} extensions, special escapes
|
|
In addition, this version of @command{sed} supports several escape characters
|
|
(some of which are multi-character) to insert non-printable characters
|
|
in scripts (@code{\a}, @code{\c}, @code{\d}, @code{\o}, @code{\r},
|
|
@code{\t}, @code{\v}, @code{\x}). These can cause similar problems
|
|
with scripts written for other @command{sed}s.
|
|
|
|
@item @option{-i} clobbers read-only files
|
|
@cindex In-place editing
|
|
@cindex @value{SSEDEXT}, in-place editing
|
|
@cindex Non-bugs, in-place editing
|
|
|
|
In short, @samp{sed -i} will let you delete the contents of
|
|
a read-only file, and in general the @option{-i} option
|
|
(@pxref{Invoking sed, , Invocation}) lets you clobber
|
|
protected files. This is not a bug, but rather a consequence
|
|
of how the Unix filesystem works.
|
|
|
|
The permissions on a file say what can happen to the data
|
|
in that file, while the permissions on a directory say what can
|
|
happen to the list of files in that directory. @samp{sed -i}
|
|
will not ever open for writing a file that is already on disk.
|
|
Rather, it will work on a temporary file that is finally renamed
|
|
to the original name: if you rename or delete files, you're actually
|
|
modifying the contents of the directory, so the operation depends on
|
|
the permissions of the directory, not of the file. For this same
|
|
reason, @command{sed} does not let you use @option{-i} on a writeable file
|
|
in a read-only directory, and will break hard or symbolic links when
|
|
@option{-i} is used on such a file.
|
|
|
|
@item @code{0a} does not work (gives an error)
|
|
@cindex @code{0} address
|
|
@cindex @acronym{GNU} extensions, @code{0} address
|
|
@cindex Non-bugs, @code{0} address
|
|
|
|
There is no line 0. 0 is a special address that is only used to treat
|
|
addresses like @code{0,/@var{RE}/} as active when the script starts: if
|
|
you write @code{1,/abc/d} and the first line includes the word @samp{abc},
|
|
then that match would be ignored because address ranges must span at least
|
|
two lines (barring the end of the file); but what you probably wanted is
|
|
to delete every line up to the first one including @samp{abc}, and this
|
|
is obtained with @code{0,/abc/d}.
|
|
|
|
@ifclear PERL
|
|
@item @code{[a-z]} is case insensitive
|
|
@cindex Non-bugs, localization-related
|
|
|
|
You are encountering problems with locales. POSIX mandates that @code{[a-z]}
|
|
uses the current locale's collation order -- in C parlance, that means using
|
|
@code{strcoll(3)} instead of @code{strcmp(3)}. Some locales have a
|
|
case-insensitive collation order, others don't: one of those that have
|
|
problems is Estonian.
|
|
|
|
Another problem is that @code{[a-z]} tries to use collation symbols.
|
|
This only happens if you are on the @acronym{GNU} system, using
|
|
@acronym{GNU} libc's regular expression matcher instead of compiling the
|
|
one supplied with @acronym{GNU} sed. In a Danish locale, for example,
|
|
the regular expression @code{^[a-z]$} matches the string @samp{aa},
|
|
because this is a single collating symbol that comes after @samp{a}
|
|
and before @samp{b}; @samp{ll} behaves similarly in Spanish
|
|
locales, or @samp{ij} in Dutch locales.
|
|
|
|
To work around these problems, which may cause bugs in shell scripts, set
|
|
the @env{LC_COLLATE} and @env{LC_CTYPE} environment variables to @samp{C}.
|
|
|
|
@item @code{s/.*//} does not clear pattern space
|
|
@cindex Non-bugs, localization-related
|
|
@cindex @value{SSEDEXT}, emptying pattern space
|
|
@cindex Emptying pattern space
|
|
|
|
This happens if your input stream includes invalid multibyte
|
|
sequences. @sc{posix} mandates that such sequences
|
|
are @emph{not} matched by @samp{.}, so that @samp{s/.*//} will not clear
|
|
pattern space as you would expect. In fact, there is no way to clear
|
|
sed's buffers in the middle of the script in most multibyte locales
|
|
(including UTF-8 locales). For this reason, @value{SSED} provides a `z'
|
|
command (for `zap') as an extension.
|
|
|
|
To work around these problems, which may cause bugs in shell scripts, set
|
|
the @env{LC_COLLATE} and @env{LC_CTYPE} environment variables to @samp{C}.
|
|
@end ifclear
|
|
@end table
|
|
|
|
|
|
@node Extended regexps
|
|
@appendix Extended regular expressions
|
|
@cindex Extended regular expressions, syntax
|
|
|
|
The only difference between basic and extended regular expressions is in
|
|
the behavior of a few characters: @samp{?}, @samp{+}, parentheses,
|
|
braces (@samp{@{@}}), and @samp{|}. While basic regular expressions
|
|
require these to be escaped if you want them to behave as special
|
|
characters, when using extended regular expressions you must escape
|
|
them if you want them @emph{to match a literal character}. @samp{|}
|
|
is special here because @samp{\|} is a GNU extension -- standard
|
|
basic regular expressions do not provide its functionality.
|
|
|
|
@noindent
|
|
Examples:
|
|
@table @code
|
|
@item abc?
|
|
becomes @samp{abc\?} when using extended regular expressions. It matches
|
|
the literal string @samp{abc?}.
|
|
|
|
@item c\+
|
|
becomes @samp{c+} when using extended regular expressions. It matches
|
|
one or more @samp{c}s.
|
|
|
|
@item a\@{3,\@}
|
|
becomes @samp{a@{3,@}} when using extended regular expressions. It matches
|
|
three or more @samp{a}s.
|
|
|
|
@item \(abc\)\@{2,3\@}
|
|
becomes @samp{(abc)@{2,3@}} when using extended regular expressions. It
|
|
matches either @samp{abcabc} or @samp{abcabcabc}.
|
|
|
|
@item \(abc*\)\1
|
|
becomes @samp{(abc*)\1} when using extended regular expressions.
|
|
Backreferences must still be escaped when using extended regular
|
|
expressions.
|
|
@end table
|
|
|
|
@ifset PERL
|
|
@node Perl regexps
|
|
@appendix Perl-style regular expressions
|
|
@cindex Perl-style regular expressions, syntax
|
|
|
|
@emph{This part is taken from the @file{pcre.txt} file distributed together
|
|
with the free @sc{pcre} regular expression matcher; it was written by Philip Hazel.}
|
|
|
|
Perl introduced several extensions to regular expressions, some
|
|
of them incompatible with the syntax of regular expressions
|
|
accepted by Emacs and other @acronym{GNU} tools (whose matcher was
|
|
based on the Emacs matcher). @value{SSED} implements
|
|
both kinds of extensions.
|
|
|
|
@iftex
|
|
Summarizing, we have:
|
|
|
|
@itemize @bullet
|
|
@item
|
|
A backslash can introduce several special sequences
|
|
|
|
@item
|
|
The circumflex, dollar sign, and period characters behave specially
|
|
with regard to new lines
|
|
|
|
@item
|
|
Strange uses of square brackets are parsed differently
|
|
|
|
@item
|
|
You can toggle modifiers in the middle of a regular expression
|
|
|
|
@item
|
|
You can specify that a subpattern does not count when numbering backreferences
|
|
|
|
@item
|
|
@cindex Greedy regular expression matching
|
|
You can specify greedy or non-greedy matching
|
|
|
|
@item
|
|
You can have more than ten back references
|
|
|
|
@item
|
|
You can do complex look aheads and look behinds (in the spirit of
|
|
@code{\b}, but with subpatterns).
|
|
|
|
@item
|
|
You can often improve performance by avoiding that @command{sed} wastes
|
|
time with backtracking
|
|
|
|
@item
|
|
You can have if/then/else branches
|
|
|
|
@item
|
|
You can do recursive matches, for example to look for unbalanced parentheses
|
|
|
|
@item
|
|
You can have comments and non-significant whitespace, because things can
|
|
get complex...
|
|
@end itemize
|
|
|
|
Most of these extensions are introduced by the special @code{(?}
|
|
sequence, which gives special meanings to parenthesized groups.
|
|
@end iftex
|
|
@menu
|
|
Other extensions can be roughly subdivided in two categories
|
|
On one hand Perl introduces several more escaped sequences
|
|
(that is, sequences introduced by a backslash). On the other
|
|
hand, it specifies that if a question mark follows an open
|
|
parentheses it should give a special meaning to the parenthesized
|
|
group.
|
|
|
|
* Backslash:: Introduces special sequences
|
|
* Circumflex/dollar sign/period:: Behave specially with regard to new lines
|
|
* Square brackets:: Are a bit different in strange cases
|
|
* Options setting:: Toggle modifiers in the middle of a regexp
|
|
* Non-capturing subpatterns:: Are not counted when backreferencing
|
|
* Repetition:: Allows for non-greedy matching
|
|
* Backreferences:: Allows for more than 10 back references
|
|
* Assertions:: Allows for complex look ahead matches
|
|
* Non-backtracking subpatterns:: Often gives more performance
|
|
* Conditional subpatterns:: Allows if/then/else branches
|
|
* Recursive patterns:: For example to match parentheses
|
|
* Comments:: Because things can get complex...
|
|
@end menu
|
|
|
|
@node Backslash
|
|
@appendixsec Backslash
|
|
@cindex Perl-style regular expressions, escaped sequences
|
|
|
|
There are a few difference in the handling of backslashed
|
|
sequences in Perl mode.
|
|
|
|
First of all, there are no @code{\o} and @code{\d} sequences.
|
|
@sc{ascii} values for characters can be specified in octal
|
|
with a @code{\@var{xxx}} sequence, where @var{xxx} is a
|
|
sequence of up to three octal digits. If the first digit
|
|
is a zero, the treatment of the sequence is straightforward;
|
|
just note that if the character that follows the escaped digit
|
|
is itself an octal digit, you have to supply three octal digits
|
|
for @var{xxx}. For example @code{\07} is a @sc{bel} character
|
|
rather than a @sc{nul} and a literal @code{7} (this sequence is
|
|
instead represented by @code{\0007}).
|
|
|
|
@cindex Perl-style regular expressions, backreferences
|
|
The handling of a backslash followed by a digit other than 0
|
|
is complicated. Outside a character class, @command{sed} reads it
|
|
and any following digits as a decimal number. If the number
|
|
is less than 10, or if there have been at least that many
|
|
previous capturing left parentheses in the expression, the
|
|
entire sequence is taken as a back reference. A description
|
|
of how this works is given later, following the discussion
|
|
of parenthesized subpatterns.
|
|
|
|
Inside a character class, or if the decimal number is
|
|
greater than 9 and there have not been that many capturing
|
|
subpatterns, @command{sed} re-reads up to three octal digits following
|
|
the backslash, and generates a single byte from the
|
|
least significant 8 bits of the value. Any subsequent digits
|
|
stand for themselves. For example:
|
|
|
|
@example
|
|
\040 @i{@r{is another way of writing a space}}
|
|
\40 @i{@r{is the same, provided there are fewer than 40}}
|
|
@i{@r{previous capturing subpatterns}}
|
|
\7 @i{@r{is always a back reference}}
|
|
\011 @i{@r{is always a tab}}
|
|
\11 @i{@r{might be a back reference, or another way of writing a tab}}
|
|
\0113 @i{@r{is a tab followed by the character @samp{3}}}
|
|
\113 @i{@r{is the character with octal code 113 (since there}}
|
|
@i{@r{can be no more than 99 back references)}}
|
|
\377 @i{@r{is a byte consisting entirely of 1 bits (@sc{ascii} 255)}}
|
|
\81 @i{@r{is either a back reference, or a binary zero}}
|
|
@i{@r{followed by the two characters @samp{81}}}
|
|
@end example
|
|
|
|
Note that octal values of 100 or greater must not be introduced
|
|
by a leading zero, because no more than three octal
|
|
digits are ever read. Note that this applies only to the LHS
|
|
pattern; it is not possible yet to specify more than 9 backreferences
|
|
on the RHS of the `s' command.
|
|
|
|
All the sequences that define a single byte value can be
|
|
used both inside and outside character classes. In addition,
|
|
inside a character class, the sequence @code{\b} is interpreted
|
|
as the backspace character (hex 08). Outside a character
|
|
class it has a different meaning (see below).
|
|
|
|
In addition, there are four additional escapes specifying
|
|
generic character classes (like @code{\w} and @code{\W} do):
|
|
|
|
@cindex Perl-style regular expressions, character classes
|
|
@table @samp
|
|
@item \d
|
|
Matches any decimal digit
|
|
|
|
@item \D
|
|
Matches any character that is not a decimal digit
|
|
@end table
|
|
|
|
In Perl mode, these character type sequences can appear both inside and
|
|
outside character classes. Instead, in @sc{posix} mode these sequences
|
|
(as well as @code{\w} and @code{\W}) are treated as two literal characters
|
|
(a backslash and a letter) inside square brackets.
|
|
|
|
Escaped sequences specifying assertions are also different in
|
|
Perl mode. An assertion specifies a condition that has to be met
|
|
at a particular point in a match, without consuming any
|
|
characters from the subject string. The use of subpatterns
|
|
for more complicated assertions is described below. The
|
|
backslashed assertions are
|
|
|
|
@cindex Perl-style regular expressions, assertions
|
|
@table @samp
|
|
@item \b
|
|
Asserts that the point is at a word boundary.
|
|
A word boundary is a position in the subject string where
|
|
the current character and the previous character do not both
|
|
match @code{\w} or @code{\W} (i.e. one matches @code{\w} and
|
|
the other matches @code{\W}), or the start or end of the string
|
|
if the first or last character matches @code{\w}, respectively.
|
|
|
|
@item \B
|
|
Asserts that the point is not at a word boundary.
|
|
|
|
@item \A
|
|
Asserts the matcher is at the start of pattern space (independent
|
|
of multiline mode).
|
|
|
|
@item \Z
|
|
Asserts the matcher is at the end of pattern space,
|
|
or at a newline before the end of pattern space (independent of
|
|
multiline mode)
|
|
|
|
@item \z
|
|
Asserts the matcher is at the end of pattern space (independent
|
|
of multiline mode)
|
|
@end table
|
|
|
|
These assertions may not appear in character classes (but
|
|
note that @code{\b} has a different meaning, namely the
|
|
backspace character, inside a character class).
|
|
Note that Perl mode does not support directly assertions
|
|
for the beginning and the end of word; the @acronym{GNU} extensions
|
|
@code{\<} and @code{\>} achieve this purpose in @sc{posix} mode
|
|
instead.
|
|
|
|
The @code{\A}, @code{\Z}, and @code{\z} assertions differ
|
|
from the traditional circumflex and dollar sign (described below)
|
|
in that they only ever match at the very start and end of the
|
|
subject string, whatever options are set; in particular @code{\A}
|
|
and @code{\z} are the same as the @acronym{GNU} extensions
|
|
@code{\`} and @code{\'} that are active in @sc{posix} mode.
|
|
|
|
@node Circumflex/dollar sign/period
|
|
@appendixsec Circumflex, dollar sign, period
|
|
@cindex Perl-style regular expressions, newlines
|
|
|
|
Outside a character class, in the default matching mode, the
|
|
circumflex character is an assertion which is true only if
|
|
the current matching point is at the start of the subject
|
|
string. Inside a character class, the circumflex has an entirely
|
|
different meaning (see below).
|
|
|
|
The circumflex need not be the first character of the pattern if
|
|
a number of alternatives are involved, but it should be the
|
|
first thing in each alternative in which it appears if the
|
|
pattern is ever to match that branch. If all possible alternatives,
|
|
start with a circumflex, that is, if the pattern is
|
|
constrained to match only at the start of the subject, it is
|
|
said to be an @dfn{anchored} pattern. (There are also other constructs
|
|
structs that can cause a pattern to be anchored.)
|
|
|
|
A dollar sign is an assertion which is true only if the
|
|
current matching point is at the end of the subject string,
|
|
or immediately before a newline character that is the last
|
|
character in the string (by default). A dollar sign need not be the
|
|
last character of the pattern if a number of alternatives
|
|
are involved, but it should be the last item in any branch
|
|
in which it appears. A dollar sign has no special meaning in a
|
|
character class.
|
|
|
|
@cindex Perl-style regular expressions, multiline
|
|
The meanings of the circumflex and dollar sign characters are
|
|
changed if the @code{M} modifier option is used. When this is
|
|
the case, they match immediately after and immediately
|
|
before an internal @code{\n} character, respectively, in addition
|
|
to matching at the start and end of the subject string. For
|
|
example, the pattern @code{/^abc$/} matches the subject string
|
|
@samp{def\nabc} in multiline mode, but not otherwise. Consequently,
|
|
patterns that are anchored in single line mode
|
|
because all branches start with @code{^} are not anchored in
|
|
multiline mode.
|
|
|
|
@cindex Perl-style regular expressions, multiline
|
|
Note that the sequences @code{\A}, @code{\Z}, and @code{\z}
|
|
can be used to match the start and end of the subject in both
|
|
modes, and if all branches of a pattern start with @code{\A}
|
|
is it always anchored, whether the @code{M} modifier is set or not.
|
|
|
|
@cindex Perl-style regular expressions, single line
|
|
Outside a character class, a dot in the pattern matches any
|
|
one character in the subject, including a non-printing character,
|
|
but not (by default) newline. If the @code{S} modifier is used,
|
|
dots match newlines as well. Actually, the handling of
|
|
dot is entirely independent of the handling of circumflex
|
|
and dollar sign, the only relationship being that they both
|
|
involve newline characters. Dot has no special meaning in a
|
|
character class.
|
|
|
|
@node Square brackets
|
|
@appendixsec Square brackets
|
|
@cindex Perl-style regular expressions, character classes
|
|
|
|
An opening square bracket introduces a character class, terminated
|
|
by a closing square bracket. A closing square bracket on its own
|
|
is not special. If a closing square bracket is required as a
|
|
member of the class, it should be the first data character in
|
|
the class (after an initial circumflex, if present) or escaped with a backslash.
|
|
|
|
A character class matches a single character in the subject;
|
|
the character must be in the set of characters defined by
|
|
the class, unless the first character in the class is a circumflex,
|
|
in which case the subject character must not be in
|
|
the set defined by the class. If a circumflex is actually
|
|
required as a member of the class, ensure it is not the
|
|
first character, or escape it with a backslash.
|
|
|
|
For example, the character class [aeiou] matches any lower
|
|
case vowel, while [^aeiou] matches any character that is not
|
|
a lower case vowel. Note that a circumflex is just a convenient
|
|
venient notation for specifying the characters which are in
|
|
the class by enumerating those that are not. It is not an
|
|
assertion: it still consumes a character from the subject
|
|
string, and fails if the current pointer is at the end of
|
|
the string.
|
|
|
|
@cindex Perl-style regular expressions, case-insensitive
|
|
When caseless matching is set, any letters in a class
|
|
represent both their upper case and lower case versions, so
|
|
for example, a caseless @code{[aeiou]} matches uppercase
|
|
and lowercase @samp{A}s, and a caseless @code{[^aeiou]}
|
|
does not match @samp{A}, whereas a case-sensitive version would.
|
|
|
|
@cindex Perl-style regular expressions, single line
|
|
@cindex Perl-style regular expressions, multiline
|
|
The newline character is never treated in any special way in
|
|
character classes, whatever the setting of the @code{S} and
|
|
@code{M} options (modifiers) is. A class such as @code{[^a]} will
|
|
always match a newline.
|
|
|
|
The minus (hyphen) character can be used to specify a range
|
|
of characters in a character class. For example, @code{[d-m]}
|
|
matches any letter between d and m, inclusive. If a minus
|
|
character is required in a class, it must be escaped with a
|
|
backslash or appear in a position where it cannot be interpreted
|
|
as indicating a range, typically as the first or last
|
|
character in the class.
|
|
|
|
It is not possible to have the literal character @code{]} as the
|
|
end character of a range. A pattern such as @code{[W-]46]} is
|
|
interpreted as a class of two characters (@code{W} and @code{-})
|
|
followed by a literal string @code{46]}, so it would match
|
|
@samp{W46]} or @samp{-46]}. However, if the @code{]} is escaped
|
|
with a backslash it is interpreted as the end of range, so
|
|
@code{[W-\]46]} is interpreted as a single class containing a
|
|
range followed by two separate characters. The octal or
|
|
hexadecimal representation of @code{]} can also be used to end a range.
|
|
|
|
Ranges operate in @sc{ascii} collating sequence. They can also be
|
|
used for characters specified numerically, for example
|
|
@code{[\000-\037]}. If a range that includes letters is used when
|
|
caseless matching is set, it matches the letters in either
|
|
case. For example, a caseless @code{[W-c]} is equivalent to
|
|
@code{[][\^_`wxyzabc]}, matched caselessly, and if character
|
|
tables for the French locale are in use, @code{[\xc8-\xcb]}
|
|
matches accented E characters in both cases.
|
|
|
|
Unlike in @sc{posix} mode, the character types @code{\d},
|
|
@code{\D}, @code{\s}, @code{\S}, @code{\w}, and @code{\W}
|
|
may also appear in a character class, and add the characters
|
|
that they match to the class. For example, @code{[\dABCDEF]} matches any
|
|
hexadecimal digit. A circumflex can conveniently be used
|
|
with the upper case character types to specify a more restricted
|
|
set of characters than the matching lower case type.
|
|
For example, the class @code{[^\W_]} matches any letter or digit,
|
|
but not underscore.
|
|
|
|
All non-alphameric characters other than @code{\}, @code{-},
|
|
@code{^} (at the start) and the terminating @code{]}
|
|
are non-special in character classes, but it does no harm
|
|
if they are escaped.
|
|
|
|
Perl 5.6 supports the @sc{posix} notation for character classes, which
|
|
uses names enclosed by @code{[:} and @code{:]} within the enclosing
|
|
square brackets, and @value{SSED} supports this notation as well.
|
|
For example,
|
|
|
|
@example
|
|
[01[:alpha:]%]
|
|
@end example
|
|
|
|
@noindent
|
|
matches @samp{0}, @samp{1}, any alphabetic character, or @samp{%}.
|
|
The supported class names are
|
|
|
|
@table @code
|
|
@item alnum
|
|
Matches letters and digits
|
|
|
|
@item alpha
|
|
Matches letters
|
|
|
|
@item ascii
|
|
Matches character codes 0 - 127
|
|
|
|
@item cntrl
|
|
Matches control characters
|
|
|
|
@item digit
|
|
Matches decimal digits (same as \d)
|
|
|
|
@item graph
|
|
Matches printing characters, excluding space
|
|
|
|
@item lower
|
|
Matches lower case letters
|
|
|
|
@item print
|
|
Matches printing characters, including space
|
|
|
|
@item punct
|
|
Matches printing characters, excluding letters and digits
|
|
|
|
@item space
|
|
Matches white space (same as \s)
|
|
|
|
@item upper
|
|
Matches upper case letters
|
|
|
|
@item word
|
|
Matches ``word'' characters (same as \w)
|
|
|
|
@item xdigit
|
|
Matches hexadecimal digits
|
|
@end table
|
|
|
|
The names @code{ascii} and @code{word} are extensions valid only in
|
|
Perl mode. Another Perl extension is negation, which is
|
|
indicated by a circumflex character after the colon. For example,
|
|
|
|
@example
|
|
[12[:^digit:]]
|
|
@end example
|
|
|
|
@noindent
|
|
matches @samp{1}, @samp{2}, or any non-digit.
|
|
|
|
@node Options setting
|
|
@appendixsec Options setting
|
|
@cindex Perl-style regular expressions, toggling options
|
|
@cindex Perl-style regular expressions, case-insensitive
|
|
@cindex Perl-style regular expressions, multiline
|
|
@cindex Perl-style regular expressions, single line
|
|
@cindex Perl-style regular expressions, extended
|
|
|
|
The settings of the @code{I}, @code{M}, @code{S}, @code{X}
|
|
modifiers can be changed from within the pattern by
|
|
a sequence of Perl option letters enclosed between @code{(?}
|
|
and @code{)}. The option letters must be lowercase.
|
|
|
|
For example, @code{(?im)} sets caseless, multiline matching. It is
|
|
also possible to unset these options by preceding the letter
|
|
with a hyphen; you can also have combined settings and unsettings:
|
|
@code{(?im-sx)} sets caseless and multiline matching,
|
|
while unsets single line matching (for dots) and extended
|
|
whitespace interpretation. If a letter appears both before
|
|
and after the hyphen, the option is unset.
|
|
|
|
The scope of these option changes depends on where in the
|
|
pattern the setting occurs. For settings that are outside
|
|
any subpattern (defined below), the effect is the same as if
|
|
the options were set or unset at the start of matching. The
|
|
following patterns all behave in exactly the same way:
|
|
|
|
@example
|
|
(?i)abc
|
|
a(?i)bc
|
|
ab(?i)c
|
|
abc(?i)
|
|
@end example
|
|
|
|
which in turn is the same as specifying the pattern abc with
|
|
the @code{I} modifier. In other words, ``top level'' settings
|
|
apply to the whole pattern (unless there are other
|
|
changes inside subpatterns). If there is more than one setting
|
|
of the same option at top level, the rightmost setting
|
|
is used.
|
|
|
|
If an option change occurs inside a subpattern, the effect
|
|
is different. This is a change of behaviour in Perl 5.005.
|
|
An option change inside a subpattern affects only that part
|
|
of the subpattern @emph{that follows} it, so
|
|
|
|
@example
|
|
(a(?i)b)c
|
|
@end example
|
|
|
|
@noindent
|
|
matches abc and aBc and no other strings (assuming
|
|
case-sensitive matching is used). By this means, options can
|
|
be made to have different settings in different parts of the
|
|
pattern. Any changes made in one alternative do carry on
|
|
into subsequent branches within the same subpattern. For
|
|
example,
|
|
|
|
@example
|
|
(a(?i)b|c)
|
|
@end example
|
|
|
|
@noindent
|
|
matches @samp{ab}, @samp{aB}, @samp{c}, and @samp{C},
|
|
even though when matching @samp{C} the first branch is
|
|
abandoned before the option setting.
|
|
This is because the effects of option settings happen at
|
|
compile time. There would be some very weird behaviour otherwise.
|
|
|
|
@ignore
|
|
There are two PCRE-specific options PCRE_UNGREEDY and PCRE_EXTRA
|
|
that can be changed in the same way as the Perl-compatible options by
|
|
using the characters U and X respectively. The (?X) flag
|
|
setting is special in that it must always occur earlier in
|
|
the pattern than any of the additional features it turns on,
|
|
even when it is at top level. It is best put at the start.
|
|
@end ignore
|
|
|
|
|
|
@node Non-capturing subpatterns
|
|
@appendixsec Non-capturing subpatterns
|
|
@cindex Perl-style regular expressions, non-capturing subpatterns
|
|
|
|
Marking part of a pattern as a subpattern does two things.
|
|
On one hand, it localizes a set of alternatives; on the other
|
|
hand, it sets up the subpattern as a capturing subpattern (as
|
|
defined above). The subpattern can be backreferenced and
|
|
referenced in the right side of @code{s} commands.
|
|
|
|
For example, if the string @samp{the red king} is matched against
|
|
the pattern
|
|
|
|
@example
|
|
the ((red|white) (king|queen))
|
|
@end example
|
|
|
|
@noindent
|
|
the captured substrings are @samp{red king}, @samp{red},
|
|
and @samp{king}, and are numbered 1, 2, and 3.
|
|
|
|
The fact that plain parentheses fulfil two functions is not
|
|
always helpful. There are often times when a grouping
|
|
subpattern is required without a capturing requirement. If an
|
|
opening parenthesis is followed by @code{?:}, the subpattern does
|
|
not do any capturing, and is not counted when computing the
|
|
number of any subsequent capturing subpatterns. For example,
|
|
if the string @samp{the white queen} is matched against the pattern
|
|
|
|
@example
|
|
the ((?:red|white) (king|queen))
|
|
@end example
|
|
|
|
@noindent
|
|
the captured substrings are @samp{white queen} and @samp{queen},
|
|
and are numbered 1 and 2. The maximum number of captured
|
|
substrings is 99, while the maximum number of all subpatterns,
|
|
both capturing and non-capturing, is 200.
|
|
|
|
As a convenient shorthand, if any option settings are
|
|
equired at the start of a non-capturing subpattern, the
|
|
option letters may appear between the @code{?} and the
|
|
@code{:}. Thus the two patterns
|
|
|
|
@example
|
|
(?i:saturday|sunday)
|
|
(?:(?i)saturday|sunday)
|
|
@end example
|
|
|
|
@noindent
|
|
match exactly the same set of strings. Because alternative
|
|
branches are tried from left to right, and options are not
|
|
reset until the end of the subpattern is reached, an option
|
|
setting in one branch does affect subsequent branches, so
|
|
the above patterns match @samp{SUNDAY} as well as @samp{Saturday}.
|
|
|
|
|
|
@node Repetition
|
|
@appendixsec Repetition
|
|
@cindex Perl-style regular expressions, repetitions
|
|
|
|
Repetition is specified by quantifiers, which can follow any
|
|
of the following items:
|
|
|
|
@itemize @bullet
|
|
@item
|
|
a single character, possibly escaped
|
|
|
|
@item
|
|
the @code{.} special character
|
|
|
|
@item
|
|
a character class
|
|
|
|
@item
|
|
a back reference (see next section)
|
|
|
|
@item
|
|
a parenthesized subpattern (unless it is an assertion; @pxref{Assertions})
|
|
@end itemize
|
|
|
|
The general repetition quantifier specifies a minimum and
|
|
maximum number of permitted matches, by giving the two
|
|
numbers in curly brackets (braces), separated by a comma.
|
|
The numbers must be less than 65536, and the first must be
|
|
less than or equal to the second. For example:
|
|
|
|
@example
|
|
z@{2,4@}
|
|
@end example
|
|
|
|
@noindent
|
|
matches @samp{zz}, @samp{zzz}, or @samp{zzzz}. A closing brace on its own
|
|
is not a special character. If the second number is omitted,
|
|
but the comma is present, there is no upper limit; if the
|
|
second number and the comma are both omitted, the quantifier
|
|
specifies an exact number of required matches. Thus
|
|
|
|
@example
|
|
[aeiou]@{3,@}
|
|
@end example
|
|
|
|
@noindent
|
|
matches at least 3 successive vowels, but may match many
|
|
more, while
|
|
|
|
@example
|
|
\d@{8@}
|
|
@end example
|
|
|
|
@noindent
|
|
matches exactly 8 digits. An opening curly bracket that
|
|
appears in a position where a quantifier is not allowed, or
|
|
one that does not match the syntax of a quantifier, is taken
|
|
as a literal character. For example, @{,6@} is not a quantifier,
|
|
but a literal string of four characters.@footnote{It
|
|
raises an error if @option{-R} is not used.}
|
|
|
|
The quantifier @samp{@{0@}} is permitted, causing the expression to
|
|
behave as if the previous item and the quantifier were not
|
|
present.
|
|
|
|
For convenience (and historical compatibility) the three
|
|
most common quantifiers have single-character abbreviations:
|
|
|
|
@table @code
|
|
@item *
|
|
is equivalent to @{0,@}
|
|
|
|
@item +
|
|
is equivalent to @{1,@}
|
|
|
|
@item ?
|
|
is equivalent to @{0,1@}
|
|
@end table
|
|
|
|
It is possible to construct infinite loops by following a
|
|
subpattern that can match no characters with a quantifier
|
|
that has no upper limit, for example:
|
|
|
|
@example
|
|
(a?)*
|
|
@end example
|
|
|
|
Earlier versions of Perl used to give an error at
|
|
compile time for such patterns. However, because there are
|
|
cases where this can be useful, such patterns are now
|
|
accepted, but if any repetition of the subpattern does in
|
|
fact match no characters, the loop is forcibly broken.
|
|
|
|
@cindex Greedy regular expression matching
|
|
@cindex Perl-style regular expressions, stingy repetitions
|
|
By default, the quantifiers are @dfn{greedy} like in @sc{posix}
|
|
mode, that is, they match as much as possible (up to the maximum
|
|
number of permitted times), without causing the rest of the
|
|
pattern to fail. The classic example of where this gives problems
|
|
is in trying to match comments in C programs. These appear between
|
|
the sequences @code{/*} and @code{*/} and within the sequence, individual
|
|
@code{*} and @code{/} characters may appear. An attempt to match C
|
|
comments by applying the pattern
|
|
|
|
@example
|
|
/\*.*\*/
|
|
@end example
|
|
|
|
@noindent
|
|
to the string
|
|
|
|
@example
|
|
/* first command */ not comment /* second comment */
|
|
@end example
|
|
|
|
@noindent
|
|
|
|
fails, because it matches the entire string owing to the
|
|
greediness of the @code{.*} item.
|
|
|
|
However, if a quantifier is followed by a question mark, it
|
|
ceases to be greedy, and instead matches the minimum number
|
|
of times possible, so the pattern @code{/\*.*?\*/}
|
|
does the right thing with the C comments. The meaning of the
|
|
various quantifiers is not otherwise changed, just the preferred
|
|
number of matches. Do not confuse this use of question
|
|
mark with its use as a quantifier in its own right.
|
|
Because it has two uses, it can sometimes appear doubled, as in
|
|
|
|
@example
|
|
\d??\d
|
|
@end example
|
|
|
|
which matches one digit by preference, but can match two if
|
|
that is the only way the rest of the pattern matches.
|
|
|
|
Note that greediness does not matter when specifying addresses,
|
|
but can be nevertheless used to improve performance.
|
|
|
|
@ignore
|
|
If the PCRE_UNGREEDY option is set (an option which is not
|
|
available in Perl), the quantifiers are not greedy by
|
|
default, but individual ones can be made greedy by following
|
|
them with a question mark. In other words, it inverts the
|
|
default behaviour.
|
|
@end ignore
|
|
|
|
When a parenthesized subpattern is quantified with a minimum
|
|
repeat count that is greater than 1 or with a limited maximum,
|
|
more store is required for the compiled pattern, in
|
|
proportion to the size of the minimum or maximum.
|
|
|
|
@cindex Perl-style regular expressions, single line
|
|
If a pattern starts with @code{.*} or @code{.@{0,@}} and the
|
|
@code{S} modifier is used, the pattern is implicitly anchored,
|
|
because whatever follows will be tried against every character
|
|
position in the subject string, so there is no point in
|
|
retrying the overall match at any position after the first.
|
|
PCRE treats such a pattern as though it were preceded by \A.
|
|
|
|
When a capturing subpattern is repeated, the value captured
|
|
is the substring that matched the final iteration. For example,
|
|
after
|
|
|
|
@example
|
|
(tweedle[dume]@{3@}\s*)+
|
|
@end example
|
|
|
|
@noindent
|
|
has matched @samp{tweedledum tweedledee} the value of the
|
|
captured substring is @samp{tweedledee}. However, if there are
|
|
nested capturing subpatterns, the corresponding captured
|
|
values may have been set in previous iterations. For example,
|
|
after
|
|
|
|
@example
|
|
/(a|(b))+/
|
|
@end example
|
|
|
|
matches @samp{aba}, the value of the second captured substring is
|
|
@samp{b}.
|
|
|
|
@node Backreferences
|
|
@appendixsec Backreferences
|
|
@cindex Perl-style regular expressions, backreferences
|
|
|
|
Outside a character class, a backslash followed by a digit
|
|
greater than 0 (and possibly further digits) is a back
|
|
reference to a capturing subpattern earlier (i.e. to its
|
|
left) in the pattern, provided there have been that many
|
|
previous capturing left parentheses.
|
|
|
|
However, if the decimal number following the backslash is
|
|
less than 10, it is always taken as a back reference, and
|
|
causes an error only if there are not that many capturing
|
|
left parentheses in the entire pattern. In other words, the
|
|
parentheses that are referenced need not be to the left of
|
|
the reference for numbers less than 10. @ref{Backslash}
|
|
for further details of the handling of digits following a backslash.
|
|
|
|
A back reference matches whatever actually matched the capturing
|
|
subpattern in the current subject string, rather than
|
|
anything matching the subpattern itself. So the pattern
|
|
|
|
@example
|
|
(sens|respons)e and \1ibility
|
|
@end example
|
|
|
|
@noindent
|
|
matches @samp{sense and sensibility} and @samp{response and responsibility},
|
|
but not @samp{sense and responsibility}. If caseful
|
|
matching is in force at the time of the back reference, the
|
|
case of letters is relevant. For example,
|
|
|
|
@example
|
|
((?i)blah)\s+\1
|
|
@end example
|
|
|
|
@noindent
|
|
matches @samp{blah blah} and @samp{Blah Blah}, but not
|
|
@samp{BLAH blah}, even though the original capturing
|
|
subpattern is matched caselessly.
|
|
|
|
There may be more than one back reference to the same subpattern.
|
|
Also, if a subpattern has not actually been used in a
|
|
particular match, any back references to it always fail. For
|
|
example, the pattern
|
|
|
|
@example
|
|
(a|(bc))\2
|
|
@end example
|
|
|
|
@noindent
|
|
always fails if it starts to match @samp{a} rather than
|
|
@samp{bc}. Because there may be up to 99 back references, all
|
|
digits following the backslash are taken as part of a potential
|
|
back reference number; this is different from what happens
|
|
in @sc{posix} mode. If the pattern continues with a digit
|
|
character, some delimiter must be used to terminate the back
|
|
reference. If the @code{X} modifier option is set, this can be
|
|
whitespace. Otherwise an empty comment can be used, or the
|
|
following character can be expressed in hexadecimal or octal.
|
|
Note that this applies only to the LHS pattern; it is
|
|
not possible yet to specify more than 9 backreferences on the
|
|
RHS of the `s' command.
|
|
|
|
A back reference that occurs inside the parentheses to which
|
|
it refers fails when the subpattern is first used, so, for
|
|
example, @code{(a\1)} never matches. However, such references
|
|
can be useful inside repeated subpatterns. For example, the
|
|
pattern
|
|
|
|
@example
|
|
(a|b\1)+
|
|
@end example
|
|
|
|
@noindent
|
|
matches any number of @samp{a}s and also @samp{aba}, @samp{ababbaa},
|
|
etc. At each iteration of the subpattern, the back reference matches
|
|
the character string corresponding to the previous iteration. In
|
|
order for this to work, the pattern must be such that the first
|
|
iteration does not need to match the back reference. This can be
|
|
done using alternation, as in the example above, or by a
|
|
quantifier with a minimum of zero.
|
|
|
|
@node Assertions
|
|
@appendixsec Assertions
|
|
@cindex Perl-style regular expressions, assertions
|
|
@cindex Perl-style regular expressions, asserting subpatterns
|
|
|
|
An assertion is a test on the characters following or
|
|
preceding the current matching point that does not actually
|
|
consume any characters. The simple assertions coded as @code{\b},
|
|
@code{\B}, @code{\A}, @code{\Z}, @code{\z}, @code{^} and @code{$}
|
|
are described above. More complicated assertions are coded as
|
|
subpatterns. There are two kinds: those that look ahead of the
|
|
current position in the subject string, and those that look behind it.
|
|
|
|
@cindex Perl-style regular expressions, lookahead subpatterns
|
|
An assertion subpattern is matched in the normal way, except
|
|
that it does not cause the current matching position to be
|
|
changed. Lookahead assertions start with @code{(?=} for positive
|
|
assertions and @code{(?!} for negative assertions. For example,
|
|
|
|
@example
|
|
\w+(?=;)
|
|
@end example
|
|
|
|
@noindent
|
|
matches a word followed by a semicolon, but does not include
|
|
the semicolon in the match, and
|
|
|
|
@example
|
|
foo(?!bar)
|
|
@end example
|
|
|
|
@noindent
|
|
matches any occurrence of @samp{foo} that is not followed by
|
|
@samp{bar}.
|
|
|
|
Note that the apparently similar pattern
|
|
|
|
@example
|
|
(?!foo)bar
|
|
@end example
|
|
|
|
@noindent
|
|
@cindex Perl-style regular expressions, lookbehind subpatterns
|
|
finds any occurrence of @samp{bar} even if it is preceded by
|
|
@samp{foo}, because the assertion @code{(?!foo)} is always true
|
|
when the next three characters are @samp{bar}. A lookbehind
|
|
assertion is needed to achieve this effect.
|
|
Lookbehind assertions start with @code{(?<=} for positive
|
|
assertions and @code{(?<!} for negative assertions. So,
|
|
|
|
@example
|
|
(?<!foo)bar
|
|
@end example
|
|
|
|
achieves the required effect of finding an occurrence of
|
|
@samp{bar} that is not preceded by @samp{foo}. The contents of a
|
|
lookbehind assertion are restricted
|
|
such that all the strings it matches must have a fixed
|
|
length. However, if there are several alternatives, they do
|
|
not all have to have the same fixed length. This is an extension
|
|
compared with Perl 5.005, which requires all branches to match
|
|
the same length of string. Thus
|
|
|
|
@example
|
|
(?<=dogs|cats|)
|
|
@end example
|
|
|
|
@noindent
|
|
is permitted, but the apparently equivalent regular expression
|
|
|
|
@example
|
|
(?<!dogs?|cats?)
|
|
@end example
|
|
|
|
@noindent
|
|
causes an error at compile time. Branches that match different
|
|
length strings are permitted only at the top level of
|
|
a lookbehind assertion: an assertion such as
|
|
|
|
@example
|
|
(?<=ab(c|de))
|
|
@end example
|
|
|
|
@noindent
|
|
is not permitted, because its single top-level branch can
|
|
match two different lengths, but it is acceptable if rewritten
|
|
to use two top-level branches:
|
|
|
|
@example
|
|
(?<=abc|abde)
|
|
@end example
|
|
|
|
All this is required because lookbehind assertions simply
|
|
move the current position back by the alternative's fixed
|
|
width and then try to match. If there are
|
|
insufficient characters before the current position, the
|
|
match is deemed to fail. Lookbehinds, in conjunction with
|
|
non-backtracking subpatterns can be particularly useful for
|
|
matching at the ends of strings; an example is given at the end
|
|
of the section on non-backtracking subpatterns.
|
|
|
|
Several assertions (of any sort) may occur in succession.
|
|
For example,
|
|
|
|
@example
|
|
(?<=\d@{3@})(?<!999)foo
|
|
@end example
|
|
|
|
@noindent
|
|
matches @samp{foo} preceded by three digits that are not @samp{999}.
|
|
Notice that each of the assertions is applied independently
|
|
at the same point in the subject string. First there is a
|
|
check that the previous three characters are all digits, and
|
|
then there is a check that the same three characters are not
|
|
@samp{999}. This pattern does not match @samp{foo} preceded by six
|
|
characters, the first of which are digits and the last three
|
|
of which are not @samp{999}. For example, it doesn't match
|
|
@samp{123abcfoo}. A pattern to do that is
|
|
|
|
@example
|
|
(?<=\d@{3@}...)(?<!999)foo
|
|
@end example
|
|
|
|
@noindent
|
|
This time the first assertion looks at the preceding six
|
|
characters, checking that the first three are digits, and
|
|
then the second assertion checks that the preceding three
|
|
characters are not @samp{999}. Actually, assertions can be
|
|
nested in any combination, so one can write this as
|
|
|
|
@example
|
|
(?<=\d@{3@}(?!999)...)foo
|
|
@end example
|
|
|
|
or
|
|
|
|
@example
|
|
(?<=\d@{3@}...(?<!999))foo
|
|
@end example
|
|
|
|
@noindent
|
|
both of which might be considered more readable.
|
|
|
|
Assertion subpatterns are not capturing subpatterns, and may
|
|
not be repeated, because it makes no sense to assert the
|
|
same thing several times. If any kind of assertion contains
|
|
capturing subpatterns within it, these are counted for the
|
|
purposes of numbering the capturing subpatterns in the whole
|
|
pattern. However, substring capturing is carried out only
|
|
for positive assertions, because it does not make sense for
|
|
negative assertions.
|
|
|
|
Assertions count towards the maximum of 200 parenthesized
|
|
subpatterns.
|
|
|
|
@node Non-backtracking subpatterns
|
|
@appendixsec Non-backtracking subpatterns
|
|
@cindex Perl-style regular expressions, non-backtracking subpatterns
|
|
|
|
With both maximizing and minimizing repetition, failure of
|
|
what follows normally causes the repeated item to be evaluated
|
|
again to see if a different number of repeats allows the
|
|
rest of the pattern to match. Sometimes it is useful to
|
|
prevent this, either to change the nature of the match, or
|
|
to cause it fail earlier than it otherwise might, when the
|
|
author of the pattern knows there is no point in carrying
|
|
on.
|
|
|
|
Consider, for example, the pattern @code{\d+foo} when applied to
|
|
the subject line
|
|
|
|
@example
|
|
123456bar
|
|
@end example
|
|
|
|
After matching all 6 digits and then failing to match @samp{foo},
|
|
the normal action of the matcher is to try again with only 5
|
|
digits matching the @code{\d+} item, and then with 4, and so on,
|
|
before ultimately failing. Non-backtracking subpatterns
|
|
provide the means for specifying that once a portion of the
|
|
pattern has matched, it is not to be re-evaluated in this way,
|
|
so the matcher would give up immediately on failing to match
|
|
@samp{foo} the first time. The notation is another kind of special
|
|
parenthesis, starting with @code{(?>} as in this example:
|
|
|
|
@example
|
|
(?>\d+)bar
|
|
@end example
|
|
|
|
This kind of parenthesis ``locks up'' the part of the pattern
|
|
it contains once it has matched, and a failure further into
|
|
the pattern is prevented from backtracking into it.
|
|
Backtracking past it to previous items, however, works as
|
|
normal.
|
|
|
|
Non-backtracking subpatterns are not capturing subpatterns. Simple
|
|
cases such as the above example can be thought of as a maximizing
|
|
repeat that must swallow everything it can. So,
|
|
while both @code{\d+} and @code{\d+?} are prepared to adjust the number of
|
|
digits they match in order to make the rest of the pattern
|
|
match, @code{(?>\d+)} can only match an entire sequence of digits.
|
|
|
|
This construction can of course contain arbitrarily complicated
|
|
subpatterns, and it can be nested.
|
|
|
|
@cindex Perl-style regular expressions, lookbehind subpatterns
|
|
Non-backtracking subpatterns can be used in conjunction with look-behind
|
|
assertions to specify efficient matching at the end
|
|
of the subject string. Consider a simple pattern such as
|
|
|
|
@example
|
|
abcd$
|
|
@end example
|
|
|
|
@noindent
|
|
when applied to a long string which does not match. Because
|
|
matching proceeds from left to right, @command{sed} will look for
|
|
each @samp{a} in the subject and then see if what follows matches
|
|
the rest of the pattern. If the pattern is specified as
|
|
|
|
@example
|
|
^.*abcd$
|
|
@end example
|
|
|
|
@noindent
|
|
the initial @code{.*} matches the entire string at first, but when
|
|
this fails (because there is no following @samp{a}), it backtracks
|
|
to match all but the last character, then all but the
|
|
last two characters, and so on. Once again the search for
|
|
@samp{a} covers the entire string, from right to left, so we are
|
|
no better off. However, if the pattern is written as
|
|
|
|
@example
|
|
^(?>.*)(?<=abcd)
|
|
@end example
|
|
|
|
there can be no backtracking for the .* item; it can match
|
|
only the entire string. The subsequent lookbehind assertion
|
|
does a single test on the last four characters. If it fails,
|
|
the match fails immediately. For long strings, this approach
|
|
makes a significant difference to the processing time.
|
|
|
|
When a pattern contains an unlimited repeat inside a subpattern
|
|
that can itself be repeated an unlimited number of
|
|
times, the use of a once-only subpattern is the only way to
|
|
avoid some failing matches taking a very long time
|
|
indeed.@footnote{Actually, the matcher embedded in @value{SSED}
|
|
tries to do something for this in the simplest cases,
|
|
like @code{([^b]*b)*}. These cases are actually quite
|
|
common: they happen for example in a regular expression
|
|
like @code{\/\*([^*]*\*)*\/} which matches C comments.}
|
|
|
|
The pattern
|
|
|
|
@example
|
|
(\D+|<\d+>)*[!?]
|
|
@end example
|
|
|
|
([^0-9<]+<(\d+>)?)*[!?]
|
|
|
|
@noindent
|
|
matches an unlimited number of substrings that either consist
|
|
of non-digits, or digits enclosed in angular brackets, followed by
|
|
an exclamation or question mark. When it matches, it runs quickly.
|
|
However, if it is applied to
|
|
|
|
@example
|
|
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
|
|
@end example
|
|
|
|
@noindent
|
|
it takes a long time before reporting failure. This is
|
|
because the string can be divided between the two repeats in
|
|
a large number of ways, and all have to be tried.@footnote{The
|
|
example used @code{[!?]} rather than a single character at the end,
|
|
because both @value{SSED} and Perl have an optimization that allows
|
|
for fast failure when a single character is used. They
|
|
remember the last single character that is required for a
|
|
match, and fail early if it is not present in the string.}
|
|
|
|
If the pattern is changed to
|
|
|
|
@example
|
|
((?>\D+)|<\d+>)*[!?]
|
|
@end example
|
|
|
|
sequences of non-digits cannot be broken, and failure happens
|
|
quickly.
|
|
|
|
@node Conditional subpatterns
|
|
@appendixsec Conditional subpatterns
|
|
@cindex Perl-style regular expressions, conditional subpatterns
|
|
|
|
It is possible to cause the matching process to obey a subpattern
|
|
conditionally or to choose between two alternative
|
|
subpatterns, depending on the result of an assertion, or
|
|
whether a previous capturing subpattern matched or not. The
|
|
two possible forms of conditional subpattern are
|
|
|
|
@example
|
|
(?(@var{condition})@var{yes-pattern})
|
|
(?(@var{condition})@var{yes-pattern}|@var{no-pattern})
|
|
@end example
|
|
|
|
If the condition is satisfied, the yes-pattern is used; otherwise
|
|
the no-pattern (if present) is used. If there are more than two
|
|
alternatives in the subpattern, a compile-time error occurs.
|
|
|
|
There are two kinds of condition. If the text between the
|
|
parentheses consists of a sequence of digits, the condition
|
|
is satisfied if the capturing subpattern of that number has
|
|
previously matched. The number must be greater than zero.
|
|
Consider the following pattern, which contains non-significant
|
|
white space to make it more readable (assume the @code{X} modifier)
|
|
and to divide it into three parts for ease of discussion:
|
|
|
|
@example
|
|
( \( )? [^()]+ (?(1) \) )
|
|
@end example
|
|
|
|
The first part matches an optional opening parenthesis, and
|
|
if that character is present, sets it as the first captured
|
|
substring. The second part matches one or more characters
|
|
that are not parentheses. The third part is a conditional
|
|
subpattern that tests whether the first set of parentheses
|
|
matched or not. If they did, that is, if subject started
|
|
with an opening parenthesis, the condition is true, and so
|
|
the yes-pattern is executed and a closing parenthesis is
|
|
required. Otherwise, since no-pattern is not present, the
|
|
subpattern matches nothing. In other words, this pattern
|
|
matches a sequence of non-parentheses, optionally enclosed
|
|
in parentheses.
|
|
|
|
@cindex Perl-style regular expressions, lookahead subpatterns
|
|
If the condition is not a sequence of digits, it must be an
|
|
assertion. This may be a positive or negative lookahead or
|
|
lookbehind assertion. Consider this pattern, again containing
|
|
non-significant white space, and with the two alternatives
|
|
on the second line:
|
|
|
|
@example
|
|
(?(?=...[a-z])
|
|
\d\d-[a-z]@{3@}-\d\d |
|
|
\d\d-\d\d-\d\d )
|
|
@end example
|
|
|
|
The condition is a positive lookahead assertion that matches
|
|
a letter that is three characters away from the current point.
|
|
If a letter is found, the subject is matched against the first
|
|
alternative @samp{@var{dd}-@var{aaa}-@var{dd}} (where @var{aaa} are
|
|
letters and @var{dd} are digits); otherwise it is matched against
|
|
the second alternative, @samp{@var{dd}-@var{dd}-@var{dd}}.
|
|
|
|
|
|
@node Recursive patterns
|
|
@appendixsec Recursive patterns
|
|
@cindex Perl-style regular expressions, recursive patterns
|
|
@cindex Perl-style regular expressions, recursion
|
|
|
|
Consider the problem of matching a string in parentheses,
|
|
allowing for unlimited nested parentheses. Without the use
|
|
of recursion, the best that can be done is to use a pattern
|
|
that matches up to some fixed depth of nesting. It is not
|
|
possible to handle an arbitrary nesting depth. Perl 5.6 has
|
|
provided an experimental facility that allows regular
|
|
expressions to recurse (amongst other things). It does this
|
|
by interpolating Perl code in the expression at run time,
|
|
and the code can refer to the expression itself. A Perl pattern
|
|
tern to solve the parentheses problem can be created like
|
|
this:
|
|
|
|
@example
|
|
$re = qr@{\( (?: (?>[^()]+) | (?p@{$re@}) )* \)@}x;
|
|
@end example
|
|
|
|
The @code{(?p@{...@})} item interpolates Perl code at run time,
|
|
and in this case refers recursively to the pattern in which it
|
|
appears. Obviously, @command{sed} cannot support the interpolation of
|
|
Perl code. Instead, the special item @code{(?R)} is provided for
|
|
the specific case of recursion. This pattern solves the
|
|
parentheses problem (assume the @code{X} modifier option is used
|
|
so that white space is ignored):
|
|
|
|
@example
|
|
\( ( (?>[^()]+) | (?R) )* \)
|
|
@end example
|
|
|
|
First it matches an opening parenthesis. Then it matches any
|
|
number of substrings which can either be a sequence of
|
|
non-parentheses, or a recursive match of the pattern itself
|
|
(i.e. a correctly parenthesized substring). Finally there is
|
|
a closing parenthesis.
|
|
|
|
This particular example pattern contains nested unlimited
|
|
repeats, and so the use of a non-backtracking subpattern for
|
|
matching strings of non-parentheses is important when applying
|
|
the pattern to strings that do not match. For example, when
|
|
it is applied to
|
|
|
|
@example
|
|
(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
|
|
@end example
|
|
|
|
it yields a ``no match'' response quickly. However, if a
|
|
standard backtracking subpattern is not used, the match runs
|
|
for a very long time indeed because there are so many different
|
|
ways the @code{+} and @code{*} repeats can carve up the subject,
|
|
and all have to be tested before failure can be reported.
|
|
|
|
The values set for any capturing subpatterns are those from
|
|
the outermost level of the recursion at which the subpattern
|
|
value is set. If the pattern above is matched against
|
|
|
|
@example
|
|
(ab(cd)ef)
|
|
@end example
|
|
|
|
@noindent
|
|
the value for the capturing parentheses is @samp{ef}, which is
|
|
the last value taken on at the top level.
|
|
|
|
@node Comments
|
|
@appendixsec Comments
|
|
@cindex Perl-style regular expressions, comments
|
|
|
|
The sequence (?# marks the start of a comment which continues
|
|
ues up to the next closing parenthesis. Nested parentheses
|
|
are not permitted. The characters that make up a comment
|
|
play no part in the pattern matching at all.
|
|
|
|
@cindex Perl-style regular expressions, extended
|
|
If the @code{X} modifier option is used, an unescaped @code{#} character
|
|
outside a character class introduces a comment that continues
|
|
up to the next newline character in the pattern.
|
|
@end ifset
|
|
|
|
|
|
@page
|
|
@node Concept Index
|
|
@unnumbered Concept Index
|
|
|
|
This is a general index of all issues discussed in this manual, with the
|
|
exception of the @command{sed} commands and command-line options.
|
|
|
|
@printindex cp
|
|
|
|
@page
|
|
@node Command and Option Index
|
|
@unnumbered Command and Option Index
|
|
|
|
This is an alphabetical list of all @command{sed} commands and command-line
|
|
options.
|
|
|
|
@printindex fn
|
|
|
|
@contents
|
|
@bye
|
|
|
|
@c XXX FIXME: the term "cycle" is never defined...
|