Sed stream editor

Introduction

Sed stands for Stream EDitor. In traditional editors such as vi or emails, the user moves the cursor around the file and interactively update the data. Stream editors, on the other hand, apply a set of commands to a stream of data read from either the standard input or file.

This page will give you a brief introduction to sed and its main use cases.

How Sed Works

This is how sed processes data:

  1. Reads next line from the input stream, and copy it to the pattern space
  2. Executes all sed commands in order,
    • Some commands will make changes to the pattern space
    • Some commands will only execute when a condition is met
  3. Writes result of pattern space to the output stream
  4. Goes back to step 1 until all lines consumed

It should be noted that sed does not change the original line.

All changes happen in the pattern space and outputted to standard output.

Sed Command-Line Syntax

This is the syntax of sed:

Options:

-nSuppresses the output.

By default sed displays all lines to the standard output. When setting the -n  flag, sed only outputs the transformed line when when using the p  command, or the p  flag with the s  command.

sed -n '/second/p' lines.data

-eAdds a script to the commands to be executed. Useful when executing more than one command

sed -e 's/milk/mojitos/' -e 's/like/love/' test.data

commandsone or several sed commands to execute
inputfileThe input file to process. If not provided, then it will use the standard input
-f scriptfileFile containing the sed commands to execute

Invoking Sed

This section contains several examples showing the basic usage of sed making use of the s  command (search and replace).

The first example reads the contents from the input stream and writes them to the output stream after replacing “milk” for “mojitos”.

You can alternatively provide an input file.

Sed can also execute multiple commands separated by semi-colon:

Or you could also provide several scripts using the -e  flag:

Alternatively, you could also separate each command into a different line in the command prompt to make it more readable:

Finally, you could also store the sed script into a file:

Sed Commands and Addresses

These are the most commonly used commands in sed. For a full list of commands, check the sed manual.

CommandDescription
sSearch and replace
yTranslates characters. For example, translate all instances of lower case a to upper case A
dDeletes a line
=Displays the line number
pPrints the line result (useful when using sed -n  to control what is printed to standard output
nSkips the current line and go to the next line
NAppends the next line to the end of the pattern space. The new line is separated by a newline

A command can have an address and arguments.  If the command does not have any address, then it is always executed. If the command has an address, then it is only executed when the address condition is met (more on addresses in the next section).

You can also specify multiple commands for an address by separating them with a semi-colon

You could also link multiple commands to an address by using curly brackets (make sure there are no spaces after the brackets):

Addresses

The address determines whether the command is to be applied to the line.  A command with no address is applied to all lines.

An address can be a line number, a range of lines, the symbol $  to indicate the last line, or a regular expression (using /slashes/ ).  If the address is followed by the ! symbol, then it applies to all lines that do not match the address.

Let’s see a few examples

AddressExampleDescription
no addresssed ‘s/milk/mojito/’ test.dataChanges all lines
1sed ‘1s/milk/mojito/’ test.dataChanges the first line only
$sed ‘$s/milk/mojito/’ test.dataChanges the last line
1,$sed ‘1,$s/milk/mojito/’ test.dataChanges all lines
2!sed ‘2!s/milk/mojito/’ test.dataChanges all lines but number 2
/drinking/sed ‘/drinking/s/milk/mojito/’ test.dataChanges all lines that match the regular expression /drinking/
/start/,/end/!sed ‘/start/,/end/!s/milk/mojito/’ test.dataChange all lines except the ones between the regular expressions /start/ and /end/

s command: Search and Replace

The  s command is the most commonly used command in sed, used to search and replace within a file.  The s command has a search and a replace section. The below  example replaces milk for mojitos.

The forward slash delimiter ( / ) is typically used to separate the search pattern from the replace part. We could however use alternative delimiters. This is particularly useful when working with path names.

The backwards slash ( \) can also be used to match special characters such as \s, \w, \n, \t. The following example uses \n  to transform each word into a separate line:

When using regular expressions, you can refer to the matched pattern using the ampersand ( &).

You could also group multiple regular expressions enclosing them in parenthesis, and then reference them in the replace section using \1 , \2 , \3,…  Note that the parenthesis needs to be escaped (e.g. \(  and \) )

By default, sed replaces only the first match:

You can however use the g  flag to replace all matches in the line:

You could alternatively specify what match number to use:

Note that sed does not support POSIX extended regular expressions (so you can’t use + , ?  or |).

y command: Translate Characters

The y command replaces any character in the search section by the character in the same position in the replace section.

The following example converts lowercase to uppercase:

d command: Delete Lines

The d command deletes all lines that match the address.

For example, let’s assume we have the following file:

The following command deletes the third line:

The following command deletes all lines between pattern /second/ and /third/:

A useful usage of the d command is to delete empty lines in a file using the /^$/  address:

p command: Print Lines

The p command prints the pattern space to the standard output.

By default the line is always printed at the end of processing. The p  command is typically used in combination with sed’s -n flag to selectively print lines to the standard output.

If we didn’t use the -n flag, then the second line would be printed twice:

= command: Print Line numbers

The = command prints out the line numbers.

If you want to output the line number in the same line as the contents, then you can use the following trick using the N multi line command (see next section):

You could also combine it with the -n flag and the p command to selectively print out lines.  The following example selects all lines between  /second/ and /third/ patterns, and prints out the line number.

n and N commands: Read Next Line

The n command reads the next line into the pattern space for processing. This command is useful for processing multiple lines at the same time.

For example, let’s assume we have a file with names and ages split in two lines, and each record separated by a dotted line:

The following multi-line sed script searches for lines with the “Name” word, prints out the name, reads the following line ( n command), and finally prints out the age:

The N command is slightly different to n command, in that it appends the next line into the pattern space.  The first and newly appended line are separated by a new line.

Bibliography

The following two tabs change content below.

Eduard Manas

Eduard is a senior IT consultant with over 15 years in the financial sector. He is an experienced developer in Java, C#, Python, Wordpress, Tibco EMS/RV, Oracle, Sybase and MySQL.Outside of work, he likes spending time with family, friends, and watching football.

Latest posts by Eduard Manas (see all)

Leave a Reply