Introduction

The command name sed is derived from stream editor. Here, stream refers to data being passed via shell pipes. Thus, the command's primary functionality is to act as a text editor for stdin data with stdout as the output target. It is now common to use sed on file inputs as well as in-place file editing.

This chapter will show how to install the latest sed version followed by details related to documentation. Then, you'll get an introduction to the substitute command, which is the most commonly used feature. The chapters to follow will add more details to the substitute command, discuss other commands and command line options. Cheatsheet, summary and exercises are also included at the end of these chapters.

Installation

If you are on a Unix-like system, you will most likely have some version of sed already installed. This book is primarily about GNU sed. As there are syntax and feature differences between various implementations, make sure to use GNU sed to follow along the examples presented in this book.

GNU sed is part of the text creation and manipulation tools and comes by default on GNU/Linux distributions. To install a particular version, visit gnu: sed software. See also release notes for an overview of changes between versions and bug list if you think some command isn't working as expected.

$ wget https://ftp.gnu.org/gnu/sed/sed-4.9.tar.xz
$ tar -Jxf sed-4.9.tar.xz
$ cd sed-4.9/
# see https://askubuntu.com/q/237576 if you get compiler not found error
$ ./configure
$ make
$ sudo make install

$ sed --version | head -n1
sed (GNU sed) 4.9

If you are not using a Linux distribution, you may be able to access GNU sed using an option below:

• Git for Windows — provides a Bash emulation used to run Git from the command line
• Windows Subsystem for Linux — compatibility layer for running Linux binary executables natively on Windows
• brew — Package Manager for macOS (or Linux)

Documentation

It is always good to know where to find documentation. From the command line, you can use man sed for a short manual and info sed for the full documentation. I prefer using the online gnu sed manual, which feels much easier to use and navigate.

$ man sed
SED(1)                          User Commands                         SED(1)  

NAME
       sed - stream editor for filtering and transforming text

SYNOPSIS
       sed [-V] [--version] [--help] [-n] [--quiet] [--silent]
           [-l N] [--line-length=N] [-u] [--unbuffered]
           [-E] [-r] [--regexp-extended]
           [-e script] [--expression=script]
           [-f script-file] [--file=script-file]
           [script-if-no-other-script]
           [file...]

DESCRIPTION
       Sed  is  a  stream  editor.  A stream editor is used to perform basic
       text transformations on an input stream (a file or input from a pipe‐
       line).   While  in  some  ways  similar  to  an  editor which permits
       scripted edits (such as ed), sed works by making only one  pass  over
       the  input(s),  and  is consequently more efficient.  But it is sed's
       ability to filter text in a pipeline which particularly distinguishes
       it from other types of editors.

Options overview

For a quick overview of all the available options, use sed --help from the command line.

# only partial output shown here
$ sed --help
  -n, --quiet, --silent
                 suppress automatic printing of pattern space
      --debug
                 annotate program execution
  -e script, --expression=script
                 add the script to the commands to be executed
  -f script-file, --file=script-file
                 add the contents of script-file to the commands to be executed
  --follow-symlinks
                 follow symlinks when processing in place
  -i[SUFFIX], --in-place[=SUFFIX]
                 edit files in place (makes backup if SUFFIX supplied)
  -l N, --line-length=N
                 specify the desired line-wrap length for the 'l' command
  --posix
                 disable all GNU extensions.
  -E, -r, --regexp-extended
                 use extended regular expressions in the script
                 (for portability use POSIX -E).
  -s, --separate
                 consider files as separate rather than as a single,
                 continuous long stream.
      --sandbox
                 operate in sandbox mode (disable e/r/w commands).
  -u, --unbuffered
                 load minimal amounts of data from the input files and flush
                 the output buffers more often
  -z, --null-data
                 separate lines by NUL characters
      --help     display this help and exit
      --version  output version information and exit

If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret.  All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.

Editing standard input

sed has various commands to manipulate text. The substitute command is the most commonly used operation, which helps to replace matching portions with something else. The syntax is s/REGEXP/REPLACEMENT/FLAGS where:

• s stands for the substitute command
• / is an idiomatic delimiter character to separate various portions of the command
• REGEXP stands for regular expression
• REPLACEMENT refers to the replacement string
• FLAGS are options to change the default behavior of the command

For now, it is enough to know that the s command is used for search and replace operations.

# input data that'll be passed to sed for editing
$ printf '1,2,3,4\na,b,c,d\n'
1,2,3,4
a,b,c,d

# for each input line, change only the first ',' to '-'
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/'
1-2,3,4
a-b,c,d

# change all matches by adding the 'g' flag
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/g'
1-2-3-4
a-b-c-d

In the above example, the input data is created using the printf command to showcase stream editing. By default, sed processes the input line by line. The newline character \n is the line separator by default. The first sed command replaces only the first occurrence of , with -. The second command replaces all occurrences as the g flag is also used (g stands for global).

As a good practice, always use single quotes around the script argument. Examples requiring shell interpretation will be discussed later.

If your input file has \r\n (carriage return and newline characters) as the line ending, convert the input file to Unix-style before processing. See stackoverflow: Why does my tool output overwrite itself and how do I fix it? for a detailed discussion and mitigation methods.

# Unix style
$ printf '42\n' | file -
/dev/stdin: ASCII text

# DOS style
$ printf '42\r\n' | file -
/dev/stdin: ASCII text, with CRLF line terminators

Editing file input

Although sed derives its name from stream editing, it is common to use sed for file editing. To do so, you can pass one or more input filenames as arguments. You can use - to represent stdin as one of the input sources. By default, the modified data will go to the stdout stream and the input files are not modified. In-place file editing chapter will discuss how to apply the changes back to the source files.

The example_files directory has all the files used in the examples.

$ cat greeting.txt
Hi there
Have a nice day

# for each line, change the first occurrence of 'day' with 'weekend'
$ sed 's/day/weekend/' greeting.txt
Hi there
Have a nice weekend

# change all occurrences of 'e' to 'E'
# redirect the modified data to another file
$ sed 's/e/E/g' greeting.txt > out.txt
$ cat out.txt
Hi thErE
HavE a nicE day

In the previously seen examples, every input line had matched the search expression. The first sed command here searched for day, which did not match the first line of greeting.txt file. By default, even if a line doesn't satisfy the search expression, it will be part of the output. You'll see how to get only the modified lines in the Print command section.

Cheatsheet and summary

This introductory chapter covered installation process, documentation and how to search and replace text using sed from the command line. In the coming chapters, you'll learn many more commands and features that make sed an important tool when it comes to command line text processing. One such feature is editing files in-place, which will be discussed in the next chapter.

Interactive exercises

I wrote a TUI app to help you solve some of the exercises from this book interactively. See SedExercises repo for installation steps and app_guide.md for instructions on using this app.

Here's a sample screenshot:

Exercises

All the exercises are also collated together in one place at Exercises.md. For solutions, see Exercise_solutions.md.

The exercises directory has all the files used in this section.

1) Replace only the first occurrence of 5 with five for the given stdin source.

$ echo 'They ate 5 apples and 5 mangoes' | sed ##### add your solution here
They ate five apples and 5 mangoes

2) Replace all occurrences of 5 with five.

$ echo 'They ate 5 apples and 5 mangoes' | sed ##### add your solution here
They ate five apples and five mangoes

3) Replace all occurrences of 0xA0 with 0x50 and 0xFF with 0x7F for the given input file.

$ cat hex.txt
start address: 0xA0, func1 address: 0xA0
end address: 0xFF, func2 address: 0xB0

$ sed ##### add your solution here
start address: 0x50, func1 address: 0x50
end address: 0x7F, func2 address: 0xB0

4) The substitute command searches and replaces sequences of characters. When you need to map one or more characters with another set of corresponding characters, you can use the y command. Quoting from the manual:

y/src/dst/ Transliterate any characters in the pattern space which match any of the source-chars with the corresponding character in dest-chars.

Use the y command to transform the given input string to get the output string as shown below.

$ echo 'goal new user sit eat dinner' | sed ##### add your solution here
gOAl nEw UsEr sIt EAt dInnEr

5) Why does the following command produce an error? How'd you fix it?

$ echo 'a sunny day' | sed s/sunny day/cloudy day/
sed: -e expression #1, char 7: unterminated `s' command

# expected output
$ echo 'a sunny day' | sed ##### add your solution here
a cloudy day