Introduction

The command name sed is derived from stream editor. Here, stream refers to data being passed via shell pipes. Thus, the command's primary functionality is to act as a text editor for stdin data with stdout as the output target. Over the years, functionality was added to edit file input and save the changes back to the same file.

This chapter will cover how to install/upgrade sed followed by details related to documentation. Then, you'll get an introduction to the substitute command, which is the most commonly used sed feature. The chapters to follow will add more details to the substitute command, discuss other commands and command line options. Cheatsheet, summary and exercises are also included at the end of these chapters.

Installation

If you are on a Unix like system, you are most likely to already have some version of sed installed. This book is primarily for GNU sed. As there are syntax and feature differences between various implementations, please make sure to follow along with what is presented here. GNU sed is part of text creation and manipulation commands provided by GNU and comes by default on GNU/Linux. To install newer or particular version, visit gnu: software. Check release notes for an overview of changes between versions. See also bug list.

$ # use a dir, say ~/Downloads/sed_install before following the steps below
$ wget https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
$ tar -Jxf sed-4.8.tar.xz
$ cd sed-4.8/
$ ./configure 
$ make
$ sudo make install

$ type -a sed
sed is /usr/local/bin/sed
sed is /bin/sed
$ sed --version | head -n1
sed (GNU sed) 4.8

If you are not using a Linux distribution, you may be able to access GNU sed using below options:

Documentation and options overview

It is always a good idea to know where to find the documentation. From command line, you can use man sed for a short manual and info sed for full documentation. I prefer using the online GNU sed manual which feels much easier to use and navigate.

$ man sed
SED(1)                          User Commands                         SED(1)  

NAME
       sed - stream editor for filtering and transforming text

SYNOPSIS
       sed [OPTION]... {script-only-if-no-other-script} [input-file]...

DESCRIPTION
       Sed  is  a  stream  editor.  A stream editor is used to perform basic
       text transformations on an input stream (a file or input from a pipe‐
       line).   While  in  some  ways  similar  to  an  editor which permits
       scripted edits (such as ed), sed works by making only one  pass  over
       the  input(s),  and  is consequently more efficient.  But it is sed's
       ability to filter text in a pipeline which particularly distinguishes
       it from other types of editors.

For a quick overview of all the available options, use sed --help from the command line. Most of them will be explained in the coming chapters.

$ # only partial output shown here
$ sed --help
  -n, --quiet, --silent
                 suppress automatic printing of pattern space
      --debug
                 annotate program execution
  -e script, --expression=script
                 add the script to the commands to be executed
  -f script-file, --file=script-file
                 add the contents of script-file to the commands to be executed
  --follow-symlinks
                 follow symlinks when processing in place
  -i[SUFFIX], --in-place[=SUFFIX]
                 edit files in place (makes backup if SUFFIX supplied)
  -l N, --line-length=N
                 specify the desired line-wrap length for the 'l' command
  --posix
                 disable all GNU extensions.
  -E, -r, --regexp-extended
                 use extended regular expressions in the script
                 (for portability use POSIX -E).
  -s, --separate
                 consider files as separate rather than as a single,
                 continuous long stream.
      --sandbox
                 operate in sandbox mode (disable e/r/w commands).
  -u, --unbuffered
                 load minimal amounts of data from the input files and flush
                 the output buffers more often
  -z, --null-data
                 separate lines by NUL characters
      --help     display this help and exit
      --version  output version information and exit

If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret.  All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.

Editing standard input

sed has various commands to manipulate text input. substitute command is most commonly used, which will be briefly discussed in this chapter. It is used to replace matching text with something else. The syntax is s/REGEXP/REPLACEMENT/FLAGS where

s stands for substitute command
/ is an idiomatic delimiter character to separate various portions of the command
REGEXP stands for regular expression
REPLACEMENT specifies the replacement string
FLAGS are options to change default behavior of the command

For now, it is enough to know that the s command is used for search and replace operations.

$ # sample command output for stream editing
$ printf '1,2,3,4\na,b,c,d\n'
1,2,3,4
a,b,c,d

$ # for each input line, change only the first ',' to '-'
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/'
1-2,3,4
a-b,c,d

$ # change all matches by adding the 'g' flag
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/g'
1-2-3-4
a-b-c-d

Here sample input is created using printf command to showcase stream editing. By default, sed processes input line by line. To determine a line, sed uses the \n newline character. The first sed command replaces only the first occurrence of , with -. The second command replaces all occurrences as the g flag is also used (g stands for global).

If you have a file with a different line ending style, you'll need to preprocess it first. For example, a text file downloaded from internet or a file originating from Windows OS would typically have lines ending with \r\n (carriage return + line feed). Modern text editors, IDEs and word processors can handle both styles easily. But every character matters when it comes to command line text processing. See stackoverflow: Why does my tool output overwrite itself and how do I fix it? for a detailed discussion and mitigation methods.

As a good practice, always use single quotes around the script to prevent shell interpretation. Other variations will be discussed later.

Editing file input

Although sed derives its name from stream editing, it is common to use sed for file editing. To do so, append one or more input filenames to the command. You can also specify stdin as a source by using - as the filename. By default, output will go to stdout and the input files will not be modified. In-place file editing chapter will discuss how to apply the changes back to the source file(s).

Sample input files used in examples are available from learn_gnused repo.

$ cat greeting.txt
Hi there
Have a nice day

$ # for each line, change first occurrence of 'day' with 'weekend'
$ sed 's/day/weekend/' greeting.txt
Hi there
Have a nice weekend

$ # change all 'e' to 'E' and save changed text to another file
$ sed 's/e/E/g' greeting.txt > out.txt
$ cat out.txt
Hi thErE
HavE a nicE day

In the stdin section examples, every input line had matched the search expression. The first sed command here searched for day, which did not match the first line of greeting.txt file input. By default, even if a line didn't satisfy the search expression, it will be part of the output. You'll see how to get only the modified lines in Print command section.

Cheatsheet and summary

Note	Description
`man sed`	brief manual
`sed --help`	brief description of all the command line options
`info sed`	comprehensive manual
online GNU sed manual	well formatted, easier to read and navigate
`s/REGEXP/REPLACEMENT/FLAGS`	syntax for substitute command
`sed 's/,/-/'`	replace first `,` with `-`
`sed 's/,/-/g'`	replace all `,` with `-`

This introductory chapter covered installation process, documentation and how to search and replace basic text using sed from the command line. In coming chapters, you'll learn many more commands and features that make sed an important tool when it comes to command line text processing. One such feature is editing files in-place, which will be discussed in the next chapter.

Exercises

Exercise related files are available from exercises folder of learn_gnused repo.

All the exercises are also collated together in one place at Exercises.md. For solutions, see Exercise_solutions.md.

a) Replace 5 with five for the given stdin source.

$ echo 'They ate 5 apples' | sed ##### add your solution here
They ate five apples

b) Replace all occurrences of 0xA0 with 0x50 and 0xFF with 0x7F for the given input file.

$ cat hex.txt
start address: 0xA0, func1 address: 0xA0
end address: 0xFF, func2 address: 0xB0

$ sed ##### add your solution here
start address: 0x50, func1 address: 0x50
end address: 0x7F, func2 address: 0xB0

c) The substitute command searches and replaces sequences of characters. When you need to map one or more characters with another set of corresponding characters, you can use the y command. Quoting from the manual:

y/src/dst/ Transliterate any characters in the pattern space which match any of the source-chars with the corresponding character in dest-chars.

Use the y command to transform the given input string to get the output string as shown below.

$ echo 'goal new user sit eat dinner' | sed ##### add your solution here
gOAl nEw UsEr sIt EAt dInnEr

GNU SED