Introduction
The command name sed
is derived from stream editor. Here, stream refers to data being passed via shell pipes. Thus, the command's primary functionality is to act as a text editor for stdin data with stdout as the output target. It is now common to use sed
on file inputs as well as in-place file editing.
This chapter will show how to install the latest sed
version followed by details related to documentation. Then, you'll get an introduction to the substitute command, which is the most commonly used feature. The chapters to follow will add more details to the substitute command, discuss other commands and command line options. Cheatsheet, summary and exercises are also included at the end of these chapters.
Installation
If you are on a Unix-like system, you will most likely have some version of sed
already installed. This book is primarily about GNU sed
. As there are syntax and feature differences between various implementations, make sure to use GNU sed
to follow along the examples presented in this book.
GNU sed
is part of the text creation and manipulation tools and comes by default on GNU/Linux distributions. To install a particular version, visit gnu: sed software. See also release notes for an overview of changes between versions and bug list if you think some command isn't working as expected.
$ wget https://ftp.gnu.org/gnu/sed/sed-4.9.tar.xz
$ tar -Jxf sed-4.9.tar.xz
$ cd sed-4.9/
# see https://askubuntu.com/q/237576 if you get compiler not found error
$ ./configure
$ make
$ sudo make install
$ sed --version | head -n1
sed (GNU sed) 4.9
If you are not using a Linux distribution, you may be able to access GNU sed
using an option below:
- Git for Windows — provides a Bash emulation used to run Git from the command line
- Windows Subsystem for Linux — compatibility layer for running Linux binary executables natively on Windows
- brew — Package Manager for macOS (or Linux)
Documentation
It is always good to know where to find documentation. From the command line, you can use man sed
for a short manual and info sed
for the full documentation. I prefer using the online gnu sed manual, which feels much easier to use and navigate.
$ man sed
SED(1) User Commands SED(1)
NAME
sed - stream editor for filtering and transforming text
SYNOPSIS
sed [-V] [--version] [--help] [-n] [--quiet] [--silent]
[-l N] [--line-length=N] [-u] [--unbuffered]
[-E] [-r] [--regexp-extended]
[-e script] [--expression=script]
[-f script-file] [--file=script-file]
[script-if-no-other-script]
[file...]
DESCRIPTION
Sed is a stream editor. A stream editor is used to perform basic
text transformations on an input stream (a file or input from a pipe‐
line). While in some ways similar to an editor which permits
scripted edits (such as ed), sed works by making only one pass over
the input(s), and is consequently more efficient. But it is sed's
ability to filter text in a pipeline which particularly distinguishes
it from other types of editors.
Options overview
For a quick overview of all the available options, use sed --help
from the command line.
# only partial output shown here
$ sed --help
-n, --quiet, --silent
suppress automatic printing of pattern space
--debug
annotate program execution
-e script, --expression=script
add the script to the commands to be executed
-f script-file, --file=script-file
add the contents of script-file to the commands to be executed
--follow-symlinks
follow symlinks when processing in place
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if SUFFIX supplied)
-l N, --line-length=N
specify the desired line-wrap length for the 'l' command
--posix
disable all GNU extensions.
-E, -r, --regexp-extended
use extended regular expressions in the script
(for portability use POSIX -E).
-s, --separate
consider files as separate rather than as a single,
continuous long stream.
--sandbox
operate in sandbox mode (disable e/r/w commands).
-u, --unbuffered
load minimal amounts of data from the input files and flush
the output buffers more often
-z, --null-data
separate lines by NUL characters
--help display this help and exit
--version output version information and exit
If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret. All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.
Editing standard input
sed
has various commands to manipulate text. The substitute command is the most commonly used operation, helps to replace matching text with something else. The syntax is s/REGEXP/REPLACEMENT/FLAGS
where
s
stands for the substitute command/
is an idiomatic delimiter character to separate various portions of the commandREGEXP
stands for regular expressionREPLACEMENT
refers to the replacement stringFLAGS
are options to change the default behavior of the command
For now, it is enough to know that the s
command is used for search and replace operations.
# input data that'll be passed to sed for editing
$ printf '1,2,3,4\na,b,c,d\n'
1,2,3,4
a,b,c,d
# for each input line, change only the first ',' to '-'
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/'
1-2,3,4
a-b,c,d
# change all matches by adding the 'g' flag
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/g'
1-2-3-4
a-b-c-d
In the above example, the input data is created using the printf
command to showcase stream editing. By default, sed
processes the input line by line. The newline character \n
is the line separator by default. The first sed
command replaces only the first occurrence of ,
with -
. The second command replaces all occurrences as the g
flag is also used (g
stands for global
).
As a good practice, always use single quotes around the script argument. Examples requiring shell interpretation will be discussed later.
If your input file has
\r\n
(carriage return and newline characters) as the line ending, convert the input file to Unix-style before processing. See stackoverflow: Why does my tool output overwrite itself and how do I fix it? for a detailed discussion and mitigation methods.# Unix style $ printf '42\n' | file - /dev/stdin: ASCII text # DOS style $ printf '42\r\n' | file - /dev/stdin: ASCII text, with CRLF line terminators
Editing file input
Although sed
derives its name from stream editing, it is common to use sed
for file editing. To do so, you can pass one or more input filenames as arguments. You can use -
to represent stdin as one of the input sources. By default, the modified data will go to the stdout
stream and the input files are not modified. In-place file editing chapter will discuss how to apply the changes back to the source file(s).
The example_files directory has all the files used in the examples.
$ cat greeting.txt
Hi there
Have a nice day
# for each line, change the first occurrence of 'day' with 'weekend'
$ sed 's/day/weekend/' greeting.txt
Hi there
Have a nice weekend
# change all occurrences of 'e' to 'E'
# redirect modified data to another file
$ sed 's/e/E/g' greeting.txt > out.txt
$ cat out.txt
Hi thErE
HavE a nicE day
In the previous section examples, every input line had matched the search expression. The first sed
command here searched for day
, which did not match the first line of greeting.txt
file. By default, even if a line doesn't satisfy the search expression, it will be part of the output. You'll see how to get only the modified lines in the Print command section.
Cheatsheet and summary
Note | Description |
---|---|
man sed | brief manual |
sed --help | brief description of all the command line options |
info sed | comprehensive manual |
online gnu sed manual | well formatted, easier to read and navigate |
s/REGEXP/REPLACEMENT/FLAGS | syntax for the substitute command |
sed 's/,/-/' | for each line, replace first , with - |
sed 's/,/-/g' | replace all , with - |
This introductory chapter covered installation process, documentation and how to search and replace text using sed
from the command line. In the coming chapters, you'll learn many more commands and features that make sed
an important tool when it comes to command line text processing. One such feature is editing files in-place, which will be discussed in the next chapter.
Interactive exercises
I wrote a TUI app to help you solve some of the exercises from this book interactively. See SedExercises repo for installation steps and app_guide.md for instructions on using this app.
Here's a sample screenshot:
Exercises
All the exercises are also collated together in one place at Exercises.md. For solutions, see Exercise_solutions.md.
The exercises directory has all the files used in this section.
1) Replace only the first occurrence of 5
with five
for the given stdin source.
$ echo 'They ate 5 apples and 5 mangoes' | sed ##### add your solution here
They ate five apples and 5 mangoes
2) Replace all occurrences of 5
with five
.
$ echo 'They ate 5 apples and 5 mangoes' | sed ##### add your solution here
They ate five apples and five mangoes
3) Replace all occurrences of 0xA0
with 0x50
and 0xFF
with 0x7F
for the given input file.
$ cat hex.txt
start address: 0xA0, func1 address: 0xA0
end address: 0xFF, func2 address: 0xB0
$ sed ##### add your solution here
start address: 0x50, func1 address: 0x50
end address: 0x7F, func2 address: 0xB0
4) The substitute command searches and replaces sequences of characters. When you need to map one or more characters with another set of corresponding characters, you can use the y
command. Quoting from the manual:
y/src/dst/ Transliterate any characters in the pattern space which match any of the source-chars with the corresponding character in dest-chars.
Use the y
command to transform the given input string to get the output string as shown below.
$ echo 'goal new user sit eat dinner' | sed ##### add your solution here
gOAl nEw UsEr sIt EAt dInnEr
5) Why does the following command produce an error? How'd you fix it?
$ echo 'a sunny day' | sed s/sunny day/cloudy day/
sed: -e expression #1, char 7: unterminated `s' command
# expected output
$ echo 'a sunny day' | sed ##### add your solution here
a cloudy day