Introduction
The command name sed
is derived from stream editor. Here, stream refers to data being passed via shell pipes. Thus, the command's primary functionality is to act as a text editor for stdin data with stdout as the output target. Over the years, functionality was added to edit file input and save the changes back to the same file.
This chapter will cover how to install/upgrade sed
followed by details related to documentation. Then, you'll get an introduction to the substitute command, which is the most commonly used sed
feature. The chapters to follow will add more details to the substitute command, discuss other commands and command line options. Cheatsheet, summary and exercises are also included at the end of these chapters.
Installation
If you are on a Unix like system, you are most likely to already have some version of sed
installed. This book is primarily for GNU sed
. As there are syntax and feature differences between various implementations, please make sure to follow along with what is presented here. GNU sed
is part of text creation and manipulation commands provided by GNU
and comes by default on GNU/Linux. To install newer or particular version, visit gnu: software. Check release notes for an overview of changes between versions. See also bug list.
$ # use a dir, say ~/Downloads/sed_install before following the steps below
$ wget https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
$ tar -Jxf sed-4.8.tar.xz
$ cd sed-4.8/
$ ./configure
$ make
$ sudo make install
$ type -a sed
sed is /usr/local/bin/sed
sed is /bin/sed
$ sed --version | head -n1
sed (GNU sed) 4.8
If you are not using a Linux distribution, you may be able to access GNU sed
using below options:
Documentation and options overview
It is always a good idea to know where to find the documentation. From command line, you can use man sed
for a short manual and info sed
for full documentation. I prefer using the online GNU sed manual which feels much easier to use and navigate.
$ man sed
SED(1) User Commands SED(1)
NAME
sed - stream editor for filtering and transforming text
SYNOPSIS
sed [OPTION]... {script-only-if-no-other-script} [input-file]...
DESCRIPTION
Sed is a stream editor. A stream editor is used to perform basic
text transformations on an input stream (a file or input from a pipe‐
line). While in some ways similar to an editor which permits
scripted edits (such as ed), sed works by making only one pass over
the input(s), and is consequently more efficient. But it is sed's
ability to filter text in a pipeline which particularly distinguishes
it from other types of editors.
For a quick overview of all the available options, use sed --help
from the command line. Most of them will be explained in the coming chapters.
$ # only partial output shown here
$ sed --help
-n, --quiet, --silent
suppress automatic printing of pattern space
--debug
annotate program execution
-e script, --expression=script
add the script to the commands to be executed
-f script-file, --file=script-file
add the contents of script-file to the commands to be executed
--follow-symlinks
follow symlinks when processing in place
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if SUFFIX supplied)
-l N, --line-length=N
specify the desired line-wrap length for the 'l' command
--posix
disable all GNU extensions.
-E, -r, --regexp-extended
use extended regular expressions in the script
(for portability use POSIX -E).
-s, --separate
consider files as separate rather than as a single,
continuous long stream.
--sandbox
operate in sandbox mode (disable e/r/w commands).
-u, --unbuffered
load minimal amounts of data from the input files and flush
the output buffers more often
-z, --null-data
separate lines by NUL characters
--help display this help and exit
--version output version information and exit
If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret. All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.
Editing standard input
sed
has various commands to manipulate text input. substitute command is most commonly used, which will be briefly discussed in this chapter. It is used to replace matching text with something else. The syntax is s/REGEXP/REPLACEMENT/FLAGS
where
s
stands for substitute command/
is an idiomatic delimiter character to separate various portions of the commandREGEXP
stands for regular expressionREPLACEMENT
specifies the replacement stringFLAGS
are options to change default behavior of the command
For now, it is enough to know that the s
command is used for search and replace operations.
$ # sample command output for stream editing
$ printf '1,2,3,4\na,b,c,d\n'
1,2,3,4
a,b,c,d
$ # for each input line, change only the first ',' to '-'
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/'
1-2,3,4
a-b,c,d
$ # change all matches by adding the 'g' flag
$ printf '1,2,3,4\na,b,c,d\n' | sed 's/,/-/g'
1-2-3-4
a-b-c-d
Here sample input is created using printf
command to showcase stream editing. By default, sed
processes input line by line. To determine a line, sed
uses the \n
newline character. The first sed
command replaces only the first occurrence of ,
with -
. The second command replaces all occurrences as the g
flag is also used (g
stands for global
).
If you have a file with a different line ending style, you'll need to preprocess it first. For example, a text file downloaded from internet or a file originating from Windows OS would typically have lines ending with
\r\n
(carriage return + line feed). Modern text editors, IDEs and word processors can handle both styles easily. But every character matters when it comes to command line text processing. See stackoverflow: Why does my tool output overwrite itself and how do I fix it? for a detailed discussion and mitigation methods.
As a good practice, always use single quotes around the script to prevent shell interpretation. Other variations will be discussed later.
Editing file input
Although sed
derives its name from stream editing, it is common to use sed
for file editing. To do so, append one or more input filenames to the command. You can also specify stdin
as a source by using -
as the filename. By default, output will go to stdout
and the input files will not be modified. In-place file editing chapter will discuss how to apply the changes back to the source file(s).
Sample input files used in examples are available from learn_gnused repo.
$ cat greeting.txt
Hi there
Have a nice day
$ # for each line, change first occurrence of 'day' with 'weekend'
$ sed 's/day/weekend/' greeting.txt
Hi there
Have a nice weekend
$ # change all 'e' to 'E' and save changed text to another file
$ sed 's/e/E/g' greeting.txt > out.txt
$ cat out.txt
Hi thErE
HavE a nicE day
In the stdin
section examples, every input line had matched the search expression. The first sed
command here searched for day
, which did not match the first line of greeting.txt
file input. By default, even if a line didn't satisfy the search expression, it will be part of the output. You'll see how to get only the modified lines in Print command section.
Cheatsheet and summary
Note | Description |
---|---|
man sed | brief manual |
sed --help | brief description of all the command line options |
info sed | comprehensive manual |
online GNU sed manual | well formatted, easier to read and navigate |
s/REGEXP/REPLACEMENT/FLAGS | syntax for substitute command |
sed 's/,/-/' | replace first , with - |
sed 's/,/-/g' | replace all , with - |
This introductory chapter covered installation process, documentation and how to search and replace basic text using sed
from the command line. In coming chapters, you'll learn many more commands and features that make sed
an important tool when it comes to command line text processing. One such feature is editing files in-place, which will be discussed in the next chapter.
Exercises
Exercise related files are available from exercises folder of learn_gnused repo.
All the exercises are also collated together in one place at Exercises.md. For solutions, see Exercise_solutions.md.
a) Replace 5
with five
for the given stdin source.
$ echo 'They ate 5 apples' | sed ##### add your solution here
They ate five apples
b) Replace all occurrences of 0xA0
with 0x50
and 0xFF
with 0x7F
for the given input file.
$ cat hex.txt
start address: 0xA0, func1 address: 0xA0
end address: 0xFF, func2 address: 0xB0
$ sed ##### add your solution here
start address: 0x50, func1 address: 0x50
end address: 0x7F, func2 address: 0xB0
c) The substitute command searches and replaces sequences of characters. When you need to map one or more characters with another set of corresponding characters, you can use the y
command. Quoting from the manual:
y/src/dst/ Transliterate any characters in the pattern space which match any of the source-chars with the corresponding character in dest-chars.
Use the y
command to transform the given input string to get the output string as shown below.
$ echo 'goal new user sit eat dinner' | sed ##### add your solution here
gOAl nEw UsEr sIt EAt dInnEr