z, s and f command line options

This chapter covers the -z, -s and -f command line options. These come in handy for specific use cases. For example, the -z option helps to process input separated by ASCII NUL character and the -f option allows you to pass sed commands from a file.

NUL separated lines

The -z option will cause sed to separate lines based on the ASCII NUL character instead of the newline character. Just like normal newline based processing, the NUL character is removed (if present) from the input line and added back accordingly when the processed line is printed.

info NUL separation is very useful to process filenames as newline is a valid character for filenames but NUL character isn't. For example, grep -Z and find -print0 will print NUL separated filenames whereas grep -z and xargs -0 will accept NUL separated input.

$ printf 'earn xerox\0at\nmare\npart\0learn eye\n' | sed -nz '/are/p'
at
mare
part

$ # \0 at end of output depends on input having \0 character
$ printf 'earn xerox\0at\nmare\npart\0learn eye\n' | sed -nz '/are/p' | od -c
0000000   a   t  \n   m   a   r   e  \n   p   a   r   t  \0
0000015
$ printf 'earn xerox\0at\nmare\npart\0learn eye\n' | sed -nz '/eye/p' | od -c
0000000   l   e   a   r   n       e   y   e  \n
0000012

If input doesn't have NUL characters, then -z option is handy to process the entire input as a single string. This is effective only for files small enough to fit your available machine memory. It would also depend on the regular expression, as some patterns have exponential relationship with respect to data size. As input doesn't have NUL character, output also wouldn't have NUL character, unlike GNU grep which always adds the line separator to the output.

$ # adds ; to previous line if current line starts with c
$ printf 'cater\ndog\ncoat\ncutter\nmat\n' | sed -z 's/\nc/;&/g'
cater
dog;
coat;
cutter
mat

info See my blog post Multiline fixed string search and replace for more examples with -z option.

Separate files

The -s option will cause sed to treat multiple input files separately instead of treating them as single concatenated input. This helps if you need line number addressing to be effective for each input file.

info If the -i option is being used, -s is also implied.

$ # without -s, there is only one first line for all files combined
$ # F command inserts filename of current file at the given address
$ sed '1F' cols.txt 5.txt
cols.txt
1:2:3:4
a:b:c:d
five
1five

$ # with -s, each file has its own address
$ # this is like 'cat' command but also prints filename as header
$ sed -s '1F' cols.txt 5.txt
cols.txt
1:2:3:4
a:b:c:d
5.txt
five
1five

File as source of sed commands

If your sed commands does not easily fit the command line, you have the option of putting the commands in a file and use -f option to specify that file as the source of commands to execute. This method also provides benefits like:

  • literal newline can be used to separate the commands
  • single quotes can be freely used as it will no longer clash as a shell metacharacter
  • comments can be specified after the # character

Consider the following input file and a sed script written into a plain text file.

$ cat sample.txt
Hi there
cats and dogs running around
Have a nice day

$ cat word_mapping.sed
# word mappings
s/cat/Cat/g
s/dog/Dog/g
s/Hi/Hey/g
# appending another line
/there|running/ s/$/\n----------/

The two lines starting with # character are comment lines. Comments can also be added at end of a command line if required, just like other programming languages. Use the -f option to pass the contents of the file as sed commands. Any other command line option like -n, -z, -E, etc have to mentioned along with sed invocation, just like you've done so far.

$ # the first 3 substitutions will work
$ # but not the last one, as | is not a metacharacter with BRE
$ sed -f word_mapping.sed sample.txt
Hey there
Cats and Dogs running around
Have a nice day

$ # | now works as ERE is enabled using -E option
$ sed -E -f word_mapping.sed sample.txt
Hey there
----------
Cats and Dogs running around
----------
Have a nice day

Similar to making an executable bash or perl or python script, you can add a shebang (see wikipedia: shebang for details) line to a sed script file.

$ # to get full path of the command
$ type sed
sed is /usr/local/bin/sed

$ # sed script with shebang, note the use of -f after command path
$ cat executable.sed
#!/usr/local/bin/sed -f
s/cats\|dogs/'&'/g

$ # add execute permission to the script
$ chmod +x executable.sed

$ # executing the script
$ ./executable.sed sample.txt
Hi there
'cats' and 'dogs' running around
Have a nice day

warning Adding any other command line option like -n, -z, -E, etc depends on a lot of factors. See stackoverflow: usage of options along with shebang for details.

info See also sed manual: Some Sample Scripts and Sokoban game written in sed

Cheatsheet and summary

NoteDescription
-zchange line separator from newline to ASCII NUL character
grep -Z and find -print0 are examples for NUL separated input
-z also useful to process entire input as single string if it doesn't
contain NUL characters, assuming small enough input file to fit memory
-sseparate addressing for multiple file inputs
-s is implied if -i is being used
-fsupply commands from a file
literal newline can be used to separate the commands
single quotes can be freely used
comments can be specified after the # character
Fcommand to insert current filename at the given address

This chapter covered three command line options that come in handy for specific situations. You also saw a few examples of sed being used as part of a solution with other commands in a pipeline or a shell script. In the next chapter, you'll learn three commands that are also specialized for particular use cases.

Exercises

a) Replace any character other than word characters and . character with _ character for the sample filenames shown below.

$ mkdir test_dir && cd $_
$ touch 'file with spaces.txt' $'weird$ch\nars.txt' '!f@oo.txt'
$ # > at start of line indicates continuation of multiline shell command
$ for file in *; do
>   new_name=$(printf '%s' "$file" | sed ##### add your solution here)
>   mv "$file" "$new_name"
> done

$ ls
file_with_spaces.txt  _f_oo.txt  weird_ch_ars.txt
$ cd .. && rm -r test_dir

b) Print only the third line, if any, from these input files: addr.txt, para.txt and copyright.txt

$ sed ##### add your solution here
This game is good
project you always wanted
bla bla bla

c) For the input file hex.txt, use content from replace.txt to perform search and replace operations. Each line in replace.txt starts with the search term, followed by a space and then followed by the replace term. Assume that these terms do not contain any sed metacharacters.

$ cat hex.txt
start address: 0xA0, func1 address: 0xA0
end address: 0xFF, func2 address: 0xB0
$ cat replace.txt
0xA0 0x5000
0xB0 0x6000
0xFF 0x7000

$ sed -f <(sed ##### add your solution here) hex.txt
start address: 0x5000, func1 address: 0x5000
end address: 0x7000, func2 address: 0x6000