z, s and f command line options

This chapter covers the -z, -s and -f command line options. These come in handy for specific use cases. For example, the -z option helps to process input separated by the ASCII NUL character and the -f option allows you to pass sed commands from a file.

The example_files directory has all the files used in the examples.

NUL separated lines

The -z option causes lines to be separated based on the ASCII NUL character instead of the newline character. Just like newline based processing, the NUL character is removed (if present) from the input line and added back accordingly when the output line is printed.

NUL separation is very useful to separate filenames since the names cannot have the NUL character. For example, grep -Z and find -print0 will print NUL separated filenames whereas grep -z and xargs -0 will accept NUL separated input.

$ printf 'earn xerox\0at\nmare\npart\0learn eye\n' | sed -nz '/are/p'
at
mare
part

# \0 at end of the output depends on the input having the \0 character
$ printf 'earn xerox\0at\nmare\npart\0learn eye\n' | sed -nz '/are/p' | od -c
0000000   a   t  \n   m   a   r   e  \n   p   a   r   t  \0
0000015
$ printf 'earn xerox\0at\nmare\npart\0learn eye\n' | sed -nz '/eye/p' | od -c
0000000   l   e   a   r   n       e   y   e  \n
0000012

If the input doesn't have NUL characters, then the -z option is handy to process the entire input as a single string. Output also wouldn't have NUL characters (unlike GNU grep which always adds the line separator to the output). This is effective only for files small enough to fit your available machine memory. It would also depend on the regular expression, as some patterns have exponential relationship with respect to the data size.

# add ; to the previous line if the current line starts with c
$ printf 'cater\ndog\ncoat\ncutter\nmat\n' | sed -z 's/\nc/;&/g'
cater
dog;
coat;
cutter
mat

See my blog post Multiline fixed string search and replace for more examples.

Separate files

The -s option will cause sed to treat multiple input files separately instead of treating them as single concatenated input. This helps if you need line number addressing to be effective for each input file. When the -i option is active, -s is also implied.

# without -s, there is only one first line for the concatenated contents
# F command inserts filename of the current file at the given address
$ sed '1F' greeting.txt ip.txt
greeting.txt
Hi there
Have a nice day
    * sky
    * apple

# with -s, each file has its own address
# this is like the 'cat' command but including the filename as the header
$ sed -s '1F' greeting.txt ip.txt
greeting.txt
Hi there
Have a nice day
ip.txt
    * sky
    * apple

File as the source of sed commands

If your sed commands does not easily fit the command line, you can put the commands in a file and use the -f option to specify that file as the source of commands to execute. This method also provides benefits like:

• literal newline can be used to separate the commands
• single quotes can be freely used as it will no longer clash as a shell metacharacter
• comments can be specified after the # character
  • See also sed manual: Often-Used Commands for details about using comments

Consider the following input file and a sed script stored in a plain text file.

$ cat sample.txt
Hi there
cats and dogs running around
Have a nice day

# sed script saved in a file
$ cat word_mapping.sed
# word mappings
s/cat/Cat/g
s/dog/Dog/g
s/Hi/Hey/g
# appending another line
/there|running/ s/$/\n----------/

The two lines starting with # character are comment lines. Comments can also be added at end of a command line if required, just like other programming languages. Use the -f option to pass the contents of the file as sed commands. Any other command line option like -n, -z, -E, etc have to mentioned along with the sed invocation, just like you've done so far.

# the first 3 substitutions will work
# but not the last one, as | is not a BRE metacharacter
$ sed -f word_mapping.sed sample.txt
Hey there
Cats and Dogs running around
Have a nice day

# | now works as ERE is enabled using the -E option
$ sed -E -f word_mapping.sed sample.txt
Hey there
----------
Cats and Dogs running around
----------
Have a nice day

Similar to making an executable Bash or Perl or Python script, you can add a shebang (see wikipedia: shebang for details) line to a sed script file.

# to get the full path of the command
$ type sed
sed is /usr/local/bin/sed

# sed script with shebang, note the use of -f after the command path
$ cat executable.sed
#!/usr/local/bin/sed -f
s/cats\|dogs/'&'/g

# add execute permission to the script
$ chmod +x executable.sed

# executing the script
$ ./executable.sed sample.txt
Hi there
'cats' and 'dogs' running around
Have a nice day

Adding any other command line option like -n, -z, -E, etc depends on a lot of factors. See stackoverflow: usage of options along with shebang for details.

See also sed manual: Some Sample Scripts and Sokoban game written in sed.

Cheatsheet and summary

This chapter covered three command line options that come in handy for specific situations. You also saw a few examples of sed being used as part of a solution with other commands in a pipeline or a shell script. In the next chapter, you'll learn some more specialized commands.

Exercises

The exercises directory has all the files used in this section.

1) Replace any character other than word and . characters with the _ character for the sample filenames shown below.

$ mkdir test_dir && cd $_
$ touch 'file with spaces.txt' $'weird$ch\nars.txt' '!f@oo.txt'
# > at start of line indicates continuation of multiline shell command
$ for file in *; do
>   new_name=$(printf '%s' "$file" | sed ##### add your solution here)
>   mv "$file" "$new_name"
> done

$ ls
file_with_spaces.txt  _f_oo.txt  weird_ch_ars.txt
$ cd .. && rm -r test_dir

2) Print only the third line, if any, from these input files: addr.txt, para.txt and copyright.txt

$ sed ##### add your solution here
This game is good
project you always wanted
bla bla bla

3) For the input file hex.txt, use content from replace.txt to perform search and replace operations. Each line in replace.txt starts with the search term, followed by a space and then followed by the replace term. Assume that these terms do not contain any sed metacharacters.

$ cat hex.txt
start address: 0xA0, func1 address: 0xA0
end address: 0xFF, func2 address: 0xB0
$ cat replace.txt
0xA0 0x5000
0xB0 0x6000
0xFF 0x7000

##### add your solution here
start address: 0x5000, func1 address: 0x5000
end address: 0x7000, func2 address: 0x6000

4) For the input file nul_separated, use the ASCII NUL character as the line separator and display lines containing fig. Also, change NUL characters in the output to the newline character.

$ sed ##### add your solution here
apple
fig
mango
icecream

5) For the input file addr.txt, change a newline character that comes before an uppercase letter to . followed by a space character. Assume that the input file doesn't have ASCII NUL characters.

$ sed ##### add your solution here
Hello World. How are you. This game is good. Today is sunny
12345. You are funny