This post shows how you can use the ripgrep, perl and sd commands to perform multiline fixed string search and replace operations from the command line. Solutions with GNU sed is also discussed, along with its limitations.


Fixed string matching🔗

The below sample input file will be used in the examples in this post.

$ cat ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.

ripgrep🔗

ripgrep supports the -U option to allow multiline matching. The -F option turns off regexp matching, i.e. the search string is treated literally. In the bash shell (and likely most other shells), you can press enter key to insert literal newline character for quoted values. When you do so, the next line starts with the secondary prompt PS2, which is usually > and a space character. This isn't shown in the examples below to make it easier to copy-paste the commands.

$ rg -UF 'like . () * [] $ {}
| ^ + ? \ and' ip.txt
4:like . () * [] $ {}
5:| ^ + ? \ and ' and so on.

# use the -l option to get only the filename instead of matching lines
$ rg -lUF 'like . () * [] $ {}
| ^ + ? \ and' ip.txt
ip.txt

You'll have an issue if your search string itself contains single quote characters. Avoid using double quotes as a workaround, as that has its own set of special characters. You can work around by concatenating multiple strings next to each other, along with escaped single quote characters as needed.

# the -N option disables line number prefix
$ rg -NUF 'like . () * [] $ {}
| ^ + ? \ and '\'' and' ip.txt
like . () * [] $ {}
| ^ + ? \ and ' and so on.

If your search string starts with the - character, you'll have to use -- before the search argument.

$ rg -NUF -- '-string multiline
search' ip.txt
-string multiline
search with cli tools.

perl🔗

You can use the -0777 option with perl to slurp the entire input as a single string. Another advantage with perl is that you can use files to pass the search and replace strings. Thus, you don't have to worry about any character that may clash with shell metacharacters. See my Perl One-Liners Guide if you are not familiar with using perl from the command line.

$ cat search_1.txt
like . () * [] $ {}
| ^ + ? \ and ' and so on.

# display filename if the given search string matches
$ perl -0777 -nE '!$#ARGV ? $s=$_ :
                  /\Q$s/ && say $ARGV' search_1.txt ip.txt
ip.txt

However, you'll have to make sure the file doesn't end with a newline if you are providing partial lines for searching, or take care of it within the perl script.

$ cat search_2.txt
-string multiline
search

# no output because there's a newline at the end of search_2.txt file
$ perl -0777 -nE '!$#ARGV ? $s=$_ :
                  /\Q$s/ && say $ARGV' search_2.txt ip.txt

# this will remove newline from the end of file before assigning to $s
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /\Q$s/ && say $ARGV' search_2.txt ip.txt
ip.txt

By default, ripgrep gives entire matching lines. To get rest of the line with perl, you'll have to explicitly add a pattern around the search string.

# $& variable has the entire matching portion
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /\Q$s/ && say $&' search_2.txt ip.txt
-string multiline
search

# use 'say $& while /.*\Q$s\E.*/g' if there are multiple matches
$ perl -0777 -nE '!$#ARGV ? $s=s/\n\z//r :
                  /.*\Q$s\E.*/ && say $&' search_2.txt ip.txt
-string multiline
search with cli tools.

Fixed string substitution🔗

ripgrep🔗

ripgrep also supports replacing the matched string with something else using the -r option. By default, you'll see only matched lines in the output. Use the --passthru option to display all the input lines, even if they do not match the given search string. See my blog post for more details about the -r option and various ways you can use it for substitution requirements.

$ rg --passthru -NUF 'like . () * [] $ {}
| ^ + ? \ and' -r '====
----
====' ip.txt
This is a multiline
sample input with lots
of special characters
====
----
==== ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.

Apart from having to workaround single quote, you'll have to use $$ instead of $ as it is used for backreferences in the replacement section.

$ echo 'sample input' | rg --passthru -F 'in' -r '$a'
sample put
$ echo 'sample input' | rg --passthru -F 'in' -r '$$a'
sample $aput

perl🔗

With perl, you can use files for both the search and replace strings. And, you can easily choose to replace the first or all occurrences, unlike ripgrep where it always replaces all the matches.

$ cat replace.txt
---------------------
$& = $1 + $2 / 3 \ 4
=====================

$ perl -0777 -ne '$#ARGV==1 ? $s=$_ : $#ARGV==0 ? $r=$_ :
                  print s/\Q$s/$r/gr' search_1.txt replace.txt ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.

As seen before, you'll have to remove newline from the search string for partial line matching.

# use $r=s/\n\z//r to avoid trailing newline from replace.txt
$ perl -0777 -ne '$#ARGV==1 ? $s=s/\n\z//r : $#ARGV==0 ? $r=$_ :
                  print s/\Q$s/$r/gr' search_2.txt replace.txt ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
=====================
 with cli tools.

sd🔗

sd supports a fixed string option and Rust regexp based substitution. Unlike ripgrep, the -s option for fixed string will apply to both the search and replacement sections. sd does in-place editing for file inputs by default, you can use -p to preview results on the terminal. Multiline matching is automatically performed by default.

$ echo 'sample input' | sd -s 'in' '$a'
sample $aput

$ sd -ps 'like . () * [] $ {}
| ^ + ? \ and' '====
----
====' ip.txt
This is a multiline
sample input with lots
of special characters
====
----
==== ' and so on.
This post shows how
you can do fixed
-string multiline
search with cli tools.

Saving file contents to a variable🔗

Trailing newlines and ASCII NUL characters will be lost if you wish to save contents of a file as bash variables using the var=$(< filename) command. See stackoverflow: pitfalls of reading file into shell variable for more details.

$ printf '\na\0b\n123\n\n\n\n\n\n\n\n' > t1
$ a=$(< t1)
bash: warning: command substitution: ignored null byte in input

# NUL character is lost after the assignment
# all the trailing newlines are lost as well
$ printf '%b' "$a" | cat -A
$
ab$
123

ripgrep🔗

If your search string doesn't have multiple trailing newlines or ASCII NUL characters, then you can save file contents to variables and then pass them to ripgrep. Single trailing newline will not normally cause an issue for searching operations as ripgrep will append a newline while displaying results anyway. If you want to make sure input file also contains the trailing newline, then you can manually concatenate a newline character to the search string.

$ s=$(< search_1.txt)
# use "$s"$'\n' if you want to match trailing newline as well
$ rg -NUF "$s" ip.txt
like . () * [] $ {}
| ^ + ? \ and ' and so on.

# use -- if the search string starts with a - character
$ s=$(< search_2.txt)
$ rg -NUF -- "$s" ip.txt
-string multiline
search with cli tools.

For substitution operations, you'll have to preprocess the replacement file to replace $ with $$.

$ s=$(< search_1.txt)
$ r=$(sed 's/\$/$$/g' replace.txt)

# here, removal of trailing newline doesn't cause an issue,
# as it evens out between the search and replace strings
$ rg --passthru -NUF "$s" -r "$r" ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.

Here, partial line has to be matched. So, $() assignment works well for the search string. If the trailing newline of the replacement string isn't needed, then $() assignment again is good enough. Otherwise, you can modify the replacement string as -r "$r"$'\n'

$ s=$(< search_2.txt)
$ r=$(sed 's/\$/$$/g' replace.txt)

$ rg --passthru -NUF -r "$r" -- "$s" ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
===================== with cli tools.

sd🔗

As mentioned before, the -s option for sd applies to both the search and replacement sections. So, the usage is lot simpler compared to ripgrep.

# -- is needed here because replace.txt starts with a - character
$ sd -ps -- "$(< search_1.txt)" "$(< replace.txt)" ip.txt
This is a multiline
sample input with lots
of special characters
---------------------
$& = $1 + $2 / 3 \ 4
=====================
This post shows how
you can do fixed
-string multiline
search with cli tools.

GNU sed🔗

To follow a similar approach with GNU sed, you'll have to preprocess the strings to escape metacharacters. Assuming input doesn't have ASCII NUL characters, you can use -z option to slurp the entire input as a single string.

Here's an example for multiline search.

# escape all BRE metacharacters
# replace literal newlines with \n
$ s=$(sed -z 's#[[^$*.\/]#\\&#g; s/\n/\\n/g' search_1.txt)

# since newlines are replaced with \n,
# trailing newlines will be preserved here
$ echo "$s"
like \. () \* \[] \$ {}\n| \^ + ? \\ and ' and so on\.\n

# display filename if input matches the given multiline search string
# tr is used to change the NUL character after filename to newline
$ sed -nz '/'"$s"'/F' ip.txt | tr '\0' '\n'
ip.txt

And here's an example for multiline substitution.

# last newline is removed here to allow partial line matching
$ s=$(sed -z 's#[[^$*.\/]#\\&#g; s/\n$//; s/\n/\\n/g' search_2.txt)

# escape all replacement section metacharacters
# and prefix \ character to literal newlines, except the last line
$ r=$(sed 's:[\\/&]:\\&:g; $!s/$/\\/' replace.txt)
$ echo "$r"
---------------------\
$\& = $1 + $2 \/ 3 \\ 4\
=====================

# if you need the trailing newline from replace.txt,
# use sed -z 's/'"$s"'/'"$r"'\n/g'
$ sed -z 's/'"$s"'/'"$r"'/g' ip.txt
This is a multiline
sample input with lots
of special characters
like . () * [] $ {}
| ^ + ? \ and ' and so on.
This post shows how
you can do fixed
---------------------
$& = $1 + $2 / 3 \ 4
===================== with cli tools.

Linux CLI ebooks🔗

Check out my ebooks if you are interested in learning more about Linux CLI basics, coreutils, text processing tools like GNU grep, GNU sed, GNU awk and perl.