In-place file editing

In the examples presented so far, the output from awk was displayed on the terminal. This chapter will discuss how to write back the changes to the input files using the -i command line option. You can also choose to create backups of the original files.

The example_files directory has all the files used in the examples.

Without backup

The -i option allows you to load libraries (see gawk manual: -i option for details). The inplace library comes by default with the awk installation. Use -i inplace to indicate that you want to modify the original input itself. Use this option with caution, preferably after testing that the code is working as intended.

$ cat greet.txt
Hi there
Have a nice day
Good bye

# prefix line numbers
$ awk -i inplace '{print NR ". " $0}' greet.txt
$ cat greet.txt
1. Hi there
2. Have a nice day
3. Good bye

Multiple input files are treated separately and changes are written back to the respective files.

$ cat f1.txt
I ate 3 apples
$ cat f2.txt
I bought two balls and 3 bats

$ awk -i inplace '{gsub(/\<3\>/, "three")} 1' f1.txt f2.txt
$ cat f1.txt
I ate three apples
$ cat f2.txt
I bought two balls and three bats

With backup

You can provide a backup extension by setting the inplace::suffix special variable. For example, if the input file is ip.txt and inplace::suffix='.orig' is used, the backup file will be named as ip.txt.orig.

$ cat f3.txt
  Name    Physics  Maths
 Moe  76  82
Raj  56  64

$ awk -i inplace -v inplace::suffix='.bkp' -v OFS=, '{$1=$1} 1' f3.txt
$ cat f3.txt
Name,Physics,Maths
Moe,76,82
Raj,56,64

# original file will be preserved in 'f3.txt.bkp'
$ cat f3.txt.bkp
  Name    Physics  Maths
 Moe  76  82
Raj  56  64

In earlier versions of awk, the INPLACE_SUFFIX variable was used instead of inplace::suffix. Also, you can use inplace::enable variable to dynamically control whether files should be in-placed or not. See gawk manual: Enabling In-Place File Editing for more details.

Security implications

By default, when you use the -i inplace option, the awk command will look for a file named inplace or inplace.awk in the current working directory. If such files aren't found, then awk will look for them in the installation directories, which is what you'd usually want.

For secure applications, you shouldn't rely on the -i inplace option. Instead, you could either use the absolute path of the inplace file from the installation directory, or manipulate AWKPATH (environment variable that controls the behavior of searching for files to be loaded) to be restricted to secure paths only. See this unix.stackexchange thread for more details about this issue and workarounds.

Summary

This chapter discussed about the -i inplace option which is useful when you need to edit a file in-place. This is particularly useful in automation scripts. But, do ensure that you have tested the awk command before applying changes to the actual files if you need to use this option without creating backups.

The next chapter will discuss the use of shell variables in more detail.

Exercises

The exercises directory has all the files used in this section.

1) For the input file copyright.txt, replace copyright: 2018 with copyright: 2020 and write back the changes to copyright.txt itself. The original contents should get saved to copyright.txt.orig

$ cat copyright.txt
bla bla 2015 bla
blah 2018 blah
bla bla bla
copyright: 2018
$ awk ##### add your solution here

$ cat copyright.txt
bla bla 2015 bla
blah 2018 blah
bla bla bla
copyright: 2020
$ cat copyright.txt.orig
bla bla 2015 bla
blah 2018 blah
bla bla bla
copyright: 2018

2) For the input files nums1.txt and nums2.txt, retain only the second and third lines and write back the changes to their respective files. No need to create backups.

$ cat nums1.txt
3.14
4201
777
0323012
$ cat nums2.txt
-45.4
-2
54316.12
0x231

$ awk ##### add your solution here
$ cat nums1.txt
4201
777
$ cat nums2.txt
-2
54316.12