Shell Scripting

This chapter will cover basics of shell scripting with bash. You'll learn about declaring variables, control structures, working with arguments passed to a script, getting user input and so on.

The example_files directory has all the shell scripts discussed in this chapter. However, it is recommended that you type the scripts manually using your favorite text editor and refer to the example_files/shell_scripting directory only if necessary.

Need for scripting

From wikipedia: Scripting language:

A scripting language or script language is a programming language for a runtime system that automates the execution of tasks that would otherwise be performed individually by a human operator. Scripting languages are usually interpreted at runtime rather than compiled.

Typical scripting languages are intended to be very fast to learn and write in, either as short source code files or interactively in a read–eval–print loop (REPL, language shell). This generally implies relatively simple syntax and semantics; typically a "script" (code written in the scripting language) is executed from start to finish, as a "script", with no explicit entry point.

From wikipedia: Shell script:

A shell script is a computer program designed to be run by the Unix shell, a command-line interpreter. The various dialects of shell scripts are considered to be scripting languages. Typical operations performed by shell scripts include file manipulation, program execution, and printing text. A script which sets up the environment, runs the program, and does any necessary cleanup or logging, is called a wrapper.

See also Difference between scripting and programming languages.

Executable script

There are several ways you can execute commands from a file. This section shows an example of creating an executable script. Consider this sample script saved in a file named hello.sh:

#!/bin/bash

echo "Hello $USER"
echo "Today is $(date -u +%A)"
echo 'Have a nice day'

The first line in the above script has two parts:

• /bin/bash is the path of bash interpreter
  • you can use type bash to get the path on your system
• #! is known as shebang or hashbang which directs the program loader to use the interpreter path provided
  • see also stackoverflow: comparison between #!/usr/bin/env and #!/bin/bash?
  • the # character starts a comment, #! is only special at the start of the script

Use chmod to add executable permission to the file and then run the script:

$ chmod +x hello.sh

$ ./hello.sh 
Hello learnbyexample
Today is Wednesday
Have a nice day

If you want to use just the script name to execute it, the file has to be located in one of the PATH folders. Otherwise, you'll have to provide the script's path (absolute or relative) in order to execute it (as shown in the above illustration).

.sh is typically used as the file extension for shell scripts. It is also common to not have an extension at all, especially for executable scripts.

Passing file argument to bash

You can also just pass a regular file as an argument to the bash command. In this case, the shebang isn't needed (though it wouldn't cause any issues either, since it will be treated as a comment).

$ cat greeting.sh
echo 'hello'
echo 'have a nice day'

$ bash greeting.sh
hello
have a nice day

Sourcing script

Yet another way to execute a script is to source it using the source (or .) builtin command. A major difference from the previous methods is that the script is executed in the current shell environment context instead of a sub-shell. A common use case is sourcing ~/.bashrc and alias/functions (if they are saved in a separate file).

Here's an example:

$ cat prev_cmd.sh
prev=$(fc -ln -2 | sed 's/^\s*//; q')
echo "$prev"

# 'echo' here is just a sample command for illustration purposes
$ echo 'hello'
hello
# sourcing the script correctly gives the previous command
$ source prev_cmd.sh
echo 'hello'

$ echo 'hello'
hello
# no output when the script is executed in a sub-shell
$ bash prev_cmd.sh

fc is a builtin command to manipulate the history of commands you've used from the terminal. See bash manual: History Builtins for more details.

Comments

Single line comments can be inserted after the # character, either at the start of a line or after an instruction.

$ cat comments.sh
# this is a comment on its own line
echo 'hello' # and this is a comment after a command

$ bash comments.sh
hello

See this unix.stackexchange thread for emulating multiline comments.

Variables

Here's a basic example of assigning a variable and accessing its value:

$ name='learnbyexample'

$ echo "$name"
learnbyexample

As seen above, you need to use the $ prefix while accessing the value stored in a variable. You can use ${variable} syntax to distinguish between the variable and other parts of the string. Using appropriate quotes is recommended, unless otherwise necessary.

You can append to a variable by using the += operator. Here's an example:

$ colors='blue'
$ echo "$colors"
blue

$ colors+=' green'
$ echo "$colors"
blue green

You can use the declare builtin to add attributes to variables. For example, the -i option for treating the variable as an integer, -r option for readonly, etc. These attributes can change the behavior of operators like = and += for those variables. See bash manual: Shell-Parameters and bash manual: declare for more details.

$ declare -i num=5
$ echo "$num"
5
$ num+=42
$ echo "$num"
47

$ declare -r color='brown'
$ echo "$color"
brown
$ color+=' green'
bash: color: readonly variable

Assigning variables is one of the most common source for errors. Unlike most programming languages, spaces are not allowed around the = sign. That is because space is a shell metacharacter. Another common issue is using quotes (or not) around the value. Here are some examples:

$ num = 42
num: command not found

$ greeting=hello world
world: command not found
$ greeting='hello world'
$ echo "$greeting"
hello world

# using quotes is NOT desirable here
$ dir_path=~/reports
$ echo "$dir_path"
/home/learnbyexample/reports
$ dir_path='~/reports'
$ echo "$dir_path"
~/reports

Arrays

From bash manual: Arrays:

Bash provides one-dimensional indexed and associative array variables. Any variable may be used as an indexed array; the declare builtin will explicitly declare an array. There is no maximum limit on the size of an array, nor any requirement that members be indexed or assigned contiguously. Indexed arrays are referenced using integers and are zero-based; associative arrays use arbitrary strings.

Here's an example of assigning an indexed array and various ways of accessing the elements:

$ fruits=('apple' 'fig' 'mango')

# first element
$ echo "${fruits[0]}"
apple

# last element
$ echo "${fruits[-1]}"
mango

# all elements (example with for loop will be discussed later on)
$ echo "${fruits[@]}"
apple fig mango
$ printf '%s\n' "${fruits[@]}"
apple
fig
mango

Parameter Expansion

Bash provides several useful ways to extract and modify contents of parameters and variables (including arrays). Some of these features will be discussed in this section.

1) Substring extraction using ${parameter:offset} syntax to get all characters from the given index:

$ city='Lucknow'

# all characters from index 4 onwards
# indexing starts from 0
$ echo "${city:4}"
now

# last two characters
# space before the negative sign is compulsory here,
# since ${parameter:-word} is a different feature
$ echo "${city: -2}"
ow

When applied to arrays, substring extraction will give you those elements:

$ fruits=('apple' 'fig' 'mango')

# all elements from index 1
$ echo "${fruits[@]:1}"
fig mango

2) Substring extraction using ${parameter:offset:length} syntax to get specific number of characters from the given index:

$ city='Lucknow'

# 4 characters starting from index 0
# can also use: echo "${city::4}"
$ echo "${city:0:4}"
Luck

# 2 characters starting from index -4 (4th character from the end)
$ echo "${city: -4:2}"
kn

# except last 2 characters
$ echo "${city::-2}"
Luckn

3) ${#parameter} will give you the length of the string and ${#array[@]} will give you the number of elements in the array:

$ city='Lucknow'
$ echo "${#city}"
7

$ fruits=('apple' 'fig' 'mango')
$ echo "${#fruits[@]}"
3

4) ${parameter#glob} will remove the shortest match from the start of the string. You can also use extended globs if enabled via shopt builtin. ${parameter##glob} will remove the longest match from the start of the string. Here are some examples:

$ s='this is his life history'

# shortest match is deleted
$ echo "${s#*is}"
 is his life history
# longest match is deleted
$ echo "${s##*is}"
tory

# assuming extglob is enabled
$ echo "${s#+([^ ])}"
his is his life history
$ echo "${s##+([^ ])}"
 is his life history

# for arrays, the processing is applied to each element
$ fruits=('apple' 'fig' 'mango')
$ echo "${fruits[@]#*[aeiou]}"
pple g ngo

5) You can use ${parameter%glob} to remove the shortest match from the end of the string. ${parameter%%glob} will remove the longest match from the end of the string. Here are some examples:

$ s='this is his life history'

$ echo "${s%is*}"
this is his life h
$ echo "${s%%is*}"
th

$ fruits=('apple' 'fig' 'mango')
$ echo "${fruits[@]%[aeiou]*}"
appl f mang

6) ${parameter/glob/string} replaces the first matching occurrence with the given replacement string and ${parameter//glob/string} will replace all the matching occurrences. You can leave out the /string portion when you want to delete the matching occurrences. The glob will match the longest portion, similar to greedy behavior in regular expressions. Here are some examples:

$ ip='this is a sample string'

# first occurrence of 'is' is replaced with '123'
$ echo "${ip/is/123}"
th123 is a sample string
# all occurrences of 'is' are replaced with '123'
$ echo "${ip//is/123}"
th123 123 a sample string

# replace all occurrences of 'am' or 'in' with '-'
$ echo "${ip//@(am|in)/-}"
this is a s-ple str-g

# matches from the first 'is' to the last 's' in the input
$ echo "${ip/is*s/ X }"
th X tring

# deletes first occurrence of 's'
$ echo "${ip/s}"
thi is a sample string
# deletes all occurrences of 's'
$ echo "${ip//s}"
thi i a ample tring

7) You can use ${parameter/#glob/string} to match only at the start of the string and ${parameter/%glob/string} to match only at the end of the string.

$ ip='spare'

# remove only from the start of the string
$ echo "${ip/#sp}"
are
$ echo "${ip/#par}"
spare
# example with replacement string
$ echo "${ip/#sp/fl}"
flare

# remove only from the end of the string
$ echo "${ip/%re}"
spa
$ echo "${ip/%par}"
spare

8) ${parameter^glob} can change only the first character to uppercase if matched by the glob. ${parameter^^glob} changes all the matching characters to uppercase (anywhere in the input string). You should provide a glob that only matches one character in length. If the glob is omitted, entire parameter will be matched. These rules also apply to the lowercase and swap case versions discussed later.

$ fruit='apple'

# uppercase the first character
$ echo "${fruit^}"
Apple
# uppercase the entire parameter
$ echo "${fruit^^}"
APPLE

# first character doesn't match the 'g-z' range, so no change
$ echo "${fruit^[g-z]}"
apple
# uppercase all letters in the 'g-z' range
$ echo "${fruit^^[g-z]}"
aPPLe
# uppercase all letters in the 'a-e' or 'j-m' ranges
$ echo "${fruit^^[a-ej-m]}"
AppLE

# this won't work since 'sky-' is not a single character
$ color='sky-rose'
$ echo "${color^^*-}"
sky-rose

9) To change the characters to lowercase, use , and ,, as shown below:

$ fruit='APPLE'

$ echo "${fruit,}"
aPPLE
$ echo "${fruit,,}"
apple

$ echo "${fruit,,[G-Z]}"
ApplE

10) To swap case, use ~ and ~~ as shown below. Note that this seems to be deprecated, since it is no longer mentioned in the bash manual.

$ fruit='aPPle'

# swap case only for the first character
$ echo "${fruit~}"
APPle
# swap case for all the characters
$ echo "${fruit~~}"
AppLE

# swap case for characters matching the given character set
$ echo "${fruit~~[g-zG-Z]}"
appLe

See bash manual: Shell Parameter Expansion for more details and other types of expansions.

Command Line Arguments

Command line arguments passed to a script (or a function) are saved in positional parameters starting with 1, 2, 3 etc. 0 contains the name of the shell or shell script. @ contains all the positional parameters starting from 1. Use # to get the number of positional parameters. Similar to variables, you need to use a $ prefix to get the value stored in these parameters. If the parameter number requires more than a single digit, you have to necessarily enclose them in {} (for example, ${12} to get the value of the twelfth parameter).

Here's an example script that accepts two arguments:

$ cat command_line_arguments.sh
echo "No. of lines in '$1' is $(wc -l < "$1")"
echo "No. of lines in '$2' is $(wc -l < "$2")"

$ seq 12 > 'test file.txt'

$ bash command_line_arguments.sh hello.sh test\ file.txt 
No. of lines in 'hello.sh' is 5
No. of lines in 'test file.txt' is 12

Further Reading

• unix.stackexchange: shell script choking on whitespace or other special characters
• bash manual: Special Parameters

Conditional Expressions

You can test a condition within [[ and ]] to get a success (0) or failure (1 or higher) exit status and take action accordingly. Bash provides several options and operators that you can use. Space is required after [[ and before ]] for this compound command to function.

Operators ;, && and || will be used in this section to keep the examples terser. if-else and other control structures will be discussed later.

Options

The -e option checks if the given path argument exists or not. Add a ! prefix to negate the condition.

# change to the 'example_files/shell_scripting' directory for this section

$ [[ -e hello.sh ]] && echo 'found' || echo 'not found'
found

$ [[ -e xyz.txt ]] && echo 'found' || echo 'not found'
not found

# exit status
$ [[ -e hello.sh ]] ; echo $?
0
$ [[ -e xyz.txt ]] ; echo $?
1
$ [[ ! -e xyz.txt ]] ; echo $?
0

You can use -d and -f to check if the path is a valid directory and file respectively. The -s option checks if the file exists and its size is greater than zero. The -x option checks if the file exists and is executable. See help test and bash manual: Conditional Expressions for a complete list of such options.

String comparisons

• s1 = s2 or s1 == s2 checks if two strings are equal
  • unquoted portions of s2 will be treated as a wildcard while testing against s1
  • extglob would be considered as enabled for such comparisons
• s1 != s2 checks if strings are not equal
  • unquoted portions of s2 will be treated as a wildcard while testing against s1
  • extglob would be considered as enabled for such comparisons
• s1 < s2 checks if s1 sorts before s2 lexicographically
• s1 > s2 checks if s1 sorts after s2 lexicographically
• s1 =~ s2 checks if s1 matches the POSIX extended regular expression provided by s2
  • exit status will be 2 if s2 is not a valid regexp

Here are some examples for equal and not-equal comparisons:

$ fruit='apple'
$ [[ $fruit == 'apple' ]] && echo 'true' || echo 'false'
true
$ [[ $fruit == 'banana' ]] && echo 'true' || echo 'false'
false

# glob should be constructed to match the entire string
$ [[ hello == h* ]] && echo 'true' || echo 'false'
true
# don't quote the glob!
$ [[ hello == 'h*' ]] && echo 'true' || echo 'false'
false

# another example to emphasize that the glob should match the entire string
$ [[ hello == e*o ]] && echo 'true' || echo 'false'
false
$ [[ hello == *e*o ]] && echo 'true' || echo 'false'
true

$ [[ hello != *a* ]] && echo 'true' || echo 'false'
true
$ [[ hello != *e* ]] && echo 'true' || echo 'false'
false

Here are some examples for greater-than and less-than comparisons:

$ [[ apple < banana ]] && echo 'true' || echo 'false'
true
$ [[ par < part ]] && echo 'true' || echo 'false'
true

$ [[ mango > banana ]] && echo 'true' || echo 'false'
true
$ [[ sun > moon && fig < papaya ]] && echo 'true' || echo 'false'
true

# don't use this to compare numbers!
$ [[ 20 > 3 ]] && echo 'true' || echo 'false'
false
# -gt and other such operators will be discussed later
$ [[ 20 -gt 3 ]] && echo 'true' || echo 'false'
true

Here are some examples for regexp comparison. You can use the special array BASH_REMATCH to retrieve specific portions of the string that was matched. Index 0 gives entire matched portion, 1 gives the portion matched by the first capture group and so on.

$ fruit='apple'
$ [[ $fruit =~ ^a ]] && echo 'true' || echo 'false'
true
$ [[ $fruit =~ ^b ]] && echo 'true' || echo 'false'
false

# entire matched portion
$ [[ $fruit =~ a.. ]] && echo "${BASH_REMATCH[0]}"
app
# portion matched by the first capture group
$ [[ $fruit =~ a(..) ]] && echo "${BASH_REMATCH[1]}"
pp

Numeric comparisons

• n1 -eq n2 checks if two numbers are equal
• n1 -ne n2 checks if two numbers are not equal
• n1 -gt n2 checks if n1 is greater than n2
• n1 -ge n2 checks if n1 is greater than or equal to n2
• n1 -lt n2 checks if n1 is less than n2
• n1 -le n2 checks if n1 is less than or equal to n2

These operators support only integer comparisons.

$ [[ 20 -gt 3 ]] && echo 'true' || echo 'false'
true

$ n1='42'
$ n2='25'
$ [[ $n1 -gt 30 && $n2 -lt 12 ]] && echo 'true' || echo 'false'
false

Numeric arithmetic operations and comparisons can also be performed within the (( and )) compound command. Here are some sample comparisons:

$ (( 20 > 3 )) && echo 'true' || echo 'false'

$ n1='42'
$ n2='25'
$ (( n1 > 30 && n2 < 12 )) && echo 'true' || echo 'false'
false

Note that the $ prefix was not used for variables in the above example. See bash manual: Shell Arithmetic for more details.

Accepting user input interactively

You can use the read builtin command to accept input from the user interactively. If multiple variables are given as arguments to the read command, values will be assigned based on whitespace separation by default. Any pending values will be assigned to the last variable. Here are some examples:

# press 'Enter' after the 'read' command
# and also after you've finished entering the input
$ read color
light green
$ echo "$color"
light green

# example with multiple variables
$ read fruit qty
apple 10
$ echo "${fruit}: ${qty}"
apple: 10

The -p option helps you to add a user prompt. Here is an example of getting two arguments from the user:

$ cat user_input.sh
read -p 'Enter two integers separated by spaces: ' num1 num2
sum=$(( num1 + num2 ))
echo "$num1 + $num2 = $sum"

$ bash user_input.sh
Enter two integers separated by spaces: -2 42
-2 + 42 = 40

You can use the -a option to assign an array, the -d option to specify a custom delimiter instead of newline for terminating user input and so on. See help read and bash manual: Builtins for more details.

if then else

The keywords needed to construct an if control structure are if, then, fi and optionally else and elif. You can use compound commands like [[ and (( to provide the test condition. You can also directly use a command's exit status. Here's an example script:

$ cat if_then_else.sh
if (( $# != 1 )) ; then
    echo 'Error! One file argument expected.' 1>&2
    exit 1
else
    if [[ ! -f $1 ]] ; then
        printf 'Error! %q is not a valid file\n' "$1" 1>&2
        exit 1
    else
        echo "No. of lines in '$1' is $(wc -l < "$1")"
    fi
fi

1>&2 is used in the above script to redirect error messages to the stderr stream. Sample script invocations are shown below:

$ bash if_then_else.sh
Error! One file argument expected.
$ echo $?
1

$ bash if_then_else.sh xyz.txt
Error! xyz.txt is not a valid file
$ echo $?
1

$ bash if_then_else.sh hello.sh
No. of lines in 'hello.sh' is 5
$ echo $?
0

Sometimes you just need to know if the intended command operation was successful or not and then take an action depending on the outcome. In such cases, you can provide the command directly after the if keyword. Note that stdout and stderr of the command will still be active unless redirected or suppressed using appropriate options.

For example, the grep command supports -q option to suppress stdout. Here's a script using that feature:

$ cat search.sh 
read -p 'Enter a search pattern: ' search

if grep -q "$search" hello.sh ; then
    echo "match found"
else
    echo "match not found"
fi

Sample invocations for the above script:

$ bash search.sh
Enter a search pattern: echo
match found

$ bash search.sh
Enter a search pattern: xyz
match not found

for loop

To construct a for loop, you'll need the for, do and done keywords. Here are some examples:

# iterate over numbers generated using brace expansion
$ for num in {2..4}; do echo "$num"; done
2
3
4

# iterate over files matched using wildcards
# echo is used here for dry run testing
$ for file in [gh]*.sh; do echo mv "$file" "$file.bkp"; done
mv greeting.sh greeting.sh.bkp
mv hello.sh hello.sh.bkp

As seen in the above examples, the space separated arguments provided after the in keyword are automatically assigned to the variable provided after the for keyword during each iteration.

Here's a modified example of the last example that accepts user provided command line arguments:

$ cat for_loop.sh
for file in "$@"; do 
    echo mv "$file" "$file.bkp"
done

$ bash for_loop.sh [gh]*.sh
mv greeting.sh greeting.sh.bkp
mv hello.sh hello.sh.bkp

$ bash for_loop.sh report.log ip.txt fruits.txt
mv report.log report.log.bkp
mv ip.txt ip.txt.bkp
mv fruits.txt fruits.txt.bkp

Here's an example of iterating over an array:

$ files=('report.log' 'pass_list.txt')
$ for f in "${files[@]}"; do echo "$f"; done
report.log
pass_list.txt

You can use continue and break to alter the loop flow depending on specific conditions. See bash manual: Bourne Shell Builtins for more details.

for file; is same as for file in "$@"; since in "$@" is the default. I'd recommend using the explicit version.

while loop

Here's a simple while loop construct. You'll see a more practical example later in this chapter.

$ cat while_loop.sh
i="$1"
while (( i > 0 )) ; do
    echo "$i"
    (( i-- ))
done

$ bash while_loop.sh 3
3
2
1

Reading a file

The while loop combined with the read builtin helps you to process the content of a file. Here's an example of reading input contents line by line:

$ cat read_file_lines.sh
while IFS= read -r line; do
    # do something with each line
    wc -l "$line"
done < "$1"

$ printf 'hello.sh\ngreeting.sh\n' > files.txt
$ bash read_file_lines.sh files.txt
5 hello.sh
2 greeting.sh

The intention in the above script is to treat each input line literally. So, the IFS (input field separator) special variable is set to empty string to prevent stripping of leading and trailing whitespaces. The -r option to the read builtin allows \ in input to be treated literally. Note that the input filename is accepted as the first command line argument and redirected as stdin to the while loop. You also need to make sure that the last line of input ends with a newline character, otherwise the last line won't be processed.

You can change IFS to split the input line into different fields and specify appropriate number of variables to the read builtin. Here's an example:

$ cat read_file_fields.sh
while IFS=' : ' read -r field1 field2; do
    echo "$field2,$field1"
done < "$1"

$ bash read_file_fields.sh <(printf 'apple : 3\nfig : 100\n')
3,apple
100,fig

You can pass a number to the -n option for the read builtin to process the input that many characters at a time. Here's an example:

$ while read -r -n2 ip; do echo "$ip"; done <<< '\word'
\w
or
d

The xargs command can also be used for some of the cases discussed above. See unix.stackexchange: parse each line of a text file as a command argument for examples.

Functions

From bash manual: Shell Functions:

Shell functions are a way to group commands for later execution using a single name for the group. They are executed just like a "regular" command. When the name of a shell function is used as a simple command name, the list of commands associated with that function name is executed. Shell functions are executed in the current shell context; no new process is created to interpret them.

You can use either of the syntax shown below to declare functions:

fname () compound-command [ redirections ]

function fname [()] compound-command [ redirections ]

Arguments to functions are passed in the same manner as those discussed earlier for shell scripts. Here's an example:

$ cat functions.sh
add_border ()
{
    size='10'
    color='grey'
    if (( $# == 1 )) ; then
        ip="$1"
    elif (( $# == 2 )) ; then
        if [[ $1 =~ ^[0-9]+$ ]] ; then
            size="$1"
        else
            color="$1"
        fi
        ip="$2"
    else
        size="$1"
        color="$2"
        ip="$3"
    fi

    op="${ip%.*}_border.${ip##*.}"
    echo convert -border "$size" -bordercolor "$color" "$ip" "$op"
}

add_border flower.png
add_border 5 insect.png
add_border red lake.png
add_border 20 blue sky.png

In the above example, echo is used to display the command that will be executed. Remove echo if you want this script to actually create new images with the given parameters. The function accepts one to three arguments and uses default values when some of the arguments are not passed. Here's the output:

$ bash functions.sh
convert -border 10 -bordercolor grey flower.png flower_border.png
convert -border 5 -bordercolor grey insect.png insect_border.png
convert -border 10 -bordercolor red lake.png lake_border.png
convert -border 20 -bordercolor blue sky.png sky_border.png

Use mogrify instead of convert if you want to modify the input image inplace instead of creating a new image. These image manipulation commands are part of the ImageMagick suite. As an exercise, modify the above function to generate error if the arguments passed do not match the expected usage. You can also accept output image name (or perhaps a different suffix) as an additional argument.

The shell script and user defined functions (which in turn might call itself or another function) can both have positional arguments. In such cases, the shell takes cares of restoring positional arguments to the earlier state once a function completes its tasks.

Functions have exit status as well, which is based on the last executed command by default. You can use the return builtin to provide your own custom exit status.

Debugging

You can use the following bash options for debugging purposes:

• -x print commands and their arguments as they are executed
• -v verbose option, print shell input lines as they are read

Here's an example with bash -x option:

$ bash -x search.sh
+ read -p 'Enter a search pattern: ' search
Enter a search pattern: xyz
+ grep -q xyz hello.sh
+ echo 'match not found'
match not found

The lines starting with + show the command being executed with expanded values if applicable (the search variable to grep -q for example). Multiple + will be used if there are multiple expansions. Here's how bash -xv would behave for the same script:

$ bash -xv search.sh
read -p 'Enter a search pattern: ' search
+ read -p 'Enter a search pattern: ' search
Enter a search pattern: xyz

if grep -q "$search" hello.sh ; then
    echo "match found"
else
    echo "match not found"
fi
+ grep -q xyz hello.sh
+ echo 'match not found'
match not found

You can also use set -x or set -v or set -xv from within the script to debug from a particular point onwards. You can turn off such debugging by using + instead of - as the option prefix (for example, set +x).

shellcheck

shellcheck is a static analysis tool that gives warnings and suggestions for scripts. You can use it online or install the tool for offline use. Given the various bash gotchas, this tool is highly recommended for both beginners and advanced users.

Consider this script:

$ cat bad_script.sh
#!/bin/bash

greeting = 'hello world'
echo "$greeting"

Here's how shellcheck reports the issue:

$ shellcheck bad_script.sh

In bad_script.sh line 3:
greeting = 'hello world'
         ^-- SC1068: Don't put spaces around the = in assignments
                     (or quote to make it literal).

For more information:
  https://www.shellcheck.net/wiki/SC1068 -- Don't put spaces around the = in ...

If the script doesn't have a shebang, you can use the -s option (shellcheck -s bash for example) to specify the shell application.

Note that shellcheck will not catch all types of issues. And suggestions should not be blindly accepted without understanding if that makes sense in the given context.

Resource lists

Here are some more learning resources:

Shell Scripting

• Bash Guide — aspires to teach good practice techniques for using Bash, and writing simple scripts
• Bash Scripting Tutorial — solid foundation in how to write Bash scripts, to get the computer to do complex, repetitive tasks for you
• bash-handbook — for those who want to learn Bash without diving in too deeply
• Serious Shell Programming — focuses on POSIX-compliant Bourne Shell for portability

Handy tools, tips and reference

• shellcheck — linting tool to avoid common mistakes and improve your script
• Bash reference cheatsheet — nicely formatted and explained well
• Bash scripting cheatsheet — quick reference to getting started with Bash scripting
• Comprehensive lists on mywiki.wooledge.org website:
  • Bash FAQ
  • Bash Practices
  • Bash Pitfalls
• Google shell style guide
• Reliability and robustness
  • safe ways to do things in bash
  • better scripting
  • robust scripting

Specific topics

• Reading file(s)
  • Reading file
  • Loop through the lines of two files in parallel
• arrays
• nameref
  • also see this FAQ
• getopts
  • getopts tutorial
  • handling command-line arguments
  • stackoverflow: getopts example
• Sending and Trapping Signals

Exercises

Use a temporary working directory before attempting the exercises. You can delete such practice directories afterwards.

1) What's wrong with the script shown below? Also, will the error go away if you use bash try.sh instead?

$ printf '    \n!#/bin/bash\n\necho hello\n' > try.sh
$ chmod +x try.sh
$ ./try.sh
./try.sh: line 2: !#/bin/bash: No such file or directory
hello

# expected output
$ ./try.sh
hello

2) Will the command shown below work? If so, what would be the output?

$ echo echo hello | bash

3) When would you source a script instead of using bash or creating an executable using shebang?

4) How would you display the contents of a variable with shake appended?

$ fruit='banana'

$ echo # ???
bananashake

5) What changes would you make to the code shown below to get the expected output?

# default behavior
$ n=100
$ n+=100
$ echo "$n"
100100

# expected output
$ echo "$n"
200

6) Is the following code valid? If so, what would be the output of the echo command?

$ declare -a colors
$ colors[3]='green'
$ colors[1]='blue'

$ echo "${colors[@]}"
# ???

7) How would you get the last three characters of a variable's contents?

$ fruit='banana'

# ???
ana

8) Will the second echo command give an error? If not, what will be the output?

$ fruits=('apple' 'fig' 'mango')
$ echo "${#fruits[@]}"
3

$ echo "${#fruits}"
# ???

9) For the given array, use parameter expansion to remove characters until first/last space.

$ colors=('green' 'dark brown' 'deep sky blue white')

# remove till first space
$ printf '%s\n' # ???
green
brown
sky blue white

# remove till last space
$ printf '%s\n' # ???
green
brown
white

10) Use parameter expansion to get the expected outputs shown below.

$ ip='apple:banana:cherry:dragon'

$ echo # ???
apple:banana:cherry

$ echo # ???
apple

11) Is it possible to achieve the expected outputs shown below using parameter expansion? If so, how?

$ ip='apple:banana:cherry:dragon'

$ echo # ???
apple 42 dragon

$ echo # ???
fig:banana:cherry:dragon

$ echo # ???
apple:banana:cherry:end

12) For the given input, change case as per the expected outputs shown below.

$ ip='This is a Sample STRING'

$ echo # ???
THIS IS A SAMPLE STRING

$ echo # ???
this is a sample string

$ echo # ???
tHIS IS A sAMPLE string

13) Why does the conditional expression shown below fail?

$ touch ip.txt
$ [[-f ip.txt]] && echo 'file exists'
[[-f: command not found

14) What is the difference between == and =~ string comparison operators?

15) Why does the conditional expression used below show failed both times? Modify the expressions such that the first one correctly says matched instead of failed.

$ f1='1234.txt'
$ f2='report_2.txt'

$ [[ $f1 == '+([0-9]).txt' ]] && echo 'matched' || echo 'failed'
failed
$ [[ $f2 == '+([0-9]).txt' ]] && echo 'matched' || echo 'failed'
failed

16) Extract the digits that follow a : character for the given variable contents.

$ item='chocolate:50'
# ???
50

$ item='50 apples, fig:100, books-12'
# ???
100

17) Modify the expression shown below to correctly report true instead of false.

$ num=12345
$ [[ $num > 3 ]] && echo 'true' || echo 'false'
false

18) Write a shell script named array.sh that accepts array input from the user followed by another input as index. Display the corresponding value at that index. Couple of examples are shown below.

$ bash array.sh
enter array elements: apple banana cherry
enter array index: 1
element at index '1' is: banana

$ bash array.sh
enter array elements: dragon unicorn centaur
enter array index: -1
element at index '-1' is: centaur

19) Write a shell script named case.sh that accepts exactly two command line arguments. The first argument can be lower, upper or swap and this should be used to transform the contents of the second argument. Examples script invocations are shown below, including what should happen if the command line arguments do not meet the script expectations.

$ ./case.sh upper 'how are you?'
HOW ARE YOU?

$ ./case.sh lower PineAPPLE
pineapple

$ ./case.sh swap 'HeLlo WoRlD'
hElLO wOrLd

$ ./case.sh lower
Error! Two arguments expected.
$ echo $?
1

$ ./case.sh upper apple fig
Error! Two arguments expected.

$ ./case.sh lowercase DRAGON
Error! 'lowercase' command not recognized.
$ echo $?
1

$ ./case.sh apple lower 2> /dev/null
$ echo $?
1

20) Write a shell script named loop.sh that displays the number of lines for each of the files passed as command line arguments.

$ printf 'apple\nbanana\ncherry\n' > items_1.txt
$ printf 'dragon\nowl\nunicorn\ntroll\ncentaur\n' > items_2.txt

$ bash loop.sh items_1.txt
number of lines in 'items_1.txt' is: 3

$ bash loop.sh items_1.txt items_2.txt
number of lines in 'items_1.txt' is: 3
number of lines in 'items_2.txt' is: 5

21) Write a shell script named read_file.sh that reads a file line by line to be passed as argument to the paste -sd, command. Can you also write a solution using the xargs command instead of a script?

$ printf 'apple\nbanana\ncherry\n' > items_1.txt
$ printf 'dragon\nowl\nunicorn\ntroll\ncentaur\n' > items_2.txt
$ printf 'items_1.txt\nitems_2.txt\n' > list.txt

$ bash read_file.sh list.txt
apple,banana,cherry
dragon,owl,unicorn,troll,centaur

$ xargs # ???
apple,banana,cherry
dragon,owl,unicorn,troll,centaur

22) Write a function named add_path which prefixes the path of the current working directory to the arguments it receives and displays the results. Examples are shown below.

$ add_path() # ???

$ cd
$ pwd
/home/learnbyexample
$ add_path ip.txt report.log
/home/learnbyexample/ip.txt /home/learnbyexample/report.log

$ cd cli-computing
$ pwd
/home/learnbyexample/cli-computing
$ add_path f1
/home/learnbyexample/cli-computing/f1

23) What do the options bash -x and bash -v do?

24) What is shellcheck and when would you use it?