Shell Scripting Master Guide

A Complete Reference from Basics to Advanced Concepts

Table of Contents

●​ Introduction to Shell Scripting

●​ Shell Types
●​ Shebang (#!)
●​ Variables & Data Types
●​ User Input
●​ Conditional Statements
●​ Loops
●​ Functions
●​ Arrays
●​ String Manipulation
●​ Command-Line Arguments
●​ File Operations
●​ Error Handling
●​ Regular Expressions
●​ Pipelines & Redirection
●​ Debugging
●​ Subshells & Process Control
●​ Environment Variables
●​ Practical Scripts

SHELL SCRIPTING

In Linux, a shell is a command-line interface that allows users to interact with the operating
system and execute various commands. It's both a user interface and a scripting language that
provides a way to communicate with the computer system through textual commands. The shell
is a type of program called an interpreter.

SHELL

●​ BASH​

●​ PYTHON​
 ●​ SH​

●​ ZSH​

Shebang

The shebang (#!) is a special sequence of characters at the beginning of a script file that tells the
operating system which interpreter should be used to execute the script. It's followed by the path
to the interpreter binary. This is particularly useful for shell scripts to ensure they're executed by
the correct shell, even if the user's default shell is different.

For example, in a Bash shell script, the shebang line would be:

#!/bin/bash

Let's break down this example:

●​ #!: This is the shebang sequence that indicates the start of the shebang line.​

●​ /bin/bash: This is the path to the Bash interpreter binary. It tells the system to use the
Bash interpreter to execute the script.​

Here are a few points to note about the shebang:

1.​ The shebang line is not required for all scripts, but it's a good practice to include it,
especially for shell scripts.​

2.​ The shebang line must be the very first line of the script file.​

3.​ The path after the shebang (/bin/bash in the example) points to the interpreter that
should be used. It should be the absolute path to the interpreter.​

4.​ The shebang line doesn't introduce a comment. It's a directive for the system to
determine how to execute the script.​

5.​ Different shells or interpreters may have their own paths in the shebang line, such as
/usr/bin/python for Python scripts or /usr/bin/env node for Node.js scripts.​

Including the appropriate shebang line ensures that your script is executed by the correct
interpreter, regardless of the default shell of the user running the script.

Basic Example of Shell Script!

#!/bin/bash​
# Prompt the user for their name​
echo "Hello! What's your name?"​
read name​
# Greet the user​
echo "Nice to meet you, $name!"

To use this script:

1.​ Save it to a file (e.g., greeting_script.sh).​

2.​ Make it executable using chmod +x greeting_script.sh.​

3.​ Run the script using ./greeting_script.sh.​

To install Docker using a shell script, you can use the following script for a Linux system. This
example is based on installing Docker on Ubuntu using the official Docker repository:

#!/bin/bash
​
# Update package index and install required dependencies​
sudo apt update​
sudo apt install -y apt-transport-https ca-certificates curl
software-properties-common
​
# Add Docker's official GPG key​
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor
-o /usr/share/keyrings/docker-archive-keyring.gpg
​
# Add Docker repository​
echo "deb [arch=$(dpkg --print-architecture)
signed-by=/usr/share/keyrings/docker-archive-keyring.gpg]
https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo
tee /etc/apt/sources.list.d/docker.list > /dev/null
 ​
# Install Docker​
sudo apt update​
sudo apt install -y docker-ce docker-ce-cli containerd.io

# Add the current user to the 'docker' group to run Docker commands without
'sudo'​
sudo usermod -aG docker $USER
​
# Start and enable Docker service​
sudo systemctl start docker​
sudo systemctl enable docker
​
# Print Docker version​
docker --version

Explanation:

1.​ Shebang (#!/bin/bash): Specifies the shell to be used for interpreting the script.​

2.​ Update Package Index: Updates the package index and installs necessary
dependencies.​

3.​ Add Docker's GPG Key: Adds Docker's official GPG key for secure package installation.​

4.​ Add Docker Repository: Adds Docker's official repository to the system's sources list.​

5.​ Install Docker: Installs Docker Community Edition (CE) packages.​

6.​ Add User to Docker Group: Adds the current user to the 'docker' group to run Docker
commands without needing 'sudo'.​

7.​ Start and Enable Docker: Starts the Docker service and enables it to start on system
boot.​

8.​ Print Docker Version: Prints the installed Docker version.​

To use the script:

1.​ Save it to a file (e.g., install_docker.sh).​

2.​ Make it executable using chmod +x install_docker.sh.​

3.​ Run the script with superuser privileges using sudo ./install_docker.sh.​
Please note that this script is tailored for Ubuntu. If you're using a different Linux distribution, you
might need to adjust the package management commands and repository configuration
accordingly. Always verify scripts from reliable sources and review them before execution,
especially when dealing with system-level changes.

SHELL-SCRIPTING HANDS-ON

Variables and Data Types in Shell Scripting, Along with Examples:

1.​ Declaring Variables: In shell scripting, you can declare variables without specifying
their data types. Variables are case-sensitive and can consist of letters, numbers, and
underscores, but they must start with a letter or underscore. Here's how you declare a
variable:​

variable_name=value

2.​ Variable Types: Shell scripting doesn't have strict data types like other programming
languages. Variables are treated as strings by default, but you can manipulate them to
behave as different types.​

3.​ Variable Assignment and Manipulation: You can assign values to variables using the
assignment operator =. Here are examples of various types of variable assignments and
manipulations:​

●​ String Variable:​

name="John"​
echo "Hello, $name!" # Output: Hello, John!

●​ Integer Variable:​

age=25​
echo "Age: $age years" # Output: Age: 25 years
 ●​ Arithmetic Operations:​

x=10​
y=5​
sum=$((x + y))​
echo "Sum: $sum" # Output: Sum: 15​

●​ Concatenation:​

greeting="Hello"​
subject="World"​
message="$greeting, $subject!"​
echo $message # Output: Hello, World!

●​ String Length:​

string="Shell Scripting"​
length=${#string}​
echo "Length: $length" # Output: Length: 15

●​ Substring Extraction:​

substring=${string:0:4} # Extracts first 4 characters​

echo "Substring: $substring" # Output: Substring: Shell

4.​ Command Substitution: You can capture the output of a command and assign it to a
variable using command substitution. There are two ways to do this:​

●​ Using Backticks:​

current_date=`date`​
echo "Current date: $current_date"

●​ Using $(...):​

current_time=$(date +%H:%M:%S)​
echo "Current time: $current_time"

5.​ Readonly Variables: You can declare variables as readonly to prevent their values from
being changed after initial assignment:​

readonly pi=3.14159​
pi=3.14 # This will result in an error

6.​ Unsetting Variables: You can unset (delete) a variable using the unset command:​

bash

unset variable_name

7.​ Quoting Variables: Quoting variables properly is essential to preserve spaces and
special characters:​

variable="Hello World"​
echo "Using double quotes: $variable"​
echo 'Using single quotes: $variable'​
echo Using no quotes: $variable​
echo "Using double quotes: '$variable'"

8.​ Escaping Characters: If you need to include special characters within a variable, you
can escape them using backslashes:​
 special_char="\$"​
echo "Variable: $special_char" # Output: Variable: $

Reading User Input and Input Validation in Shell Scripts

Shell scripts are a series of commands that are executed in a sequence. User input allows
scripts to interact with users and make decisions based on that input. Input validation ensures
that the provided input meets certain criteria.

Reading User Input

To read user input in a shell script, you can use the read command. It reads input from the user
until the Enter key is pressed and assigns the input to a variable. Here's an example:

#!/bin/bash​
# Prompt the user for their name​
echo "Please enter your name:"​
read name​
# Display a greeting using the user's input​
echo "Hello, $name! Nice to meet you."

In this example, the user's input is stored in the name variable, and the script uses that input to
display a greeting.

Input with Prompting

You can also use the read command with a prompt message directly, eliminating the need for
separate echo commands:

#!/bin/bash​
# Read input with a prompt message​
read -p "Enter your favorite color: " color​
echo "Your favorite color is $color."​

Using read Options

The read command has various options to customize its behavior. For example:

●​ -p specifies a prompt message.​

 ●​ -r disables interpreting backslashes, useful for reading file paths.​

●​ -t sets a timeout for input.​

●​ -s hides input (useful for passwords).​

Example of reading a password without echoing characters:

#!/bin/bash​
# Read password without echoing characters​
read -s -p "Enter your password: " password​
echo "Password entered."​

Example of reading input with a timeout:

# Read input with timeout​

read -t 5 -p "Enter something in 5 seconds: " timed_input​
echo "You entered: $timed_input"​

Conditional Statements

Conditional statements allow your script to make decisions and execute different code paths
based on certain conditions.

1.​ if, else, elif:​

The if statement is used to execute code block(s) conditionally. You can use else to define
what should be done if the condition is not met, and elif to add more conditions.

#!/bin/bash​
num=10​
if [ $num -gt 10 ]; then​
echo "Number is greater than 10"​
elif [ $num -eq 10 ]; then​
echo "Number is equal to 10"​
else​
echo "Number is less than 10"​
fi
 2.​ Case Statements (Switch):​

The case statement allows you to compare a variable against multiple values and execute code
based on the match.

#!/bin/bash​
fruit="apple"​
case $fruit in​
"apple")​
echo "It's an apple"​
;;​
"banana")​
echo "It's a banana"​
;;​
"orange")​
echo "It's an orange"​
;;​
*)​
echo "Unknown fruit"​
;;​
esac

Looping

Looping structures help you repeat a set of commands multiple times.

1.​ for Loop:​

The for loop iterates over a list of items and performs the specified commands for each item.
​

#!/bin/bash​
for color in red green blue; do​
echo "Color: $color"​
done

2.​ while Loop:​

The while loop repeatedly executes a block of code as long as a condition is true.

#!/bin/bash​
count=1​
while [ $count -le 5 ]; do​
echo "Count: $count"​
((count++))​
done

3.​ until Loop:​

The until loop is similar to the while loop but continues executing until a condition becomes
true.

#!/bin/bash​
num=0​
until [ $num -ge 5 ]; do​
echo "Number: $num"​
((num++))​
done

Declaring and Using Functions:

In shell scripting, functions allow you to group a series of commands together and give
them a name, making your code more organized and modular. To declare a function, you
use the following syntax:

function_name() { ​
# Commands to be executed ​
}
Here's an example of a simple function that prints a greeting:

# Declare a function named greet ​

greet() { ​
echo "Hello, how are you?" ​
} ​
# Call the greet function ​
greet

Function Arguments and Return Values:

Shell functions can accept arguments just like command-line scripts. You access these
arguments using special variables: $1 for the first argument, $2 for the second, and so
on. To access all the arguments, you use $@ or $*.

Here's an example of a function that takes two arguments and prints them:

# Declare a function named print_args ​

print_args() { ​
echo "First argument: $1" ​
echo "Second argument: $2" ​
} ​
# Call the print_args function with arguments ​
print_args "Hello" "World"

Shell functions can't directly return values like functions in some other programming
languages. However, you can use the exit status of a function to convey success (0) or
failure (non-zero). If you need to pass values back from a function, you can print them and
capture the output using command substitution.

Here's an example of a function that calculates the sum of two numbers and returns it
through the exit status and output:

# Declare a function named calculate_sum ​

calculate_sum() { ​
local num1="$1" ​
local num2="$2" ​
local sum=$((num1 + num2)) ​
echo "$sum" ​
return $sum ​
 } ​
# Call the calculate_sum function and capture the output ​
result=$(calculate_sum 10 20) ​
echo "Sum: $result"​

Scope of Variables:
Variables declared within a function have local scope, meaning they're only accessible
within that function. To create local variables, use the local keyword. Variables declared
outside functions have global scope and can be accessed from anywhere.

Here's an example illustrating local and global variables:

# Declare a global variable ​

global_var="I'm global" ​
# Declare a function with local variables ​
local_variables() { ​
local local_var="I'm local" ​
echo "Inside function: $local_var" ​
echo "Inside function: $global_var" ​
} ​
# Call the local_variables function ​
local_variables ​
# Access global variable outside the function ​
echo "Outside function: $global_var" ​
# Attempting to access local_var here will result in an error

In the example above, the local_var is accessible only within the local_variables
function, while global_var is accessible both inside and outside the function.

Remember that each instance of a function call has its own set of local variables,
ensuring that they don't interfere with each other.

String Manipulation in Shell Scripts:

String manipulation in shell scripting involves various operations on strings, such as
concatenation, substring extraction, searching, and replacing. These operations are
commonly used when working with text data in scripts. Let's explore each operation with
examples:

1.​ Concatenation: Concatenation involves combining two or more strings to create a

new string. In shell scripting, you can achieve this using variables and the
concatenation operator (.).​
#!/bin/bash ​
string1="Hello, " ​
string2="World!" ​
result=$string1$string2​
echo "Concatenated string: $result"

Output:

Concatenated string: Hello, World!

2.​ Substring Extraction: You can extract a portion of a string using substrings. In
shell scripting, you use parameter expansion to achieve this.​

#!/bin/bash ​
string="Hello, World!" ​
substring=${string:7:5} # Starting from index 7, extract 5 characters​
echo "Substring: $substring"

Output:

Substring: World

3.​ Searching and Replacing: Searching involves finding a specific substring within a
string, and replacing involves substituting one substring with another.​

#!/bin/bash ​
string="Hello, World! Hello!" ​
search="Hello" ​
replace="Hi" ​
result=${string//$search/$replace} # Replace all occurrences ​
echo "Original string: $string" ​
echo "Result after replacement: $result"

Output:
 Original string: Hello, World! Hello!​
Result after replacement: Hi, World! Hi!

You can also use parameter expansion to replace the first occurrence or perform
case-insensitive replacements. Here's a script that combines all three operations:

#!/bin/bash ​
string1="Hello, " ​
string2="World!" ​
concatenated=$string1$string2 ​
original="Hello, World! Hello!" ​
search="Hello" ​
replace="Hi" ​
replaced=${original//$search/$replace} ​
echo "Concatenated string: $concatenated" ​
echo "Original string: $original" ​
echo "Replaced string: $replaced"​
substring=${replaced:0:5} ​
echo "Extracted substring: $substring"

Output:

Concatenated string: Hello, World!​

Original string: Hello, World! Hello!​
Replaced string: Hi, World! Hi!​
Extracted substring: Hi, W

Arrays
1.​ Declaring and Using Arrays: In shell scripting, you can declare an array by
assigning values to consecutive indices. Arrays in shell scripts are 0-indexed.​

#!/bin/bash ​
# Declare an array with values ​
fruits=("Apple" "Banana" "Orange") ​
# Access array elements ​
echo "First fruit: ${fruits[0]}" ​
echo "Second fruit: ${fruits[1]}" ​
echo "Third fruit: ${fruits[2]}"​

Output:

First fruit: Apple​

Second fruit: Banana​
Third fruit: Orange

2.​ Looping Through Arrays: You can use loops to iterate through array elements.​

#!/bin/bash ​
fruits=("Apple" "Banana" "Orange") ​
# Loop through array using for loop ​
echo "Using for loop:" ​
for fruit in "${fruits[@]}"; do ​
echo "Fruit: $fruit" ​
done ​
# Loop through array using while loop and index ​
echo "Using while loop:" ​
index=0 ​
while [ $index -lt ${#fruits[@]} ]; do ​
echo "Fruit at index $index: ${fruits[$index]}" ​
index=$((index + 1)) ​
done

Output:

Using for loop:​

Fruit: Apple​
Fruit: Banana​
Fruit: Orange​
​
Using while loop:​
Fruit at index 0: Apple​
Fruit at index 1: Banana​
Fruit at index 2: Orange​
3. Array Manipulation: You can modify arrays by adding, removing, and updating
elements.​

#!/bin/bash ​
fruits=("Apple" "Banana" "Orange") ​
# Adding an element ​
fruits+=("Grapes") ​
# Updating an element ​
fruits[1]="Mango" ​
# Removing an element ​
unset fruits[0] ​
# Display the modified array ​
echo "Modified array:" ​
for fruit in "${fruits[@]}"; do ​
echo "Fruit: $fruit" ​
done

Output:

Modified array:​
Fruit: Mango​
Fruit: Orange​
Fruit: Grapes

Note: Some shell variants, like sh, might not support all these features, especially more
advanced array manipulation. For extensive array manipulation, consider using a shell
like bash.

Shell Scripting with Command-Line Arguments:

Command-line arguments are values or options provided to a script or program when it's
executed. Shell scripts can access these arguments to customize their behavior. Here's
how you can work with command-line arguments in shell scripting, along with examples
and explanations:

Accessing Command-Line Arguments:

In a shell script, command-line arguments are accessible using special variables:

●​ $0 represents the script's name itself.​

 ●​ $1, $2, ... represent the first, second, and so on, arguments.​

●​ $# gives the total number of arguments.​

●​ $@ represents all the arguments as a list.​

●​ $* represents all the arguments as a single string.​

Example: Script to Print Command-Line Arguments

#!/bin/bash ​
echo "Script name: $0" ​
echo "First argument: $1" ​
echo "Second argument: $2" ​
echo "Total number of arguments: $#" ​
echo "All arguments as list: $@" ​
echo "All arguments as string: $*"

Executing the Script:

Assuming the script is saved as args_script.sh:

$ chmod +x args_script.sh ​
$ ./args_script.sh arg1 arg2 arg3

Output:

Script name: ./args_script.sh​

First argument: arg1​
Second argument: arg2​
Total number of arguments: 3​
All arguments as list: arg1 arg2 arg3​
All arguments as string: arg1 arg2 arg3​

Argument Parsing Libraries:

When command-line arguments become complex, using argument parsing libraries can
simplify the process of handling arguments, options, and flags. These libraries offer
features like long and short option parsing, default values, help messages, and more.
Some popular libraries include:
 ●​ getopt​

●​ getopts​

●​ argparse (Python-based library usable in shell scripts)​

Error Handling
Error handling in shell scripting is crucial to ensure your scripts handle unexpected
situations gracefully. This involves dealing with exit codes, error messages, and trapping
signals. Let's break down each of these aspects with examples and outputs:

1. Exit Codes:

Exit codes are numeric values returned by commands upon completion. Conventionally,
an exit code of 0 indicates success, while non-zero values indicate errors.

Example Program (example_exit_codes.sh):

#!/bin/bash ​
echo "Starting script..." ​
ls /nonexistent-directory ​
if [ $? -eq 0 ]; then ​
echo "Directory exists." ​
else ​
echo "Directory does not exist." ​
fi ​
echo "Script finished."​

Output:

Starting script... ​
ls: cannot access '/nonexistent-directory': No such file or directory ​
Directory does not exist. ​
Script finished.

In this example, the ls command fails to list the contents of a nonexistent directory. The
exit code is checked using $?, and the script handles the failure by printing an error
message.

2. Error Messages:

Custom error messages help users understand what went wrong. You can use echo to
print error messages along with relevant information.

Example Program (example_error_messages.sh):

 #!/bin/bash ​
file="nonexistent-file.txt" ​
if [ ! -f "$file" ]; then ​
echo "Error: File '$file' does not exist." ​
exit 1 ​
fi ​
echo "File '$file' exists."

Output:

Error: File 'nonexistent-file.txt' does not exist.

Here, the script checks for the existence of a file. Since the file doesn't exist, an error
message is printed, and the script exits with an exit code of 1.

3. Trap for Handling Signals:

The trap command allows you to specify actions to be taken when certain signals are
received, like when the script receives Ctrl+C (SIGINT).

Example Program (example_trap.sh):

#!/bin/bash ​
cleanup() { ​
echo "Cleaning up..." ​
# Additional cleanup steps can be added here ​
exit 1 ​
} ​
trap cleanup INT ​
echo "Running..." ​
sleep 10

Output:

Running... ​
^C​
Cleaning up...

In this example, the script sets up a cleanup function using the trap command. When the
script receives the SIGINT signal (Ctrl+C), the cleanup function is executed, allowing you
to perform cleanup actions before exiting.

These examples showcase how to handle errors in shell scripts using exit codes, error
messages, and signal trapping. Effective error handling improves the reliability and
usability of your scripts, making them more robust in handling unexpected situations.

Regular Expressions
Regular expressions (regex or regexp) are powerful tools for pattern matching and text
manipulation. They allow you to define complex patterns to search for and manipulate
strings based on certain criteria.

Regular expressions are used in various Unix command-line tools like grep, sed, and awk
to perform pattern matching and text transformations.

Let's dive into each tool with examples:

1. grep:

grep is a command-line tool used to search for specific patterns in text files.

Example: Searching for a Pattern Suppose you have a file named sample.txt with the
following content:

apple

banana

cherry

date

grape

You can use grep to search for lines containing the word "banana":

grep "banana" sample.txt

Output:

banana

2. sed:

sed (Stream Editor) is a command-line tool for performing basic text transformations on
an input stream.

Example: Replacing a Pattern Suppose you have the same sample.txt file. You can use
sed to replace all occurrences of "cherry" with "orange":

bash

sed 's/cherry/orange/' sample.txt​

Output:

apple

banana

orange

date

grape

3. awk:

awk is a versatile tool for text processing and reporting. It can also handle pattern
matching and manipulation.

Example: Extracting Specific Fields Suppose you have a file named data.txt with the
following content:

John 25

Alice 30

Bob 28

Eve 22

You can use awk to print only the names of people who are older than 25:

awk '$2 > 25 { print $1 }' data.txt

Output:

Alice

Bob
Using Regular Expressions
Regular expressions add more flexibility and power to these tools.

Example: Using grep with Regular Expression Suppose you have a file named
emails.txt with email addresses, and you want to find all Gmail addresses:

grep "@gmail\.com" emails.txt

Output:

john@gmail.com

alice@gmail.com

In this example, the dot (.) is a special character in regex, so we escape it with a
backslash (\) to match a literal dot.

Example: Using sed with Regular Expression Suppose you want to replace all
occurrences of dates in the format "dd/mm/yyyy" with "mm/dd/yyyy" in a file named
dates.txt:

sed 's/\([0-9]\{2\}\)\/\([0-9]\{2\}\)\/\([0-9]\{4\}\)/\2\/\1\/\3/'
dates.txt

Example: Using awk with Regular Expression Suppose you have a file named log.txt
with lines containing dates and events. You want to print lines with events that contain the
word "error":

awk '/error/ { print }' log.txt

Pipeline and Redirection in Shell Scripts

In shell scripting, pipelines and redirection are powerful concepts that allow you to
manipulate input and output streams of commands to achieve more complex and flexible
operations.

1. Piping Commands Together:

Piping commands involves sending the output of one command as the input to another
command. This is achieved using the | (pipe) operator.
Example 1: List files and directories, and then filter for specific files using grep:

bash

ls -l | grep ".txt"

In this example, the ls -l command lists files and directories in long format, and the
output is piped to grep to filter and display only the lines containing .txt.

Example 2: Count the number of lines in a file using wc:

cat file.txt | wc -l

Here, cat reads the content of the file, and its output is piped to wc -l, which counts the
number of lines.

2. Redirecting Input and Output:

Redirection involves changing the source or destination of input or output streams of a

command. The operators used for redirection are > (output) and < (input).

Example 1: Redirect Output to a File:

ls > file_list.txt

This redirects the output of the ls command to the file file_list.txt instead of
printing it to the terminal.

Example 2: Append Output to a File:

echo "New content" >> file_list.txt

The >> operator appends the output to the end of the file.

Example 3: Redirect Input from a File:

sort < unsorted.txt > sorted.txt

This takes the content of unsorted.txt as input to the sort command and redirects the
sorted output to sorted.txt.

Example 4: Combining Redirection and Piping:

cat file.txt | grep "keyword" > filtered.txt

This combines piping and redirection. The cat command reads the content of file.txt,
pipes it to grep to filter lines containing "keyword", and then redirects the filtered output
to filtered.txt.

Example 5: Using /dev/null to Discard Output:

ls non_existent_folder 2> /dev/null

The 2> operator redirects the error output to /dev/null, effectively discarding the error
message.

Example 6: Redirecting Input from Here Document:

grep -f - << EOF ​

pattern1 ​
pattern2 ​
EOF

This uses a here document to provide input to the grep command. The -f - flag tells
grep to use patterns from standard input.

Outputs:
Let's see the outputs for some of the examples:

Example 1:

$ ls -l | grep ".txt"​
-rw-r--r-- 1 user user 123 Aug 14 10:00 file1.txt​
-rw-r--r-- 1 user user 456 Aug 14 11:00 file2.txt

Example 2:
$ cat file.txt | wc -l

10

Example 3:

$ sort < unsorted.txt > sorted.txt

Example 4:

$ cat file.txt | grep "keyword" > filtered.txt

Example 5:

$ ls non_existent_folder 2> /dev/null

Example 6:

$ grep -f - << EOF

> pattern1

> pattern2

> EOF

In these examples, you can see how piping and redirection are used to manipulate input
and output streams, making your shell scripts more versatile and efficient.

Shell Script Debugging:

Debugging shell scripts involves identifying and fixing errors in your scripts.
Proper debugging techniques can help you locate issues quickly and efficiently.
Two common techniques for debugging shell scripts are printing/debugging
techniques and using set -x and set -e.

1. Printing/Debugging Techniques:

One of the simplest ways to debug shell scripts is to use print statements to
display variable values, execution paths, and other relevant information. You can
use echo, printf, or logger commands to print messages to the console or system
logs.
Example Script:

#!/bin/bash ​
name="John" ​
age=25 ​
echo "Debug: Starting script..." ​
echo "Debug: Name is $name" ​
echo "Debug: Age is $age" ​
result=$((age * 2)) ​
echo "Debug: Result is $result" ​
echo "Debug: Script completed."​

Output:

Debug: Starting script...

Debug: Name is John
Debug: Age is 25
Debug: Result is 50
Debug: Script completed.

While this technique is simple, it can become overwhelming for larger scripts and
might not provide a structured view of the script's execution flow.

2. Using set -x and set -e:

The set -x command, when added at the beginning of a script, enables the debugging
mode. It prints each line before executing it, allowing you to see the exact
commands being executed.​
The set -e command, when added, makes the script exit immediately if any command
returns a non-zero exit status. This helps catch errors early in the script
execution.

Example Script:

#!/bin/bash ​
set -x ​
set -e ​
name="John" ​
age=25 ​
echo "Debug: Starting script..." ​
echo "Debug: Name is $name" ​
echo "Debug: Age is $age"​
result=$((age * 2)) ​
echo "Debug: Result is $result" ​
# Introducing an intentional error to demonstrate set -e ​
ls /nonexistent_directory ​
echo "Debug: Script completed."
Output:

+ name=John ​
+ age=25 ​
+ echo 'Debug: Starting script...' ​
Debug: Starting script... ​
+ echo 'Debug: Name is John' ​
Debug: Name is John ​
+ echo 'Debug: Age is 25' ​
Debug: Age is 25 ​
+ result=50 ​
+ echo 'Debug: Result is 50' ​
Debug: Result is 50 ​
+ ls /nonexistent_directory ​
ls: cannot access '/nonexistent_directory': No such file or directory

In this example, the set -x command shows each executed command with a + sign, and
the set -e command causes the script to exit immediately after the ls command
fails.

Notes:

●​ While set -x and set -e are powerful debugging tools, they may not be
suitable for all scenarios. For instance, set -e might lead to unexpected
exits if you're intentionally handling errors.​

●​ To turn off debugging mode, use set +x.​

●​ Debugging tools like set -x and set -e are usually used for development and
testing purposes. In production, it's recommended to minimize debugging
outputs and handle errors more gracefully.​

Remember that effective debugging involves understanding the script's logic,

carefully analyzing error messages, and iteratively refining your code based on
the feedback you receive.

Subshells and Process Control:

In Unix-like operating systems, a subshell is a separate instance of the shell
that is spawned to execute a command or a group of commands. Subshells are useful
for various purposes, including isolating variables and managing process control.
Additionally, process control involves managing background and foreground
processes to efficiently multitask and manage system resources.

Running Commands in a Subshell:

Running commands in a subshell is achieved by enclosing the commands within

parentheses ( ). This creates a new shell instance to execute the commands.

Example Script 1: Running Commands in a Subshell

#!/bin/bash​
# Running commands in a subshell ​
echo "Current working directory: $(pwd)" ​
echo "Number of files in /tmp: $(ls /tmp | wc -l)"

Output:

Current working directory: /home/user

Number of files in /tmp: 10

In the example above, the $( ) syntax is used to run the commands within a
subshell. The output of the commands is captured and inserted into the echo
statements.

Background and Foreground Processes:

Processes can run in the background or foreground. Background processes run

independently of the shell, allowing you to continue using the shell for other
tasks. Foreground processes run in the shell itself and typically require user
interaction.

Example Script 2: Running a Background Process

#!/bin/bash ​
# Running a command in the background ​
sleep 5 & ​
echo "Background process started." ​
# Wait for the background process to finish ​
wait ​
echo "Background process completed."​

Output:

Background process started.

[1]+ Done sleep 5
Background process completed.

In the example above, the & symbol is used to run the sleep 5 command in the
background. The wait command ensures that the script waits for the background
process to complete before proceeding.

Example Script 3: Running a Foreground Process

#!/bin/bash ​
# Running a command in the foreground ​
echo "Enter your name:" ​
read name ​
echo "Hello, $name!"

Output:

Enter your name:

John
Hello, John!

In the example above, the script prompts the user for their name using the read
command, which requires user interaction. This is an example of a foreground
process.

Combining Subshells and Process Control:

Subshells can be combined with process control techniques to manage complex

scenarios.

Example Script 4: Combining Subshells and Process Control

#!/bin/bash ​
# Running commands in a subshell and background process ​
( ​
echo "Subshell working directory: $(pwd)" ​
sleep 3 & ​
echo "Subshell background process started." ​
wait ​
echo "Subshell background process completed." ​
) ​
echo "Main shell continues."

Output:

Subshell working directory: /home/user

Subshell background process started.
[1]+ Done sleep 3
Subshell background process completed.
Main shell continues.
In the example above, the subshell is used to run commands, including a background
process. The main shell continues executing while the subshell is running. This
demonstrates the isolation and concurrent execution of subshells.

In summary, subshells allow you to isolate commands and variables within a

separate shell instance. Process control techniques like running processes in the
background or foreground enhance the efficiency and usability of shell scripts. By
combining these concepts, you can create powerful and flexible shell scripts for
various tasks.

Environment and Configuration:

Environment Variables and Their Usage:

Environment variables are dynamic values that affect the behavior of processes
running on a system. They provide a way to pass information from the shell to
processes when they are created. Here are some commonly used environment variables
and their usage:

●​ PATH: Contains a colon-separated list of directories where the shell looks

for executable files. It determines which commands can be run without
specifying their full path.​

●​ HOME: Points to the current user's home directory.​

●​ USER or LOGNAME: Represents the current username.​

●​ SHELL: Specifies the default shell for the user.​

●​ PWD: Holds the current working directory.​

●​ LANG or LC_ALL: Determines the language and locale settings for the user
interface.​

●​ PS1: Defines the primary prompt string used by the shell.​

●​ PS2: Defines the secondary prompt string used when input spans multiple
lines.​

Example:

echo "PATH: $PATH"

echo "Home Directory: $HOME"
echo "Username: $USER"
echo "Current Directory: $PWD"
echo "Language: $LANG"
echo "Primary Prompt: $PS1"

Sourcing Configuration Files:

Configuration files are used to set environment variables and customize behavior
for specific applications or the entire system. The source or . command in the
shell is used to execute commands from a file within the current shell session.
Common configuration files include .bashrc, .bash_profile, and /etc/profile.

Example .bashrc content:

export MY_VARIABLE="Hello, World!"

Example:

source .bashrc ​
echo $MY_VARIABLE

Example using bc for floating-point math:

num1=10.5 ​
num2=3.2 ​
result=$(echo "$num1 + $num2" | bc) ​
echo "Result: $result"

Interview Scripts
BACKUP DIRECTORY:

Shell script that you can use to backup a directory using the tar command:

#!/bin/bash ​
# Source directory to backup ​
source_dir="/path/to/source/directory" ​
# Backup destination directory ​
backup_dir="/path/to/backup/directory" ​
# Backup filename with date ​
backup_filename="backup_$(date +%Y%m%d%H%M%S).tar.gz" ​
# Create the backup directory if it doesn't exist ​
mkdir -p "$backup_dir" ​
# Create the backup using tar ​
tar -czvf "$backup_dir/$backup_filename" "$source_dir" ​
# Check if the backup was successful ​
if [ $? -eq 0 ]; then ​
echo "Backup successful: $backup_filename created in $backup_dir" ​
else ​
echo "Backup failed" ​
fi​

Make sure to replace /path/to/source/directory with the actual path of the

directory you want to backup and /path/to/backup/directory with the desired
backup destination. You can save this script to a file (e.g., backup_script.sh),
give it execute permissions (chmod +x backup_script.sh), and then run it using
./backup_script.sh. This script creates a compressed tar archive of the source
directory and saves it in the backup directory with a filename that includes the
current date and time. After the backup is created, it provides feedback on
whether the backup was successful or not.

DEPLOYMENT-SCRIPT:

Automating deployment using a shell script is a common practice in software

development to streamline the deployment process and ensure consistency. You can
use a shell script to automate tasks like pulling code from a repository, building
the project, and deploying it to a server. Here's a general outline of how you
could structure such a shell script:

#!/bin/bash ​
# Define variables ​
REPO_URL="https://github.com/yourusername/yourrepository.git" ​
TARGET_DIR="/path/to/deployment/directory" ​
BRANCH="main" # or the branch you want to deploy ​
BUILD_DIR="/path/to/build/directory" ​
# Update the code from the repository ​
echo "Updating code from the repository..." ​
cd "$TARGET_DIR" || exit ​
git pull origin "$BRANCH" ​
# Build the project (if needed) ​
echo "Building the project..." ​
cd "$BUILD_DIR" || exit ​
# Insert build commands here ​
# Deploy the project ​
echo "Deploying the project..." ​
# Insert deployment commands here ​
echo "Deployment complete!"

FIND & REPLACE IN FILES:

Shell script to perform a find and replace operation in multiple files using tools
like sed (stream editor) or grep (with Perl regex). Below is an example using sed
to perform a find and replace operation:

#!/bin/bash ​
# Define variables ​
 SEARCH_STRING="old_string" ​
REPLACE_STRING="new_string" ​
FILES_DIR="/path/to/files/directory" ​
# Perform find and replace using sed ​
find "$FILES_DIR" -type f -exec sed -i "s/$SEARCH_STRING/$REPLACE_STRING/g" {} + ​
echo "Find and replace complete!"

Monitoring System Resources with Shell:

Shell script to monitor system resources such as CPU usage, memory usage, disk
space, and more. The script can use built-in Unix commands like top, free, df, and
others to gather information about system resources. Here's an example of a simple
shell script to monitor system resources:

#!/bin/bash ​
while true; do ​
clear # Clear the terminal ​
echo "System Resource Monitoring" ​
echo "--------------------------" ​
# Display CPU usage ​
echo "CPU Usage:" ​
top -n 1 -b | grep "Cpu" ​
# Display memory usage ​
echo -e "\nMemory Usage:" ​
free -h ​
# Display disk space usage ​
echo -e "\nDisk Space Usage:" ​
df -h ​
sleep 5 # Wait for 5 seconds before the next update ​
done

Here's what this script does:

1.​ Shebang (#!/bin/bash): Specifies the shell to be used for interpreting the
script.​

2.​ while Loop: Creates an infinite loop that repeatedly gathers and displays
system resource information.​

3.​ clear: Clears the terminal screen for a cleaner display.​

4.​ Display CPU Usage: Uses the top command with the -n 1 flag to display a
single iteration of CPU usage information. The grep "Cpu" command filters
out the relevant line.​
 5.​ Display Memory Usage: Uses the free command with the -h flag to display
memory usage in a human-readable format.​

6.​ Display Disk Space Usage: Uses the df command with the -h flag to display
disk space usage in a human-readable format.​

7.​ sleep 5: Pauses the loop for 5 seconds before gathering and displaying
resource information again.​

To use the script, save it to a file (e.g., monitor_resources.sh), make it

executable using chmod +x monitor_resources.sh, and then run it using
./monitor_resources.sh. The script will continually update and display system
resource information in the terminal. Keep in mind that this script is a basic example
and may not provide real-time or highly detailed resource monitoring. For more
comprehensive and advanced resource monitoring, you might consider using
specialized tools like htop, nmon, sysstat, or other monitoring solutions.

Parsing & Text Processing

Let's say you have a text file named data.txt with the following content:

John Doe|25|Engineer ​
Jane Smith|30|Designer ​
Michael Johnson|28|Manager

And you want to parse and process this data using a shell script:

#!/bin/bash​
# Read the text file line by line​
while IFS='|' read -r name age profession; do​
echo "Name: $name"​
echo "Age: $age"​
echo "Profession: $profession"​
echo "---"​
done < data.txt
Explanation:

1.​ Shebang (#!/bin/bash): Specifies the shell to be used for interpreting the
script.​

2.​ while Loop: Reads the text file line by line using the read command. The
IFS='|' sets the input field separator to | so that the line is split into fields
based on the | character.​

3.​ read -r name age profession: Reads the fields from each line and assigns
them to the variables name, age, and profession.​

4.​ Display Information: Displays the extracted information using echo.​

5.​ < data.txt: Redirects the content of data.txt to the while loop's input. When
you run the script, it will output:​

Name: John Doe ​

Age: 25 ​
Profession: Engineer ​
--- ​
Name: Jane Smith ​
Age: 30 ​
Profession: Designer ​
--- ​
Name: Michael Johnson ​
Age: 28 ​
Profession: Manager ​
---

This is a simple example of parsing and processing a text file. Depending on your
needs, you can include more advanced logic within the loop to perform actions like
filtering data, performing calculations, or updating the file content.

Automation User Management with Shell Script

Automating user management tasks using a shell script can save time and ensure
consistency when dealing with user accounts on a Unix-like system. Below is an
example of a shell script that demonstrates how to automate user creation,
modification, and deletion:

#!/bin/bash​
# Define variables​
ACTION="$1" # First argument: create, modify, or delete​
USERNAME="$2" # Second argument: username​
​
# Function to create a new user​
create_user() {​
if [ -z "$USERNAME" ]; then​
echo "Usage: $0 create <username>"​
exit 1​
fi​
useradd -m -s /bin/bash "$USERNAME"​
echo "User $USERNAME created."​
}​
​
# Function to modify an existing user​
modify_user() {​
if [ -z "$USERNAME" ]; then​
echo "Usage: $0 modify <username>"​
exit 1​
fi​
usermod -s /bin/bash "$USERNAME"​
echo "User $USERNAME modified."​
}​
​
# Function to delete an existing user​
delete_user() {​
if [ -z "$USERNAME" ]; then​
echo "Usage: $0 delete <username>"​
exit 1​
fi​
userdel -r "$USERNAME"​
echo "User $USERNAME deleted."​
}​
​
# Main script​
case "$ACTION" in​
create)​
 create_user​
;;​
modify)​
modify_user​
;;​
delete)​
delete_user​
;;​
*)​
echo "Usage: $0 {create|modify|delete} <username>"​
exit 1​
;;​
esac

Explanation:

1.​ Shebang (#!/bin/bash): Specifies the shell to be used for interpreting the
script.​

2.​ Variables: The script takes two arguments: an action (create, modify, or
delete) and a username.​

3.​ Functions: Separate functions are defined for each user management action
(create, modify, delete). Each function includes relevant commands to
perform the desired action using useradd, usermod, and userdel.​

4.​ Case Statement: The main part of the script uses a case statement to
determine which user management action to perform based on the provided
argument ($ACTION). It calls the corresponding function depending on the
action.​

To use the script, save it to a file (e.g., manage_users.sh), make it executable

using chmod +x manage_users.sh, and then you can run it with the desired
action and username as arguments. For example:

./manage_users.sh create johndoe ​

./manage_users.sh modify johndoe ​
./manage_users.sh delete johndoe
Be cautious when performing user management tasks, especially deletions, as they
can have significant consequences. Always test your script in a controlled
environment before using it in a production environment.

DATABASE-BACKUP

Automating database backups using a shell script is a common practice to ensure

data integrity and disaster recovery. The example below demonstrates how to create
a shell script to automate the backup of a MySQL database. This example assumes
you have the mysqldump tool available, which is commonly used to create database
backups.

#!/bin/bash​
# Database credentials​
DB_USER="your_username"​
DB_PASS="your_password"​
DB_NAME="your_database"​
​
# Backup directory​
BACKUP_DIR="/path/to/backup/directory"​
​
# Date format for the backup file​
DATE=$(date +"%Y-%m-%d_%H-%M-%S")​
BACKUP_FILE="$BACKUP_DIR/$DB_NAME-backup-$DATE.sql"​
​
# Create backup directory if it doesn't exist​
mkdir -p "$BACKUP_DIR"​
​
# Perform database backup using mysqldump​
mysqldump -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" > "$BACKUP_FILE"​
​
# Check if the backup was successful​
if [ $? -eq 0 ]; then​
echo "Database backup successful: $BACKUP_FILE"​
else​
echo "Database backup failed"​
fi
Explanation:

1.​ Shebang (#!/bin/bash): Specifies the shell to be used for interpreting the
script.​

2.​ Database Credentials: Replace your_username, your_password, and

your_database with your actual database credentials and database name.​

3.​ Backup Directory: Set the BACKUP_DIR variable to the directory where you
want to store your database backups.​

4.​ Date Format: Generates a timestamp in the format YYYY-MM-DD_HH-MM-SS

to be used in the backup file name.​

5.​ Create Backup Directory: Creates the backup directory if it doesn't exist
using mkdir -p.​

6.​ Database Backup: Uses the mysqldump command to create a backup of the
specified database. The -u flag specifies the user, -p specifies the password
(without a space), and the database name is provided. The output is
redirected to the backup file.​

7.​ Check Backup Status: Checks the exit status of the mysqldump command. If
the exit status is 0, the backup is considered successful. If it's not 0, the
backup is considered failed.​

Remember to secure your shell script by setting appropriate permissions (chmod +x

scriptname.sh) and ensuring that the database credentials are kept secure. You
can use additional options with mysqldump to specify more backup options such as
excluding certain tables, using compression, and so on, based on your needs.

Test the script in a controlled environment before using it to automate database

backups in a production environment.

File exists or Not

Shell script that checks if a specified file exists and prints a message if it does:

#!/bin/bash​
# Specify the file path​
 FILE_PATH="/path/to/your/file.txt"​
​
# Check if the file exists​
if [ -f "$FILE_PATH" ]; then​
echo "The file $FILE_PATH exists."​
else​
echo "The file $FILE_PATH does not exist."​
fi

Explanation:

1.​ Shebang (#!/bin/bash): Specifies the shell to be used for interpreting the
script.​

2.​ FILE_PATH: Set the variable to the path of the file you want to check.​

3.​ if [ -f "$FILE_PATH" ]; then: This line checks if the file exists using the -f flag
with the test command (also known as [).​

4.​ If the file exists, the script prints a message indicating that the file exists;
otherwise, it prints a message indicating that the file does not exist.​

To use the script:

1.​ Save it to a file (e.g., check_file.sh).​

2.​ Make it executable using chmod +x check_file.sh.​

3.​ Update FILE_PATH with the actual path of the file you want to check.​

4.​ Run the script using ./check_file.sh.​

The script will output the appropriate message based on whether the
specified file exists or not.​

Interview Questions

Sure, here are 20 commonly asked shell scripting interview questions along with brief
explanations:

1.​ What is a shell script?​

○​ Explanation: A shell script is a series of commands written in a text file

that can be executed by a shell interpreter. It allows automation of tasks
in a Unix or Linux environment.​

2.​ How do you comment in a shell script?​

○​ Explanation: Comments in a shell script start with a # character.

Anything following # on a line is treated as a comment and is ignored
by the interpreter.​

3.​ What is the shebang line (#!) used for in a shell script?​

○​ Explanation: The shebang line (#!) at the beginning of a script tells the
system which interpreter should be used to execute the script. For
example, #!/bin/bash specifies that the script should be interpreted
using the Bash shell.​

4.​ How do you pass arguments to a shell script?​

○​ Explanation: Arguments can be passed to a shell script when it's called

from the command line. These arguments are accessed inside the
script using positional parameters like $1, $2, etc., which represent the
first, second, etc., arguments respectively.​

5.​ What is the difference between $@ and $* in shell scripting?​

○​ Explanation: $@ and $* both represent all the arguments passed to the

script, but $@ treats each argument as a separate entity, while $* treats
all arguments as a single string.​

6.​ Explain the use of the if-else statement in shell scripting.​

○​ Explanation: The if-else statement is used for conditional execution in a

shell script. It allows the script to perform different actions based on
whether a condition is true or false.​

7.​ What is a for loop in shell scripting?​

○​ Explanation: A for loop is used to iterate over a range of values or a list

of items. It executes a block of code for each item in the list.​
8.​ How do you read user input in a shell script?​

○​ Explanation: User input can be read in a shell script using the read
command followed by a variable name. For example, read name will
read input from the user and store it in the variable name.​

9.​ Explain the purpose of the case statement in shell scripting.​

○​ Explanation: The case statement is used for multi-way branching in

shell scripts. It allows the script to choose from a list of options based
on a given condition.​

10.​What is command substitution in shell scripting?​

○​ Explanation: Command substitution is a process that allows you to

replace a command with its output. It's achieved by enclosing the
command in backticks (`) or using $(command).​

11.​How do you check if a file exists in a shell script?​

○​ Explanation: The existence of a file can be checked using conditional

statements and file testing operators like -e or -f.​

12.​Explain how to redirect output in a shell script.​

○​ Explanation: Output redirection allows you to send the output of a

command to a file or another command. For example, command >
output.txt will redirect the output of command to the file
output.txt.​

13.​What is a function in shell scripting?​

○​ Explanation: A function in shell scripting is a reusable block of code that

performs a specific task. It can be defined using the function
keyword or simply as name() { ... }.​

14.​How do you use grep in a shell script?​

○​ Explanation: grep is used for pattern matching in text. In a script, it can

be used to search for specific patterns in files or input streams.​

15.​Explain the purpose of the awk command in shell scripting.​

 ○​ Explanation: awk is a powerful text processing tool that allows you to
manipulate and analyze data in a file. It's often used for tasks like
pattern matching and reporting.​

16.​How do you handle errors in a shell script?​

○​ Explanation: Errors can be handled in a shell script using conditional

statements (if-else), checking return codes of commands, and using
the trap command to catch signals.​

17.​What is the purpose of the sed command in shell scripting?​

○​ Explanation: sed (stream editor) is used for text processing and is

particularly useful for search and replace operations in a script.​

18.​How do you declare and use variables in shell scripting?​

○​ Explanation: Variables are declared by assigning a value to them. They

can be accessed using the variable name with a $ prefix (e.g.,
$variable).​

19.​Explain the difference between && and || in shell scripting.​

○​ Explanation: && is used to execute a command only if the preceding

command succeeds, while || is used to execute a command only if the
preceding command fails.​

20.​What is the purpose of the export command in shell scripting?​

○​ Explanation: export is used to make a variable available to child

processes (subshells). It's commonly used for environment variables.​

Remember, in an interview, it's important not only to know the answers but also to be
able to explain your thought process and demonstrate your understanding.