Shell Scripting for DevOps (Week 2)

Part 1 - The Essentials: Shebang, Shell Types, Basic Syntax, Variables & User Input

By Ashish — Learn in Public DevOps Journey (Week 2)
🔗 LinkedIn: https://www.linkedin.com/in/ashish360/
Github: https://github.com/ashish0360/devops-learn-in-public/tree/main/shell-scripting-for-devops

📘 Table of Contents

1. Why Shell Scripting Matters in DevOps
2. What is a Shell? (bash, sh, dash)
3. The Shebang (#!)
4. Running a Shell Script
5. Basic Syntax: echo, comments, variables
6. Reading User Input (read)
7. Script Arguments ($0, $1, $2)
8. Debugging & Error Handling (set -x, set -e, set -o pipefail)
9. Essential Commands You’ll Use Daily

10. Summary & What’s Next (Part 2 Preview)

11. 🚀 Why Shell Scripting Matters in DevOps
    Shell scripting is the foundation of DevOps automation.
    Every time you:
    Deploy an application
    Configure a server
    Build a container
    Parse logs
    Create CI/CD pipelines
    Automate infrastructure
    Trigger AWS CLI, Azure CLI, or gcloud
    Write Kubernetes helpers

…you are essentially writing or using shell scripts.
If you master shell scripting → you unlock real automation power.

1. 🧠 What Is a Shell? (bash vs sh vs dash) A shell executes your commands. Common shells: Shell Path Notes bash /bin/bash Default for Linux servers, DevOps standard sh /bin/sh Lightweight shell, usually a link to dash dash /bin/dash Very fast, used in system-level scripts zsh/fish optional For interactive use

➡️ DevOps uses bash the most.

1. 🔥 The Shebang (#!) The first line of every script defines which shell should run it. Most common: #!/bin/bash

System-level (mapped to dash on many systems):

!/bin/sh

Why it matters:
bash has more features than sh/dash
Some expressions don’t work in dash
CI pipelines may break if the wrong shebang is used

1. ▶️ How to Run a Shell Script Step 1 — Create a script touch script.sh

Step 2 — Add a shebang

!/bin/bash

Step 3 — Give execute permission
chmod +x script.sh

Step 4 — Run
./script.sh

Or:
bash script.sh
sh script.sh # if POSIX compatible

1. 📝 Basic Syntax Every DevOps Engineer Must Know echo — print output echo "Hello DevOps"

Comments

single-line comment

Block comments:
<<comment
This is a block comment
comment

Variables
User-defined:
name="Ashish"
echo "Hello $name"

System variables:
echo $HOME
echo $PATH
echo $USER

1. 🔡 Reading User Input (read) read -p "Enter your name: " username echo "Welcome $username"

read -p "Enter your age: " age
echo "You entered $age"

Used for:
interactive scripts

validation

server setup tools

menu-based scripts

1. 🧩 Script Arguments ($0, $1, $2 …) When running: ./deploy.sh prod v2

Inside script:
echo "Script name: $0"
echo "Environment: $1"
echo "Version: $2"

Used for:
dynamic deployments
folder creation
automation scripts
CI/CD parameter passing

1. 🔍 Debugging & Error Handling Essentials set -x — debug mode (prints commands) set -x

set -e — exit when a command fails
set -e

set -o pipefail — stop when any pipe command fails
set -o pipefail

Most common DevOps combo
set -exo pipefail

1. 🛠 Essential Shell Commands DevOps Engineers Use Daily Process management ps -ef ps -ef | grep nginx ps -ef | grep python | wc -l ps -ef | grep python | awk -F" " '{print $2}'

curl vs wget
curl https://api.com/data # fetch data
wget https://example.com/file # download file

find
find / -name app.log

These commands become the backbone of automation scripts.

1. 🎯 Part 1 Completed You now understand: What shells DevOps engineers use bash vs sh differences The importance of the shebang How to create and run scripts Variables, input, arguments Debugging with set flags Essential shell commands This is the foundation needed to automate real systems.

=================================================================

Part 2 - Conditions, Expressions, If/Else, Case, Loops (for/while/until) — The Logic Layer of Automation

By Ashish — Learn in Public DevOps Journey (Week 2)
🔗 LinkedIn: https://www.linkedin.com/in/ashish360/

📘 Table of Contents

1. Why Logic Matters in DevOps Automation
2. Expressions & Operators (-gt, -eq, -lt…)
3. If/Else — The Backbone of Script Decisions
4. Nested Conditions
5. Case Statement (Menu-Based Scripts)
6. For Loops — Iteration for Automation
7. While Loops — Event-Based Automation
8. Until Loops — Run Until Condition Becomes True
9. Practical DevOps Examples

10. Summary & Next Part Preview

11. 🚀 Why Logic Matters in DevOps Automation
    Every meaningful DevOps script makes decisions.
    For example:
    “Is Nginx running? If not, restart it.”
    “If the age is >= 18, allow access.”
    “If the directory already exists, exit the script.”
    “Loop through 100 log files and clean them.”
    “Run this until the server becomes healthy.”

Without if, loops, conditions, comparisons, shell scripting is basically just a list of commands.

This section gives you the logic power needed for real automation.

1. 🧠 Understanding Expressions & Operators

Shell scripts don't use normal symbols like > or <.
They use operators inside [[ ]].

🔹 Common numeric operators:
Operator Meaning
-gt greater than
-lt less than
-eq equal
-ne not equal
-ge greater or equal
-le less or equal

Example:

if [[ $age -ge 18 ]]; then
echo "Adult"
fi

1. 🧩 If / Else — Decision Making in Scripts Basic structure: if [[ condition ]]; then # code elif [[ another_condition ]]; then # code else # fallback fi

Example:
a=10
b=5

if [[ $a -gt $b ]]; then
echo "a is greater than b"
else
echo "a is smaller than b"
fi

1. 🎯 Practical Example — Voting Eligibility Script #!/bin/bash

read -p "Enter your age: " age
read -p "Are you Indian? yes/no: " nation

if [[ $age -ge 18 ]]; then
echo "You can vote because you are $age"
elif [[ $nation == "yes" ]]; then
echo "You are Indian, you can vote"
else
echo "You cannot vote because you are $age"
fi

✔ Demonstrates numeric + string evaluations.

1. 🔄 Case Statement — Build Menu-Based Tools

Case is perfect for DevOps automation menus.

read -p "Choose option: " choice

case $choice in
1) echo "Start service" ;;
2) echo "Stop service" ;;
3) echo "Check status" ;;
*) echo "Invalid option" ;;
esac

Used for:
deployment menus
CI/CD utility scripts
service management tools

1. 🔁 For Loop — Iterate Through Numbers, Files, Commands Example 1 — Simple counter: for ((i=1; i<=10; i++)); do echo "Number: $i" done

Example 2 — Create multiple folders:
for (( num=$1 ; num<=$2 ; num++ )); do
mkdir "$3$num"
done

Run:

./folder.sh 1 50 project

Creates:
project1, project2, ..., project50
Perfect for:
log directories
batch processing
repeating commands over files

1. ⏳ While Loop — Run While Condition Is True Example: read -p "Enter a number between 1-8: " num

while [[ $(( num%2 )) -eq 0 && $num -le 10 ]]; do
echo "$num"
num=$(( num+2 ))
done

Useful for:
monitors
waiting loops
retry logic

1. 🔁 Until Loop — Opposite of While

Runs until the condition becomes true.

Example:

n=1

until [[ $n -gt 5 ]]; do
echo "n = $n"
n=$((n+1))
done

1. 🧰 FUNCTIONS (Logic + Reusability)
   function vote() {
   read -p "Enter your age: " age
   read -p "Are you Indian? yes/no: " nation

   if [[ $age -ge 18 ]]; then
   echo "You can vote"
   elif [[ $nation == "yes" ]]; then
   echo "You can vote"
   else
   echo "You cannot vote"
   fi
   }
   vote

Functions are essential in DevOps for:
deployments
checks
re-used scripts
AWS CLI automation

1. 🛠 Real DevOps Use Case — Create Folder via Script #!/bin/bash

create_directory(){
mkdir test
}

if ! create_directory; then
echo "Directory already exists — exiting."
exit 1
fi
echo "Directory created successfully"

✔ Demonstrates if-condition
✔ Demonstrates error handling
✔ Demonstrates function use

1. 🎯 Real DevOps Logic Example — Process Counting ps -ef | grep python | wc -l

Add logic:

count=$(ps -ef | grep python | wc -l)

if [[ $count -gt 1 ]]; then
echo "Python service is running"
else
echo "Python service is DOWN!"
fi

These checks form the basis of:
monitoring
health checks
auto-restarts

1. 📌 Part 2 Complete

You now understand the full logic system in shell scripting:

✔ Numeric & string comparisons
✔ if / else / elif
✔ case (menu-driven automation)
✔ for, while, until loops
✔ functions for reusable automation

You now have the building blocks for real-world DevOps scripts.

=================================================================

**Part 3 - Error Handling, Debugging Flags, Exit Codes & Real DevOps Automation Scripts

**
By Ashish — Learn in Public DevOps Journey (Week 2)
🔗 LinkedIn: https://www.linkedin.com/in/ashish360/

📘 Table of Contents

1. Why Error Handling Matters in DevOps
2. Exit Codes (0, non-zero, $? usage)
3. set -e, set -x, set -o pipefail, set -exo
4. Error Handling Patterns
5. Debugging Shell Scripts
6. Writing Safe & Reliable DevOps Scripts
7. Practical DevOps Examples
8. Directory creation with error handling
9. Process validation
10. Automated deployment script
11. Django Notes App deployment
12. AWS EC2 creation through script

13. Summary & Next Part Preview

14. 🚨 Why Error Handling Is Critical in DevOps

DevOps engineers write scripts that automate:
deployments
backups
AWS/GCP/Azure infra creation
CI/CD steps
service monitoring
log processing
A single silent error can:
break deployments
corrupt environments
delete production data
take servers offline
So DevOps scripts must never fail silently.
That’s why error handling and debugging flags matter.

1. 🧯 Understanding Exit Codes Every command in Linux returns an exit code.

✔ Exit Code Meaning:
Code Meaning
0 Success
1-255 Error / Failure
Check exit code:
echo $?

Example:
ls /not_found
echo $? # 2 → means failure

As a DevOps engineer, you must use exit codes to stop scripts when something breaks.

1. 🛑 set Flags — Your Safety Net in Shell Scripts

These are critical for production-grade scripts.

3.1 set -e → Exit Immediately on Error

If any command fails, the script stops.
set -e
mkdir test
cp abc.txt xyz.txt # if this fails → script stops
echo "Will NOT run"

When DevOps uses it:
CI/CD pipelines
AWS provisioning
Database backup scripts
Deployment scripts

3.2 set -x → Show Commands as They Execute (Debugging)
set -x
echo "Hello"
ls -l

Output looks like:

• echo Hello
• ls -l Helps debug scripts line-by-line.

3.3 set -o pipefail → Fail If Any Command in a Pipeline Fails
Without this:
command1 | command2 | command3

❌ Only command3 exit code is checked
✔ Earlier failures are ignored silently

With pipefail:

set -o pipefail

Pipeline fails if any command fails — perfect for DevOps pipelines.

3.4 set -exo — The DevOps Gold Standard
set -euo pipefail

Meaning:
-e → Exit on errors
-u → Undefined variables cause errors
-o pipefail → Pipelines must succeed
This is the correct way to write most DevOps automation scripts.

1. ⚙️ Error Handling Patterns (Must Learn) Pattern 1 — Use || for Manual Error Catching mkdir demo || { echo "Failed to create directory" exit 1 }

Pattern 2 — Use a Function and Catch Its Error
create_directory() {
mkdir demo
}

if ! create_directory; then
echo "Directory already exists"
exit 1
fi

Pattern 3 — Use Combined Flags
set -euo pipefail

deploy() {
echo "Deploying..."
cp app.conf /etc/app/
}

deploy || {
echo "Deployment failed"
exit 1
}

1. 🐞 Debugging Your Script (DevOps Way) Use set -x during debugging: set -x

Or wrap only the part you want to debug:
set -x
command1
command2
set +x

Print variables to trace execution:
echo "User: $USER"
echo "Directory: $PWD"

Debugging is a non-negotiable DevOps skill.

1. 🔥 Real DevOps Scripts with Error Handling Below are real scripts you wrote with me—now rewritten cleanly, documented, and production-ready.

✔ Example 1 — Safe Directory Creation Script

!/bin/bash

create_directory() {
mkdir test
}

if ! create_directory; then
echo "Directory already exists — exiting."
exit 1
fi

echo "Directory created successfully."

Used for:
log directories
deployment folders

✔ Example 2 — Count Running Processes
count=$(ps -ef | grep python | wc -l)

if [[ $count -gt 1 ]]; then
echo "Python service is running"
else
echo "Python service is DOWN!"
fi

Used in:
service monitors
cron jobs
health checks

✔ Example 3 — Django Notes App Deployment (Full Script)
Your complete script, now cleaned and formatted:

!/bin/bash

echo "********** DEPLOYMENT STARTED *********"
code_clone() {
echo "Cloning Django app..."
if [ -d "django-notes-app" ]; then
echo "Directory exists, skipping clone."
else
git clone https://github.com/LondheShubham153/django-notes-app.git || {
echo "Clone failed"; return 1;
}
fi
}

install_requirements() {
echo "Installing dependencies..."
sudo apt-get update && sudo apt-get install -y docker.io nginx docker-compose || {
echo "Dependency installation failed"; return 1;
}
}

required_restarts() {
echo "Performing restarts..."
sudo chown "$USER" /var/run/docker.sock || {
echo "Failed to change docker.sock ownership"; return 1;
}
}

deploy() {
echo "Building and deploying the app..."
docker build -t notes-app . && docker-compose up -d || {
echo "Deployment failed"; return 1;
}
}

if ! code_clone; then cd django-notes-app || exit 1; fi
if ! install_requirements; then exit 1; fi
if ! required_restarts; then exit 1; fi
if ! deploy; then
echo "Deployment failed — notifying admin..."
exit 1
fi

echo "********** DEPLOYMENT DONE *********"

This is real DevOps-grade code.

✔ Example 4 — AWS EC2 Creation Script (Professional Version)

Your script, polished for publication:

!/bin/bash

set -euo pipefail

check_awscli() {
if ! command -v aws &>/dev/null; then
echo "AWS CLI is not installed."
exit 1
fi
}

install_awscli() {
echo "Installing AWS CLI v2..."
curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip
sudo apt-get install -y unzip
unzip awscliv2.zip
sudo ./aws/install
rm -rf aws awscliv2.zip
aws --version
}

wait_for_instance() {
local instance_id="$1"

echo "Waiting for EC2 instance..."
while true; do
    state=$(aws ec2 describe-instances \
        --instance-ids "$instance_id" \
        --query 'Reservations[0].Instances[0].State.Name' \
        --output text)

    [[ "$state" == "running" ]] && break
    sleep 10
done

}

create_ec2_instance() {
instance_id=$(aws ec2 run-instances \
--image-id "$1" \
--instance-type "$2" \
--key-name "$3" \
--subnet-id "$4" \
--security-group-ids "$5" \
--tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$6}]" \
--query 'Instances[0].InstanceId' \
--output text)

echo "Created instance: $instance_id"
wait_for_instance "$instance_id"

}

main() {
check_awscli

AMI_ID=""
INSTANCE_TYPE="t2.micro"
KEY_NAME=""
SUBNET_ID=""
SECURITY_GROUP_IDS=""
INSTANCE_NAME="Shell-Script-EC2-Demo"

create_ec2_instance "$AMI_ID" "$INSTANCE_TYPE" "$KEY_NAME" "$SUBNET_ID" "$SECURITY_GROUP_IDS" "$INSTANCE_NAME"

}

main "$@"

Used in:
DevOps automation
Infrastructure creation
AWS provisioning
Demo/testing environments

1. 🧾 Part 3 Complete You now understand: ✔ Exit codes ✔ Error handling (||, functions, manual checks) ✔ set -e, set -x, set -o pipefail, set -exo ✔ Debugging shell scripts ✔ Crafting production-grade automation ✔ Real-world scripts for EC2, deployments, directories, processes

This is the level of scripting expected from a DevOps engineer.

=================================================================

Trap, Signals, Cron Jobs & Background Automation (Professional DevOps Guide)

By Ashish — Learn in Public DevOps Journey (Week 2)
🔗 LinkedIn: https://www.linkedin.com/in/ashish360/

📘 Table of Contents

1. Understanding Signals in Linux
2. trap — The DevOps Lifesaver
3. Common Signals Every DevOps Engineer Must Know
4. Automating Cleanup with trap
5. Running Background Processes
6. nohup for long-running jobs
7. Scheduling with Cron (Beginner → Advanced)
8. Practical DevOps Automation Examples
9. Production-Grade Script Templates

10. Summary & What’s Coming in Part 5

11. ⚠️ Understanding Linux Signals (Critical DevOps Concept)
    Linux processes constantly interact with signals — notifications sent by users, other programs, the kernel, or system events.

Most commonly used signals:
Signal Code Meaning
SIGINT 2 Interrupt (Ctrl + C)
SIGTERM 15 Request to terminate
SIGKILL 9 Force kill (cannot be trapped)
SIGHUP 1 Hangup (restart daemons)
SIGQUIT 3 Quit & dump core
SIGUSR1/2 — Custom signals

As a DevOps engineer, you MUST handle signals safely because:
Deployments shouldn’t quit mid-way
Cleanup must always run
Temporary files must be removed
Background tasks must be reliable
That’s where… 👇

1. 🔗 trap — The Most Underrated DevOps Tool trap executes a command when a signal is received.

Syntax:
trap "commands" SIGNALS

1. 🧹 Example 1 — Cleanup Temporary Files on Exit #!/bin/bash

tmp_file="/tmp/myapp.log"

trap "echo 'Cleaning...'; rm -f $tmp_file" EXIT

echo "Working..."
touch $tmp_file
sleep 5

echo "Done."

What it does:
Creates temp file
Removes it even if you press Ctrl+C
Ensures the script never leaves junk behind
This is production-safe scripting.

1. ❗ Example 2 — Trap SIGINT (Ctrl + C) trap "echo 'Ctrl+C detected. Stopping safely...'; exit" SIGINT

while true; do
echo "Running..."
sleep 2
done

Useful for:
long-running operations
loops
automation tasks

1. 🛑 Example 3 — Prevent Script from Dying Unexpectedly trap "echo 'Script terminated unexpectedly'; exit 1" SIGTERM

Used in:
CI/CD pipelines
systemd scripts
deployment hooks

1. 🪝 Example 4 — Multiple Signals at Once trap "cleanup" SIGINT SIGTERM EXIT

cleanup() {
echo "Cleaning up resources..."
rm -rf /tmp/deploy
}

1. ⚙️ Background Jobs (Essential for DevOps) You learned loops, conditions, debugging… Now let’s add background jobs.

7.1 Run Command in Background
command &

Example:

python server.py &

Check jobs:
jobs

Resume background job:
bg %1

Bring to foreground:
fg %1

7.2 nohup — Keep Process Alive After Logout

Used heavily in servers, SSH sessions, and EC2 instances.
nohup python app.py &

You close the terminal → the process still runs.
Output saved to:
nohup.out

1. ⏰ Cron Jobs — Scheduling in Linux (DevOps Mandatory Skill) Cron = Scheduler for Linux automation. Open cron: crontab -e

8.1 Cron Syntax

• * * * * command | | | | | | | | | └── day of week (0–6) | | | └──── month (1–12) | | └────── day of month (1–31) | └──────── hour (0–23) └────────── minute (0–59)

8.2 Examples DevOps Engineers Use Daily
Run backup every night:
0 0 * * * /home/ashish/backup.sh

Run a health check every 5 minutes:
*/5 * * * * /home/ashish/check_service.sh

Clear logs weekly:
0 0 * * 0 rm -rf /var/log/*.gz

8.3 Example — Cron + Error Handling

!/bin/bash

set -euo pipefail

logfile="/var/log/health.log"

if ! systemctl is-active --quiet nginx; then
echo "$(date) — Nginx DOWN" >> $logfile
systemctl restart nginx
fi

Cron entry:
*/2 * * * * /home/ashish/health.sh

1. ⚡ Real-World DevOps Automation Examples ✔ Example: Kill Hanging Process & Log the Event #!/bin/bash trap "echo 'Terminated!' >> /var/log/proc.log" SIGTERM

pid=$(pgrep python)

if [[ -z "$pid" ]]; then
echo "No python process running"
else
kill -9 $pid
echo "Killed python process $pid"
fi

✔ Example: Auto-Restart Docker on Failure

!/bin/bash

trap "echo 'Docker restarted due to failure'" EXIT

if ! docker ps >/dev/null; then
systemctl restart docker
fi

✔ Example: Cleanup Docker Resources Automatically

!/bin/bash

set -e

trap "docker system prune -f" EXIT

echo "Running build..."
docker build -t webapp .

1. 📦 Production Template — “Safe DevOps Script” Use this for all future scripts:

!/bin/bash

set -euo pipefail

trap "echo 'Script exited unexpectedly'; cleanup" EXIT

cleanup() {
echo "Performing cleanup..."
}

main() {
echo "Starting task..."
}

main "$@"

This template = best practice.

🎉 Part 4 Complete
You now understand advanced automation fundamentals:
trap
Signal handling
Background tasks
nohup
Cron jobs
Cleanup automation
Production-safe scripting patterns
This is the professional level expected in DevOps interviews & real jobs.

=================================================================

**File Operations, find, grep, awk, sed, Automation Scripts

**
By Ashish — Learn in Public DevOps Journey (Week 2)
🔗 LinkedIn: https://www.linkedin.com/in/ashish360/

📘 Table of Contents

1. File Operations in Shell Scripting (Beginner → Pro)
2. Reading/Writing Files in Bash
3. Using find like a DevOps Engineer
4. DevOps-Grade grep, sed, awk Integration
5. File Permissions Automation
6. Bulk Folder Creation & Organization
7. Building Real DevOps Automation Scripts
8. Log Automation, Cleanup Scripts, CI/CD Helpers
9. Production Template — File-Handling Script

10. Summary & Next Steps

11. 📁 File Operations in Shell Scripting (Practical Guide)
    File handling is at the core of DevOps:
    Editing configs
    Cleaning logs
    Managing backups
    Processing monitoring data
    Updating YAML/JSON
    Generating environment files
    Here are the must-know techniques.

1.1 Create a File
touch filename.txt

Inside script:

!/bin/bash

touch report.log

1.2 Write to a File
echo "Hello DevOps" > file.txt

Overwrite? → yes.

Append instead:
echo "New entry" >> file.txt

1.3 Read a File Line-by-Line
while read line; do
echo "Line: $line"
done < file.txt

Use-case:
processing logs, configs, CSVs, access logs

1.4 Read a specific line number
sed -n '5p' file.txt

1.5 Replace text in a file
sed -i 's/old/new/g' config.yaml

Used in:
CI/CD variable changes
Environment promotion (dev → prod)
Auto-fixing broken configs

1. 🔍 find — The Most Important DevOps File Tool find is used everywhere: troubleshoot disk usage locate configs find failing scripts cleanup logs manage permissions

2.1 Find file by name
find / -name "nginx.conf"

2.2 Find directories
find /var -type d -name "log*"

2.3 Find files modified recently

Last 24 hours:
find /var/log -mtime -1

2.4 Delete files older than X days

Critical DevOps script:
find /var/log -type f -mtime +7 -delete

Use this for:
disk pressure issues
log rotation

CI runners storing artifacts

2.5 Find files > 100MB
find / -type f -size +100M

2.6 Find and execute a command (🔥 powerful)
find /tmp -type f -name "*.log" -exec rm -f {} \;

1. 🎯 Integrating grep + sed + awk (DevOps-Level)

These 3 tools together =
90% of DevOps text processing automation.

3.1 grep → find matching lines

Extract errors from logs:
grep -i "error" app.log

3.2 sed → modify lines

Bulk edit config values:
sed -i 's/ENV=dev/ENV=prod/g' .env

3.3 awk → extract fields/reports

Extract IP & status:
awk '{print $1, $9}' access.log

1. 🗄 Automating File Permissions

Fix “permission denied” errors:
sudo chown -R $USER:$USER /path

Change permission in script:
chmod +x deploy.sh

1. 🧬 Bulk Folder & File Creation 5.1 Create 100 folders automatically for i in {1..100}; do mkdir "folder$i" done

5.2 Create folders using arguments

!/bin/bash

for (( i=$1 ; i<=$2 ; i++ ))
do
mkdir "$3$i"
done

Run:

./create.sh 1 50 week

5.3 Create multiple files based on list
for f in $(cat list.txt); do
touch "$f.txt"
done

1. ⚡Real DevOps Automation Scripts

These are the scripts DevOps engineers actually use daily.

6.1 Cleanup Logs Older Than X Days

!/bin/bash

LOG_DIR="/var/log"
DAYS=7

find $LOG_DIR -type f -mtime +$DAYS -delete

echo "Old logs cleaned."

6.2 Disk Pressure Emergency Script

!/bin/bash

du -sh /* | sort -h | tail
df -h

6.3 Find top 10 IPs hitting your server
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head

6.4 Auto-Restart Service When It Goes Down

!/bin/bash

if ! systemctl is-active --quiet nginx; then
echo "Nginx down! Restarting..."
systemctl restart nginx
fi

6.5 Check all Python Processes
ps -ef | grep python | awk '{print $2}'

6.6 Delete specific type of files safely
find . -name "*.tmp" -exec rm -f {} \;

6.7 Backup a directory
tar -cvzf backup.tar.gz /etc

6.8 Parse AWS EC2 instance data
aws ec2 describe-instances | jq '.Reservations[].Instances[].InstanceId'

1. 🏗 Production-Grade File Automation Script Template #!/bin/bash set -euo pipefail

trap "echo 'Exiting safely'; cleanup" EXIT

cleanup() {
echo "Performing cleanup..."
}

log() {
echo "$(date) — $1"
}

process_files() {
for file in *.log; do
log "Processing $file"
grep -i "error" "$file" >> errors.txt
done
}

main() {
log "Script started"
process_files
log "Script completed"
}

main "$@"

This is the structure companies expect.

🎉 Part 5 Complete - Advanced Automation Unlocked
You now know:
Full file operations
find for DevOps scenarios
Combining grep + awk + sed
Bulk file & folder creation
Cleanup, backup, monitoring scripts
Production-safe file automation templates
This is professional-grade shell scripting, not beginner-level theory.

=================================================================

**Part 6 - DevOps Project Scripts, AWS Automation, Error Handling & Deployment

**
By Ashish — Learn in Public DevOps Journey (Week 2)
🔗 LinkedIn: https://www.linkedin.com/in/ashish360/

📘 Table of Contents

Why DevOps Engineers Automate Cloud Tasks
Understanding AWS CLI for Automation
Installing AWS CLI via Shell Script
AWS EC2 Creation Script (Deep-Dive)
AWS Instance Waiters (Production-Grade)
Error Handling, Logging & Exit Codes
Automating Docker + Nginx Deployment
Django App Deployment Script (Your Notes Included)
Real DevOps Cloud Automation Examples
CI/CD Integration Patterns
Summary & Next Part

1. 🚀 Why DevOps Engineers Automate Cloud Tasks

Modern DevOps is automation-first.
A DevOps engineer must be able to:
spin up servers
install dependencies
run deployments
debug errors
restart services
scale infra
perform health checks
all using shell scripting, not just clicking on AWS Console.

Your Week-2 work (EC2 script, error handling, Django deploy script) reflects exactly what real engineers do.

1. ⚙️ Understanding AWS CLI for Automation To automate AWS from your shell scripts, AWS CLI must be: installed configured authenticated tested with simple commands

Check if installed:
aws --version

If output is missing → install inside the script.

1. 🧰 Script: Install AWS CLI Automatically

This is the improved, production-level version of your script.

!/bin/bash

set -euo pipefail

install_aws_cli() {
echo "Installing AWS CLI v2..."

curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt-get install -y unzip &>/dev/null
unzip -q awscliv2.zip

sudo ./aws/install
aws --version

rm -rf aws awscliv2.zip

}

check_aws() {
if ! command -v aws &>/dev/null; then
install_aws_cli
else
echo "AWS CLI already installed."
fi
}

set -euo pipefail is essential in DevOps:
-e → exit on error
-u → undefined variables error
-o pipefail → detect pipeline failures

1. 🖥️ EC2 Instance Creation Script (Full Breakdown) This is the exact version DevOps teams use — simple, readable, powerful.

create_ec2_instance() {
local ami_id="$1"
local instance_type="$2"
local key_name="$3"
local subnet_id="$4"
local sg_id="$5"
local name="$6"

instance_id=$(aws ec2 run-instances \
    --image-id "$ami_id" \
    --instance-type "$instance_type" \
    --key-name "$key_name" \
    --subnet-id "$subnet_id" \
    --security-group-ids "$sg_id" \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$name}]" \
    --query 'Instances[0].InstanceId' \
    --output text)

if [[ -z "$instance_id" ]]; then
    echo "ERROR: Instance not created!"
    exit 1
fi

echo "Instance created: $instance_id"

}

1. ⏳ EC2 Waiters — Wait Until Instance Becomes Running

This is pro-level — prevents CI/CD failures due to boot delay.

wait_for_instance() {
local instance_id="$1"

echo "Waiting for instance $instance_id to enter 'running' state..."

while true; do
    state=$(aws ec2 describe-instances \
            --instance-ids "$instance_id" \
            --query 'Reservations[0].Instances[0].State.Name' --output text)

    if [[ "$state" == "running" ]]; then
        echo "Instance is now running."
        break
    fi

    sleep 10
done

}

1. 🐳 Docker + Django Deployment Script (Your Notes → Polished Version) Here is the polished, production-grade version of the Django deployment script you captured in your notes.

!/bin/bash

set -euo pipefail

repo="https://github.com/LondheShubham153/django-notes-app.git"

log() {
echo "$(date) - $1"
}

code_clone() {
log "Checking code directory..."
if [[ ! -d django-notes-app ]]; then
log "Cloning repository..."
git clone "$repo" || return 1
else
log "Repo already exists. Skipping clone."
fi
}

install_requirements() {
log "Installing dependencies..."
sudo apt-get update
sudo apt-get install -y docker.io docker-compose nginx || return 1
}

configure_docker() {
log "Configuring Docker..."
sudo chown "$USER" /var/run/docker.sock || return 1
}

deploy() {
log "Deploying Django app..."

docker build -t notes-app . || return 1
docker-compose up -d || return 1

log "Application deployed successfully."

}

main() {
log "Deployment started."
code_clone || cd django-notes-app
install_requirements
configure_docker
deploy
log "Deployment finished."
}

main "$@"

This is a superb Week-2 DevOps script — you should be proud.

1. 🧨 Error Handling in DevOps Scripts (Industry Level)

Use the pattern:

if ! command; then
echo "Error occurred"
exit 1
fi

Example:

if ! mkdir demo; then
echo "Directory exists! Exiting..."
exit 1
fi

This prevents:
partial deployment
corrupted services
half-created resources
CI/CD failures

1. 🛠 Real DevOps Cloud Automation Examples

These are practical, real-world scripts DevOps engineers write constantly.

8.1 Backup your EC2 metadata
aws ec2 describe-instances > instances.json

8.2 Restart EC2 instance using script
aws ec2 reboot-instances --instance-ids i-123456

8.3 Get EC2 public IP automatically
aws ec2 describe-instances \
--instance-ids i-123 \
--query 'Reservations[0].Instances[0].PublicIpAddress' \
--output text

8.4 Upload files to S3 automatically
aws s3 cp backup.tar.gz s3://mybucket/

8.5 Check if AWS credentials are expired
aws sts get-caller-identity || echo "AWS credentials expired!"

1. 🧩 CI/CD Integration — Best Practices You’ll use your scripts in:

GitHub Actions
Jenkins
GitLab CI
Bitbucket Pipelines
AWS CodePipeline

Checklist:
✔ All scripts must return correct exit codes
✔ Avoid echoing secrets
✔ Never hardcode AWS keys
✔ Use IAM role-based credentials
✔ Scripts must work in non-interactive mode
✔ Include retries & timeouts

🎉 Part 6 Complete — AWS + DevOps Automation Achieved
You now know how to:
✔ Create, manage, and automate AWS EC2
✔ Build error-proof deployment scripts
✔ Integrate Docker + Nginx automation
✔ Apply real-world debugging
✔ Use production-grade error handling
✔ Follow DevOps scripting best practices

This Part brings your skill to junior → mid-level DevOps engineer level.

=================================================================

**Part 7 - Real-World DevOps Projects, Interview-Grade Tasks & Advanced Shell Scripting Patterns

**
By Ashish — Learn in Public DevOps Journey (Week 2)
🔗 LinkedIn: https://www.linkedin.com/in/ashish360/

📘 Table of Contents

1. Why Real-World Shell Scripting Matters for DevOps
2. Practical DevOps Project Scenarios (Beginner → Advanced)
3. File Automation & Log Processing Scripts
4. System Health, Monitoring & Alert Automation
5. Networking, Ports & Service Debug Scripts
6. Infrastructure Automation (AWS, Docker, Nginx)
7. CI/CD Pipeline Shell Tasks
8. Interview-Grade Shell Scripting Problems
9. Optimization, Best Practices & Production Standards

10. Final Summary + Week-2 Completion

11. 🚀 Why Real-World Shell Scripting Matters in DevOps
    Shell scripting is not about writing “scripts”;
    it’s about automating infrastructure.

In real DevOps jobs, shell scripts are used to automate:
✔ deployments
✔ server provisioning
✔ log parsing
✔ monitoring
✔ backups
✔ docker builds
✔ pipeline steps
✔ cloud tasks
✔ alerts

This chapter focuses on everything interviewers expect AND everything real DevOps teams use daily.

1. 🧩 Practical DevOps Project Scenarios

(Beginner → Intermediate → Advanced)

✅ Scenario 1 — Rotate & Archive Logs Daily (Cron + Shell)

!/bin/bash

log_dir="/var/log/nginx"
backup_dir="/backup/nginx"

timestamp=$(date +%F-%H-%M)
mkdir -p $backup_dir

tar -czf "$backup_dir/nginx-$timestamp.tar.gz" $log_dir/*

echo "Log backup completed: $timestamp"

Add cronjob:

0 1 * * * /home/ashish/rotate_logs.sh

✅ Scenario 2 — Clean Old System Logs Automatically

!/bin/bash

find /var/log -type f -mtime +7 -delete

Interviewers LOVE this one.

✅ Scenario 3 — Find Top CPU/Memory Processes & Alert

!/bin/bash

cpu_limit=80

ps aux --sort=-%cpu | awk -v limit=$cpu_limit '
NR>1 && $3 > limit {print "HIGH CPU:", $2, $3"%", $11}'

✅ Scenario 4 — Backup Database Daily

!/bin/bash

mysqldump -u root -pPASSWORD dbname > backup_$(date +%F).sql

With cron:
0 2 * * * /home/ashish/db_backup.sh

✅ Scenario 5 — Restart a Service if It’s Down
This script can literally SAVE a production outage:

!/bin/bash

service="nginx"

if ! systemctl is-active --quiet $service; then
echo "$(date): $service DOWN — restarting"
systemctl restart $service
fi

✅ Scenario 6 — Auto-Deploy Build Artifacts (CI/CD Ready)

!/bin/bash

rsync -avz ./build/ user@server:/var/www/app/
systemctl restart nginx

Perfect for GitHub Actions, Jenkins, GitLab CI.

1. 🧪 File Automation & Log Processing Scripts 🔍 Extract all unique IPs from Nginx logs awk '{print $1}' access.log | sort | uniq

🛠 Count HTTP 500 errors
grep " 500 " access.log | wc -l

📂 Find largest files on server (production debugging)
find / -type f -exec du -Sh {} + | sort -rh | head -n 20

🧹 Delete files older than X days
find /tmp -type f -mtime +3 -delete

Very common DevOps interview question.

1. 🖥 System Health, Monitoring & Alerts Memory alert script (interview favorite) #!/bin/bash threshold=80

used=$(free | awk '/Mem/ {print int($3/$2 * 100)}')

if [[ $used -gt $threshold ]]; then
echo "WARNING: RAM usage is ${used}%"
fi

Disk space alert

!/bin/bash

df -h | awk '$5+0 > 85 {print "HIGH DISK USAGE:", $0}'

CPU alert
top -bn1 | grep "Cpu(s)"

1. 🌐 Networking, Ports & Service Debug Scripts Check if a port is open nc -zv localhost 8080

Find which process is using a port
sudo lsof -i :8080

Check DNS resolution in script
nslookup google.com

1. ☁️ AWS Automation Scripts (Real DevOps Work) 🔸 Create EC2 🔸 Wait until ready 🔸 Deploy Docker app 🔸 Restart on failure

You already wrote EC2 creation in Part 6.
Here are two more production scripts:

🛢 Get EC2 instance public IP
aws ec2 describe-instances \
--instance-ids "$1" \
--query "Reservations[0].Instances[0].PublicIpAddress" \
--output text

🧹 Clean unused EBS snapshots (AWS cost savings)
aws ec2 describe-snapshots --owner self \
| jq -r '.Snapshots[] | select(.StartTime < "2024-01-01") | .SnapshotId' \
| xargs -I {} aws ec2 delete-snapshot --snapshot-id {}

1. 🔁 CI/CD Pipeline Shell Scripting Patterns

Almost every CI/CD pipeline uses:

✔ mkdir
✔ cp, rsync
✔ sed for config editing
✔ grep/log parsing
✔ exit codes
✔ if conditions
✔ docker build & run

Example: extract version from package.json

version=$(grep version package.json | cut -d '"' -f4)
echo "Deploying version: $version"

1. 🎯 Interview-Grade Shell Scripting Questions (Your Week 2 Version)

Here are the exact problems companies ask:

1. Print Fibonacci series up to N
2. Reverse a file content
3. Find the largest of 3 numbers
4. Monitor a log file for a keyword and alert
5. Parse CSV and print a column
6. Archive and rotate logs
7. Validate if a service is running
8. Extract IP address from logs
9. Write your own “ls” command using shell
10. Automate daily backups

If you want, I can generate solutions to all 10 (optional).

1. 📝 Optimization, Best Practices & Production Standards ✔ Use set -euo pipefail ✔ Use functions ✔ Use clear naming ✔ Modular scripts ✔ Avoid hardcoding secrets ✔ Validate inputs ✔ Use logging function ✔ Test scripts with shellcheck

Example:

log() {
echo "$(date) — $1"
}

🎉 Part 7 Complete — You’ve Reached DevOps-Ready Shell Scripting Level

By completing Part 7, you’ve achieved:

✔ Hands-on DevOps automation skills
✔ Cloud scripting knowledge
✔ Error-safe, production-grade syntax
✔ Ability to write monitoring & alerting scripts
✔ Confidence for interviews
✔ CI/CD scripting proficiency
✔ AWS automation capability
✔ Understanding real infrastructure problems

This completes Week 2 of your Learn-in-Public DevOps Journey.