Shivakumar

Posted on Jan 19

Shell script for Devops

#shellscript #devops #tutorial #programming

Here is a comprehensive, blog-style set of notes on Shell Scripting.

These notes cover the fundamentals, syntax, and logic control required to write effective automation scripts in Linux/Unix environments.

The Ultimate Guide to Shell Scripting

Shell scripting is the art of chaining together Linux commands into a reusable file to automate repetitive tasks. It is the backbone of DevOps, System Administration, and backend automation.

1. What is a Shell?

Before scripting, we must understand the Shell. The Shell is a program that takes commands from the keyboard and gives them to the operating system to perform.

The Kernel: The core of the OS that talks to the hardware (CPU, RAM).
The Shell: The interface between You and the Kernel.
Bash (Bourne Again SHell): The most common shell used in Linux and macOS.

2. Anatomy of a Script

Every shell script follows a specific structure.

The Shebang (`#!`)

The very first line of your script tells the system which interpreter to use to run the code.

#!/bin/bash

#!: The "Shebang" (Sharp-Bang).
/bin/bash: The path to the interpreter. You might also see #!/bin/sh or #!/usr/bin/python.

Permissions

By default, a new file is just text. You must make it executable.

chmod +x myscript.sh
./myscript.sh

3. Variables

Variables are containers for storing data. In Bash, they are untyped (everything is treated as a string usually).

Defining and Using Variables

No spaces around the = sign.
Use $ to access the value.

#!/bin/bash

# Defining variables
NAME="John"
AGE=25

# Using variables
echo "Hello, my name is $NAME and I am $AGE years old."

Special Variables (Arguments)

These are reserved for handling inputs passed to the script (e.g., ./script.sh input1 input2).

Variable	Description
`$0`	The name of the script itself.
`$1`	The first argument passed to the script.
`$2`	The second argument passed.
`$#`	The total number of arguments provided.
`$@`	All arguments passed (as a list).
`$?`	The exit status of the last command (0 = Success, Non-zero = Failure).

4. User Input

How to make your script interactive.

#!/bin/bash

echo "What is your website?"
read WEBSITE

echo "Pinging $WEBSITE now..."
ping -c 1 $WEBSITE

5. Conditionals (If/Else)

Logic allows your script to make decisions. Pay close attention to the spacing inside the brackets [ ... ].

Syntax

#!/bin/bash

echo "Enter your age:"
read AGE

if [ $AGE -ge 18 ]; then
    echo "You are an adult."
elif [ $AGE -eq 17 ]; then
    echo "Almost there."
else
    echo "You are a minor."
fi

Comparison Operators

Numbers: -eq (equal), -ne (not equal), -gt (greater than), -lt (less than), -ge (greater or equal).
Strings: == (equal), != (not equal).
Files:
-f file.txt: Checks if file exists.
-d /folder: Checks if directory exists.

6. Loops

Loops allow you to repeat actions.

The `For` Loop

Best for iterating over a list or numbers.

#!/bin/bash

# Loop through a list of names
NAMES="Alice Bob Charlie"
for PERSON in $NAMES; do
    echo "Hello $PERSON"
done

# Loop through a range of numbers
for i in {1..5}; do
    echo "Count: $i"
done

The `While` Loop

Runs as long as a condition is true.

#!/bin/bash

COUNT=1
while [ $COUNT -le 5 ]; do
    echo "Line $COUNT"
    ((COUNT++))  # Increment the counter
done

7. Functions

Functions allow you to write code once and reuse it.

#!/bin/bash

# Define the function
check_status() {
    if ping -c 1 google.com > /dev/null; then
        echo "Internet is UP"
    else
        echo "Internet is DOWN"
    fi
}

# Call the function
check_status

8. Best Practices for Professional Scripts

Always use comments (#): Explain why you are doing something, not just what you are doing.
Exit on Error: Add set -e at the top of your script. This stops the script immediately if any command fails, preventing a snowball effect of errors.
Use meaningful variable names: Use FILENAME instead of F.
Quote your variables: Use "$VAR" instead of $VAR to prevent bugs if the variable contains spaces.

Summary Table

Concept	Command / Syntax	Purpose
Shebang	`#!/bin/bash`	Defines the interpreter.
Execution	`chmod +x script.sh`	Makes the file runnable.
Output	`echo "Text"`	Prints to the screen.
Input	`read VAR_NAME`	Takes input from user.
Variables	`VAR="Value"` / `$VAR`	Stores and retrieves data.
Condition	`if [ condition ]; then`	Branching logic.
Loop	`for i in list; do`	Repeating tasks.

To move from a "beginner" to an "intermediate" scripter, you need to master three key concepts: Functions, Arrays, and Case Statements.

These tools allow you to write scripts that are modular, organized, and capable of handling complex lists of data.

1. Functions (Don't Repeat Yourself)

If you find yourself copy-pasting the same code block twice, you should turn it into a Function. Functions make your script cleaner and easier to debug.

Syntax:

function_name() {
    # Code goes here
    echo "Argument 1 is $1"
}

Example: A flexible Logger
Instead of typing echo "$(date) ..." every time, create a function.

#!/bin/bash

# Define the function
log_msg() {
    local LEVEL=$1
    local MESSAGE=$2
    TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
    echo "[$TIMESTAMP] [$LEVEL] $MESSAGE"
}

# Call the function
log_msg "INFO" "Starting the script..."
log_msg "ERROR" "Database connection failed!"

Note: local variables only exist inside the function, keeping your global variables safe.

2. Arrays (Handling Lists)

In DevOps, you often need to loop through a list of servers, packages, or users. Instead of creating $SERVER1, $SERVER2, etc., use an Array.

Syntax:

Create: MY_LIST=("item1" "item2" "item3")
Access All: ${MY_LIST[@]}
Access One: ${MY_LIST[0]} (Index starts at 0)
Count Items: ${#MY_LIST[@]}

Example: Installing Multiple Packages

#!/bin/bash

# Define a list of packages to install
PACKAGES=("git" "curl" "nginx" "htop")

echo "We need to install ${#PACKAGES[@]} packages."

# Loop through the array
for PKG in "${PACKAGES[@]}"; do
    echo "Installing $PKG..."
    sudo apt-get install -y "$PKG"
done

3. Case Statements (The "Menu" Logic)

If you have a script that needs to handle many different options (like start, stop, restart, status), using 10 different if/else statements is messy. Use case instead.

Example: A Service Manager Script
Run this script like: ./manage_service.sh start

#!/bin/bash

ACTION=$1

case "$ACTION" in
    "start")
        echo "🟢 Starting application..."
        # systemctl start myapp
        ;;
    "stop")
        echo "🔴 Stopping application..."
        # systemctl stop myapp
        ;;
    "restart")
        echo "🔄 Restarting..."
        # systemctl restart myapp
        ;;
    *)  # The "Default" or "Catch-all" option
        echo "❌ Error: Invalid option."
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac

4. Debugging (How to fix broken scripts)

When a complex script fails, it can be hard to see why. You can turn on "Debug Mode" to print every command before it executes.

Option A: Add -x to the shebang line: #!/bin/bash -x
Option B: Run the script with bash: bash -x myscript.sh

Output Example:

+ ACTION=start
+ case "$ACTION" in
+ echo '🟢 Starting application...'
🟢 Starting application...

The lines starting with + show you exactly what Bash is doing.

5. The "Master Script": Putting it all together

Here is a professional-grade script that combines Functions, Arrays, Checks, and Logging. This is the level of scripting expected in a DevOps interview.

Scenario: A "System Provisioner" script that sets up a new server.

#!/bin/bash
set -e  # Exit on error

# --- Configuration ---
LOG_FILE="/var/log/provision.log"
PACKAGES=("vim" "git" "nginx" "jq")

# --- Functions ---
log() {
    echo "[$(date +'%T')] $1" | tee -a "$LOG_FILE"
}

check_root() {
    if [ "$EUID" -ne 0 ]; then
        log "❌ Error: Please run as root."
        exit 1
    fi
}

install_packages() {
    log "📦 Updating package repositories..."
    apt-get update -y > /dev/null 2>&1

    for PKG in "${PACKAGES[@]}"; do
        log "   -> Installing $PKG..."
        apt-get install -y "$PKG" > /dev/null 2>&1
    done
}

configure_firewall() {
    log "shield: Configuring Firewall..."
    # Check if ufw is installed first
    if command -v ufw > /dev/null; then
        ufw allow 22/tcp
        ufw allow 80/tcp
        ufw --force enable
    else
        log "⚠️ Warning: UFW not found, skipping firewall."
    fi
}

# --- Main Execution Flow ---
check_root
log "🚀 Starting System Provisioning..."

install_packages
configure_firewall

log "✅ Provisioning Complete!"

Here are 4 real-world shell scripts used frequently in actual DevOps projects. These cover Monitoring, Cleanup, Backups, and CI/CD Automation.

In a real job, these usually run as Cron Jobs (scheduled tasks) or steps in a Jenkins/GitHub Actions pipeline.

1. The "Disk Space Alerter" (Monitoring)

Scenario: Servers often crash because logs fill up the disk. You need a script that runs every hour, checks disk usage, and sends a specific Slack notification if it crosses a dangerous threshold (e.g., 80%).

Key Concepts: df, awk, curl (for API calls).

#!/bin/bash

# Configuration
THRESHOLD=80
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T000/B000/XXXX"
HOSTNAME=$(hostname)

# Get current disk usage percentage (stripping the % sign)
# df -h / gives usage of root partition. 
# awk 'NR==2 {print $5}' gets the percentage column from the second line.
# sed 's/%//g' removes the percentage sign so we can do math.
CURRENT_USAGE=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//g')

if [ "$CURRENT_USAGE" -gt "$THRESHOLD" ]; then
    echo "⚠️ Disk space critical: ${CURRENT_USAGE}% detected."

    # Send Alert to Slack
    MESSAGE="🚨 *CRITICAL ALERT* \nServer: $HOSTNAME \nDisk Usage: ${CURRENT_USAGE}% \nPlease clean up immediately!"

    curl -X POST -H 'Content-type: application/json' \
        --data "{\"text\": \"$MESSAGE\"}" \
        "$SLACK_WEBHOOK_URL"
else
    echo "✅ Disk space normal: ${CURRENT_USAGE}%"
fi

2. The "Docker Garbage Collector" (Maintenance)

Scenario: CI/CD runners (like Jenkins agents) build thousands of Docker images. Eventually, the disk fills up with unused "dangling" images. This script cleans them up safely.

Key Concepts: docker, if/else, Exit Codes.

#!/bin/bash

echo "🐳 Starting Docker Cleanup..."

# 1. Prune stopped containers (older than 24h) to prevent deleting active work
# -f means force (don't ask for confirmation)
# --filter "until=24h" creates a safety buffer
docker container prune -f --filter "until=24h"

# 2. Prune unused images (dangling)
docker image prune -f

# 3. Prune unused volumes (be careful with this in production!)
docker volume prune -f

# Check how much space is left
FREE_SPACE=$(df -h / | awk 'NR==2 {print $4}')
echo "✅ Cleanup Complete. Free Space: $FREE_SPACE"

3. The "Log Rotator & S3 Uploader" (Cloud Ops)

Scenario: You need to keep application logs for legal reasons (Compliance), but you can't keep them on the server forever because they are expensive. This script compresses yesterday's logs and pushes them to AWS S3 (Cheap storage).

Key Concepts: tar (compression), date math, aws cli.

#!/bin/bash
set -e # Exit if any command fails

# Variables
LOG_DIR="/var/log/myapp"
ARCHIVE_DIR="/tmp/log_archives"
S3_BUCKET="s3://my-company-logs-backup"
YESTERDAY=$(date -d "yesterday" +'%Y-%m-%d')
ARCHIVE_NAME="logs-$YESTERDAY.tar.gz"

mkdir -p $ARCHIVE_DIR

echo "📦 Compressing logs for $YESTERDAY..."

# Find logs from yesterday and compress them
# We assume logs are named like 'app-2024-01-01.log'
tar -czf "$ARCHIVE_DIR/$ARCHIVE_NAME" -C "$LOG_DIR" . 

echo "☁️ Uploading to AWS S3..."
aws s3 cp "$ARCHIVE_DIR/$ARCHIVE_NAME" "$S3_BUCKET/$ARCHIVE_NAME"

if [ $? -eq 0 ]; then
    echo "✅ Upload Successful. Deleting local archive..."
    rm "$ARCHIVE_DIR/$ARCHIVE_NAME"

    # Optional: Delete the original log files from the server to save space
    # find "$LOG_DIR" -type f -name "*$YESTERDAY*" -delete
else
    echo "❌ Upload Failed!"
    exit 1
fi

4. The "Semantic Version Tag Generator" (CI/CD)

Scenario: In a CI pipeline (GitLab CI or Jenkins), you want to automatically generate a new version number (e.g., v1.2.5) every time a build passes, based on the previous tag.

Key Concepts: String manipulation, git.

#!/bin/bash

# Get the latest tag (e.g., v1.2.4). If no tag, default to v1.0.0
LATEST_TAG=$(git describe --tags `git rev-list --tags --max-count=1` 2>/dev/null || echo "v1.0.0")

echo "Current Version: $LATEST_TAG"

# Split the string by "."
# v1.2.4 -> MAJOR=v1, MINOR=2, PATCH=4
MAJOR=$(echo $LATEST_TAG | awk -F. '{print $1}')
MINOR=$(echo $LATEST_TAG | awk -F. '{print $2}')
PATCH=$(echo $LATEST_TAG | awk -F. '{print $3}')

# Increment the Patch version (v1.2.4 -> v1.2.5)
NEW_PATCH=$((PATCH + 1))
NEW_TAG="$MAJOR.$MINOR.$NEW_PATCH"

echo "🚀 New Version Detected: $NEW_TAG"

# Apply the tag (In a real pipeline, you would push this back to git)
# git tag $NEW_TAG
# git push origin $NEW_TAG

How to use these in an Interview?

If asked, "What shell scripts have you written?", pick Script #1 (Monitoring) or Script #3 (Backup).

Say this:

"I wrote a script to automate our log retention policy. It ran as a cron job every night, compressed the application logs, and uploaded them to an S3 bucket for long-term storage, which saved us about 20% on EBS disk costs."

DEV Community

Shell script for Devops

The Ultimate Guide to Shell Scripting

1. What is a Shell?

2. Anatomy of a Script

The Shebang (`#!`)

Permissions

3. Variables

Defining and Using Variables

Special Variables (Arguments)

4. User Input

5. Conditionals (If/Else)

Syntax

Comparison Operators

6. Loops

The `For` Loop

The `While` Loop

7. Functions

8. Best Practices for Professional Scripts

Summary Table

1. Functions (Don't Repeat Yourself)

2. Arrays (Handling Lists)

3. Case Statements (The "Menu" Logic)

4. Debugging (How to fix broken scripts)

5. The "Master Script": Putting it all together

1. The "Disk Space Alerter" (Monitoring)

2. The "Docker Garbage Collector" (Maintenance)

3. The "Log Rotator & S3 Uploader" (Cloud Ops)

4. The "Semantic Version Tag Generator" (CI/CD)

How to use these in an Interview?

Top comments (0)

The Ultimate Guide to Shell Scripting

1. What is a Shell?

2. Anatomy of a Script

The Shebang (#!)

Permissions

3. Variables

Defining and Using Variables

Special Variables (Arguments)

4. User Input

5. Conditionals (If/Else)

Syntax

Comparison Operators

6. Loops

The For Loop

The While Loop

7. Functions

8. Best Practices for Professional Scripts

Summary Table

1. Functions (Don't Repeat Yourself)

2. Arrays (Handling Lists)

3. Case Statements (The "Menu" Logic)

4. Debugging (How to fix broken scripts)

5. The "Master Script": Putting it all together

1. The "Disk Space Alerter" (Monitoring)

2. The "Docker Garbage Collector" (Maintenance)

3. The "Log Rotator & S3 Uploader" (Cloud Ops)

4. The "Semantic Version Tag Generator" (CI/CD)

How to use these in an Interview?

The Shebang (`#!`)

The `For` Loop

The `While` Loop