How to automate your boring tasks with scripts — a beginner tutorial for engineers

Automating Repetitive Development Tasks: A Practical Tutorial

Every developer wastes hours on repetitive work: cleaning cache folders, resizing images, backing up databases, scaffolding new projects, running the same test/deploy commands. This tutorial shows you how to automate these tasks with scripting, file manipulation, batch operations, database tasks, deployment helpers, and CI/CD automation-with real examples in Python, Bash, and Makefile.

Why Automation Matters

Tasks that take 5-10 minutes each add up quickly. Do them 10 times a week and you've lost 2-4 hours. A simple script reduces each task to a single command, saving hours over months.

Choosing the Right Tool for the Job

Tool	Best For	Strengths	Limitations
Bash	Quick automation, cron jobs, server workflows, cross-platform scripting	Complex logic, loops, conditionals; uses standard shell commands; very portable	Manual dependency tracking; imperative style
Makefile	Build/test workflows, standardized team commands, dependency tracking	Automatic dependency tracking; make help for onboarding; declarative syntax	Needs make installed; tab-sensitive syntax
Python	API calls, image processing, complex logic, database operations	Rich libraries; cross-platform; better for complex automation	Slower for simple shell tasks; needs dependencies

Rule of thumb: Use Makefile for repeatable build/test workflows, Bash for advanced logic/one-off automation, and Python for tasks needing APIs, image processing, or complex logic. Modern teams often combine all three: Makefile calls Bash scripts for advanced logic.

Part 1: Bash Scripts for Quick Automation

Script 1: Clean Up Development Folders

Remove junk files (.DS_Store, node_modules, Python cache, build folders, logs):

#!/bin/bash
### cleanup.sh - Remove junk files from development folders

echo "Cleaning up development folders..."

### Remove .DS_Store files (macOS)
find . -name ".DS_Store" -type f -delete
echo "Removed .DS_Store files"

### Remove node_modules folders
find . -name "node_modules" -type d -prune -exec rm -rf {} + 2>/dev/null
echo "Removed node_modules folders"

### Remove Python cache
find . -name "__pycache__" -type d -prune -exec rm -rf {} + 2>/dev/null
find . -name "*.pyc" -type f -delete
echo "Removed Python cache"

### Remove .next build folders
find . -name ".next" -type d -prune -exec rm -rf {} + 2>/dev/null
echo "Removed .next build folders"

### Remove log files
find . -name "*.log" -type f -delete
echo "Removed log files"

echo "Cleanup complete!"

Save as cleanup.sh, make executable with chmod +x cleanup.sh, and run anywhere.

Script 2: Create Project Structure

Scaffold a new project with your preferred structure:

#!/bin/bash
### newproject.sh - Create a new project structure

if [ -z "$1" ]; then
  echo "Usage: newproject.sh <project-name>"
  exit 1
fi

PROJECT_NAME=$1

### Create main directory
mkdir -p "$PROJECT_NAME"/{src/{components,hooks,utils,styles},public/images,tests,docs}

### Create base files
touch "$PROJECT_NAME/README.md"
touch "$PROJECT_NAME/.gitignore"
touch "$PROJECT_NAME/src/index.ts"
touch "$PROJECT_NAME/src/styles/globals.css"

### Add content to .gitignore
cat > "$PROJECT_NAME/.gitignore" << EOF
node_modules/
.next/
.env.local
.env
*.log
.DS_Store
dist/
build/
EOF

echo "Project $PROJECT_NAME created successfully!"

Run with ./newproject.sh my-app.

Script 3: Batch Rename Files

#!/bin/bash
### rename.sh - Batch rename files with pattern

if [ -z "$1" ] || [ -z "$2" ]; then
  echo "Usage: rename.sh <search-pattern> <replace-pattern>"
  exit 1
fi

SEARCH=$1
REPLACE=$2
COUNT=0

for file in *"$SEARCH"*; do
  if [ -f "$file" ]; then
    newname=$(echo "$file" | sed "s/$SEARCH/$REPLACE/g")
    mv "$file" "$newname"
    echo "Renamed: $file -> $newname"
    ((COUNT++))
  fi
done

echo "Renamed $COUNT files."

Script 4: Archive Log Files

#!/bin/bash
### archive_logs.sh - Move log files to archive folder

mkdir -p ~/log_archive
mv ~/logs/*.log ~/log_archive/
echo "Logs archived successfully."

Part 2: Python Scripts for Complex Automation

Script 5: Resize Images for Web

#!/usr/bin/env python3
"""resize_images.py - Batch resize images for web"""

import os
import sys
from pathlib import Path

try:
    from PIL import Image
except ImportError:
    print("Install Pillow: pip install Pillow")
    sys.exit(1)

def resize_image(input_path: str, output_path: str, max_width: int = 1200):
    """Resize image maintaining aspect ratio."""
    with Image.open(input_path) as img:
        if img.width > max_width:
            ratio = max_width / img.width
            new_height = int(img.height * ratio)
            img = img.resize((max_width, new_height), Image.LANCZOS)

        if img.mode in ("RGBA", "P"):
            img = img.convert("RGB")

        img.save(output_path, "JPEG", quality=85, optimize=True)

def process_folder(input_folder: str, output_folder: str, max_width: int = 1200):
    """Process all images in a folder."""
    input_path = Path(input_folder)
    output_path = Path(output_folder)
    output_path.mkdir(parents=True, exist_ok=True)

    image_extensions = {".jpg", ".jpeg", ".png", ".gif", ".webp"}
    images = [f for f in input_path.iterdir()
              if f.suffix.lower() in image_extensions]

    print(f"Processing {len(images)} images...")

    for i, img_file in enumerate(images, 1):
        output_file = output_path / f"{img_file.stem}.jpg"
        try:
            resize_image(str(img_file), str(output_file), max_width)
            original_size = img_file.stat().st_size / 1024
            new_size = output_file.stat().st_size / 1024
            reduction = ((original_size - new_size) / original_size) * 100
            print(f"[{i}/{len(images)}] {img_file.name}: {original_size:.1f}KB -> {new_size:.1f}KB ({reduction:.1f}% smaller)")
        except Exception as e:
            print(f"[{i}/{len(images)}] Failed: {img_file.name} - {e}")

if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: python resize_images.py <input-folder> <output-folder> [max-width]")
        sys.exit(1)

    input_folder = sys.argv
    output_folder = sys.argv
    max_width = int(sys.argv) if len(sys.argv) > 3 else 1200
    process_folder(input_folder, output_folder, max_width)

Script 6: Database Backup (PostgreSQL)

#!/usr/bin/env python3
"""db_backup.py - Backup PostgreSQL database to file"""

import os
import subprocess
import sys
from datetime import datetime
from pathlib import Path

def backup_postgres(database_url: str, output_dir: str = "./backups"):
    """Create a PostgreSQL backup."""
    output_path = Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)

    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    backup_file = output_path / f"backup_{timestamp}.sql"

    print(f"Starting backup...")

    try:
        result = subprocess.run(
            ["pg_dump", database_url, "-f", str(backup_file)],
            capture_output=True,
            text=True
        )

        if result.returncode == 0:
            size = backup_file.stat().st_size / 1024 / 1024
            print(f"Backup successful: {backup_file}")
            print(f"Size: {size:.2f} MB")
            cleanup_old_backups(output_path, keep=7)
        else:
            print(f"Backup failed: {result.stderr}")

    except FileNotFoundError:
        print("pg_dump not found. Install PostgreSQL client tools.")

def cleanup_old_backups(backup_dir: Path, keep: int = 7):
    """Remove old backups, keeping only the most recent ones."""
    backups = sorted(backup_dir.glob("backup_*.sql"), reverse=True)
    for old_backup in backups[keep:]:
        old_backup.unlink()
        print(f"Removed old backup: {old_backup.name}")

if __name__ == "__main__":
    database_url = os.environ.get("DATABASE_URL")
    if not database_url:
        print("Set DATABASE_URL environment variable")
        sys.exit(1)

    output_dir = sys.argv if len(sys.argv) > 1 else "./backups"
    backup_postgres(database_url, output_dir)

PostgreSQL client tools include pg_dump for backing up databases and pg_restore for restoring them.

Script 7: Run Tests in CI/CD

#!/usr/bin/env python3
"""run_tests.py - Automate pytest in CI/CD"""

import subprocess
import sys

def run_tests():
    """Run pytest on the current directory."""
    result = subprocess.run(['pytest'], text=True)
    if result.returncode != 0:
        print("Some tests failed!")
        sys.exit(result.returncode)
    else:
        print("All tests passed!")

if __name__ == "__main__":
    run_tests()

Script 8: Build Docker Image

#!/usr/bin/env python3
"""build_docker.py - Build Docker image"""

import subprocess

def build_docker_image(image_name: str, dockerfile_path: str = '.'):
    """Build a Docker image using the provided Dockerfile."""
    command = ['docker', 'build', '-t', image_name, dockerfile_path]
    try:
        subprocess.run(command, check=True)
        print(f"Docker image '{image_name}' built successfully!")
    except subprocess.CalledProcessError as e:
        print("Error building Docker image.")
        print(e)

Part 3: Makefile for Project Automation

Create a Makefile in your project root:

coverage: ## Run tests with coverage
    coverage erase
    coverage run --include=podsearch/* -m pytest -ra
    coverage report -m

deps: ## Install dependencies
    pip install black coverage flake8 mypy pylint pytest tox

lint: ## Lint and static-check
    flake8 podsearch
    pylint podsearch
    mypy podsearch

push: ## Push code with tags
    git push && git push --tags

test: ## Run tests
    pytest -ra

help: ## Show all available commands
    @awk 'BEGIN {FS=":"} \
    /^#/ {comment=substr($$0,3)} \
    /^[a-zA-Z0-9_-]+:/ {printf "\033[36m%-20s\033[0m %s\n", $$1, comment}' Makefile

Run with make lint coverage or make test. Discover all tasks with make help.

Makefile Features

• Task steps: Each target can include multiple steps
• Dependencies: test: lint runs lint before test
• Parameters: Use bind ?= localhost for default values ### Part 4: CI/CD Automation with GitHub Actions

Create .github/workflows/ci-cd.yml:

name: CI/CD Pipeline
on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest

      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Log in to Docker Hub
        run: echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u "${{ secrets.DOCKER_USERNAME }}" --password-stdin

      - name: Push Docker image
        run: |
          docker push ${{ secrets.DOCKER_USERNAME }}/myapp:${{ github.sha }}

This pipeline triggers on every push to main, installs dependencies, runs tests, builds a Docker image, and pushes to Docker Hub. Add secrets via Settings > Secrets and variables > Actions.

Part 5: Patterns for Composable Automation

Pattern 1: Action + Verification

Build the health check before the action script. Never just act-verify the result.

Pattern 2: Auto-Heal Common Failures

Handle the top 3 failure modes inline instead of logging errors for "later".

Pattern 3: Structured Logging

Every automation writes a structured log. Never debug the same thing twice.

Best Practices

Practice	Why
Use version control (Git) for scripts	Track changes
Add logging and error handling	Diagnose issues
Test scripts in a safe environment	Avoid breaking production
Use virtual environments for Python	Manage dependencies
Set set -euo pipefail in Bash	Safer script defaults
Don't over-automate	If you do something once a month, a checklist beats a script
Log everything from day one	Future-you will thank present-you

Part 6: Making Scripts Easy to Use

Add Scripts to Your PATH

### Add to ~/.bashrc or ~/.zshrc
export PATH="$HOME/scripts:$PATH"

Now you can run scripts from anywhere.

Create Aliases

### Add to ~/.bashrc or ~/.zshrc
alias cleanup="~/scripts/cleanup.sh"
alias newproj="~/scripts/newproject.sh"
alias resize="python ~/scripts/resize_images.py"
alias backup="python ~/scripts/db_backup.py"

Schedule with Cron

### Edit crontab
crontab -e

### Add daily backup at 2 AM
0 2 * * * /usr/bin/python3 ~/scripts/db_backup.py >> ~/logs/backup.log 2>&1

### Weekly cleanup on Sunday at 3 AM
0 3 * * 0 ~/scripts/cleanup.sh >> ~/logs/cleanup.log 2>&1

Real-World Example: Full Deployment Helper

#!/bin/bash
### deploy.sh - Parameterized deployment with health checks and rollbacks

set -euo pipefail
ENVIRONMENT="${1:-staging}"
APP_DIR="/srv/myapp"
REPO_URL="[e-mail]:org/myapp.git"

log() { echo "[$(date +'%F %T')] [$ENVIRONMENT] $*"; }

log "Updating code..."
if [ ! -d "$APP_DIR/.git" ]; then
  git clone "$REPO_URL" "$APP_DIR"
fi
cd "$APP_DIR"
git fetch --all
git checkout main
git pull --ff-only

log "Installing dependencies..."
npm ci

log "Running tests..."
npm test

log "Building..."
npm run build

log "Deployment complete!"

This includes safer script defaults (set -euo pipefail), parameterized deployments, and logging.

Start Small

1. Identify a task you do repeatedly
2. Write a simple script to automate it
3. Refine as you use it

The 30 minutes you spend writing a script today can save you hours over the coming months. Over time, you'll build a personal toolkit of scripts that make you significantly more productive.

---

Rizwan Saleem — https://rizwansaleem.co