The Solo DevOps Engineer Survival Guide: Running 24/7 Without a Team

Being the only DevOps person at a company is not a job. It is a survival game. Here is how I kept systems running 24/7 without burning out, with the exact tools and scripts I used.

The Reality of Solo DevOps

You are the on-call engineer. The deployment specialist. The security person. The monitoring guy. The incident responder. And you still have to ship features.

The math does not work: 168 hours in a week, you can work maybe 50. That means systems run unmonitored 118 hours per week. Unless you automate.

Principle 1: Automate Before You Sleep

Every task you do twice should be automated the third time. No exceptions.

import subprocess
import json
from datetime import datetime

class AutoDeploy:
    def __init__(self):
        self.steps = []

    def add_step(self, name, command):
        self.steps.append({"name": name, "command": command})

    def run(self):
        results = []
        for step in self.steps:
            start = datetime.now()
            try:
                output = subprocess.run(step["command"], shell=True, capture_output=True, text=True, timeout=300)
                results.append({
                    "name": step["name"],
                    "status": "success" if output.returncode == 0 else "failed",
                    "duration": str(datetime.now() - start),
                    "output": output.stdout[:500]
                })
                if output.returncode != 0:
                    self.alert(f"Deploy step failed: {step['name']}", output.stderr)
                    break
            except subprocess.TimeoutExpired:
                results.append({"name": step["name"], "status": "timeout"})
                self.alert(f"Deploy step timed out: {step['name']}")
                break
        return results

    def alert(self, message, details=""):
        print(f"ALERT: {message}")
        if details:
            print(f"Details: {details}")

Principle 2: Monitor Everything, Alert on What Matters

Not all alerts are equal. A disk at 85% is information. A disk at 95% is urgent. A disk at 99% is a 3am page.

import shutil
import socket
import time

def check_system_health():
    # Disk usage
    disk = shutil.disk_usage("/")
    disk_pct = (disk.used / disk.total) * 100

    # Memory
    with open("/proc/meminfo") as f:
        meminfo = f.readlines()
    mem_total = int(meminfo[0].split()[1])
    mem_available = int(meminfo[2].split()[1])
    mem_pct = ((mem_total - mem_available) / mem_total) * 100

    # CPU load
    load_avg = float(open("/proc/loadavg").read().split()[0])

    # Network connectivity
    try:
        socket.create_connection(("8.8.8.8", 53), timeout=3)
        network = "ok"
    except:
        network = "down"

    return {
        "disk_pct": round(disk_pct, 1),
        "mem_pct": round(mem_pct, 1),
        "load_avg": load_avg,
        "network": network,
        "timestamp": time.time()
    }

def should_alert(metrics):
    if metrics["disk_pct"] > 90: return ("critical", "Disk usage above 90%")
    if metrics["mem_pct"] > 90: return ("critical", "Memory usage above 90%")
    if metrics["load_avg"] > 4: return ("warning", "Load average above 4")
    if metrics["network"] == "down": return ("critical", "Network is down")
    return (None, None)

Principle 3: Document Like You Will Forget Everything Tomorrow

Because you will. At 3am, you will not remember why the database failover script has a 30-second sleep.

def generate_runbook(alert_type):
    runbooks = {
        "disk_full": {
            "severity": "critical",
            "steps": [
                "Check largest directories: du -sh /* | sort -h",
                "Clean old logs: find /var/log -name '*.gz' -mtime +7 -delete",
                "Clean docker: docker system prune -a --volumes",
                "If still full: identify and move large files to S3",
                "Update disk size if recurring: aws ec2 modify-volume"
            ],
            "estimated_time": "15 minutes",
            "escalation": "If not resolved in 30 min, page infrastructure lead"
        },
        "high_load": {
            "severity": "warning",
            "steps": [
                "Check top processes: ps aux --sort=-%cpu | head -10",
                "Check for runaway queries: SELECT * FROM pg_stat_activity WHERE state='active' ORDER BY query_start",
                "Restart problematic service: systemctl restart <service>",
                "Scale horizontally if traffic spike: kubectl scale deploy <name> --replicas=4"
            ],
            "estimated_time": "10 minutes",
            "escalation": "If load stays above 8 for 15 min, scale cluster"
        }
    }
    return runbooks.get(alert_type, {"error": "No runbook for this alert type"})

Principle 4: Schedule Everything

If it is not scheduled, it does not happen. Backups, updates, security scans, log rotation.

import schedule
import time

def daily_backup():
    # Dump database, upload to S3, verify integrity
    pass

def security_scan():
    # Run vulnerability scanner, check SSL certs, review access logs
    pass

def log_rotation():
    # Compress old logs, delete logs older than 30 days
    pass

def dependency_update():
    # Check for security patches, update packages, run tests
    pass

# Schedule everything
schedule.every().day.at("02:00").do(daily_backup)
schedule.every().day.at("04:00").do(log_rotation)
schedule.every().monday.at("03:00").do(security_scan)
schedule.every().wednesday.at("03:00").do(dependency_update)

Principle 5: Have a Life

The most important principle. If you burn out, the systems go down anyway.

• Set working hours and stick to them
• Phone on silent after 10pm (unless on call)
• Take weekends off (automate everything that used to need weekend work)
• Use your vacation days (test your automation by being unreachable)
• Exercise, sleep, eat properly (your health IS your infrastructure)

The Solo DevOps Toolkit

Category	Tool	Why
Monitoring	Prometheus + Grafana	Free, powerful, self-hosted
Alerting	AlertManager	Routes alerts to the right channel
CI/CD	GitHub Actions	Free for small projects
Backups	restic + S3	Fast, encrypted, deduplicated
Secrets	Vault or SOPS	Never commit secrets to git
Documentation	MkDocs	Simple, version-controlled docs
Communication	Slack + PagerDuty	Alerts that actually wake you up

---

For complete DevOps automation templates, check out the Ops Starter Kit.