I Replaced Our Entire On-Call Rotation With 4 Python Scripts (And Slept Through 3 Incidents)

I Replaced Our Entire On-Call Rotation With 4 Python Scripts

Last quarter, our 4-person team was woken up 47 times for after-hours alerts. This quarter: 3 times. Here is exactly what changed.

The Problem: Alert Fatigue Is Real

We had a PagerDuty rotation. We had runbooks. We had a Slack channel dedicated to incidents. And yet, every engineer on our team was burning out.

The issue wasn't the alerts themselves — it was that 80% of them required the same response: check the service, restart if needed, verify recovery, log the incident. A human doing this takes 15-20 minutes. A script takes 4 seconds.

So I wrote 4 scripts. Here they are.

---

Script 1: The Auto-Triage Engine

This script receives a webhook from our monitoring system, classifies the alert, and decides whether to auto-remediate or escalate.

import json, subprocess, re
from datetime import datetime

ALERT_RULES = {
    "service_down": {"action": "restart", "auto_remediate": True},
    "high_latency": {"action": "check_cache", "auto_remediate": True},
    "disk_full": {"action": "clean_logs", "auto_remediate": True},
    "security_breach": {"action": "escalate", "auto_remediate": False},
}

def triage_alert(webhook_payload):
    alert_type = classify_alert(webhook_payload)
    rule = ALERT_RULES.get(alert_type, {"action": "escalate", "auto_remediate": False})

    if rule["auto_remediate"]:
        result = execute_remediation(rule["action"], webhook_payload)
        log_incident(webhook_payload, result, auto=True)
        return {"status": "auto_resolved", "action": rule["action"]}
    else:
        notify_on_call(webhook_payload)
        log_incident(webhook_payload, "escalated", auto=False)
        return {"status": "escalated", "reason": "manual_required"}

def classify_alert(payload):
    message = payload.get("message", "").lower()
    if "down" in message or "unreachable" in message:
        return "service_down"
    if "latency" in message or "slow" in message:
        return "high_latency"
    if "disk" in message or "space" in message:
        return "disk_full"
    if "security" in message or "breach" in message:
        return "security_breach"
    return "unknown"

def execute_remediation(action, payload):
    if action == "restart":
        svc = payload.get("service", "nginx")
        subprocess.run(["systemctl", "restart", svc], capture_output=True)
        return f"restarted {svc}"
    elif action == "clean_logs":
        subprocess.run(["find", "/var/log", "-name", "*.gz", "-delete"], capture_output=True)
        return "cleaned old logs"
    elif action == "check_cache":
        subprocess.run(["redis-cli", "flushall"], capture_output=True)
        return "flushed cache"
    return "no action taken"

def log_incident(payload, result, auto):
    entry = {
        "timestamp": datetime.now().isoformat(),
        "alert": payload.get("message", ""),
        "result": result,
        "auto_remmediate": auto,
    }
    with open("/var/log/incidents.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

Impact: 38 out of 47 alerts auto-resolved without waking anyone.

---

Script 2: The Log Pattern Detector

This one runs every 5 minutes and looks for error patterns that precede outages. It caught 12 incidents before they became alerts.

import re
from collections import Counter, deque
from datetime import datetime, timedelta

class LogPatternDetector:
    def __init__(self, log_path, window_minutes=5):
        self.log_path = log_path
        self.window = timedelta(minutes=window_minutes)
        self.error_history = deque(maxlen=100)
        self.threshold = 3  # 3 similar errors in window = alert

    def scan(self):
        recent_errors = self._get_recent_errors()
        patterns = self._cluster_errors(recent_errors)

        for pattern, count in patterns.items():
            if count >= self.threshold:
                self._alert(pattern, count)
                return {"pattern": pattern, "count": count, "action": "pre-emptive_alert"}

        self.error_history.extend(recent_errors)
        return {"status": "healthy", "errors_scanned": len(recent_errors)}

    def _get_recent_errors(self):
        errors = []
        with open(self.log_path) as f:
            for line in f:
                if "ERROR" in line or "WARN" in line:
                    errors.append(line.strip())
        return errors[-50:]

    def _cluster_errors(self, errors):
        normalized = []
        for err in errors:
            clean = re.sub(r'\d{4}-\d{2}-\d{2}.*?Z', '', err)
            clean = re.sub(r'0x[0-9a-f]+', '0xADDR', clean)
            clean = re.sub(r'\b\d+\b', 'N', clean)
            normalized.append(clean.strip())
        return Counter(normalized)

    def _alert(self, pattern, count):
        print(f"PRE-EMPTIVE ALERT: {count} occurrences of pattern: {pattern[:100]}")

Impact: We stopped 12 incidents before users ever noticed.

---

Script 3: The Post-Mortem Generator

After every incident, this script collects timeline data, correlates events, and drafts a post-mortem document. What used to take 2 hours now takes 2 minutes.

import json
from datetime import datetime, timedelta

def generate_postmortem(incident_id, start_time, end_time):
    timeline = collect_timeline(start_time, end_time)
    deploy_log = get_deploys_in_window(start_time, end_time)
    metrics = get_metrics(start_time, end_time)

    report = f"""# Post-Mortem: Incident {incident_id}

## Summary
- **Start**: {start_time}
- **End**: {end_time}
- **Duration**: {end_time - start_time}
- **Impact**: {assess_impact(metrics, start_time, end_time)}

## Timeline
{format_timeline(timeline)}

## Recent Deployments
{format_deploys(deploy_log)}

## Root Cause Analysis
{analyze_root_cause(timeline, deploy_log)}

## Action Items
{generate_action_items(timeline, metrics)}
"""
    save_postmortem(incident_id, report)
    return report

def analyze_root_cause(timeline, deploys):
    for event in timeline:
        if event["type"] == "deploy":
            time_to_incident = (timeline[0]["time"] - event["time"]).total_seconds()
            if time_to_incident < 600:
                return f"Deploy of {event['version']} preceded incident by {time_to_incident}s."
    return "No clear root cause from deploy correlation. Manual investigation needed."

def generate_action_items(timeline, metrics):
    items = []
    if any(m["value"] > m["threshold"] for m in metrics):
        items.append("- [ ] Add alerting for metric threshold breach")
    if not any(e["type"] == "rollback" for e in timeline):
        items.append("- [ ] Implement automated rollback for future incidents")
    items.append("- [ ] Review and update runbook with lessons learned")
    return "\n".join(items)

Impact: Post-mortems now take 2 minutes instead of 2 hours.

---

Script 4: The Capacity Predictor

This script analyzes resource trends and predicts when you will run out of disk, memory, or connections.

import statistics

def predict_capacity_exhaustion(metrics_history, resource, days_to_predict=30):
    values = [m[resource] for m in metrics_history]

    n = len(values)
    x_mean = statistics.mean(range(n))
    y_mean = statistics.mean(values)

    numerator = sum((i - x_mean) * (v - y_mean) for i, v in enumerate(values))
    denominator = sum((i - x_mean) ** 2 for i in range(n))

    slope = numerator / denominator if denominator else 0
    intercept = y_mean - slope * x_mean

    max_capacity = get_max_capacity(resource)
    threshold = max_capacity * 0.90

    if slope <= 0:
        return {"resource": resource, "status": "healthy"}

    days_until = int((threshold - intercept) / slope - n)

    if days_until <= days_to_predict:
        return {
            "resource": resource,
            "status": "warning",
            "message": f"{resource} will hit 90% capacity in ~{days_until} days",
            "current_usage": f"{values[-1]/max_capacity*100:.1f}%",
        }
    return {"resource": resource, "status": "healthy", "days_until_threshold": days_until}

Impact: We stopped 3 out-of-disk incidents and 2 connection-pool exhaustions before they happened.

---

The Results After 90 Days

Metric	Before	After
After-hours wakeups	47	3
Mean time to resolution	18 min	4 sec (auto) / 6 min (manual)
Post-mortem completion rate	40%	100%
Capacity incidents	5	0
Engineer burnout (self-reported)	7/10	2/10

What I Learned

1. Most alerts are boring. If the response is always the same, a script should do it.
2. Pre-emption beats reaction. The log pattern detector saved us more grief than the auto-restart.
3. Post-mortems must be frictionless. If it takes 2 hours, nobody does it. If it takes 2 minutes, everyone does.
4. Capacity planning is not optional. The predictor script paid for itself in week one.

---

Want the Complete Ops Automation Library?

These 4 scripts are just the beginning. I have put together a complete Ops Mega Bundle with all 5 automation kits (34 files total) that covers:

• Incident response automation
• Log analysis and pattern detection
• Post-mortem templates and generators
• Capacity planning tools
• On-call scheduling automation
• Deployment pipeline scripts
• Security audit checklists

Get the full bundle here: Ops Mega Bundle — $49 (normally $88)

Or start smaller:

• Ops Starter Kit — $19.64 — The essential scripts and templates
• Automation Starter Pack — $19 — Pick-first workflows for small teams

Use code MEGA50 for 50% off the Mega Bundle (first 20 buyers). 30-day money-back guarantee.

---

What automation scripts have saved your team the most time? Drop a comment below.