Nobody gets paged at 2AM because the system failed. They get paged because nobody designed the failure to be survivable without a human.
After auditing on-call rotations across a dozen small teams, the pattern is always the same: the alerting exists, the dashboards exist, the runbooks are missing or 400 lines long. The pager fires, the human wakes up, and the human guesses.
Here are the seven safeguards that actually reduce pages — ranked by how little effort they take.
1. Auto-remediate the top 3 recurring pages
Pull your last 90 days of alerts. The top three causes are almost always: disk full, memory leak restart, dead worker process. Each has a five-line fix script. Write them, cron them, log every auto-action. This alone kills 40–60% of page volume.
2. Every alert links to its runbook — one click, no wiki search
If a responder has to find the runbook at 2AM, you don't have a runbook, you have a scavenger hunt. Put the deep link directly in the alert payload.
3. Severity means something
If 80% of your alerts are "critical," critical is a decoration. Three tiers maximum: page now, business hours, log only. Re-audit monthly.
4. Sleep-based suppression windows
Deploys and batch jobs generate known, transient noise. Suppress alerts on tagged services during their own deploy windows. The pager should never fire for a change you are watching anyway.
5. The 20-minute restore drill
Your backups are a theory until you restore one. Schedule a drill: pick last night's backup, restore it into a scratch environment, verify the app boots on it. Twenty minutes, monthly. The first team that runs this almost always finds their backups have been silently broken for months.
6. An escalation policy with teeth
- 0–5 min: on-call acks or auto-escalates
- 5–15 min: secondary on-call + incident channel opened
- 15 min: incident commander named, status page updated
No policy means the most junior person stares at the problem alone until sunrise.
7. Blameless postmortems with one owner and one deadline
A postmortem that produces no owned action item is creative writing. Every incident gets exactly one owner and one due date for its fix.
The theme across all seven: move judgment from the human to the system. The human at 2AM should be a supervisor of automation, not its fallback.
If you want the shortcut version — pre-built auto-remediation scripts, alert-tuning checklists, and the restore drill as a runnable playbook — we package exactly that:
👉 Agent Ops 24/7 — autonomous on-call automation kit
The full storefront (starter kits, automation packs) is here: hive80lab.gumroad.com
Ship calm. Sleep through the night.
Top comments (0)