At 11:06, a "temporary" sync job hit its error path and started retrying. By 11:19 it had run 14,000 times. It had emailed our entire customer list three times, called a paid API 200,000 times, and the developer who wrote it was in a meeting, unaware that his creation had opinions.
We stopped it in 40 seconds — because two weeks earlier we'd installed a kill switch. Before that, stopping it would have meant: SSH to the box, crontab -l, find the job, comment it out, hope there wasn't a second copy, and repeat for every server. We know, because that's how the first incident went.
Here's the runbook we now install with every piece of automation.
1. Every automation gets a kill switch reachable in one command.
Not "we could stop it" — a flag file a wrapper checks ([ -f /etc/job.pause ] && exit 0), a queue-pause command, an external iteration cap for agents, or a revoked API key for integrations. One touch stops every wrapped job; no crontab surgery under pressure. The acceptance test: a teammate who's never seen the system can stop it in under a minute using only the runbook. If stopping requires memory, the kill switch doesn't exist.
2. Ceilings live OUTSIDE the automation.
A looping script cannot be reasoned with — and neither can a looping AI agent. Telling it "please stop" via prompt is not a control. The caps that work are external: max runs per window enforced by the scheduler wrapper, max spend per day on anything that bills per call, dead-letter after N failures instead of infinite retry. Infinite retry is a second automation nobody agreed to run.
3. Idempotency is what keeps re-runs from double-charging.
The scariest runaway incidents do real work twice: two invoices per customer, two shipments, two identical apology emails (which is what the incident becomes). A job ID plus an idempotency key on every external side effect, and a check-before-act guard on stateful steps, turn re-runs into log lines instead of customer calls.
4. After the fire: the five-line checklist.
What guardrail was missing (every runaway has exactly one absent ceiling — install it this week)? What did it cost? Who got spammed or double-charged (outbound apologies beat inbound complaints)? Did the kill switch actually work? When is the quarterly re-test?
The uncomfortable truth: our runaway job had no ceiling above it. No max runs, no max spend, no dead-letter. Every guardrail I've listed is a one-hour install. The runaway version is one busy month away.
The full runbook — kill switch patterns per layer, ceiling setup, the post-incident checklist — is free:
→ https://hive80-lab.github.io/ops-notes/runaway-automation-runbook.html
The Automation Starter Pack ($19) is built around workflows with these bounds installed — flag-file wrappers, max-run ceilings, the weekly review. If part of your stack is autonomous agents, the Agent Ops 24/7 ($19) playbook covers keepalives, watchdogs, and graceful shutdowns. Launch-week pricing: 30% off with code HIVE-LAUNCH30 at https://hive80lab.gumroad.com
Free starting point: The First 30 Minutes — the one-page quick-start for any outage: https://hive80lab.gumroad.com/l/first-30-minutes
Top comments (0)