We Put Our AI Agent On-Call. It Paged Us 40 Times Before Breakfast.
There's a version of AI-agent ops nobody posts about: the agent works, the demo is great, and then you give it a real job with real integrations — and the pages start.
Ours burned through its token budget twice in one night, retried a failing webhook 400 times, and emailed the whole team at 3am "for visibility." Forty pages before breakfast. The fix wasn't smarter AI. It was the same boring ops discipline we apply to every other production system.
What actually stopped the pain
1. Budget ceilings with teeth. A hard cap at the provider, an alert at 80%. The runaway loop is the most common failure mode and it bills per token. Now a bad loop costs hours, not rent money.
2. Retry budgets, not infinite retries. Cap retries per destination (3, then stop and flag). Exponential backoff only if there's a human still watching. An agent that retries forever is just a DDoS you wrote yourself.
3. Notification routing that respects sleep. "For visibility" emails go to a dashboard, not to humans' phones. Pages are for "production is down," nothing else. The agent gets one severity level: it can ask for attention, it cannot grant itself an emergency.
4. Scope per job, not per agent. The webhook-retrying agent doesn't need the database key. Rotating one scoped credential after an incident takes a minute; untangling a shared god-key takes a week.
5. A kill switch you've actually tested. One command revokes everything. We tested it by accident once (thanks, fat-fingered teammate) and it worked in under 30 seconds. That accidental test was worth more than any runbook we never read.
The pattern underneath
Every one of these is a constraint on autonomy, not a feature of intelligence. The agent didn't get smarter. It got smaller blast radius, capped spend, and a curfew — exactly what you'd give a very fast intern on their first day.
The teams doing this well treat agents like junior staff: promising, eager, and one misconfiguration away from a very bad day. The teams doing it badly gave the agent root and went to bed.
If you're standing up agents in production, the boring paperwork matters more than the model choice. We packaged our runbooks, paging rules, and the incident comms templates into the Ops Starter Kit — Incident Response for Small Teams and the advanced volume (Ops Starter Kit Vol. 2) — plus Agent Ops 24/7 for teams that want continuous monitoring instead of 3am surprises.
The 40 pages before breakfast cost nothing after the ceilings went in. Write yours down before breakfast finds you.
Top comments (0)