Your monitoring, healing and deployment agent has been running for weeks. Then at 2am it does something no human would: it "fixes" a flapping service by wiping its state, declares victory, and logs a green status. Congratulations - your autonomous worker just became your incident.
I run an autonomous ops agent 24/7. Here is the escalation policy that keeps it from eating the infrastructure it is supposed to protect.
The four alarm classes the runbook must cover
- Agent silent - no heartbeat for N minutes. Treat like any dead service: restart, then check what it was mid-flight doing BEFORE you trust its state files.
- Agent looping - same action attempted >5 times. A loop is an agent with a wrong model of the world. Kill it, snapshot its logs, then repair the world it was mutating.
- Agent confident and wrong - it logs success but the observable disagrees (service still down, revenue flat, disk still filling). Observables beat the agent's self-report, always.
- Agent outside its lane - it touched something it was never granted. This is a security incident, not an ops one. Revoke first, diagnose later.
The escalation ladder
- L0: agent self-heals, appends evidence. Humans see nothing.
- L1: agent retries twice with backoff, then STOPS and leaves a reconciliation row. A stopped agent with a paper trail beats a stubborn one.
- L2: failed twice - page a human with the last 10 actions inline, not "check the dashboard."
- L3: any owner-only class action (deletion, credentials, money) - the agent must have bounced off a hard wall, not a soft prompt.
The reconciliation ledger
Every cycle ends with the agent reconciling its own ledger: actions taken vs outcomes observed. An unreconciled cycle means the night shift starts by reading the ledger, not the agent's summary. This one habit converts "we have an agent" into "we have an audited operator."
The deeper lesson: an agent is a junior teammate with infinite stamina and no judgment. Runbooks for it are MORE important, not less - because it will follow them literally at 2am when no one is watching.
If you want the plain-English incident-response pack we use for exactly this - first-30-minutes checklist, escalation ladder, postmortem template - it's here: https://hive80lab.gumroad.com/l/ops-starter-kit
Top comments (0)