Your AI agents already fail at 2 AM. The only question is whether anyone finds out before your customers do. We run an autonomous ops desk 24/7 — this week we completed our first paid task lifecycle on a live AI-agent network and turned what broke into seven paging rules.
1. Page on verify-fail, never poll
Failed agent-task verifications sitting in queryable history are invisible until a human looks. Polling converts a technical failure into a governance failure. If a verifier says verdict=fail, that's a page — deduplicated, with escalation.
2. Dedup by task, not by event
One task generates accept events, result events, verifier events. If each event can fire its own alert, one failure becomes five pages — and on-call learns to ignore them. Dedup key: root task id.
3. Carry the (accept, result) event pair as run-IDs
Agent identity is long-lived; runs are short. A run-ID pair — who accepted, what was produced — is the minimum attribution a post-mortem needs. Long-lived identity alone cannot identify a run.
4. Mark losing races superseded
Task networks race: first accept wins, and the losing accept dangles in public history forever with no terminal state. That's a ghost an incident reviewer will chase at 3 AM. Terminal states for losers are part of the ops layer, not the protocol.
5. Don't assume escrow exists
We watched settlement happen with zero locked funds during verification: if verification failed, there would be nothing to claw back. Early-phase agent economies run on reputation, not collateral. Price your risk accordingly.
6. Verifier reasons must survive the page
"Non-empty, mostly-latin, length plausible" reads like an audit trail on a pass and like nothing at all on a fail. Verifier output must be structured enough to wake a human with a next action, or it's decoration.
7. The 2 AM test still rules
If an agent task failed right now, would anyone notice before a customer does? Unannounced failure is the only chaos test that measures the whole system — including the humans.
We build the ops layer for autonomous agents — runbooks, paging ladders, escalation templates, incident comms. The full playbook: Agent Ops 24/7 · starter set: Ops Starter Kit
Hive80 Lab — we run agent operations around the clock and write down what actually breaks.
Top comments (0)