Everyone demos the agent demo. Nobody writes the runbook for 3am when it silently fails. I run agent workloads daily and packaged the operational side — failure taxonomy, health checks, cost controls, rollback — into a $22 kit. Here's what actually breaks.
The failure taxonomy (it's not "the model errored")
| Failure class | Symptom | Root cause |
|---|---|---|
| Silent degradation | Outputs get vaguer, nobody notices for days | Prompt/model drift, context rot |
| Cost spiral | Bill 3x's overnight | Retry loops, unbounded context, no token budgets |
| Stuck loop | Agent repeats the same action forever | Missing termination condition, tool returning errors it can't parse |
| Stale memory | Agent "forgets" decisions made yesterday | No persistence layer, or context window eviction |
| Bad tool calls | Right intent, wrong arguments | Schema drift, ambiguous tool descriptions |
The dangerous ones are silent — the agent keeps producing output, just worse output.
Health checks that catch silent failures
Don't just check "did it respond." Check:
- Output schema validation — does the response match the contract? A 200 with malformed JSON is a failure.
- Eval suite regression — run a fixed set of test prompts weekly; score drift >10% = investigate.
- Cost per task — alert at 2x rolling average, not at month-end.
- Tool-call success rate — if a tool's error rate climbs, the agent will start hallucinating workarounds.
Cost controls that actually work
- Token budgets per task type — a classification task doesn't need the same budget as a report.
- Context pruning — drop old tool results before they hit the window; summarize, don't accumulate.
- Model routing — cheap model for triage, expensive model only for the 10% that need it.
- Circuit breakers — N consecutive failures = stop and alert, don't retry forever.
The rollback problem nobody talks about
When you change a prompt and things get worse, how do you roll back? Prompt versioning is the answer — every prompt change is a git commit with an eval score attached. "It worked yesterday" is not a rollback plan.
Pre-flight checklist (60 seconds before deploy)
- [ ] Eval suite passes on the new prompt/version
- [ ] Token budget set for this task type
- [ ] Failure alerts wired to somewhere a human sees
- [ ] Rollback = revert commit, documented
- [ ] Cost ceiling for the run defined
The honest part
Runbooks don't prevent failures — they make failures cheap. The agent will still do something dumb at 3am; the question is whether you find out in minutes or weeks, and whether recovery takes a revert or a rebuild.
The kit — 12 runbooks covering failure taxonomy, health checks, cost controls, prompt versioning, observability schema, incident response, pre-flight, quarterly review — is in the store:
Store (pay in BTC, no account, instant download): https://dev.to/nexusfoundry/nexus-store-live-link-auto-updated-1918 (auto-updated current store URL)
Top comments (0)