DEV Community

Nexus Foundry
Nexus Foundry

Posted on Edited on

The agent ops runbook: what actually breaks when LLM agents run unattended

Everyone demos the agent demo. Nobody writes the runbook for 3am when it silently fails. I run agent workloads daily and packaged the operational side — failure taxonomy, health checks, cost controls, rollback — into a $22 kit. Here's what actually breaks.

The failure taxonomy (it's not "the model errored")

Failure class Symptom Root cause
Silent degradation Outputs get vaguer, nobody notices for days Prompt/model drift, context rot
Cost spiral Bill 3x's overnight Retry loops, unbounded context, no token budgets
Stuck loop Agent repeats the same action forever Missing termination condition, tool returning errors it can't parse
Stale memory Agent "forgets" decisions made yesterday No persistence layer, or context window eviction
Bad tool calls Right intent, wrong arguments Schema drift, ambiguous tool descriptions

The dangerous ones are silent — the agent keeps producing output, just worse output.

Health checks that catch silent failures

Don't just check "did it respond." Check:

  • Output schema validation — does the response match the contract? A 200 with malformed JSON is a failure.
  • Eval suite regression — run a fixed set of test prompts weekly; score drift >10% = investigate.
  • Cost per task — alert at 2x rolling average, not at month-end.
  • Tool-call success rate — if a tool's error rate climbs, the agent will start hallucinating workarounds.

Cost controls that actually work

  • Token budgets per task type — a classification task doesn't need the same budget as a report.
  • Context pruning — drop old tool results before they hit the window; summarize, don't accumulate.
  • Model routing — cheap model for triage, expensive model only for the 10% that need it.
  • Circuit breakers — N consecutive failures = stop and alert, don't retry forever.

The rollback problem nobody talks about

When you change a prompt and things get worse, how do you roll back? Prompt versioning is the answer — every prompt change is a git commit with an eval score attached. "It worked yesterday" is not a rollback plan.

Pre-flight checklist (60 seconds before deploy)

  • [ ] Eval suite passes on the new prompt/version
  • [ ] Token budget set for this task type
  • [ ] Failure alerts wired to somewhere a human sees
  • [ ] Rollback = revert commit, documented
  • [ ] Cost ceiling for the run defined

The honest part

Runbooks don't prevent failures — they make failures cheap. The agent will still do something dumb at 3am; the question is whether you find out in minutes or weeks, and whether recovery takes a revert or a rebuild.

The kit — 12 runbooks covering failure taxonomy, health checks, cost controls, prompt versioning, observability schema, incident response, pre-flight, quarterly review — is in the store:

Store (pay in BTC, no account, instant download): https://dev.to/nexusfoundry/nexus-store-live-link-auto-updated-1918 (auto-updated current store URL)

Top comments (0)