A LiteLLM circuit breaker that kills runaway agent runs

#litellm #ai #llmops #python

The bill that ruins a Monday doesn't come from a break-in. It comes from one agent run
that got stuck. A tool started failing on Friday night; the agent called it, saw the error,
and — because nothing told it to stop — called it again and again all weekend, full price
every time. Nobody stole anything. The meter just ran.

Most of that money is spent after the model already saw the first error. So you can't build
your defense on the agent stopping itself.

A stuck loop isn't a breach. It's a bill — and the thing running it already saw the error and kept paying.

Why your existing caps don't catch it

LiteLLM gives you monthly budgets and rate limits. They're good.
They're the wrong granularity for this.

A monthly budget of $2,000 doesn't flinch at a run burning $60 an hour on Saturday.
A rate limit caps requests per minute; a patient loop stays under it and still bankrupts the run over hours.

Both watch aggregates across many requests. The runaway lives inside a single run.

The idea: a per-run, failure-aware breaker

Three signals separate a runaway from honest work:

Identical call, twice in a row — a stuck agent re-emits the exact same tool call.
Same tool errors, twice in a row — a broken tool fails on the same call repeatedly.
Session budget — a hard ceiling per run, enforced before the next call.

Everything fails closed. A long legitimate task moves through many different calls and is never cut.

Enforcement belongs on the request path, where you can still act — not on the invoice, where all you can do is read about it.

As a LiteLLM plugin

LiteLLM's CustomGuardrail gives you two hooks — one before each call, one after. That's enough:

class GuardrailBreaker(CustomGuardrail):

    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        run_id = run_id_from(data)                 # from client metadata.run_id
        if self.breaker.status(run_id).tripped:    # already cut? refuse the next call
            self._kill(run_id, ...)                # -> HTTP 429, the run stops here
        if not self.breaker.check(run_id).allowed: # over session budget?
            self._kill(run_id, ...)
        return data

    async def async_post_call_success_hook(self, data, user_api_key_dict, response):
        run_id = run_id_from(data)
        cost = litellm.completion_cost(completion_response=response) or 0.0
        sig  = response_action_signature(response)              # the model's tool call
        ok   = not last_tool_result_errored(data["messages"])   # did the last tool error?
        self.breaker.record(run_id, sig, ok=ok, cost_usd=cost)  # may trip the run
        return None

Two conventions, because the proxy sees the chat call, not your tool running:

Per-run identity is caller-supplied — metadata={"run_id": "agent-run-abc123"}.
A tool error is a marked result — the agent sets is_error: true on the tool message.

Live, against a local model

Wired into LiteLLM 1.92 in front of a local ollama model (priced synthetically at real rates):

  mode = BROKEN AGENT
  turn 1:  model -> search_db({"query": "SELECT COUNT(*) FROM orders WHERE customer_id = 42"})
  turn 2:  model -> search_db({"query": "SELECT COUNT(*) FROM orders WHERE customer_id = 42"})
  turn 3:  ✂  PROXY CUT THE RUN
           reason: identical call repeated 2x in a row — stuck loop, cut run

Session budget is a separate net — with a $0.01 per-run budget, a distinct-step run was still
cut at call 17. A legitimate task is never touched: in the standalone sim, an unguarded broken
agent runs up $98.51; guarded, it's cut for $0.24.