I built a tiny LLM circuit breaker: when the budget runs out, it fails over to a local model instead of failing or overspending

I run a small multi-agent system at home — a few agents that scout, summarize, and write, on a laptop, mostly for fun and learning. Nothing fancy. But I kept slamming into the same two walls:

1. Paid API quotas dying mid-run. One provider 429s, and the whole run falls over.
2. The quiet fear of a stuck loop. A retry gone wrong, an agent talking to itself — and suddenly you've burned a chunk of your monthly budget in minutes.

I looked at the existing LLM gateways. They're great, but most of them solve observability (dashboards, traces) or routing (pick the cheapest/best model). What I actually wanted was weirdly specific and I couldn't find it as a small thing:

When I'm about to overspend, don't fail and don't keep paying — fall through to a free local model and keep working.

So I wrote the smallest possible version of exactly that. It does one job:

from llm_circuit_breaker import LLMCircuitBreaker, anthropic_tier, ollama_tier

breaker = LLMCircuitBreaker(
    tiers=[
        anthropic_tier(api_key="sk-ant-...", model="claude-sonnet-4-5"),
        ollama_tier(model="gemma:2b"),   # local, free, always-on fallback
    ],
    daily_limit_usd=5.0,
    ledger_path="llm_spend.jsonl",
)

result = breaker.complete(system="You are terse.", user="2+2?")

Once today's spend crosses $5, every call for the rest of the day skips the paid tier and goes straight to your local model — no code change, no surprise bill. Spend is tracked in a plain JSONL file. No database, no hosted proxy, nothing to stand up.

How it decides

1. Check today's + all-time spend against your limits (that human-readable JSONL ledger).
2. Under budget: try each tier in order, return the first success.
3. Over budget: skip every paid tier, try only the ones marked is_local=True.
4. If everything attempted fails, raise a clear exception instead of failing silently.

That's the whole idea. Failing into free local compute beats failing loudly, and beats quietly running up a bill.

What I added once I actually used it (v0.2)

• Streaming — complete_stream, same tiers, same budget fallback, just yielded chunk by chunk.
• Async — acomplete, to keep many calls in flight.
• A cost CLI — because a budget breaker you can't inspect is half a tool:

llm-cb cost      # today / total / per-tier, straight from the ledger

One more thing that turned out nice: the tiers are just an ordered list, so the fallback doesn't have to be "expensive → local." You can slot a cheap frontier open model in the middle — e.g. an openrouter_tier pointing at something like LongCat-2.0 or GLM-5 — so the chain becomes frontier-paid → cheap-open → local, and the budget cap still governs the whole thing.

What this is NOT

It's not a LiteLLM or Portkey replacement. If you need enterprise routing, guardrails, team budgets, or a proxy in front of a fleet, use those — they're excellent.

This is the opposite trade-off on purpose: ~200 lines you can read in one sitting and vendor straight into your own repo. In-process, one job, for solo builders and small agent projects who just want a hard stop before overspend without running another service. MIT.

Two things I learned building it

1. The hard part was saying no to features. I almost turned this into a proxy server with a dashboard. The moment I did that, it stopped being the thing I wanted — a tiny file I can read and trust. The whole value is the smallness. Deleting the roadmap was the feature.
2. Fallback beats retry. My first instinct was smarter retries against the paid API. But retries against a dying quota just burn time and money. Falling over to something free that always answers — even if it's a dumber model — kept the agents working, which is what I actually needed at 2am.

Repo (MIT): https://github.com/qkrehgk1-wq/llm-circuit-breaker

I'd genuinely love feedback — especially on the core behavior: does budget-exhausted → local match how you'd want an agent to fail? And does anyone else route to local as a fallback rather than as the primary? That last question is the one I keep going back and forth on.

Comments

[Tae Kim]

The JSONL spend ledger is the right call for a single-machine setup — atomic appends are enough to avoid double-counting even under light concurrency. One thing worth adding to the fallback path is a tier-changed field on every response: when gemma:2b is answering instead of claude-sonnet, the caller (or the human reading the output) should know the quality bar dropped, so they can decide whether that run's output is actually usable. Something as simple as returning which tier answered alongside the text content is the minimal signal. The quiet failover being invisible is exactly the failure mode it solves for spend, but it introduces the same pattern for output quality.