I Added a Circuit Breaker to My AI Agent Setup — It Caught Three Outages This Month

I have a confession: for the longest time, my local AI setup had a very embarrassing failure mode.

One of my machines goes down — a reboot, a driver update, someone trips over a power cable — and the agents that depend on it just keep firing requests at it. Every single request times out. Each one burns 30-60 seconds waiting for a connection that will never succeed. Meanwhile the task queue piles up, retries stack on retries, and by the time the machine is back, I have a swamp of failed jobs to clean up.

The fix took an afternoon: a circuit breaker pattern. If you've never heard the term, it comes from electrical engineering — when too much current flows, the breaker trips and the circuit opens. No drama, no fire. You flip it back when things are safe.

The software version does the same thing for network calls. After X consecutive failures, stop calling. Fail fast, fall back to something that works, and check periodically whether the dead service is back instead of hammering it with every request.

Since I added it, it's caught three real outages this month. Here's what I learned building it.

My Setup (The Short Version)

For context: I run a small AI lab — a Mac mini as the orchestrator, a Windows PC with a GPU for heavy inference, and an old Ubuntu box for background services. The orchestrator has agents that route tasks to the other machines: code generation goes to the GPU box, quick chat stays local, image parsing goes to the Ubuntu box, and so on.

The routing itself is a dumb little Python function with a model registry and keyword matching. It's the piece that decides where each request goes. What it didn't have until recently was any concept of whether the destination is actually alive.

The Problem, In Numbers

Here's what a typical outage looked like before:

• The GPU machine reboots for a Windows update at 14:02
• Agent jobs keep hitting it: image generation, code refactors, embeddings
• Each request has a 30-second timeout
• The agent framework retries failed requests up to 3 times
• So one failed job = up to 2 minutes of pure waiting, then a cascade of retry jobs

On a bad day, a 15-minute outage would produce 40+ queued tasks that all tried the dead endpoint, failed, and retried — burning a couple of hours of queue time on nothing. And since some of those jobs had follow-up jobs ("after generating the images, post them to X"), the failures propagated downstream. I'd come back to find half my evening pipeline in a error state because Windows decided 14:02 was a great time to update.

The worst part: the requests that should have just gone somewhere else — the Ubuntu box runs the same models, just slower on CPU — sat in the same queue behind the doomed ones.

The Circuit Breaker, Minimal Version

The pattern has three states. I stole this directly from the classic description (it's all over the resilience literature):

1. Closed — normal operation. Requests pass through. Failures are counted.
2. Open — tripped. Requests fail immediately, no network call at all. This is the "stop hammering the dead thing" state.
3. Half-open — after a cooldown, let one request through as a probe. If it succeeds, close the breaker. If it fails, stay open.

Here's roughly what I run per endpoint (simplified, but functionally what's in production):

import time

class CircuitBreaker:
    def __init__(self, name, threshold=3, cooldown=60, timeout=5):
        self.name = name
        self.threshold = threshold        # consecutive failures before tripping
        self.cooldown = cooldown          # seconds before probing again
        self.timeout = timeout
        self.failures = 0
        self.state = "closed"            # closed | open | half-open
        self.opened_at = None

    def allow_request(self):
        if self.state == "closed":
            return True
        if self.state == "open":
            if time.monotonic() - self.opened_at >= self.cooldown:
                self.state = "half-open"  # probe time
                return True
            return False                  # fail fast, no network call
        return False                      # a probe is already in flight

    def record_success(self):
        self.failures = 0
        self.state = "closed"

    def record_failure(self):
        self.failures += 1
        if self.state == "half-open" or self.failures >= self.threshold:
            self._trip()

    def _trip(self):
        self.state = "open"
        self.opened_at = time.monotonic()
        self.failures = 0

And in the request wrapper (again, simplified):

def call_endpoint(breaker, url, payload):
    if not breaker.allow_request():
        raise CircuitOpen(f"{breaker.name} is tripped, failing fast")

    try:
        resp = requests.post(url, json=payload, timeout=breaker.timeout)
        resp.raise_for_status()
        breaker.record_success()
        return resp.json()
    except Exception:
        breaker.record_failure()
        raise

One breaker instance per endpoint, keyed in a dict. The router that decides which machine handles which task now consults the breaker before doing anything else.

Note the aggressive timeout — 5 seconds, not 60. When a machine is genuinely there, a health probe responds in well under a second. Long timeouts are for slow responses, not dead machines. Distinguishing those two cases is half the value.

The Part That Made It Actually Useful: Fallbacks

A circuit breaker that just fails fast is... okay. Your errors arrive sooner. Great. Congratulations.

The real win comes from pairing the breaker with your fallback chain. My router already knew fallbacks existed — big model down? use the small one. GPU box rented out? CPU box takes over. What it lacked was the signal to switch. The breaker provides exactly that signal, in constant time, without a single wasted network call.

So the flow became:

1. Router picks the destination for a task
2. Is the breaker for that destination open? If yes, use the fallback destination immediately
3. Otherwise, call with a short timeout; failures count toward the threshold
4. If the breaker trips mid-flight, the next request reroutes — not to a retry queue, but to the fallback

A 30B code model being unavailable now means requests degrade to the 4B model on the Mac. Code quality drops slightly. The pipeline keeps flowing. Image jobs meant for the GPU get routed to the Ubuntu box and take 3 minutes instead of 20 seconds. Still flowing.

Three Outages, Zero Cascade Failures

I've had this running for a bit over a month now. Three times the breaker earned its keep:

1. The Windows Update (14:02, of course). GPU box gone for 17 minutes. Before the breaker: ~40 doomed tasks, cascading failures, an evening of cleanup. With the breaker: it tripped after 3 failures, code and image tasks rerouted to the fallback chain, and when the box came back the half-open probe re-closed it within a minute. I didn't even notice until I read the logs.

2. The flaky PCIe riser. One of my GPU cards had a flaky riser connection and the machine hard-froze. When a box is hung (not off), connections don't get refused — they just hang there. The 5-second timeout is what saved this one: three hangs, breaker trips, traffic reroutes. Without the short timeout, every request would have sat there for 60 seconds apiece, pretending to work.

3. The Ubuntu box's full disk. A background service filled the disk with logs, and the service started returning errors while the machine itself stayed up. The breaker tripped on HTTP errors just like it does on connection failures — the pattern doesn't care why the endpoint is failing, only that it is. Tasks that needed the vision service waited for the half-open probe instead of piling up garbage.

Three outages. Zero cascade failures. Zero evenings spent cleaning up retry swamps.

What I Got Wrong First Try

I counted total failures instead of consecutive ones. My first version tripped the breaker after N failures ever, which meant a busy day with a handful of one-off timeouts could trip it even though the endpoint was healthy. Consecutive-failure counting with reset-on-success is the fix. One-off failures happen; the breaker should ignore them.

My cooldown was too short. I started with 10 seconds, which meant a dead endpoint got probed every 10 seconds — fine — but a slowly recovering endpoint (say, a machine that just booted and is still loading a 30B model into VRAM) would get marked healthy by a TCP-level probe, then fail on the actual request and re-trip. 60 seconds worked better. If the first real request after recovery is heavy, give it room to finish loading.

I forgot to reset breakers on deploy. The breaker state lives in memory. When I shipped a new version of the router, all breakers came up closed while the actual machine was still down — instant trip, three failures wasted. The state should be persisted (or at least seeded from a health check at boot). It's on my list.

Fail-fast errors need to be distinguishable. When a task fails because "the breaker is open," that's a different failure than "the endpoint returned garbage." My first version raised the same generic exception for both, which made my alerting noisy and my logs confusing. One extra exception class fixed it. Now "breaker open" is not even an alert — it's the expected state during a known outage. The alert fires when a breaker trips, not on every request it rejects.

The Dashboard Bit

Because I was already logging breaker trips, I added one more thing: a daily message to my Telegram with any breakers that tripped in the last 24h and why.

That message has quietly become one of the most useful signals in my whole setup. Not because outages are frequent — they're not — but because it tells me which machine is the flaky one this month. September it was the riser. Last month it was power settings putting a box to sleep mid-job. The pattern in the trips is a pattern in my hardware.

Should You Build One?

If you have agents calling services and you have no failure isolation between them — yes. Honestly, it's the kind of thing that sounds like Enterprise Java Architecture Astronautics until the first time it saves your evening, and then it's just plumbing.

You don't need a library. The class above is ~40 lines and covers 90% of the value. If you want batteries included, there are solid packages out there — but read their retry/timeout defaults carefully, because a circuit breaker that waits 60 seconds per failure isn't protecting you from much.

The mental model that made it click for me: retries ask "will it work this time?" A circuit breaker asks "is it worth asking?" During an outage, the answer is no, and knowing that up front is worth more than any amount of persistence.

---

Have you wired failure isolation into your agents, or are you still in the retry-swamp phase? I'm curious what patterns people are using for multi-machine setups — drop a comment.

This is part of my series on running AI entirely on my own machines — a router for picking the right model per task, getting real work out of small local models, and evaluating local models before trusting them.

Comments

[Jo Do]

The half-open state needs single-flight semantics across the whole worker pool, not just inside one process. Otherwise ten agents wake after the cooldown and all become the probe. I also separate transport health from request outcome: an unknown write result should be reconciled by idempotency key before the breaker reroutes or retries it somewhere else.

[Raknaos]

"Retries ask will it work this time, a breaker asks is it worth asking" is the cleanest way I've seen the two mechanisms separated. The detail I'd add from running something similar: the half-open probe needs single-flight across the whole worker pool. Ten concurrent agents waking on the same cooldown all become the probe, the one slow failure gets counted ten times, and the breaker oscillates instead of recovering.

The other split worth making is transport health versus request outcome. A timeout tells you nothing about whether the far side did the work, so an unknown write result needs to be reconciled by an idempotency key before the breaker decides to reroute or retry it somewhere else - otherwise the outage breaker is what causes the duplicate. Which one did your three outages turn out to be: unreachable, or reachable-but-slow?