Why This Pattern Keeps Coming Up
Circuit breakers show up in almost every serious postmortem about a cascading outage, and usually in the same spot: a downstream dependency degraded, requests kept flowing to it at full volume, and the retry storm made the outage worse and longer than the original problem would have been on its own. Adding a breaker doesn't prevent the initial failure. It prevents your own system from actively making that failure worse.
The Problem Circuit Breakers Actually Solve
When a downstream API starts failing, the naive response is to keep retrying every request as it comes in. That feels productive but it's usually the opposite. Every retry against a struggling or down service adds load exactly when the service needs less of it, and a fleet of workers retrying in parallel can turn a recoverable blip into an extended outage. A circuit breaker stops that spiral by detecting the failure pattern and refusing new requests for a cooldown period, instead of hammering a service that's already down.
This guide walks through adding one to an existing API client without turning it into a distributed-systems research project.
Step 1: Define What "Failing" Means for This Client
Before writing any breaker logic, decide what counts as a failure worth tripping on. Timeouts and 5xx responses almost always qualify. A 429 rate-limit response usually shouldn't trip the same breaker, since that's a different problem with a different fix (backoff, not circuit isolation). Write this mapping down explicitly rather than trusting whatever exception type happens to bubble up.
Step 2: Track a Rolling Failure Count, Not a Lifetime One
A breaker that trips after five total failures ever, and never resets, is useless after the first bad hour. Track failures in a rolling window, for example the last 20 requests or the last 60 seconds, and trip the breaker when the failure rate in that window crosses a threshold, commonly 50%.
Keep the window small enough to react quickly to a real outage but large enough that a couple of unlucky timeouts in a row don't trip it unnecessarily. Start conservative and tune from real traffic data rather than guessing the perfect number up front. Pairing the breaker with exponential backoff on whatever retries do go through, once the breaker allows them, keeps the recovery period from turning into a second thundering herd against the service you just gave a chance to breathe.
Step 3: Implement the Three States
A circuit breaker has three states: closed (normal operation, requests flow through), open (the breaker has tripped, requests fail immediately without hitting the network), and half-open (after a cooldown, a limited number of test requests are allowed through to check if the service has recovered).
This third state is the part teams skip when they build something quick and call it a circuit breaker. Without it, you either wait a fixed cooldown and hope, or manually reset the breaker, both of which are worse than automatically testing recovery with a small, controlled trickle of requests.
Step 4: Fail Fast, Not Silently
When the breaker is open, the client should fail immediately with a clear, distinct error, not a timeout, and not the same generic exception as a real API failure. Code calling the client needs to be able to tell the difference between "the API said no" and "we didn't even try because the breaker is open," since the right response to each is different.
This distinct error is also what makes the dead letter and retry-classification logic from a broader ingestion pipeline work correctly. A request that failed because the breaker is open should generally be retried later, not routed straight to a permanent-failure path.
Step 5: Log State Transitions Loudly
Every time the breaker opens, half-opens, or closes, log it with the endpoint name and the failure rate that triggered it. This is the signal that tells you, before a customer does, that a specific integration is having a bad day. Piping these transitions into whatever alerting tool your team already uses, whether that's a paging system or a simple Slack webhook, turns a silent internal state change into visible operational awareness.
Step 6: Scope Breakers Per Dependency, Not Globally
A single global circuit breaker for "the API layer" sounds simpler but actively hides problems. If one vendor's API goes down and trips a shared breaker, every other integration that shares it goes down too, even though nothing is actually wrong with them. Scope each breaker to a specific downstream dependency so a bad day for one vendor doesn't take out the rest of your integrations.
"Circuit breakers get a reputation for being complicated because people try to build one generic breaker for everything at once. Scoped per dependency, with three states and a rolling window, it's maybe eighty lines of code that pays for itself the first time a vendor has a bad afternoon." - Dennis Traina, founder of 137Foundry
Step 7: Test the Half-Open Transition Specifically
Most teams test that the breaker trips under sustained failure, and stop there. The half-open transition is where subtle bugs hide, particularly around what happens if the test request during half-open also fails, or if two requests race into half-open simultaneously. Write a test that deliberately exercises this transition, not just the trip and the cooldown. Tools like Postman or a small mock server make it easy to script a service that fails, then recovers on a delay, so you can watch your breaker walk through all three states under a controlled scenario before it ever meets production traffic.
Step 8: Resist Adding More Than You Need
It's tempting to add adaptive thresholds, per-endpoint tuning knobs, and configurable state machines once the basic breaker works. Most teams never need any of that. A fixed rolling window, a fixed failure-rate threshold, and a fixed cooldown, each set from a reasonable starting point and adjusted only if real traffic shows it's wrong, covers the overwhelming majority of use cases without the extra complexity becoming its own maintenance burden.
A Note on Client-Side vs Server-Side Breakers
Everything above describes a client-side breaker, tripped and enforced in the code that calls the downstream API. Some infrastructure setups implement equivalent behavior at a service mesh or API gateway layer instead. Either location works, and larger organizations often end up with both, a coarse mesh-level breaker as a backstop and finer client-level breakers scoped per integration for faster, more specific reactions. If you're just getting started, client-side is the faster path to shipping something useful without a platform-level change.
A Note on Language and Framework Support
You rarely need to write a circuit breaker fully from scratch. Most mainstream languages already have a mature library implementing this pattern, resilience4j for the JVM ecosystem, Polly for .NET, and opossum for Node.js are common choices, and similar options exist for most others. Reaching for an existing, well-tested implementation of the state machine itself, and spending your own engineering time on the scoping and threshold decisions instead, is almost always the better tradeoff than hand-rolling the state transitions from scratch.
Putting the Pattern Into Practice
Circuit breakers are one piece of a larger reliability story that includes retry classification, idempotent writes, and dead-letter handling for the failures that do get through. 137Foundry's write-up on designing ingestion pipelines around partial API failures covers how those pieces fit together at the pipeline level, beyond just the individual client.
Adding a scoped, three-state circuit breaker to your highest-traffic API client is a focused project you can ship in a day, and it's usually the single highest-leverage reliability change available before reaching for anything more elaborate. 137Foundry works with engineering teams on this kind of integration reliability work directly.
Top comments (0)