Every backend team learns the same lesson, usually at 3 AM: your service did not fail because of your code. It failed because a dependency got slow, and your code kept calling it.
The payments API slows to thirty seconds. Your threads block waiting. Incoming requests queue behind those threads. Memory climbs, health checks fail, and the orchestrator restarts you in the middle of the backlog. The dependency never went down. It just got slow, and slowness is contagious.
Then your own retries finish the job. Every timed-out call gets retried, so a thousand requests become four thousand against an API that is already struggling. The dependency never gets a quiet moment to recover, because your good intentions keep hammering it. This is a retry storm, and it turns a partial outage into a full one.
Circuit breakers exist to stop this exact cascade. Here is how they work, and how to tune them so they protect you instead of annoying you.
The idea, stolen from electricians
An electrical breaker opens when too much current flows, and the house does not burn down. A software circuit breaker does the same for network calls: when failures cross a threshold, calls stop going out. The failing dependency gets room to breathe. Your service keeps its threads for work that can actually succeed.
The breaker sits on the client side, wrapped around every outbound call: databases, APIs, queues, all of them. It has three states.
The three states
Closed is normal operation, and the confusing name just means traffic flows. Calls pass through while the breaker counts failures in a rolling window. The classic rule: if half the calls fail over the last twenty, the breaker decides this is not a blip and trips open. The window matters. Five failures in a quiet hour should not trip anything, and a good breaker knows the difference between a blip and a pattern.
Open is the whole point. Every call to that dependency now fails instantly, without touching the network. No thread waits thirty seconds on a doomed call. Those threads stay free to serve the healthy parts of your system, and the fallback path takes over. Failing fast is the feature: you spend zero resources on work that cannot succeed.
Half-open is how recovery gets tested instead of assumed. After a cooldown period, the breaker lets exactly one probe call through. Success closes the circuit and traffic resumes. Failure opens it again with a longer wait. You never dump full traffic onto a dependency that just coughed, which is what stops the flapping.
The fallback is the product
An open circuit does not mean a broken page. It means the fallback takes over, and the fallback deserves real design work.
Serve the cached price list instead of the live one. Queue the write for later instead of failing the checkout. Show a sensible default instead of an error page. Users tolerate slightly stale data. They do not tolerate a down page.
Design the fallback first, and the breaker becomes invisible. Design it as an afterthought, and every trip is a user-visible incident. Most teams skip this step and then blame the pattern.
Tuning: where most teams get it wrong
Trip on failure rate, not raw counts. Ten failures mean nothing at midnight and everything at noon. Rates normalize across traffic; counts do not.
Count slow calls as failures. A call that takes thirty seconds is a failure even if it eventually returns. Use your p99 latency as the slow-call threshold, not the average, because averages hide the tail that is killing you.
Let the cooldown grow. A fixed five-second wait after every trip flaps against a dependency that needs a minute. Back off: seconds first, longer with each consecutive failure.
Tune per dependency, never global. The search service and the payment gateway have nothing in common: different latency profiles, different error budgets, different business cost of a fallback. One global config protects nothing well.
The supporting cast
Circuit breakers work best with two companions. Timeouts bound how long any single call can wait, so a closed breaker still cannot hang forever. Bulkheads isolate thread pools per dependency, so one slow downstream cannot eat every thread even before the breaker trips.
And you do not have to build any of this. Resilience4j in Java, Polly in .NET, gobreaker in Go, pybreaker in Python. The pattern is identical everywhere, so learn it once and recognize it in every stack.
The takeaway
Slow dependencies are inevitable. Retry storms are optional. Break the circuit before it breaks you: closed counts, open fails fast, half-open probes. Build the fallback before you need it, tune per dependency, and the next outage stays someone else's problem.
If you take one thing from this: wrap your next outbound call in a breaker this week. The incident you prevent will not make noise, and that is exactly the point.
Top comments (0)