Payments went fully down, and checkout was fine.
The next week payments stayed up and answered in 30 seconds instead of 40 milliseconds. The entire site went dark.
A dead dependency is a small problem. A slow one takes everything with it.
Every call still waiting is holding a thread, a database connection and a chunk of memory that the product page needed too. Nothing throws an error. The pool is just gone.
That's the cascading failure a circuit breaker exists to stop — and it's why the breaker is never installed on its own. A request meets a timeout, then a retry policy, then the breaker, then a fallback. In that order.
Most of us built that chain from a tutorial that said "open after 5 consecutive failures."
Go read your actual config.
Resilience4j, Hystrix, Polly and gobreaker don't count consecutive failures. They trip on a failure rate across a sliding window, gated by a minimum call volume. Which is better — until you notice that on a low-traffic endpoint, two of those defaults combine to leave the breaker unable to open at all. Not "slow to open." Unable.
Three more things I had wrong for years:
Retries are selfish. Exponential backoff without jitter doesn't spread the load, it just reschedules the same spike for later.
You can have full throughput and zero goodput at the same time, and every dashboard will look busy while nothing succeeds.
And AWS's own Builders' Library warns that circuit breakers add modal behavior and recovery time — its SDK ships a retry token bucket instead. Amazon has a written case against fallback code, too. The pattern everyone teaches as the answer is the one Amazon argues you should think twice about.
So I built the version that includes the numbers and the counterarguments: 36 minutes, one call from checkout to payments, all the way through.
If you take one thing: an open breaker with no alarm on its state changes is a silent partial outage.
Top comments (0)