We used to think the worst failure mode for a downstream service was an outage. It isn't. An outage is clean — the connection refuses, the request fails fast, your code hits a catch block and moves on. What actually took us down was a payment-verification service that didn't fail. It just got slow.
It started as a P2 ticket: a handful of customers reporting that checkout was "spinning." Our on-call engineer checked the payment service's health dashboard — green across the board, CPU normal, error rate at zero. The service was up. It just took 8 seconds to answer instead of 80 milliseconds, and about one request in twenty took 30+ seconds before timing out on our side.
Here's the part that turned a minor annoyance into a full outage: every request into checkout held a thread (and a database connection from our pool) for the entire time it waited on that call. We had a connection pool sized for a world where dependencies answer quickly. When the payment service degraded, our pool filled with connections stuck waiting on it. Once the pool was exhausted, unrelated requests — order lookups, cart updates, anything touching that same pool — started queuing behind them. Within 11 minutes, a slow dependency for one feature had turned into a site-wide outage for everything.
That's the core lesson: a hard failure is contained by definition. A slow failure spreads, because nothing in a typical request path is designed to give up early. Threads wait. Connections hold. Retries pile on top of requests that are already struggling. The system doesn't crash, it just gets quieter and quieter until it stops answering anything.
We fixed the immediate incident by manually killing connections to the payment service and restarting the pool, which is not a strategy, it's a stopgap. The actual fix was a circuit breaker in front of every external dependency call — payment verification included.
The idea is simple, but it took discipline to implement well: track success/failure and, crucially, latency on every call to a dependency, per dependency, in a rolling window. If the failure rate or the slow-response rate crosses a threshold, "trip" the breaker — stop even trying the call for a cooldown period, and fail fast with an explicit error instead. After the cooldown, let a small number of test requests through; if they succeed, close the breaker; if not, stay open. The mechanics aren't novel — we didn't reinvent anything Hystrix or resilience4j hadn't already solved — but wiring it in correctly mattered more than the algorithm.
Three details made the difference between a breaker that helped and one that just moved the problem:
We tripped on latency, not just errors. A dependency returning 200s at 10x normal latency will never trip a breaker that only counts failures — it looks "healthy" the whole time it's strangling your thread pool. We added a p95-latency threshold per dependency as a trip condition, not just an error-rate one.
We scoped breakers per dependency, not globally. An early draft used one shared breaker for "external calls," which meant a degraded shipping-rate API could trip protection for payment calls that were working fine. Each external call now gets its own breaker with thresholds tuned to that dependency's normal behavior, not a one-size-fits-all number.
We made the open state loud, not silent. A tripped breaker that just returns a generic 500 pushes the failure downstream to whatever's easiest — usually the customer, staring at a spinner. Ours returns a specific, typed error the caller can act on: for payment verification, checkout falls back to "we'll confirm and email you," rather than hanging or hard-failing the whole order.
None of this eliminates the underlying problem — the payment service still gets slow sometimes, for reasons outside our control. What changed is the blast radius. A slow dependency now costs us one degraded feature for a few minutes, not a site-wide outage. That distinction — degraded versus down — is the entire point of a circuit breaker, and it's cheap insurance once you've been paged for the alternative.
We think about resilience patterns like this constantly at Edilec (edilec.com), because most of the incidents that actually hurt aren't the ones where something breaks loudly — they're the ones where something just gets a little too slow to notice until it's too late.
Top comments (0)