The upstream provider did not go down. That is what made it interesting. Their API stayed up and kept returning correct responses, but at some point during a busy morning the median latency moved from about eighty milliseconds to roughly nine seconds. Everything they published afterwards was accurate. Nothing on our side handled it, and within twenty minutes our own service was refusing connections to customers who had nothing to do with that integration at all.
The mechanism is depressingly ordinary. Our calls had no timeout worth the name, so request threads sat waiting. The thread pool filled. Because the pool was shared across the whole application, work that never touched the upstream at all queued behind work that did. Then the health check, which also needed a thread, started failing, so the orchestrator killed the instances and replaced them, and the new instances immediately made the same calls and filled up in the same way. We had built something that converted a slow dependency into a total outage, and then made it self-perpetuating.
Nobody designs that deliberately. It arrives one reasonable decision at a time. A default timeout that is effectively infinite because nobody set one. A retry added after an earlier incident, with no backoff and no cap, which triples the load on a struggling partner exactly when they can least absorb it. A shared pool because separate pools felt like premature complexity. Each choice is defensible alone; together they are a design that assumes the rest of the world is either fine or absent, with nothing in between.
What we changed was mostly about limits. Every outbound call now has an explicit timeout chosen against a documented expectation, not inherited. Retries have backoff, jitter and a budget. Calls to third parties run in their own bounded pool, so their failure cannot starve unrelated work. There is a circuit breaker that stops hammering something that is clearly unwell, and, crucially, an answer to the question of what the product should do while that breaker is open, which is a decision for the business rather than for me.
Availability is not a property of your service. It is a property of your service plus everything it waits on, and waiting is the part most of us leave undefined.
– Serguey Shinder
Top comments (0)