A recommendations service, which is genuinely optional, started responding slowly. Not failing. Responding, in ninety seconds instead of eighty milliseconds. Within four minutes the product catalogue was down, then checkout, then the mobile API. An optional feature took out the entire storefront, and the recommendations service never once returned an error.
The catalogue called recommendations with a client constructed as a default instance. No connect timeout, no socket timeout, because the library's default for both is none. So each request that would have failed fast instead held a worker thread for the full ninety seconds. The pool had two hundred threads and roughly forty requests a second touching that path, which means saturation in about five seconds. Once every thread was parked, the catalogue stopped answering anything, including its own readiness probe, including the endpoints that had nothing to do with recommendations. The services calling the catalogue had the same default client, so the same thing happened to them, one hop at a time.
Slow is worse than down. A dependency returning connection refused fails in a millisecond and your code takes the fallback path it was written for. A dependency that answers eventually consumes the one resource every service is actually limited by, which is concurrency, and it does so quietly.
The remediation had a boring part and an interesting one. The boring part was auditing every outbound client in every repository and setting explicit connect and read timeouts, derived from the caller's own SLO rather than the callee's typical latency. We added a lint rule that fails the build on a default-constructed client, because this will otherwise come back with the next new service.
The interesting part was bulkheads. Calls to non-critical dependencies now run in their own bounded thread pool of twenty, with a circuit breaker that opens on a rate of timeouts, and a fallback that returns an empty recommendation list. Saturating that pool degrades one feature and cannot touch the request path. We proved it by holding the recommendations service at ninety seconds in staging and watching the storefront stay up with an empty carousel.
If you cannot name the timeout on a call, the answer is not a sensible default. It is infinity.
– Sergey Shinder
Top comments (0)