An internal configuration service went down for eleven minutes during a routine node drain. It was a small thing, owned by a team of two, serving feature settings. When it went down, seven unrelated services stopped accepting traffic, including the two we'd spent a year making highly available across three regions.
Nobody had designed that. It accumulated. A service needed a config value at startup, so it called the config service. Reasonable. Then someone added a health check that verified config was reachable, so a failed fetch marked the pod unhealthy. Also reasonable in isolation. Then Kubernetes did what it's supposed to do with unhealthy pods, and a dependency that was meant to be a convenience became a hard prerequisite for existing. No architecture review would have approved that. No architecture review ever saw it, because it arrived in six separate pull requests over two years.
The genuinely uncomfortable finding came after, when we mapped it properly. We instrumented outbound calls and drew the actual dependency graph from live traffic rather than from documentation. Four more services had the same shape: something small and unowned sitting underneath things we'd declared critical. Our redundancy was real but it was all horizontal — three replicas of a thing that all needed the same fragile thing.
The remediation was less about redundancy than about degradation. A service should be able to start and serve with a stale cached config, log loudly, and keep going. So config fetches got a local cache on disk, a long fallback TTL, and sane defaults compiled in. The health check stopped checking dependencies entirely, because a readiness probe that fails on a downstream outage converts a partial degradation into a total one and takes your capacity away exactly when you need it most.
The rule I took away: for every dependency, decide explicitly whether it's hard or soft, and write down what happens when it's gone. If the answer is "we stop," that's a choice you should make deliberately for maybe two things, not something you inherit from a health check somebody added on a Tuesday.
Draw your dependency graph from real traffic, not from the diagram. The diagram shows what you designed. The traffic shows what you built.
– Sergey Shinder
Top comments (0)