The database had a slow half hour. A vacuum on a large table, contention on an index, nothing that should have been more than degraded latency. Instead the API disappeared completely for twenty-two minutes, and the pod list showed every replica in CrashLoopBackOff with restart counts in the teens.
The liveness probe was pointed at /health. That endpoint did a SELECT 1 against the primary, plus a Redis ping, plus a check on a downstream service, because whoever wrote it reasoned that health means the service can actually do its job. Under normal conditions it returned in three milliseconds. When the database went slow, it took longer than the one second probe timeout, three consecutive times, and the kubelet killed the container. Every replica at once, since they all shared the same sick dependency. Then each new pod started, warmed nothing, hit the same slow database, and got killed again.
The distinction I had never taken seriously is that liveness answers one question only: is this process wedged in a way that only a restart can fix. Deadlock, exhausted event loop, unrecoverable internal state. A slow database is not that. Restarting made it strictly worse, because we threw away warm connection pools and added a reconnect storm to a database that was already struggling.
We rewrote both probes with that split in mind. Liveness is now a trivial in-process check that touches no dependency, on a separate port served by its own thread so it stays answerable while the worker pool is saturated. Readiness keeps the dependency checks, because a pod that cannot reach the database genuinely should not receive traffic, and readiness failure removes an endpoint rather than killing a process. We added failureThreshold of six with a ten second period on liveness, so the bar for "restart this" is a minute of unresponsiveness rather than three seconds.
The last piece was accepting that when every replica is unready, serving degraded responses beats serving none. The readiness check now tolerates a failing non-critical downstream and reports degraded through a metric instead.
A liveness probe that depends on something external converts a partner's bad minute into your outage.
– Sergey Shinder
Top comments (0)