The three replicas that turned out to be on one node

The checkout service ran three replicas. It had run three replicas for two years, and every design document that mentioned it said the number three as though it meant something. During a routine node pool upgrade, one node was cordoned and drained, and checkout was unavailable for ninety seconds.

All three pods were on that node. Nothing had gone wrong: the scheduler had placed them there because it fit, and no constraint in the manifest ever told it not to. Two of the nodes in that pool were much larger than the rest after a capacity change, and once a pod landed on a large node its siblings were perfectly happy to follow. There was also no PodDisruptionBudget, so the drain evicted all three at once without asking anybody's permission, and they came back only as fast as the scheduler could find room plus a cold JVM start.

Replica count is a statement about how many copies exist. It says nothing about where they are, and the failure domain you actually care about is the node, the zone and the rack, not the number in the YAML.

We added two things to the deployment. A topology spread constraint over kubernetes.io/hostname and another over topology.kubernetes.io/zone, both with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule, so an unschedulable pod is a visible pending pod rather than a quiet loss of redundancy. And a PodDisruptionBudget with minAvailable: 2, which makes a drain take its turn instead of taking the service.

Then we made it a rule rather than a fix. An admission policy rejects any Deployment with more than one replica that has neither a spread constraint nor an anti-affinity rule, and there is a dashboard panel showing, per workload, how many distinct nodes its pods occupy. Four other services were in the same position. One had five replicas on two nodes.

We now drain a node during working hours once a month, deliberately, with everyone watching. The first time we did it after the fix, nothing happened, which is the entire point.

– Sergey Shinder