Our managed Kubernetes version was due to fall out of support at the end of the month, so we started the node pool upgrade on a Tuesday morning expecting it to be done by lunch. By Thursday afternoon the upgrade still said in progress. Forty two of forty four nodes had moved. Two had not, and the only visible symptom was a drain step retrying every few seconds with an error about a disruption budget.
A PodDisruptionBudget tells the cluster how many pods of a workload may be voluntarily evicted at once. We had put one on every deployment through our base chart, with minAvailable set to one. For almost everything that means one pod may go, the replacement comes up elsewhere, the next may go. For two services it meant nothing could ever go, because they ran a single replica. One pod, at least one available, zero permitted disruptions. The eviction API refused, correctly, forever.
Both were small internal services that somebody had scaled down to one to save money during a cost review in the spring. The chart's budget stayed as it was, because nobody thinks about a budget when changing a replica count, and a single replica with a budget of one produces no warning anywhere. It just sits there until the day something needs to move it.
The managed upgrade respects budgets and waits, which is the right behaviour, and it reports its waiting as progress. There was no alert, because nothing was failing. We found the two pods by listing every budget whose allowed disruptions was zero, which took one command once we knew to ask it.
The base chart now derives the budget from the replica count: no budget at all for a single replica, maxUnavailable of one above that, and never minAvailable, which expresses the intent less safely when replicas change. A policy rejects any budget whose allowed disruptions would be zero at the declared scale. A metric exported from the cluster counts budgets with zero allowed disruptions and alerts if any stays there for more than an hour. And node upgrades now have a time limit, after which they fail loudly rather than wait politely.
A disruption budget is a promise the cluster will keep for you. Make sure the promise you wrote is one that can be kept by something other than never moving.
– Sergey Shinder
Top comments (0)