DEV Community

Sergey Shinder
Sergey Shinder

Posted on

We set maxUnavailable to zero, the platform team set a pod quota, and no release could start

Our checkout service runs twenty replicas, and its rolling update has maxUnavailable set to zero and maxSurge at twenty five percent. Zero unavailable was a rule we were proud of: a release never takes capacity away. It adds five new pods, waits for them to be ready, then removes five old ones.

In October the platform team introduced resource quotas on every namespace, after one team's batch job had filled a cluster. To avoid arguments, each quota was set from what the namespace was actually running that week. Checkout got a limit of twenty pods and a CPU request total to match. Nothing changed on the day, because nothing new was created.

On Thursday we released a fix for a voucher bug. The deployment created its new ReplicaSet, the ReplicaSet tried to create five pods, and quota admission refused every one: exceeded quota, the namespace already had twenty pods. With maxUnavailable at zero, the deployment could not remove an old pod until a new one was ready, and no new pod could exist until an old one was removed. Neither side could move. The rollout sat there until its progress deadline, ten minutes, and the pipeline reported a timeout.

kubectl rollout status only said it was waiting. The reason was in the new ReplicaSet's events, as FailedCreate, which nobody looks at during a release, and it took most of an hour to find. The voucher fix went out two hours late, after the quota was raised by hand.

Two settings, each chosen by people trying to make releases safe, together made releasing impossible. Neither team had been wrong about its own setting. Nobody owned the combination.

Quotas are now sized from each deployment's replicas plus its surge, not from what happens to be running, and the platform repository computes that from our manifests rather than from a snapshot of the cluster. Our pipeline checks quota headroom before a rollout starts and fails at once with the reason. A failed rollout also prints the warning events for the new ReplicaSet, so the next person sees the refusal in the pipeline log instead of a bare timeout.

Steady state is the wrong size for anything that has to change.

– Sergey Shinder

Top comments (0)