In August release 4.12 of our pricing service went out at two in the afternoon and started returning the wrong discounts. The on call engineer did what our runbook said: kubectl rollout undo. The rollback did not finish. For twenty five minutes most traffic was still served by 4.12, while the pods of 4.11 crashed on startup.
The release changed two things: the image, and the service's configuration file, which lives in a ConfigMap called pricing-config. 4.12 renamed a setting, discount_rules_path to rules.discount_path. Our pipeline applies the ConfigMap and the Deployment together, and the Deployment carries a hash of the configuration as an annotation, so a config change restarts the pods. I wrote about that annotation here a while ago.
rollout undo restores the Deployment's pod template from the previous ReplicaSet: the old image, and the old hash annotation. It does not touch the ConfigMap. Nothing in Kubernetes keeps a history of a ConfigMap's contents. So the rollback produced 4.11 pods reading 4.12's configuration, and 4.11's strict loader refused a file containing a key it did not know. The pods went into CrashLoopBackOff. A rolling update only removes old pods as new ones become ready, and these never did, so the controller kept most of the faulty 4.12 pods running. The hash annotation even looked correct. It described a configuration that no longer existed.
We recovered by reapplying the whole of the previous release from git, configuration included.
Configuration is now versioned with the image. Our kustomize build generates the ConfigMap with a hash of its contents in its name, so each release refers to its own configuration object, and the previous one stays in the cluster until a cleanup job removes those older than the last five releases. A rollout undo now goes back to the old image and the old configuration together, because the old pod template names the old ConfigMap. The runbook no longer says rollout undo. It says to run the deploy job for the previous release, which is the same path every release takes. And every renamed setting ships in two steps: first a release that reads both names, then one that drops the old name.
A rollback is only as complete as the history it can reach. The Deployment remembered its image. Nothing was remembering the file next to it.
– Sergey Shinder
Top comments (0)