A bug came back nine days after we fixed it. Not intermittently: for some users every time, for others never. Two engineers spent a day on cache invalidation theories before somebody thought to ask which version of the application the failing requests had actually been served by.
Fourteen of our forty instances were running a build from before the fix. Our deploy tool does a rolling update across the instances in the autoscaling group, waits for health checks, and reports success, which it had. What it does not do is change the launch template. The template still referenced the previous AMI, so every instance the group replaced during the following week, whether from a scale-out at lunchtime or an unhealthy host being cycled, came up on the old build. The fleet quietly drifted backwards while the pipeline history showed nothing but green.
The deploy had finished. The rollout had not, and nobody had a way to tell the difference.
Every service now exports a build_info gauge carrying the git SHA it was built from. One query counts distinct SHAs serving production traffic, and an alert fires if that count is greater than one for more than fifteen minutes outside a deploy window. It found two other services in the same state within an hour of being switched on, one of them running three versions at once.
The deploy job was changed to update the launch template first and the running instances second, so the thing that creates future instances is always the thing that was released. Refusing to declare success until the whole group reports the expected SHA turned "deploy succeeded" into a claim about the fleet rather than about the tool's own work list.
We also stopped debugging without asking that question first. The incident template now opens with the version of everything involved, because a fix that appears to have stopped working is usually a fix that is only running in some places.
Green in the pipeline means a process completed. It does not mean nothing is still serving last month's code, and unless you measure it you will find out from a bug that refuses to stay dead.
– Sergey Shinder
Top comments (0)