Two pull requests were merged into main ninety seconds apart on a Tuesday afternoon. Both were small. One of them fixed a rounding error on invoice totals, and by six that evening the rounding error was back in production with the fix sitting quietly in main, merged, green, done.
Our deploy workflow triggered on push to main. It built an image tagged with the commit SHA and ran a helm upgrade against the cluster. Two pushes meant two runs, and there was nothing anywhere telling them to take turns. The first run started with a cold layer cache and took eleven minutes. The second started warm and took four. The second one finished first and deployed the newer commit. Then the first one finished and deployed the older image over the top of it. Both runs were green. The cluster did exactly what it was told, twice, in the wrong order.
The reason it took us until the evening to see it is that nothing in our tooling knows what is deployed, only what was deployed most recently by whom. The workflow run for the newer commit said success and linked to a deployment that had since been replaced.
Three changes. The deploy job now sits in a concurrency group keyed on the branch with cancel-in-progress set to false, so a second deploy queues behind the first instead of racing it. Pull request checks use the same mechanism with cancel-in-progress set to true, because there the newest run is the only one worth paying for.
Second, the deploy step refuses to run if the SHA it is about to ship is not the current head of main. It fetches, checks whether its own commit is an ancestor of origin/main and not equal to it, and if so it exits zero with a message saying a newer deploy has superseded this one. Cheap, and it makes the race harmless even if the queue is bypassed.
Third, the running revision is written to a Kubernetes annotation and exported as a metric, and there is an alert when the deployed commit timestamp moves backwards. That alert has fired once since, during a deliberate rollback, which is the only time it should.
Green pipelines told us both deploys succeeded. Neither of them told us which one was still there.
– Sergey Shinder
Top comments (0)