A release went out on a Wednesday afternoon carrying a change that had failed review and had never been approved by anyone. It was not malicious and it was not a force push. Our pipeline had simply built a different commit from the one it tested, and it had been capable of that since the day it was written.
Each stage of the pipeline started with a fresh git clone of the branch. Test ran, took eleven minutes, and passed. Build ran next, cloned again, and by then two more merges had landed on main. The image it produced contained code no test in that run had ever seen. Deploy cloned a third time. Under normal cadence the three clones agreed, because nothing merged in the gap. On a busy afternoon before a deadline, they did not.
The tell had been there for months, in the form of image tags. We tagged images main-<build number>, which is monotonic and meaningless. Nobody could answer "what is running in production" with a commit hash, only with a build number that mapped to a time, and a time is not a version.
Rebuilding this properly took a day. The first job resolves the commit SHA once and passes it to everything downstream as an explicit input; no later stage resolves a ref. Every stage checks out that SHA in detached head. The image is tagged with the short SHA and the build metadata records the full one, so the question "what is deployed" is answered by reading a tag. The deploy step refuses to run against an image whose SHA is not the one the tests recorded, and that check has fired twice since, both times legitimately.
We also stopped building anything from a branch name. Releases build from a tag, tags are created by the pipeline, and the tag points at the tested commit.
The general shape here is that a pipeline stitched together from independent jobs will quietly disagree with itself unless something forces them to agree. Resolve mutable references exactly once, at the start, and treat everything after that as operating on an immutable artifact identified by a hash. Otherwise your green build is a statement about some code, not about this code.
– Sergey Shinder
Top comments (0)