A configuration change went through staging on a Wednesday afternoon with nothing to report, and broke every call to our payment partner ninety minutes later in production. Same commit, same pipeline, same configuration. We said that to each other for most of the evening.
Our pipeline built the image three times. Once for the test stage, once on promotion to staging, once at the production gate, each from a clean checkout of the same commit, because that is how it had been written years ago and because building is cheap. Three builds, three digests, and not one of us had ever thought to compare them.
The Dockerfile begins from a slim base referenced by a moving tag. Between the staging build at half past two and the production build at ten past four, the publisher moved that tag. The new base brought a newer TLS library that no longer offers a cipher suite our partner's gateway still insists on, so every handshake ended with no shared cipher. Nothing in our code and nothing in our change had any part in it.
I pulled both images and compared their package manifests. Forty one packages differed between an artifact that had passed every check and an artifact that was serving customers, from a single commit, two hours apart.
We build once now. The first stage builds, pushes and records a digest, and every later stage deploys that exact digest rather than a tag or a commit. Base images are pinned by digest and moved by a scheduled pull request that goes through the pipeline like any other change. The deploy step refuses any digest with no passing test record attached to it.
The rollback path was worse and nobody had noticed. Rolling back meant re running the pipeline against an older commit, which would have produced a fourth image that had never existed before, assembled from whatever the internet was publishing that morning, under the name of a release we had been running happily for a fortnight.
A commit describes source. It does not describe the thing that runs, and everything your build reaches out and collects is sitting in the gap between the two.
– Sergey Shinder
Top comments (0)