We had a production bug we could not reproduce. Same commit, same Dockerfile, same build command, and the image running in production behaved differently from the one I built locally from the identical SHA. It took most of a day to accept that the two images were genuinely not the same software.
Three things caused it, and each was a decision somebody had made deliberately. The base image was node:20-alpine, a moving tag that had picked up a new patch release between the two builds. A dependency was declared with a caret range and the lockfile was in .dockerignore, so npm install resolved fresh versions at build time. And a build argument defaulted to a value derived from the CI environment, which was simply absent on my laptop.
None of that is exotic. It's the default behaviour of a normal toolchain, and it means the artifact you tested is not necessarily the artifact you shipped. Once I understood that, a whole category of past weirdness made sense: the release that was fine in staging and broken in production, where we had rebuilt the image for the promotion instead of promoting the one we had tested.
The changes were unglamorous. Base images pinned by digest, not tag, with an automated PR to bump them so pinning doesn't rot into staleness. Lockfile committed and npm ci instead of install, so resolution happens once, in a reviewed change, rather than at every build. Build args that vary by environment removed from the image entirely and moved to runtime configuration, which is where environment differences belong anyway.
The structural change mattered more than any of them: we build once per commit and promote that exact digest through staging to production. Nothing rebuilds on promotion. The pipeline refers to artifacts by digest rather than tag from that point on, so there is no path by which "the thing we tested" and "the thing we deployed" can diverge.
Full bit-for-bit reproducibility is a harder project and we haven't done it. We didn't need it. We needed the much weaker property that a given deployment traces back to exactly one build.
If your promotion step rebuilds, your staging environment is testing a different program.
– Sergey Shinder
Top comments (0)