A colleague opened a ticket that read, in full: "the artifact CI produces is not the artifact I get locally." I assumed he'd misconfigured something. He hadn't. We ran the same command, on the same commit, and got two different binaries, and it took me two days to find out why.
The build pulled dependencies from a lockfile, which felt safe. But one of those dependencies published a native extension that compiled against whatever headers were on the machine. CI ran on a runner image with an older glibc. My laptop had a newer one. Same lockfile, same commit, different output. On top of that, our Dockerfile started from a base tagged with a floating version, so "the same build" in March and "the same build" in September pulled two different base layers without a single line of our code changing.
The uncomfortable part is that this had been true for years and nobody noticed, because the differences were usually invisible. They only became visible when a bug reproduced in production and refused to reproduce anywhere else. We'd been debugging a binary none of us could actually recreate.
The fix was to stop treating the build environment as background scenery and start treating it as an input. We pinned the base image by digest, not by tag. We moved the whole build inside a container so the runner's own toolchain stopped leaking in. We pinned the compiler and the package index snapshot. And we added a job that builds the same commit twice and compares the output hashes, which failed immediately and told us about three more sources of nondeterminism we didn't know we had, including a timestamp baked into an archive.
Reproducibility isn't an academic virtue. It's the difference between "I can investigate this bug" and "I can only investigate it in production." If your build depends on the machine that ran it, then your CI isn't building your software, it's building one particular sample of your software, and you're shipping the sample.
Pin your inputs by digest, build in a box, and make the pipeline prove it twice. If two builds of one commit disagree, you don't have a build. You have a coin flip with a version number.
– Sergey Shinder
Top comments (0)