Production was running three different versions of the same service at the same time, and every deployment manifest in git said the identical thing. The tag was :latest. Each pod had pulled it at a different moment, and the registry had been rewritten under them twice in between.
Nobody had done anything obviously wrong. CI pushed :latest on every merge to main. The deployment referenced :latest. A node got replaced during a scale event, its replacement pulled a fresh image, and now that pod was ahead of its neighbours. Another node had imagePullPolicy: IfNotPresent and a cached layer, so it stayed behind for a week. The manifests were identical, the git history was clean, and the fleet was heterogeneous.
Debugging that was miserable. An error appeared in logs from one pod and not another, on what we believed was the same code. We chased a phantom race condition for most of a day before someone ran a crictl on two nodes and compared image IDs. Different digests. The bug wasn't a race. It was two versions of the software disagreeing about a serialization format.
The fix is boring and I'd make it a hard rule anywhere now: deploy by immutable tag, ideally by digest. CI tags images with the git SHA, pushes that, and never overwrites it. The deployment manifest names the SHA. :latest can still exist as a convenience pointer for someone poking around locally, but nothing in a cluster is allowed to reference a mutable tag, and we added an admission policy that rejects manifests that do.
The deeper idea is that a Kubernetes manifest is supposed to be a complete description of desired state. The moment it contains a mutable reference, it stops describing anything specific. It says "run whatever that name points at right now," and "right now" is different for every pod, every restart, every node replacement. You've written a manifest whose meaning depends on when it's read.
Immutable artifacts are what make rollbacks, bisecting, and incident timelines possible at all. If you can't point at a digest and say "this exact thing was running," you can't reason about production. Pin the digest. Let the tag be a nickname, never an address.
– Sergey Shinder
Top comments (0)