DEV Community

Metronom
Metronom

Posted on

The Kubernetes `:latest` Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

The Kubernetes :latest Tag Problem: How a Mutable Tag Rolled Our Prod Back for 40 Minutes

Nobody deployed. Nobody ran kubectl apply. At 02:00 the orders API started throwing errors we'd fixed three weeks earlier, and it stayed broken for 40 minutes. The kubernetes latest tag problem is not theoretical. This is the incident that made me delete :latest from every manifest we own.

Symptom

On-call paged. Errors that were fixed weeks ago, live again.

2025-... POST /orders 500  InvalidStateError: order already settled
2025-... POST /orders 500
2025-... POST /orders 500
Enter fullscreen mode Exit fullscreen mode

Deploy history: empty. No release. No apply. No human near the cluster. Running code had changed anyway. First reflex was to blame a rogue actor with cluster creds. kubectl get events and the audit log both said no — nobody had written to the API server in hours. The image had moved without a single Kubernetes write.

Root cause

The Deployment referenced a mutable tag.

containers:
  - name: orders
    image: orders:latest
Enter fullscreen mode Exit fullscreen mode

Timeline, reconstructed from the kubelet events and registry push log:

  1. Someone pushed a mid-refactor build and tagged it latest. Never promoted it.
  2. Hours later a pod OOM'd and restarted.
  3. On restart the kubelet pulled latest again — now pointing at the broken build.
  4. Pod came up on code nobody chose to deploy. Prod changed its own version while we slept.

Three separate properties of latest conspired here, and the Kubernetes image docs warn about all of them:

  • latest is mutable. It points at a different digest tomorrow. You cannot answer "what is running in prod?"
  • Re-running apply is a no-op. Kubernetes rolls out only when the pod template changes. The literal string image: orders:latest never changes, so the cluster sees nothing to do even when the registry digest is completely different.
  • A restart re-pulls. imagePullPolicy defaults to Always for latest (and IfNotPresent for a pinned tag). So latest maximizes the odds of a surprise pull on every restart. The inverse trap is just as ugly: a pinned tag with IfNotPresent can stick on a stale local layer and never pull the fix you just pushed. Both are the same disease — the running image and the name you typed have drifted apart.

There's a solid breakdown of the full local-to-CI path that this class of bug forced on us at this writeup on preparing a local setup for deployment and CI. I'll stick to the fixes.

The fix

1. Immutable tags, generated in CI

Rule: every image is tagged by something that resolves to exactly one build. Short SHA, semver, or a digest pin for maximum determinism.

image: ghcr.io/acme/orders@sha256:45b23dee...
Enter fullscreen mode Exit fullscreen mode

Don't hand-write tags. docker/metadata-action emits them from the git context and writes OCI labels like org.opencontainers.image.revision. "What's in prod?" collapses from an investigation to one label read:

docker inspect ghcr.io/acme/orders:sha-abc1234 \
  --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}'
Enter fullscreen mode Exit fullscreen mode

2. One base, kill the copy-pasted k8s-prod/

The other sin: a k8s/ folder copied to k8s-prod/ months earlier. Two copies drift. Ours had, and prod's memory limit no longer matched what we'd tested. Configuration drift, cured by a single base with only the diffs layered on top.

Kustomize — built into kubectl, plain YAML, no templating engine to reason about:

# overlays/prod/kustomization.yaml
resources:
  - ../../base
namePrefix: prod-
images:
  - name: orders
    newName: ghcr.io/acme/orders
    newTag: sha-abc1234    # swap the image without touching the Deployment
patches:
  - path: replicas-patch.yaml
Enter fullscreen mode Exit fullscreen mode 
# render and eyeball it before it touches the cluster
kustomize build overlays/prod | kubectl apply -f -
Enter fullscreen mode Exit fullscreen mode

The images field is the quiet win: CI rewrites the tag without editing the Deployment YAML. Helm with a per-env values-prod.yaml lands the same place — just mind -f ordering, last file wins, and swapping the order silently changes your prod replica count.

3. A pipeline that physically cannot ship :latest

Push-based and deliberately dumb: commit → build → push → update manifest to the immutable tag → apply. GitHub Actions skeleton, using Docker's build-push-action:

name: build-and-deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/acme/orders
          tags: |
            type=sha
            type=semver,pattern={{version}}
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
      - name: Deploy
        run: |
          echo "${{ secrets.KUBECONFIG }}" > kubeconfig
          export KUBECONFIG=kubeconfig
          kubectl set image deployment/orders \
            orders=ghcr.io/acme/orders:sha-${GITHUB_SHA::7}
Enter fullscreen mode Exit fullscreen mode

The deploy step sets an immutable tag, never latest. Now the pod template genuinely differs, the rollout is honest, and a restart can't re-pull a mystery image.

Reproducing the outage on k3d first

I didn't ship any of this blind. I reproduced the exact failure on a laptop k3d cluster, then confirmed the fix killed it. The whole loop is a few minutes and costs nothing.

k3d cluster create latest-repro
# push two builds to the same mutable tag
docker build -t k3d-registry:5000/orders:latest ./good && docker push k3d-registry:5000/orders:latest
kubectl apply -k overlays/local           # Deployment references :latest
kubectl rollout status deploy/orders

# now poison latest and force a restart, no apply involved
docker build -t k3d-registry:5000/orders:latest ./broken && docker push k3d-registry:5000/orders:latest
kubectl delete pod -l app=orders          # simulate the 02:00 crash
kubectl get pod -l app=orders -o jsonpath='{.items[0].spec.containers[0].image}'
Enter fullscreen mode Exit fullscreen mode

The pod came back on the broken image with an empty deploy history — the outage, on demand. Re-run the same script against the SHA-pinned overlay and the delete/restart brings the same image back every time. That's the whole proof. Once it held, I trusted it in prod.

Before / after

Before After
Image tags :latest immutable sha-… / digest
"What's in prod?" an investigation one label read
Environments copy-pasted k8s-prod/ one base + overlays
Drift silent, unbounded the diff is the only diff
Restart may pull broken latest pinned, deterministic
Rollout on redeploy sometimes a no-op always honest

Guardrail

  • Grep CI and manifests for :latest in a pre-merge check. No merge if it hits.
  • Most local Kubernetes work carried straight to prod: same Dockerfile, same Kustomize base, same probes. Prod parity is the payoff. Prod only adds — registry, Secrets, TLS, HPA, PodDisruptionBudget.
  • Some things must never cross the line: --reload, file-sync dev accelerators, exposed debug ports. The prod overlay is the mechanism that keeps them out.
  • Pin a digest, not just a tag, for anything you truly can't afford to move — @sha256:... is the only reference in Kubernetes that a registry cannot repoint under you.

What I'd do differently

Move off the push pipeline sooner. It has three built-in weaknesses I already know will bite:

  • CI holds the kubeconfig. CI access equals prod access.
  • Drift is invisible. Someone runs kubectl edit by hand and CI never notices.
  • No automatic rollback. Ship broken, revert by hand, at 02:00.

Those three are exactly what GitOps removes. Per the CNCF OpenGitOps principles, Git is the source of truth and an in-cluster agent (Argo CD or Flux) pulls and reconciles drift. CI's job ends at pushing the image and committing the tag. That's the next rung: CI builds an immutable image → a bot bumps the tag in Git → the agent applies it.

Bottom line: the outage wasn't a Kubernetes flaw, it was a mutable tag plus a copy-pasted folder — pin an immutable SHA and collapse onto one base before your own 02:00 page.

Sources

Top comments (0)