DEV Community

Menshikov Vasil
Menshikov Vasil

Posted on

Why `kubectl apply` doesn't roll out your `:latest` image (and the bridge from local to CI)

#devops #docker #kubernetes #cicd

The counterintuitive pipeline bug: you push a new image under the same :latest tag, run kubectl apply, and nothing deploys. Kubernetes only triggers a rollout when the pod template changes — the line image: myapp:latest didn't change, so as far as the cluster is concerned there's nothing to do, even though the registry image is now different.

Takeaways

  • latest is three problems in one: it's mutable (no reproducibility); re-applying doesn't roll out (template unchanged); and a Pod restart may silently pull a different, possibly broken image. Prod changes version "by itself."
  • Use immutable tags that point to one specific build: short SHA (sha-abc1234), semver (v1.2.3), or a digest (@sha256:...) for maximum reproducibility. In CI, docker/metadata-action generates these from git context and writes OCI labels (org.opencontainers.image.revision, etc.) tying the image to a commit.
  • One base, per-environment overlays to avoid config drift: Helm (Go templates + values-prod.yaml; multiple -f merge last-wins — order matters) or Kustomize (plain YAML base/ + overlays/; the images: field swaps tags without editing the Deployment). Hybrid is common: third-party charts via Helm, your own services via Kustomize.
  • Minimal push-based CI (GitHub Actions): checkout -> buildx -> login -> metadata-action -> build-push-action -> deploy with an immutable tag (kubectl set image / helm upgrade / kustomize build | apply).
  • Be deliberate about the local/prod split. Reused as-is: same Dockerfile/image, same charts/base, probes. Prod-only: real registry + immutable tags, HPA/PDB/anti-affinity, real Secrets/TLS, monitoring. Must NOT carry over: Tilt live-update/file-sync, uvicorn --reload, exposed debug ports — they're a security hole in prod.
  • Push-CI's three weaknesses point to GitOps: CI holds the cluster keys (attack surface), Git-vs-cluster drift isn't tracked, and there's no automatic rollback.

Full article: https://dorokhovich.com/blog/local-k8s-preparing-for-deployment-and-ci?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-preparing-for-deployment-and-ci

Top comments (0)