Oleksandr Kuryzhev

Posted on Jul 29 • Originally published at kuryzhev.cloud

kubectl apply Not Updating Pods? Fix the Stale Image Pitfall

#devops

Originally published on kuryzhev.cloud

Last Monday a teammate pinged me: "deploy went green, pipeline says success, but the bug I fixed two hours ago is still live in prod." We'd both stared at kubectl apply -f deployment.yaml returning a clean deployment.apps/api configured — no errors, no warnings. And yet nothing changed. This is the classic "kubectl apply not updating" pitfall, and it's one of the most common silent failures I've seen in production Kubernetes clusters. No CrashLoopBackOff, no ImagePullBackOff, no red text anywhere. Just... old code, quietly running forever.

I've hit this exact issue on three separate teams now, always for the same reason: someone left a mutable image tag in a manifest "temporarily." Here's the full runbook — symptoms, root cause, and three fixes ranked from "unblock me right now" to "never let this happen again." Tested against Kubernetes 1.28, kubectl client 1.28.2, containerd 1.7.

Symptoms

This pitfall is deceptively quiet. You won't get a Slack alert or a red pipeline stage — that's exactly what makes it dangerous. Here's what you'll actually observe:

kubectl apply -f deployment.yaml returns deployment.apps/api configured with zero errors, but the running app still behaves like the old version — same bug, same missing feature.
kubectl get pods shows no new pod created, no restart count increment, and the same AGE value on existing pods as before you ran the "deploy."
kubectl describe pod <pod> shows the exact same Image ID (the sha256 digest) as it did before your supposed new release — this is the smoking gun that confirms the pull never happened at all.

Watch out: don't confuse this with ImagePullBackOff. That's a loud, obvious failure — bad credentials, unreachable registry, typo in the image name. This pitfall is the opposite: everything reports success. GitOps tools like ArgoCD and Flux have the exact same failure mode — a "Synced" status only confirms the spec in the cluster matches Git, not that a new image was actually pulled.

Root cause

kubectl apply performs a three-way merge diff against the live object, the last-applied config, and your new manifest. If the image string in your YAML is byte-for-byte identical to what's already running, Kubernetes sees zero diff. No new ReplicaSet gets created, regardless of what you pushed to the registry five minutes ago. This isn't a bug — it's the declarative model working exactly as designed.

The ReplicaSet hash is derived from the pod template spec, and the pod-template-hash label won't change unless something in that template actually changes. An unchanged image string means an unchanged hash means zero rollout, every single time.

Then there's imagePullPolicy. The default is Always only when the tag is literally :latest. For any other tag — :stable, :dev, :v2 — the default is IfNotPresent. If the node's containerd content store already has a layer cached under that tag, kubelet won't re-pull it even when the pod gets recreated.

Common mistake I see constantly: assuming kubectl apply -f behaves like docker run --pull always. It doesn't. One is an imperative pull command; the other is a declarative diff engine. Mutable tags break the entire mental model — the registry image changes underneath you, but nothing in the cluster spec tells Kubernetes to act on it.

Fix #1

When production is stale right now, don't touch the manifest — force a rollout directly. This is the fastest unblock:

# force a new ReplicaSet and fresh pod pulls without editing YAML
kubectl rollout restart deployment/api -n prod
kubectl rollout status deployment/api -n prod --timeout=60s

# confirm the image actually changed on the running pod
kubectl get pod -n prod -l app=api -o jsonpath='{.items[0].status.containerStatuses[0].imageID}'

kubectl rollout restart patches spec.template.metadata.annotations with a timestamp. That's the whole trick — it's a spec change, so Kubernetes creates a new ReplicaSet and reschedules pods. Verify with kubectl rollout status, then grep the Image ID in describe pod output.

Gotcha: this only helps if imagePullPolicy is Always, or if the node's cached layer for that tag has genuinely changed. If the node still has the old layer cached under the same tag and the policy is IfNotPresent, you'll restart straight back into the identical stale image. I've watched engineers restart a deployment five times in a row, convinced it "should have worked," while the node cache silently serves the same bits every time.

Fix #2

The real fix is at the source: make the manifest spec actually change on every deploy. Stop using mutable tags entirely.

# CI/CD tags with something unique per build — the git SHA works great
GIT_SHA=$(git rev-parse --short HEAD)
docker build -t myregistry/api:sha-$GIT_SHA .
docker push myregistry/api:sha-$GIT_SHA

# this actually changes the Deployment spec -> triggers a real rollout
kubectl set image deployment/api api=myregistry/api:sha-$GIT_SHA -n prod
kubectl rollout status deployment/api -n prod --timeout=90s

Because the image string now genuinely differs on every build, kubectl set image produces a real diff and a real, auditable rollout — no restart hacks needed. This is the pattern I now enforce on every pipeline I own. Bonus: unique tags make kubectl rollout undo a real rollback to a known-good image, and you get a clean audit trail of exactly what's running in each environment at any point in time.

If you're on Helm, the same problem exists: helm upgrade with an unchanged image.tag value in values.yaml won't force a rollout either. Same fix applies — pass --set image.tag=$SHA from your pipeline instead of hardcoding a tag in the chart values.

Fix #3

If your team can't fully control tag hygiene yet — legacy pipelines, shared registries, third-party images — add a safety net at the pod spec level.

kubectl patch deployment api -n prod --type='json' \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/imagePullPolicy","value":"Always"}]'

Setting imagePullPolicy: Always explicitly forces kubelet to re-check the registry digest on every scheduling event, even for a tag it's already cached. This closes the "cached stale layer" gap regardless of what tag string you're using.

Watch out: this trade-off has a real cost. Every pod start or restart now does a registry round trip. On a low-traffic service that's negligible. On a high-scale deployment with frequent horizontal autoscaling events, it adds pull latency to every scale-up and increases registry egress traffic meaningfully — I've seen this measurably slow down autoscaler response time under load. Use Fix #3 as a safety net, not a substitute for Fix #2. It papers over bad tagging; it doesn't fix it.

Prevention

Fixing one deployment doesn't stop the next engineer from copy-pasting :latest into a new service six months later. Bake prevention into the pipeline and policy layer:

Ban :latest and other mutable tags in production manifests with an admission policy — OPA Gatekeeper or Kyverno rules rejecting non-immutable tags are trivial to write and catch this before it ever reaches a cluster.
Pin CI to always tag with the commit SHA, or better, the full image digest (image@sha256:...) for anything security-sensitive. Mutable tags allow tag-reuse attacks — if a compromised image gets pushed under an existing tag, every future pull, and every future rollout restart, silently deploys the compromised image. Digest pinning closes that door entirely. The official Kubernetes image docs cover this in more depth.
Add kubectl rollout status --timeout=60s as a required pipeline step after every apply, not just kubectl apply -f and hope. A no-op "successful" apply should never be allowed to pass as a real deploy in your CI logs.

I stopped trusting green pipeline checkmarks after this bit us twice in one quarter. Now every deploy pipeline I set up has an explicit "did the image digest actually change" assertion before marking the job successful. It's a five-line check that's saved us multiple late-night "why is prod still broken" investigations. If you're building out your CI/CD gates more broadly, we've got a deeper checklist on kuryzhev.cloud covering quality gates and rollback paths worth pairing with this fix.

Bottom line: kubectl apply not updating isn't a Kubernetes bug — it's a tagging discipline problem wearing a Kubernetes costume. Fix the tags, and the symptom disappears for good.

DEV Community