Your Deployment's rollout never finished: kubectl rollout status sits on Waiting for deployment "go-api" rollout to finish: 1 old replicas are pending termination..., and kubectl get pods shows the old pod Running next to a new one in ErrImagePull or ImagePullBackOff. That almost always means the new release points at an image the node cannot pull, so Kubernetes keeps the old pod serving and waits. The fix is a rollback, but not the one most people type first. kubectl rollout undo without a revision number goes to "the previous revision", and if anyone runs it twice, the broken release comes back.
I broke a small Go service on purpose to show the whole thing on a real cluster: a release with an image tag that was never pushed, the diagnosis, the rollback, the trap, and the rollback I'd actually run on a Friday evening. Every output below is copied from that recording.
What a stuck rollout looks like
A stuck rollout is a Deployment whose new ReplicaSet never gets a Ready pod, so the rollout neither finishes nor fails right away. Before the bad release, the service ran one pod on go-api:v1.2.0, with one revision in its history:
$ kubectl get deploy go-api -n go-api -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
go-api 1/1 1 1 71s go-api go-api:v1.2.0 app=go-api,...
(I cut the selector short.) Then a new release went out with go-api:v99.9.9, a tag nobody ever pushed. The pipeline goes red, or worse, it goes green because it only checked that kubectl apply succeeded.
The first question is the one people skip: is it deployed or not? Don't ask the pipeline. Ask the rollout.
$ kubectl rollout status deploy/go-api -n go-api
Waiting for deployment "go-api" rollout to finish: 1 old replicas are pending termination...
It never returns, so I stopped it with Ctrl-C. "1 old replicas are pending termination" tells you the old pod is still there because the new one hasn't become ready. Now the pods:
$ kubectl get pods -n go-api
NAME READY STATUS RESTARTS AGE
go-api-5d8497557b-nrv9d 1/1 Running 0 2m59s
go-api-6c8fb7c7bf-v8gm4 0/1 ErrImagePull 0 39s
Two pods. The old one, running and ready. The new one, not ready, because Kubernetes could not pull its image. If you run the command a few seconds later you may see ImagePullBackOff instead: the kubelet alternates between the two as it retries, waiting longer between attempts up to five minutes (kubernetes.io, Images).
And the users? The app's version endpoint, read from the browser while the rollout was stuck, still answered with the old build:
Nothing is down. That's exactly why this failure is easy to miss.
Why users don't notice: the rolling update waits
A rolling update only removes an old pod once a new pod is Ready. The Deployment creates a new ReplicaSet for the new pod template and scales it up, and it scales the old one down only as new pods become available. With one replica and the default maxUnavailable of 25% (rounded down to zero) and maxSurge of 25% (rounded up to one), Kubernetes may add one extra pod but may not take the only available one away (kubernetes.io, Deployments: rolling update).
The new pod can't pull its image, so it never becomes Ready, so the old pod is never removed. You have a stuck deploy, not an outage. The same mechanism that protects your users also hides the failure from anyone who only watches error rates.
Find the reason in the pod's events
The pod's events name the image the kubelet tried to pull and the registry's answer. I described the new pod and paged to the Events section:
$ kubectl describe pod -n go-api go-api-6c8fb7c7bf-v8gm4
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 91s default-scheduler Successfully assigned go-api/go-api-6c8fb7c7bf-v8gm4 to k3d-snowops-agent-0
Normal BackOff 19s (x4 over 89s) kubelet spec.containers{go-api}: Back-off pulling image "go-api:v99.9.9"
Warning Failed 19s (x4 over 89s) kubelet spec.containers{go-api}: Error: ImagePullBackOff
Normal Pulling 7s (x4 over 91s) kubelet spec.containers{go-api}: Pulling image "go-api:v99.9.9"
Warning Failed 6s (x4 over 89s) kubelet spec.containers{go-api}: Failed to pull image "go-api:v99.9.9": failed to pull and unpack image "docker.io/library/go-api:v99.9.9": failed to resolve reference "docker.io/library/go-api:v99.9.9": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 6s (x4 over 89s) kubelet spec.containers{go-api}: Error: ErrImagePull
(I joined the wrapped lines and lined up the columns. The text is unchanged.)
Two things in there are worth reading slowly. The image is go-api:v99.9.9, the tag that was never pushed. And the error does not say "tag not found". It says pull access denied, repository does not exist or may require authorization. A short image name with no registry is resolved to docker.io/library/..., and Docker Hub answers a missing repository with an authorization error. So that message can mean a missing image as easily as missing credentials. Check the name and tag before you go hunting for pull secrets.
Each release of a Deployment is a ReplicaSet, and you can see both side by side with their images:
$ kubectl get rs -n go-api -o wide
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
go-api-5d8497557b 1 1 1 4m21s go-api go-api:v1.2.0 app=go-api,...
go-api-6c8fb7c7bf 1 1 0 2m1s go-api go-api:v99.9.9 app=go-api,...
The old release has its pod ready. The new one has one pod and none ready.
In the lab, a Prometheus alert on kube_pod_container_status_waiting_reason with reason=~"ImagePullBackOff|ErrImagePull" fired a minute later. That metric comes from kube-state-metrics. If your error-rate alerts stayed quiet during a failed release, an alert like this is the gap.
Read the history before you undo anything
Every change to a Deployment's pod template creates a numbered revision, and kubectl rollout history lists them (kubernetes.io, checking rollout history):
$ kubectl rollout history deploy/go-api -n go-api
deployment.apps/go-api
REVISION CHANGE-CAUSE
1 <none>
2 <none>
The list only shows numbers. To see what a revision holds, ask for it by number:
$ kubectl rollout history deploy/go-api -n go-api --revision=2
deployment.apps/go-api with revision #2
Pod Template:
...
Containers:
go-api:
Image: go-api:v99.9.9
...
(I trimmed the labels, ports, resources and probes.) Revision 2 is the bad release. Revision 1 is the one to go back to.
Why does CHANGE-CAUSE show <none>?
CHANGE-CAUSE is copied from the Deployment's kubernetes.io/change-cause annotation when a revision is created, and nothing sets that annotation unless you do. The old --record flag used to fill it and is deprecated. Set it yourself as part of the release, for example kubectl annotate deploy/go-api -n go-api kubernetes.io/change-cause="release 1.3.0, image go-api:v1.3.0" right after the change. Then your history tells you which revision is which without opening each one.
The first rollback works
kubectl rollout undo with no flags rolls the Deployment back to the previous revision:
$ kubectl rollout undo deploy/go-api -n go-api
deployment.apps/go-api rolled back
$ kubectl rollout status deploy/go-api -n go-api
deployment "go-api" successfully rolled out
$ kubectl get pods -n go-api
NAME READY STATUS RESTARTS AGE
go-api-5d8497557b-nrv9d 1/1 Running 0 6m24s
This time rollout status answers at once. And look at the pod: same name as before the release, six minutes old, zero restarts. The rollback restarted nothing. The pod template went back to the one the old ReplicaSet already had, so the Deployment kept that ReplicaSet's pod and scaled the stuck one to zero. Users never saw a thing.
Most people stop here. Look at the history first:
$ kubectl rollout history deploy/go-api -n go-api
deployment.apps/go-api
REVISION CHANGE-CAUSE
2 <none>
3 <none>
Revision 1 is gone. Undo didn't delete anything. It copied revision 1's pod template forward and made it the newest revision, number 3. Keep that in mind.
The trap: undo, twice
Now picture the Friday version of this. A colleague sees the red pipeline and runs the same rollback. Or the pipeline's retry does. Same command:
$ kubectl rollout undo deploy/go-api -n go-api
deployment.apps/go-api rolled back
$ kubectl get pods -n go-api
NAME READY STATUS RESTARTS AGE
go-api-5d8497557b-nrv9d 1/1 Running 0 7m6s
go-api-6c8fb7c7bf-tgdg8 0/1 ErrImagePull 0 7s
"Rolled back", it says, and the broken pod is back. The history shows why:
$ kubectl rollout history deploy/go-api -n go-api
deployment.apps/go-api
REVISION CHANGE-CAUSE
3 <none>
4 <none>
Undo means "the previous revision", whatever that is right now. After the first undo, the previous revision was the bad release, so the second undo restored it as revision 4. Two undos in a row put you exactly where you started. The command even reports success both times.
Roll back to a revision you checked
kubectl rollout undo --to-revision=N rolls back to the revision you name, and gives the same result no matter who ran what before you (kubectl rollout undo). So the safe order is: read the revision, then name it.
$ kubectl rollout history deploy/go-api -n go-api --revision=3
deployment.apps/go-api with revision #3
Pod Template:
Labels: app=go-api
...
pod-template-hash=5d8497557b
...
Containers:
go-api:
Image: go-api:v1.2.0
v1.2.0, the build users were on. Now roll back to it by number and prove it:
$ kubectl rollout undo deploy/go-api -n go-api --to-revision=3
deployment.apps/go-api rolled back
$ kubectl rollout status deploy/go-api -n go-api
deployment "go-api" successfully rolled out
$ kubectl get pods -n go-api
NAME READY STATUS RESTARTS AGE
go-api-5d8497557b-nrv9d 1/1 Running 0 8m31s
$ kubectl get rs -n go-api -o wide
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
go-api-5d8497557b 1 1 1 8m43s go-api go-api:v1.2.0 app=go-api,...
go-api-6c8fb7c7bf 0 0 0 6m23s go-api go-api:v99.9.9 app=go-api,...
One pod, the same one as at the very start. The good ReplicaSet has its pod. The bad one is scaled to zero but still there, because it's still in the revision history. The next plain undo could find it again, which is why the release itself still has to be fixed. The version endpoint answered v1.2.0 again, and the lab's own check passed.
Run this way, a second engineer typing the same command changes nothing. That's the property you want from anything people might run during an incident.
Questions people ask next
Does rollout undo revert ConfigMaps and Secrets?
No. A revision is created only when the pod template (.spec.template) changes, and a rollback restores only the pod template (kubernetes.io, rolling back a Deployment). If the release also changed a ConfigMap or a Secret that the pods read, undo puts the old image back but leaves the new config in place. Revert the config separately, or version it by name (for example app-config-v2) so the pod template references it and a rollback brings the old name back too.
Will Argo CD or Flux overwrite my manual rollback?
Often, yes. A GitOps controller makes the cluster match Git. With Argo CD's automated sync and self-heal on, a manual rollout undo is drift and gets reverted (Argo CD automated sync). Flux reapplies the manifests on its reconcile interval. In a GitOps setup the rollback is a revert commit in Git. If you need the cluster fixed before that merges, pause the controller first (turn off auto-sync for the Argo CD application, or flux suspend kustomization <name>), roll back by revision, then put the revert in Git and resume.
Doesn't Kubernetes give up on a stuck rollout by itself?
It reports, it doesn't act. After progressDeadlineSeconds (600 seconds by default) the Deployment gets a Progressing condition with status False and reason ProgressDeadlineExceeded, and kubectl rollout status exits with a non-zero code. Kubernetes takes no other action, so there is no automatic rollback (kubernetes.io, failed Deployment). That non-zero exit is useful: run kubectl rollout status as a gate in the pipeline and a release that never finishes turns the pipeline red instead of green.
Roll back or fix forward?
At 5pm on a Friday, roll back to a checked revision, then find out on Monday why the tag was missing. If you know the right tag and know it exists, kubectl set image deploy/go-api go-api=go-api:<known-good-tag> -n go-api fixes forward. Either way, fix the source of the release (the manifest, the Helm values, the pipeline's tag) or the next deploy brings the bad tag back.
Other causes of a stuck rollout
The same symptom, a rollout waiting on a new pod that never becomes Ready, has several causes besides an image that doesn't exist. Kubernetes lists insufficient quota, failing readiness probes, image pull errors, insufficient permissions, limit ranges and application runtime misconfiguration (kubernetes.io, failed Deployment). In practice:
-
Pull errors with a real image: a private registry without an
imagePullSecret, or a node that can't reach the registry. The event message tells these apart from a missing tag. -
The pod starts but never gets Ready: a readiness probe that fails on the new build.
describe podshowsReadiness probe failedin the events. -
The pod starts and crashes:
CrashLoopBackOffon the new pod. The rollout waits exactly the same way. I wrote up how to find the real cause of a CrashLoopBackOff separately. -
The pod is never created or never scheduled: quota or limit range errors show up in
kubectl describe rs <new-replicaset>, andPendingpods withFailedSchedulingevents mean the cluster has no room.
The diagnosis order doesn't change: rollout status, the pods, the new pod's events, then the history.
Try it yourself
On any test cluster, you can cause the same failure with one command on a throwaway Deployment:
kubectl set image deploy/<your-app> <container>=<its-image>:v99.9.9
Then work through it: rollout status, pods, events, history, undo, and undo again to see the trap.
I ran this drill in SnowOps Labs, an open-source Kubernetes lab where you inject faults like this one into a real app and fix them (GitHub). Disclosure: I build it. The same fault is also a timed challenge there, par six minutes. My run took 1 minute 34 seconds with no hints.
The ladder to remember
-
kubectl rollout status deploy/<name>: ask the rollout, not the pipeline. -
kubectl get pods: old pod Running, new podErrImagePullorImagePullBackOff. -
kubectl describe pod <new-pod>: the Events name the image and the registry's answer. -
kubectl rollout history deploy/<name> --revision=N: read the revision you want to go back to. -
kubectl rollout undo deploy/<name> --to-revision=N: never a bare undo during an incident. -
kubectl rollout statusagain, then fix the release itself.
*This is part of a series I'm writing on debugging Kubernetes in production. You can find other videos on our Youtube channel Snowops Learn.
We also created a game Snowops Night Shift in which you play as a Devops Engineer who gets new incidents daily, and you solve them based on kubernetes commands, logs and dashboards. The game is a simulation, but all logs and commands shown are from real cluster generated from Snowops Labs.



Top comments (0)