CrashLoopBackOff: how to find the real cause (and why deleting the pod won't help)

Sagar — Wed, 21 Oct 2026 11:48:23 +0000

A config change goes out. A couple of minutes later every request to the app fails, and kubectl get pods shows the status nobody wants to see:

NAME                      READY   STATUS             RESTARTS      AGE
go-api-7b45497dfd-qxqx2   0/1     CrashLoopBackOff   4 (15s ago)   115s

I broke a small Go service on purpose to walk through this one properly, from the first symptom to a fix you can prove. The commands work on any cluster. The specific bug is just one example, and I'll come back at the end to the other things that cause the same status.

What CrashLoopBackOff actually tells you

It is easy to read CrashLoopBackOff as an error. It isn't one. It is Kubernetes describing what it is doing about an error.

The container starts and then exits. The kubelet on the node restarts it, because the pod's restartPolicy says so. If it exits again, the kubelet waits before the next try, and the wait grows each time: 10 seconds, 20, 40, and so on, capped at five minutes (Kubernetes docs: container restarts). While it is waiting, the pod shows CrashLoopBackOff.

So the status answers "what is happening?" (the container keeps dying) and says nothing about why. That second question is the whole job, and the pod has already written the answer down somewhere. You just have to know where to read it.

First, check whether anyone is still being served

Before digging into the crash, it's worth thirty seconds to find out how bad things are. A crash loop during a rollout is often harmless: with a rolling update and a readiness probe, the old pods keep serving until a new one becomes Ready. If the new pod never gets there, the rollout just stalls.

A Deployment keeps one ReplicaSet per version of its pod template, so listing them tells you whether an older version is still up:

$ kubectl get rs -n go-api
NAME                DESIRED   CURRENT   READY   AGE
go-api-5d8497557b   0         0         0       5m46s
go-api-7b45497dfd   1         1         0       2m9s

Not good. The old ReplicaSet is at zero, and the new one wants a pod and has none ready. Nothing is serving. In my drill I set it up that way on purpose, but you get the same picture in real life with a Recreate strategy, or when the crash only starts after the pod has already passed its readiness check. Either way, this isn't a stuck rollout any more. It's an outage.

The browser agreed:

That message comes from the ingress. It had a route for the app and no ready pod to send the request to.

A detour worth taking: the dashboard that says everything is fine

This was the part of the drill that surprised me most. Here is the Grafana dashboard during the outage. On the left is the app's own error rate (HTTP 5xx). On the right is the failure rate seen by k6, the load generator sending traffic from outside.

The app reports zero errors. The client sees 100% of requests fail. Both are telling the truth: a process that never finishes starting can't count the requests it never received, so its own metrics go quiet rather than red. If your alerting only looks at what the service says about itself, a total outage can look like a calm afternoon. Watch at least one signal from the client side too, whether that's an ingress or load balancer metric, a synthetic check, or a probe from outside the cluster.

Read the crash's last words: `kubectl logs --previous`

Plain kubectl logs shows the current container. In a crash loop the current container has often just started, or hasn't started yet, so you get nothing useful. The container that just died is still on the node, and --previous (or -p) asks for its logs:

$ kubectl logs -n go-api go-api-7b45497dfd-qxqx2 --previous
{"level":"INFO","msg":"server starting","port":"80800","service":"go-api"}
{"level":"ERROR","msg":"server error","error":"listen tcp: address 80800: invalid port"}

(I trimmed the timestamps.) Two lines, and the second one is the answer. The server tried to listen on port 80800. Port numbers stop at 65535, so the bind fails and the process exits.

Most of the time the logs get you this far. When they don't, the next command usually does.

Confirm it from Kubernetes' side: `kubectl describe pod`

describe is long, so I pipe it into less and search for Last State:

$ kubectl describe pod -n go-api go-api-7b45497dfd-qxqx2 | less
...
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Ready:          False
    Restart Count:  4
    ...
    Environment:
      PORT:                 80800
      READINESS_FAILURE:    false
      SHUTDOWN_TIMEOUT:     10s

Three things to take from this block.

The exit code. 1 means the app chose to exit with an error. Nothing killed it. If you see 137 instead, the process got a SIGKILL (128 + 9), and a Reason of OOMKilled tells you it ran out of memory. Those two numbers send you down completely different paths, so it's worth reading this line before anything else.

The environment. The same 80800 again, now in the spec. So the bad value isn't something the app made up at runtime. Kubernetes handed it over.

The events at the bottom. In this case they only confirm the loop: Back-off restarting failed container. For other causes they carry the real clue, such as Liveness probe failed, a missing ConfigMap or Secret, or an image that can't be pulled.

The fix everyone tries first

With the pod clearly broken, deleting it is tempting. I tried:

$ kubectl delete pod -n go-api go-api-7b45497dfd-qxqx2
pod "go-api-7b45497dfd-qxqx2" deleted

$ kubectl get pods -n go-api
NAME                      READY   STATUS             RESTARTS      AGE
go-api-7b45497dfd-4c995   0/1     CrashLoopBackOff   1 (21s ago)   22s

A new name, the same crash, within seconds. The pod was never the problem. A pod owned by a Deployment is a copy of the Deployment's pod template, and the ReplicaSet will keep making fresh copies of the same mistake for as long as you delete them. The only thing deleting achieved was resetting the restart counter, which made the situation look less serious than it was.

Fix it where it lives: the pod template

So the question becomes where the template gets PORT=80800 from. describe deploy shows the template:

$ kubectl describe deploy go-api -n go-api | less
...
  Containers:
   go-api:
    Image:      go-api:v1.2.0
    Port:       8080/TCP (http)
    Liveness:   http-get http://:http/health delay=15s timeout=5s period=10s
    Readiness:  http-get http://:http/ready delay=10s timeout=3s period=5s
    Environment:
      PORT:                 80800

The container declares port 8080, both probes use that named port, and the environment variable says 80800. One zero too many. The fix goes on the Deployment:

$ kubectl -n go-api set env deploy/go-api PORT=8080
deployment.apps/go-api env updated

Changing the template starts a new rollout. In a real team you'd make the same change in whatever produced the manifest (the Helm values, the Kustomize overlay, the GitOps repo) or the next sync will put the typo straight back. kubectl set env is fine for stopping the bleeding, as long as the real fix follows it.

kubectl rollout undo deploy/go-api -n go-api would have worked too. It returns the template to the previous revision, environment included. When you're paged and you know the last change caused it, rolling back first and investigating second is often the right call.

Prove the fix instead of assuming it

Don't eyeball get pods a few seconds after the change. rollout status waits until the new pods are actually Ready:

$ kubectl -n go-api rollout status deploy/go-api
Waiting for deployment "go-api" rollout to finish: 1 old replicas are pending termination...
deployment "go-api" successfully rolled out

$ kubectl get pods -n go-api
NAME                      READY   STATUS    RESTARTS   AGE
go-api-5d8497557b-r47rn   1/1     Running   0          26s

$ kubectl get rs -n go-api
NAME                DESIRED   CURRENT   READY   AGE
go-api-5d8497557b   1         1         1       9m8s
go-api-7b45497dfd   0         0         0       5m31s

Look at the ReplicaSet names. Kubernetes didn't create a third one. The fixed template is identical to the one from before the bad change, so the Deployment scaled the old ReplicaSet back up. It's a small detail, but it's a nice confirmation that the template now matches the last version that worked.

Then check from outside, the way users see it. The health URL answered again, and the request rate on the dashboard came back.

Try it on your own cluster

You don't need anything special to reproduce this. On a test cluster, take any Deployment whose app reads its port from an environment variable and give it an impossible one:

kubectl set env deploy/<your-test-app> PORT=80800
kubectl get pods -w

With the default rolling update you'll likely see the stalled-rollout version: the old pod stays Running while the new one crash-loops. That's a useful thing to see for yourself, because it's exactly why the ReplicaSet check at the start matters. Put it back with:

kubectl rollout undo deploy/<your-test-app>

If you'd rather practise the full outage version, with traffic, dashboards, alerts and a check that tells you when you've actually fixed it, this drill is one of the incidents in SnowOps Labs, a free, open-source Kubernetes lab that runs on a laptop. Disclosure: I build it. Every screenshot above is from it, and the code is on GitHub.

Other causes of the same status

The port typo is one cause among many. The Kubernetes docs group the usual suspects into application errors, configuration errors, resource limits, and failing health checks. In practice that looks like:

A bug or a missing dependency in the app. Logs with --previous usually say so, often as a stack trace or a "connection refused" to a database that isn't there.
Missing or wrong configuration. A typo like this one, a variable renamed in one place and not the other, a config file mounted at the wrong path, or a value that's fine in staging and wrong in production.
Not enough memory. Exit code 137 with Reason: OOMKilled in describe. Raise the limit or fix the leak, but find out which one first.
A liveness probe that kills a healthy app. The app starts, the probe fails (too strict, wrong path, too short a delay for a slow start), and the kubelet restarts the container. The events say Liveness probe failed.
The container never really starts. A wrong command or args, an image built for another CPU architecture, a missing binary. Here the logs are often empty, because nothing ran long enough to write any. describe and its events carry the message instead.

One extra tip from the docs on finding why a pod failed: set terminationMessagePolicy: FallbackToLogsOnError on your containers. When a container fails without writing a termination message, Kubernetes then copies the last lines of its log into the pod status, so describe shows the error even after the logs are gone.

The short version

When a pod is in CrashLoopBackOff, this order gets you to the cause quickly, whatever the cause turns out to be:

kubectl get pods shows the symptom and the restart count.
kubectl get rs tells you whether anything is still serving, which decides how urgent this is.
kubectl logs <pod> --previous gives you the dead container's last words.
kubectl describe pod <pod> gives the exit code, the reason and the events.

Then fix the thing the pod was copied from, not the pod, and let rollout status tell you when it's really done.

DEV Community: Sagar