Tandap Noel Bansikah

Posted on Aug 31

My Pod Won't Start" Is Actually Three Different Problems

#devops #debugging #kubernetes #help

Every Kubernetes beginner hits this. You deploy something, run kubectl get pods,
and the pod is not running. So you reach for the command everyone reaches for:

kubectl logs my-pod

And you get nothing useful. Sometimes you get an error saying there are no logs
at all.

The reason is that "pod won't start" is not one problem. It is at least three
unrelated problems that happen to look similar in the output. Each one fails at
a different stage of the pod lifecycle, and each needs a different command
to diagnose.

Let me show you all three at once.

Reproducing All Three

You need any Kubernetes cluster to follow along. I used a local K3d cluster, but
minikube, kind, or a remote cluster all work the same way.

Step 1 — Create the manifest

Create a file called broken.yaml and paste in the following. It defines a
learning namespace plus three pods, each broken in a different way:

apiVersion: v1
kind: Namespace
metadata:
  name: learning
---
# Tag does not exist, so the kubelet can never pull it -> ImagePullBackOff
apiVersion: v1
kind: Pod
metadata:
  name: bad-image
  namespace: learning
spec:
  containers:
    - name: app
      image: nginx:this-tag-does-not-exist
---
# Exits non-zero immediately; restartPolicy Always retries -> CrashLoopBackOff
apiVersion: v1
kind: Pod
metadata:
  name: crash-loop
  namespace: learning
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "echo 'starting up'; exit 1"]
---
# Requests more memory than any node has -> Pending, never scheduled
apiVersion: v1
kind: Pod
metadata:
  name: unschedulable
  namespace: learning
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests:
          memory: 500Gi

Everything lands in its own learning namespace, so this cannot disturb
anything already running in your cluster, and cleanup is a single command at the
end.

Each pod breaks at a deliberately different stage:

bad-image points at a tag that does not exist, so the image can never be pulled.
crash-loop starts fine, prints a line, then exits 1.
unschedulable asks for 500Gi of memory, which no normal node can satisfy.

Step 2 — Apply it

kubectl apply -f broken.yaml

namespace/learning created
pod/bad-image created
pod/crash-loop created
pod/unschedulable created

Step 3 — Watch them fail

Give it about 20–30 seconds. The failures take a moment to appear, because
Kubernetes retries the image pull and the container restart before it reports a
backoff state:

kubectl apply -f broken.yaml
kubectl get pods -n learning

Three pods. Three different STATUS values. Three completely different causes:

Pod	Status	What failed
`bad-image`	`ImagePullBackOff`	Never downloaded the image
`crash-loop`	`Error` / `CrashLoopBackOff`	Ran, then died
`unschedulable`	`Pending`	Never got placed on a node

That STATUS column is the most important thing on the screen. It tells you
which command is worth typing next.

Notice something in the screenshot: crash-loop shows Error, not
CrashLoopBackOff. Both are normal. The pod alternates between Error
immediately after crashing and CrashLoopBackOff while it waits before the next
retry. Catch it at the wrong moment and you see a different word for the same
underlying problem.

Problem 1: ImagePullBackOff — it never got the image

The pod was scheduled to a node, but the kubelet could not download the
container image. The container never existed, so there are no logs to read.

For this one, describe is the right tool, because the story is in the events:

kubectl describe pod bad-image -n learning

Read the Events section from the bottom up:

Pulling    Pulling image "nginx:this-tag-does-not-exist"
Failed     failed to resolve reference "docker.io/library/nginx:this-tag-does-not-exist": not found
Failed     Error: ErrImagePull
BackOff    Back-off pulling image "nginx:this-tag-does-not-exist"
Failed     Error: ImagePullBackOff

That is the whole story in five lines. Kubernetes tried to pull, the registry
said the tag does not exist, so it backed off and started retrying with an
increasing delay.

Two details worth noticing:

Successfully assigned learning/bad-image to k3d-noel-lab-server-0 appears
near the top. Scheduling worked fine. This immediately rules out node capacity
and scheduling rules as the cause.

Restart Count: 0. Zero. This is the giveaway. Kubernetes never started the
container, so there was nothing to restart.

In real clusters this is almost always one of:

a typo in the image tag
a private registry with no imagePullSecrets
a rate-limited public registry

The fix: use a tag that exists.

image: nginx:1.27-alpine

Problem 2: CrashLoopBackOff — it ran and died

This is the opposite situation. The image pulled fine, the container started,
and then the process exited. Because the default restartPolicy is Always,
Kubernetes restarts it, it dies again, and the delay between attempts grows.

Here logs do exist, because the container really ran:

kubectl logs crash-loop -n learning
kubectl get pod crash-loop -n learning \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

The application printed starting up, then exited with code 1. That is an
application bug, not a Kubernetes problem. Kubernetes is doing exactly what it
was told: keep this thing running.

Exit codes are worth memorising:

Code	Meaning
`0`	Clean exit — but a `restartPolicy: Always` pod will still be restarted
`1`	General application error
`137`	SIGKILL — very often an out-of-memory kill
`143`	SIGTERM — usually a normal shutdown signal

If the container has already restarted, the current logs may be empty because
they belong to a brand new attempt. Use the previous instance instead:

kubectl logs crash-loop -n learning --previous

The fix depends on intent. If the process is supposed to keep running, fix
the crash. If it is supposed to run once and finish, it should be a Job, not a
plain pod with restartPolicy: Always.

Problem 3: Pending — it never got a node

The third pod is stuck at Pending with no restarts and no node assigned. No
container was ever created, which is why logs and describe on the container
give you nothing.

Pending is a scheduler problem, so ask the scheduler:

kubectl get events -n learning \
  --field-selector involvedObject.name=unschedulable

That message is unusually direct. Both nodes were evaluated, neither had enough
memory, and preemption (evicting lower-priority pods) would not help either.

The pod requested 500Gi of memory. No node in this cluster has that, so the
scheduler simply refuses to place it. It will wait forever.

The fix: request something the cluster actually has.

resources:
  requests:
    memory: 64Mi

Other common causes of Pending:

a nodeSelector or affinity rule that matches no node
taints without a matching toleration
a PersistentVolumeClaim that never binds

The Rule Worth Remembering

Read STATUS first. It tells you which stage failed, and therefore which command
will actually help:

Pending           -> scheduler problem    -> kubectl get events
ImagePullBackOff  -> registry problem     -> kubectl describe pod
CrashLoopBackOff  -> application problem  -> kubectl logs --previous

The restart count is a useful second signal:

0 restarts and 0/1 ready means the container never started. Logs cannot help you. Look at events.
Climbing restarts means the container did run. Logs are exactly where the answer is.

That single distinction saves a lot of time, because it stops you from running
kubectl logs against a container that never existed and concluding that
Kubernetes is broken.

Cleanup

kubectl delete namespace learning

DEV Community