DEV Community

Metronom
Metronom

Posted on

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Kubernetes ImagePullBackOff, CrashLoopBackOff, OOMKilled: The 3-Command Troubleshooting Playbook

Most Kubernetes ImagePullBackOff / CrashLoopBackOff troubleshooting is not deep. The statuses read like spaceship error codes, but nearly all of them fall out of three commands and a short list of causes. This is the playbook I run on autopilot, plus the one k3d trap that cost me a morning.

The golden set

Whatever the status, start here. Same describe / logs / exec loop the Kubernetes 'Debug Running Pods' guide recommends.

# 1. State, Reason, and the Events block at the bottom — ALWAYS first
kubectl describe pod <pod> -n myapp

# 2. App logs, and the CRASHED container's logs via --previous
kubectl logs <pod> -n myapp
kubectl logs <pod> -n myapp --previous

# 3. Namespace timeline
kubectl get events -n myapp --sort-by=.lastTimestamp
Enter fullscreen mode Exit fullscreen mode

describe hands you three fields: State, Reason, and Events. That's the loop. The rest is pattern-matching. There's a solid field guide to these same statuses at this troubleshooting writeup.

ImagePullBackOff — and the k3d containerd trap

Symptom

ErrImagePull first, then ImagePullBackOff after backoff. Pod stuck Waiting. Events line usually says it outright: not found / unauthorized / no such host.

Root cause

The boring causes are enumerated in the Kubernetes images docs: typo in name or tag, private registry with no imagePullSecrets, Docker Hub rate limit (toomanyrequests), bad registry address.

The trap that isn't boring: k3d nodes run on containerd, isolated from your Docker daemon. The image you just built with docker build — visible right there in docker images — the cluster cannot see. Docker has it; containerd doesn't. ImagePullBackOff, even though "the image is obviously built."

The fix

Two correct ways to get the image into the cluster, both in the k3d image-import docs:

# Option 1: import into the nodes directly
docker build -t myapp:dev .
k3d image import myapp:dev -c dev
Enter fullscreen mode Exit fullscreen mode 
# Option 2 (recommended): a local registry created with the cluster
k3d cluster create dev --registry-create k3d-registry.localhost:5000
Enter fullscreen mode Exit fullscreen mode 
# then the FULL name matters: address + port + tag
image: k3d-registry.localhost:5000/myapp:dev
Enter fullscreen mode Exit fullscreen mode

The classic follow-up mistake is an incomplete image name — myapp:dev instead of k3d-registry.localhost:5000/myapp:dev. The cluster looks in the wrong place and you're back in backoff. If you use Tilt, docker_build handles delivery, but the Tiltfile image name must match the manifest exactly or you get the same error.

CrashLoopBackOff — the status that hides its own cause

Symptom

Container starts, crashes, kubelet restarts it, it crashes again — with exponential backoff, roughly 10s → 20s → 40s, capped at 5 minutes. The status itself tells you nothing.

The fix

The command I wish I'd known on day one:

kubectl logs <pod> -n myapp --previous
Enter fullscreen mode Exit fullscreen mode

Without --previous you get the freshly-started container, which usually hasn't written anything. --previous pulls the stack trace from the instance that actually died. Also read Last State: Terminated and its Exit Code in describe.

Causes I check in order:

  • Startup bug — exception, non-zero exit. For a FastAPI service, missing DB_HOST / DB_PASSWORD crashes it on connect.
  • Missing env/config — a ConfigMap/Secret isn't mounted.
  • Dependency not up — Postgres hasn't started and the app doesn't wait.
  • OOM — out of memory (below).
  • Liveness too strict — kills the app before it warms up. Fix with initialDelaySeconds or a startup probe.
  • Exit code 0 — for a long-running service, "exited successfully" is still a crash loop. Usually a wrong entrypoint that runs and exits instead of starting uvicorn.

When the container dies instantly and logs are empty, keep it alive and climb inside:

# temporary, in the Deployment
command: ["sleep", "infinity"]
Enter fullscreen mode Exit fullscreen mode 
kubectl exec -it <pod> -n myapp -- sh
env | grep DB_
uvicorn app.main:app --host 0.0.0.0 --port 8080   # run by hand, read the real error
Enter fullscreen mode Exit fullscreen mode

Pending — the scheduler couldn't place it

Symptom

Pod never leaves Pending. describe → Events → FailedScheduling, usually naming the cause: Insufficient cpu / Insufficient memory.

Root cause

On a small local cluster this is the most common one. I once set requests: 4 CPUs on a 2-CPU k3d VM and the pod simply never scheduled. Other causes: nodeSelector/affinity mismatch, taints without tolerations, an unbound PVC, a taken hostPort.

The fix

Lower requests, fix the selector/PVC, or grow the cluster:

k3d node create extra --cluster dev --role agent
Enter fullscreen mode Exit fullscreen mode

For a local FastAPI service, 100m CPU / 128Mi memory is fine. Don't paste production numbers into a laptop cluster.

OOMKilled — the exit-137 signature

Symptom

Process blew past its memory limit and the kernel SIGKILL'd it. Recognize it by exit code 137 (128 + 9, SIGKILL) and Reason: OOMKilled in describe. That Reason field distinguishes a memory kill from other SIGKILLs, and OOM is often the hidden cause behind a CrashLoopBackOff.

Root cause

Two flavors:

  • Container-level — exceeded resources.limits.memory. Raise the limit or fix a leak.
  • Node-level (sneaky on k3d) — the whole Docker VM ran out of memory, so pods get OOM-killed even while each app is within its own limit. The fix isn't Kubernetes limits; it's giving the Docker VM more memory, or not overcommitting — the sum of all pods' limits shouldn't exceed the VM's memory.

The Service that's green but silent

Symptom

Pods Running, everything looks fine, requests never reach the app. Almost always one of three breaks in Service → endpoints → Pod.

The fix

Check endpoints first:

kubectl get endpointslices -n myapp -l kubernetes.io/service-name=myapp
Enter fullscreen mode Exit fullscreen mode

Empty means nothing is behind the Service. Three causes:

  1. Selector mismatch — labels are case-sensitive, app: Web ≠ app: web.
  2. Port mismatch — targetPort must equal the real containerPort.
  3. Readiness not passing — pod shows 0/1 Ready, Kubernetes pulls it from endpoints so no traffic flows, but the container doesn't restart and still reads Running.

Localize by hitting the pod directly, bypassing the Service:

kubectl port-forward <pod> -n myapp 8080:8080
curl http://localhost:8080/healthz   # works? then it's selector/endpoints/readiness, not the app
Enter fullscreen mode Exit fullscreen mode

Before / after

Status Old reflex The move
ImagePullBackOff re-docker build, panic check k3d containerd/registry gotcha; full image name
CrashLoopBackOff stare at empty logs logs --previous + exit code
Pending assume cluster broken read FailedScheduling; shrink requests
OOMKilled confused restarts spot exit 137 + Reason: OOMKilled
Service silent blame the app check endpoints/selector/readiness

Guardrail

  • The whole thing is one algorithm: get pods shows what → describe explains why in Reason/Events → logs --previous gives the crash detail. Ninety percent of cases are on the list above.
  • Watch disk creep. Kubelet image GC is lazy — it won't clean until the node's disk crosses its high-threshold, 85% by default — so images pile up on purpose. docker system df to see it, prune carefully; docker system prune -a can delete images the cluster still needs.
  • Keep the golden set in muscle memory. It's faster than any search bar.

What I'd do differently

Stop treating a red status as a mystery. Every one of them resolves to a Reason field and a short cause list. The dread was never warranted — the statuses just have unfriendly names.

Bottom line: memorize describe + logs --previous + get events, learn the five common statuses, and remember that in k3d "the image is built" and "the cluster can see it" are two different facts.

Sources

Top comments (0)