Debug Kubernetes CrashLoopBackOff: The events -> describe -> logs Chain That Ends the Flailing

To debug Kubernetes CrashLoopBackOff you do not need luck, you need a route. For too long my strategy was "run random kubectl commands until something makes sense." Then I learned there's exactly one path — the same three steps, every incident. Here's the workflow that turned on-call from panic into procedure.

Symptom

Matchmaking service myapp — FastAPI on 8080, PostgreSQL behind it — went CrashLoopBackOff in staging.

$ kubectl get pods -n myapp
NAME                    READY   STATUS             RESTARTS   AGE
myapp-7d9c4b6f8-jq2xp   0/1     CrashLoopBackOff   5          4m

I ran kubectl logs <pod> and got almost nothing — the start of a fresh boot. Assumed the logs were broken and spiraled for an hour. They weren't broken. I was reading the wrong container instance: the kubelet had already restarted the pod, and plain logs only shows the current, still-empty container. There's a solid write-up of a repeatable method at this debugging and observability breakdown.

The chain: events -> describe -> logs

Memorize the order.

1. events — what happened at the cluster level.
2. describe — the details of a specific Pod.
3. logs — what the application itself said.

kubectl get events --sort-by='.lastTimestamp' -n myapp
kubectl describe pod <pod> -n myapp
kubectl logs <pod> -n myapp

Detail that silently misled me: kubectl get events is not sorted chronologically by default — you almost always need --sort-by='.lastTimestamp'. Events also have a TTL of about an hour, so yesterday's incident is gone and you fall back to logs and resource state.

The fix: --previous cracks the crash loop

The flag that would've saved my night. In a crash loop the current container hasn't started (or just died again), so plain kubectl logs shows nothing useful. The real cause lives in the previous, already-crashed instance:

kubectl logs -p <pod> -n myapp

describe tells you what happened at the Kubernetes level ("container keeps crashing, so BackOff"). Only logs --previous tells you why the app crashed. The connection is exact: see BackOff in events → go to logs -p for the real error. Mine was a bad DB connection string throwing on startup — invisible until I read the dead container's logs.

Root cause of my confusion: phases vs status

Something that quietly confused me for a year: CrashLoopBackOff and ImagePullBackOff, the things in the STATUS column, are not Pod phases. The five real phases are Pending, Running, Succeeded, Failed, Unknown. CrashLoopBackOff is a display field kubectl assembles from container states — the underlying phase might still be Pending. Once I stopped conflating the two, the docs read cleanly. The decoder:

• CrashLoopBackOff — starts, crashes, backs off, repeats (exponential, min 100ms, capped at 5 minutes). Diagnose with logs -p + describe.
• ImagePullBackOff — image can't be pulled. On local k3d this is almost always a typo in the tag or an image never pushed to k3d-registry.localhost:5000. Cause is in the Events from describe.
• Pending + FailedScheduling — scheduler found no suitable node; locally, usually a resource shortage.

Checking from the inside

Pod is Running but you don't know if it responds or can reach the DB. Two commands:

kubectl port-forward svc/myapp 8080:80 -n myapp   # tunnel, then curl it
kubectl exec -it <pod> -n myapp -- sh             # shell in, check env / connectivity

From inside I check whether DB_HOST / DB_PASSWORD are actually what I think, and whether I can reach PostgreSQL. Quirk: the port-forward tunnel breaks when the Pod restarts — normal, just rerun it.

When the container has no shell

Production images build on distroless — minimal, no shell, far fewer CVEs — so kubectl exec ... -- sh simply fails. The fix I didn't know existed: kubectl debug attaches a temporary ephemeral container with real tools to the Pod, without restarting it:

kubectl debug <pod> -it --image=busybox --target=<container> -n myapp

--target shares the target container's process namespace so you see its processes and network. This one command retired an entire category of "I can't get into the container" tickets.

Reproducing the crash loop locally

I stopped fearing the pager by manufacturing the failure on k3d and running the chain until it was reflex. Break the DB env on purpose, then walk the route:

k3d cluster create dbg
kubectl apply -k k8s/overlays/local
kubectl set env deploy/myapp DB_PASSWORD-       # remove it, force the startup crash
kubectl get pods -w                             # watch it enter CrashLoopBackOff

# now the route, top to bottom
kubectl get events --sort-by='.lastTimestamp' -n myapp | grep -i backoff
kubectl logs -p deploy/myapp -n myapp           # the real OperationalError

Restore the var, watch it recover. Ten minutes of self-inflicted failure taught the route better than any doc — when it's muscle memory on a laptop, it holds at 2 AM.

Log flags that stopped me re-reading walls of output

Most early flailing was a logs problem — I'd dump a whole container's history and scroll. A handful of flags fixed it:

kubectl logs <pod> -c <container> -n myapp          # specific container in a multi-container Pod
kubectl logs <pod> --tail=50 --timestamps -n myapp  # last 50 lines, timestamped
kubectl logs <pod> --since=10m -n myapp             # last 10 minutes only
kubectl logs -l app=myapp --all-containers -n myapp # every matching pod at once

The -c one matters more than it looks: in a multi-container Pod, without a named container kubectl picks a default — easily not the one you want. My "empty logs" was often the wrong container entirely. Now, whenever logs look wrong, the first thing I check is whether I'm reading the right container.

k9s: the cluster in one window

Typing get pods, logs, describe, port-forward dozens of times an hour is exhausting. k9s is a terminal UI that watches the cluster and puts logs, describe, exec, and port-forward behind hotkeys:

k9s -n myapp

Framing that keeps me honest: k9s does exactly what I do by hand through kubectl, just faster. It speeds up the work; it doesn't replace understanding the chain. Juniors learn the raw commands first.

Before / after

	Before	After
CrashLoopBackOff	random guessing	logs -p finds the real error
Debug approach	ad hoc	events -> describe -> logs
Empty get events	confusing	add --sort-by
Distroless container	"can't get in"	kubectl debug
Speed	one command at a time	k9s hotkeys

Guardrail

• Everything above is debugging — reacting after the fact. Observability is watching state continuously, built on three signals — metrics, logs, traces: metrics say what, logs give details, traces say why.
• For local dev, logs + describe + events + k9s are enough. A full Prometheus/Grafana/OpenTelemetry stack is overkill until you have several interacting services.
• If the team runs Tilt, most of this collects into one web UI — logs of every resource, Docker build errors, Kubernetes events, with search. Doesn't replace the raw commands; it just tightens the inner loop.

What I'd do differently

Learn the chain before the first 2 AM page, not during it. The route is trivial to memorize and it's the difference between a two-minute check and a ruined night.

Bottom line: debug CrashLoopBackOff with one route — events, then describe, then logs --previous — and remember that kubectl logs with no -c and no -p is why the logs looked empty.

Sources

• Kubernetes docs — Debug Running Pods: kubectl debug & ephemeral containers
• Kubernetes docs — Pod Lifecycle: phases vs status (CrashLoopBackOff)
• k9s — Terminal UI for Kubernetes clusters
• OpenTelemetry — Observability primer: logs, metrics, traces
• Prometheus — Overview: metrics-based monitoring & alerting
• Local-Kubernetes debugging & observability writeup