A practical Kubernetes troubleshooting checklist

A pod is not running, or is running but serving errors, and you have a terminal and a few minutes. Trying things at random wastes them. Kubernetes already records most of what you need; the skill is reading it in the right order. This checklist moves from the cheapest, most informative checks to the more expensive ones. Commands assume kubectl is pointed at the right cluster and namespace, so confirm that first with kubectl config current-context.

Step 1: look at the state

kubectl get pods -o wide
kubectl get deploy,rs,svc

The STATUS, READY and RESTARTS columns tell you which branch of this checklist to follow. The -o wide output also shows which node each pod landed on, which matters later.

Step 2: read the status like a diagnosis

Status	What it usually means	Next check
Pending	The scheduler has not placed the pod	describe pod, events
ContainerCreating	Image pull, volume mount or network setup in progress or stuck	events
ImagePullBackOff / ErrImagePull	Wrong image name or tag, missing registry credentials, registry unreachable	events, image reference, pull secret
CrashLoopBackOff	The container starts, exits, and is restarted with growing delays	previous logs, exit code
CreateContainerConfigError	A referenced ConfigMap or Secret (or key) is missing	events
Running but not Ready	Readiness probe failing	probe config, logs
Evicted	The node removed the pod, often under resource pressure	node conditions

Note that "CrashLoopBackOff" is not itself a cause. It is the kubelet backing off between restarts of a container that keeps exiting. By default the delay starts at 10 seconds, doubles each time, and is capped at 5 minutes; it resets after the container runs for 10 minutes without problems.

Step 3: read the events

kubectl describe pod NAME
kubectl get events --sort-by=.lastTimestamp

The Events section at the bottom of describe is usually where the answer is. For a Pending pod, look for scheduler messages such as insufficient CPU or memory, unmatched node selectors or affinity, untolerated taints, or unbound PersistentVolumeClaims. Events are kept for one hour by default (--event-ttl), so check them early.

Also in describe, under Containers, read State, Last State, Reason and Exit Code. OOMKilled means the container exceeded its memory limit. Exit codes above 128 are 128 plus the signal number: 137 is SIGKILL (9), 143 is SIGTERM (15).

Step 4: read the logs, including the previous container

kubectl logs POD
kubectl logs POD --previous
kubectl logs POD -c CONTAINER
kubectl logs -l app=myapp --tail=50

For a crash loop, the current container may have just started and have nothing to show, so --previous is what you want. For pods with several containers, name the container with -c, and remember init containers: a failing init container keeps the pod from starting.

Step 5: check configuration and wiring

• Environment and config: kubectl exec POD -- env and confirm values from ConfigMaps and Secrets arrived.
• Service selectors: kubectl get endpointslices -l kubernetes.io/service-name=SVC. An empty result means no ready pod matches the Service selector. Compare kubectl get pods --show-labels with the selector.
• Ports: check targetPort on the Service against the port your container listens on.
• Probes: an aggressive liveness probe restarts a healthy but slow-starting app. A startup probe is the documented fix.
• Permissions: kubectl auth can-i list pods --as=system:serviceaccount:NAMESPACE:NAME -n TARGET_NAMESPACE.

If an image has no shell, kubectl debug -it POD --image=busybox --target=CONTAINER adds an ephemeral debug container alongside the target.

Step 6: check resources and node pressure

kubectl describe node NODE
kubectl top nodes
kubectl top pods

In describe node, read Conditions: MemoryPressure, DiskPressure or PIDPressure set to True mean the kubelet may evict pods. Also read Allocated resources: requests add up even when usage is low, so a cluster can be "full" for the scheduler while the CPUs are idle. Requests drive scheduling; limits are enforced at runtime (CPU is throttled, memory can be OOM-killed).

Step 7: check the rollout

kubectl rollout status deploy/NAME
kubectl rollout history deploy/NAME
kubectl rollout undo deploy/NAME

If the problem began with a deploy, consider rolling back first, then investigating. Restoring service and diagnosing are separate jobs.

Common mistakes

• Running kubectl logs on a crash-looping pod and stopping when it is empty. Use --previous.
• Debugging in the wrong namespace or context.
• Treating CrashLoopBackOff as the cause rather than the symptom.
• Raising a memory limit without checking whether the app leaks.
• Changing several things at once.
• Reading events too late; they are gone after an hour by default.

How to tell it is fixed

• kubectl get pods shows all containers READY and a restart count that has stopped climbing.
• kubectl rollout status reports success.
• The Service's EndpointSlice lists the pod addresses, and a request from inside the cluster succeeds.
• No new warning events for several minutes.

---

Want to check how well this sticks? Free 10-question Kubernetes skills check - scenario questions with an explanation for every answer, no sign-up: braintally.com/technical-assessments/kubernetes/check

Originally published on BrainTally.