DEV Community

Cover image for Kubernetes Troubleshooting 101 (Part 1): Pod Errors Explained
TheYAMLGuy
TheYAMLGuy

Posted on

Kubernetes Troubleshooting 101 (Part 1): Pod Errors Explained

#kubernetes #devops #softwareengineering #troubleshooting

If you’ve worked with Kubernetes for more than a day, you’ve seen a Pod get stuck in a state that isn't Running.When a production system throws an error at 2 AM, your goal isn't just to restart the Pod, it's to understand why it failed so it doesn't happen again.

Welcome to Part 1 of Kubernetes Troubleshooting 101.
In this series, we unpack common Kubernetes failure modes, translate cryptic status codes into plain English, and provide practical YAML/kubectl fixes.

Today, we cover the 4 most common Pod-level errors:

  • CrashLoopBackOff
  • ImagePullBackOff / ErrImagePull
  • OOMKilled (Exit Code 137)
  • Pending

1. CrashLoopBackOff

What it means

Kubernetes started your container, but the application inside exited or crashed immediately. Kubernetes then attempts to restart it, fails again, and enters an exponential delay loop (backoff) to prevent overloading the node.

Common Root Causes

  • Missing or misconfigured environment variables.
  • The main process exited (e.g., a script finished executing instead of running an ongoing service).
  • Misconfigured application code or broken dependencies.
  • Incorrect command or args in the Pod manifest.

The Fix Workflow

Don't just delete the Pod! First, check the logs of the failed container instance:

kubectl logs <pod-name> --previous
Enter fullscreen mode Exit fullscreen mode

Pro Tip: The --previous flag fetches logs from the container instance that just crashed, which is essential if the new instance hasn't logged anything yet.

Next, inspect the exact termination state:

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

Look under Last State:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
Enter fullscreen mode Exit fullscreen mode

If the exit code is 1, your application code threw an unhandled exception. Fix the app configuration, update your ConfigMap or Secret and redeploy.

2. ImagePullBackOff / ErrImagePull

What it means

The Kubelet cannot pull the container image specified in your Deployment manifest.

Common Root Causes

  • Typos in the image name or tag (e.g., nginx:latst).
  • The container image does not exist or was deleted from the registry.
  • Private registry authentication failure (missing imagePullSecrets).
  • Network issues or container registry rate limits (e.g., Docker Hub limits).

The Fix Workflow

Run

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

and scroll down to the Events section at the bottom. The event log tells you the exact reason:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Warning  Failed     12s (x2 over 34s)     kubelet            Failed to pull image "my-app:v1.0.4": rpc error: code = NotFound
Enter fullscreen mode Exit fullscreen mode

Fixes:

  • Fix typos: Double-check the image name and tag in your YAML.
  • Private Registries: Ensure your Pod manifest references a valid imagePullSecret: 
spec:
  imagePullSecrets:
  - name: my-registry-key
  containers:
  - name: my-app
    image: private-registry.io/my-app:v1.0.0
Enter fullscreen mode Exit fullscreen mode

3. OOMKilled (Exit Code 137)

What it means

Your container was terminated by the Linux Kernel Out-Of-Memory (OOM) killer because it exceeded its allocated memory limit defined in the Pod manifest.

Common Root Causes

  • Memory leaks in the application.
  • Memory limits set too aggressively low in the manifest.
  • Sudden traffic spikes causing high memory usage.

The Fix Workflow

Checking

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

will reveal Exit Code: 137:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
Enter fullscreen mode Exit fullscreen mode

Fixes:

Short term: Increase the limits.memory in your container spec:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi" # Increase this value
Enter fullscreen mode Exit fullscreen mode

Long term: Profile your application to identify memory leaks before simply giving it more resources.

4. Pending

What it means

The Pod has been accepted by the Kubernetes cluster, but the Scheduler cannot assign it to any node.

Common Root Causes

  • Insufficient Resources: No node in the cluster has enough CPU/Memory to satisfy the Pod's requests.
  • Unsatisfiable Constraints: Rigid nodeSelector, affinity, or tolerations rules that no active node matches.
  • Unbound PVCs: The Pod uses a PersistentVolumeClaim that isn't bound to a volume yet.

The Fix Workflow

Run

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

and check the Events section:

Events:
  Type     Reason            Age   From                Message
  ----     ------            ----  ----                -------
  Warning  FailedScheduling  45s   default-scheduler   0/3 nodes are available: 3 Insufficient memory.
Enter fullscreen mode Exit fullscreen mode

Fixes:

  • Lower your resources.requests if they are unrealistically high.
  • Scale up your cluster nodes (or allow the Cluster Autoscaler to add nodes).
  • Verify that required PersistentVolumeClaims (PVCs) exist and are bound.

Summary Cheat Sheet

Status Primary Cause First Command to Run
CrashLoopBackOff App crashed / Bad env vars kubectl logs --previous
ImagePullBackOff Wrong image name / Registry auth kubectl describe pod
OOMKilled Exceeded memory limits kubectl describe pod (Check for Code 137)
Pending No available node resources kubectl describe pod (Check Events)

Top comments (0)