DEV Community: TheYAMLGuy

Kubernetes Troubleshooting 101 (Part 1): Pod Errors Explained

TheYAMLGuy — Wed, 22 Jul 2026 14:25:53 +0000

If you’ve worked with Kubernetes for more than a day, you’ve seen a Pod get stuck in a state that isn't Running.When a production system throws an error at 2 AM, your goal isn't just to restart the Pod, it's to understand why it failed so it doesn't happen again.

Welcome to Part 1 of Kubernetes Troubleshooting 101.
In this series, we unpack common Kubernetes failure modes, translate cryptic status codes into plain English, and provide practical YAML/kubectl fixes.

Today, we cover the 4 most common Pod-level errors:

CrashLoopBackOff
ImagePullBackOff / ErrImagePull
OOMKilled (Exit Code 137)
Pending

1. CrashLoopBackOff

What it means

Kubernetes started your container, but the application inside exited or crashed immediately. Kubernetes then attempts to restart it, fails again, and enters an exponential delay loop (backoff) to prevent overloading the node.

Common Root Causes

Missing or misconfigured environment variables.
The main process exited (e.g., a script finished executing instead of running an ongoing service).
Misconfigured application code or broken dependencies.
Incorrect command or args in the Pod manifest.

The Fix Workflow

Don't just delete the Pod! First, check the logs of the failed container instance:

kubectl logs <pod-name> --previous

Pro Tip: The --previous flag fetches logs from the container instance that just crashed, which is essential if the new instance hasn't logged anything yet.

Next, inspect the exact termination state:

kubectl describe pod <pod-name>

Look under Last State:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1

If the exit code is 1, your application code threw an unhandled exception. Fix the app configuration, update your ConfigMap or Secret and redeploy.

2. ImagePullBackOff / ErrImagePull

What it means

The Kubelet cannot pull the container image specified in your Deployment manifest.

Common Root Causes

Typos in the image name or tag (e.g., nginx:latst).
The container image does not exist or was deleted from the registry.
Private registry authentication failure (missing imagePullSecrets).
Network issues or container registry rate limits (e.g., Docker Hub limits).

The Fix Workflow

Run

kubectl describe pod <pod-name>

and scroll down to the Events section at the bottom. The event log tells you the exact reason:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Warning  Failed     12s (x2 over 34s)     kubelet            Failed to pull image "my-app:v1.0.4": rpc error: code = NotFound

Fixes:

Fix typos: Double-check the image name and tag in your YAML.
Private Registries: Ensure your Pod manifest references a valid imagePullSecret:

spec:
  imagePullSecrets:
  - name: my-registry-key
  containers:
  - name: my-app
    image: private-registry.io/my-app:v1.0.0

3. OOMKilled (Exit Code 137)

What it means

Your container was terminated by the Linux Kernel Out-Of-Memory (OOM) killer because it exceeded its allocated memory limit defined in the Pod manifest.

Common Root Causes

Memory leaks in the application.
Memory limits set too aggressively low in the manifest.
Sudden traffic spikes causing high memory usage.

The Fix Workflow

Checking

kubectl describe pod <pod-name>

will reveal Exit Code: 137:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Fixes:

Short term: Increase the limits.memory in your container spec:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi" # Increase this value

Long term: Profile your application to identify memory leaks before simply giving it more resources.

4. Pending

What it means

The Pod has been accepted by the Kubernetes cluster, but the Scheduler cannot assign it to any node.

Common Root Causes

Insufficient Resources: No node in the cluster has enough CPU/Memory to satisfy the Pod's requests.
Unsatisfiable Constraints: Rigid nodeSelector, affinity, or tolerations rules that no active node matches.
Unbound PVCs: The Pod uses a PersistentVolumeClaim that isn't bound to a volume yet.

The Fix Workflow

Run

kubectl describe pod <pod-name>

and check the Events section:

Events:
  Type     Reason            Age   From                Message
  ----     ------            ----  ----                -------
  Warning  FailedScheduling  45s   default-scheduler   0/3 nodes are available: 3 Insufficient memory.

Fixes:

Lower your resources.requests if they are unrealistically high.
Scale up your cluster nodes (or allow the Cluster Autoscaler to add nodes).
Verify that required PersistentVolumeClaims (PVCs) exist and are bound.

Summary Cheat Sheet

Status	Primary Cause	First Command to Run
CrashLoopBackOff	App crashed / Bad env vars	kubectl logs --previous
ImagePullBackOff	Wrong image name / Registry auth	kubectl describe pod
OOMKilled	Exceeded memory limits	kubectl describe pod (Check for Code 137)
Pending	No available node resources	kubectl describe pod (Check Events)

5 Kubernetes YAML Anti-Patterns I Still See in Production (And How to Fix Them)

TheYAMLGuy — Tue, 21 Jul 2026 14:25:45 +0000

Unfortunately, I still see serious security risks in production environments. Therefore, I would like to highlight five common mistakes, explain their impact, and show how to fix them.

1. Using the `latest` Image Tag

Classic one.

# BAD: Using the latest tag
spec:
  containers:
  - name: app-container
    image: myregistry.azurecr.io/app:latest

Why it's a problem:

This flaw opens the door to supply chain attacks. An attacker with access to the image registry can publish an infected version; Kubernetes would then deploy that infected version, resulting in a security incident.

Additionally, it breaks deterministic deployments, two pods started on different days might run completely different code versions without you realizing it.
How to fix it:

Always pin your container images to explicit semantic versions or immutable SHA-256 digests.

# GOOD: Using explicit versioning
spec:
  containers:
  - name: app-container
    image: myregistry.azurecr.io/app:v1.4.2
    imagePullPolicy: IfNotPresent

Running Containers as Root

By default, Kubernetes containers run as the root user unless you explicitly tell them not to.

# BAD: No security context defined
spec:
  containers:
  - name: web-app
    image: nginx

Why it's a problem:

If an attacker exploits an application vulnerability (like a Remote Code Execution), they instantly get root privileges inside the container. This makes container escape attacks on the host node significantly easier.
How to fix it:

Enforce non-root execution via the securityContext.

# GOOD: Restricting privileges
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
  containers:
  - name: web-app
    image: nginxinc/nginx-unprivileged:alpine
    securityContext:
      allowPrivilegeEscalation: false

Omitting CPU and Memory Limits

Deploying workloads without setting boundaries for compute resources.

# BAD: Unbounded resource consumption
spec:
  containers:
  - name: backend
    image: backend-service:v1.0

Why it's a problem:

A single pod with a memory leak can consume all available memory on a worker node. This triggers the Node OOM killer, causing unexpected evictions of other critical pods running on the same node.
How to fix it:

Define clear requests for scheduling and limits to set an absolute ceiling.

# GOOD: Explicit resource boundaries
spec:
  containers:
  - name: backend
    image: backend-service:v1.0
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
      limits:
        memory: "512Mi"
        cpu: "500m"

Missing Liveness and Readiness Probes

Assuming Kubernetes knows your application state just because the process hasn't crashed.

# BAD: No health checks configured
spec:
  containers:
  - name: api
    image: api-service:v2.0

Why it's a problem:

If your app takes 30 seconds to boot or gets deadlocked in a database connection loop, Kubernetes will still route live user traffic to it, resulting in HTTP 500 errors for your users.
How to fix it:

Add explicit readiness and liveness endpoints.

# GOOD: Probes configured
spec:
  containers:
  - name: api
    image: api-service:v2.0
    readinessProbe:
      httpGet:
        path: /healthz/ready
        port: 8080
    livenessProbe:
      httpGet:
        path: /healthz/live
        port: 8080

Hardcoding Secrets in ConfigMaps

Putting raw API keys, tokens, or passwords directly into plain-text manifests.

# BAD: Plain-text credentials
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_PASSWORD: "NothingToSeeHere"

Why it's a problem:

ConfigMaps are stored unencrypted in etcd by default and can be read by anyone with basic namespace read access. They also frequently leak into system logs.
How to fix it:

Use native Kubernetes Secrets (encrypted at rest) or inject them via a Secret Manager (e.g., HashiCorp Vault, AWS Secrets Manager).

# GOOD: Referencing a secret object
spec:
  containers:
  - name: app
    image: app:v1.0
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

Fixing these five points, makes your cluster a bit safer.

Kubernetes: StatefulSet vs. Deployment

TheYAMLGuy — Mon, 20 Jul 2026 10:11:23 +0000

When I've started with Kubernetes it took me a long time to understand when to use a StatefulSet instead of a Deployment and whats the difference between them.

The truth is: Deployments are for things that don't care if they die. StatefulSets are for things that do.

Let's break down on when to use which and whats the difference.

The Core Difference (TL;DR)

If you remember only one thing from this article, make it this:

Deployments manage Stateless pods. Every pod is identical, anonymous and completely replaceable. Think of them like cattle
StatefulSets manage Stateful pods. Every pod has a unique identity, a persistent name and its own dedicated storage. Think of them like pets.

Feature	Deployment	StatefulSet
Pod Names	Random hash (web-89768dx878-asdff)	Sticky index (db-0, db-1, ...)
Scaling Order	All at once (Random)	One by one (0, then 1, then 2)
Storage (PVs)	Shared or ephemeral	Dedicated per pod via Templates
Best For	APIs, Frontends, Microservices	Databases, Queues, Cluster apps

When to use a Deployment (Stateless)

Go with a Deployment if your application doesn't save any data locally. If Pod A dies, Pod B spins up on a different node, your users shouldn't notice anything because the actual State lives somewhere else like an external DB or S3 Bucket.

How K8s handles it

When you scale a Deployment, Kubernetes creates a ReplicaSet, which spins up pods using random hashes at the end. It doesn't care about the order. If you delete pod-xyz, K8s replaces it with pod-abc.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
  labels:
    app: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: nginx:1.25
        ports:
        - containerPort: 80

When to use a StatefulSet (Stateful)

You have to use a StatefulSet the moment your pods need to remember who they are or need their own private disk space.

Think about a PostgreSQL cluster. You have a Primary node (writes) and Replicas (reads). The replicas need to know where the primary node is. You can't just use a random pod name that changes every time a node restarts.

The Magic behind StatefulSets

Predictable DNS: Your pods will be named postgres-1, postgres-2. etc. If postgres-0 crashes, it terminates and Kubernetes spins up a new pod with the exact same name.
Orderly Scaling: K8s won't start postgres-1 until postgres-0 is completely up and running.
Dedicated Storage(volumeClaimTemplates): Instead of all pods fighting over the same persistent volume, a StatefulSet automatically provisions a unique Persistent Volume (PV) for each pod. postgres-0 gets data-postgres-0, postgres-1 gets data-postgres-1. Even if a pod dies, when it comes back, K8s attaches the exact same disk to it.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-set
spec:
  serviceName: "postgres-headless" # Requires a Headless Service for network identity
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16
        ports:
        - containerPort: 5432
          name: db
        volumeMounts:
        - name: pgdata
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates: # This automatically creates a unique PVC per pod
  - metadata:
      name: pgdata
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

I hope that is helpful for somebody. If you're a beginner, what topics should I write about next?

DEV Community: TheYAMLGuy

Kubernetes Troubleshooting 101 (Part 1): Pod Errors Explained

1. CrashLoopBackOff

What it means

Common Root Causes

The Fix Workflow

2. ImagePullBackOff / ErrImagePull

What it means

Common Root Causes

The Fix Workflow

3. OOMKilled (Exit Code 137)

What it means

Common Root Causes

The Fix Workflow

Fixes:

4. Pending

What it means

Common Root Causes

The Fix Workflow

Fixes:

Summary Cheat Sheet

5 Kubernetes YAML Anti-Patterns I Still See in Production (And How to Fix Them)

1. Using the latest Image Tag

Kubernetes: StatefulSet vs. Deployment

The Core Difference (TL;DR)

When to use a Deployment (Stateless)

How K8s handles it

When to use a StatefulSet (Stateful)

The Magic behind StatefulSets

1. Using the `latest` Image Tag