DEV Community

Deep Fix
Deep Fix

Posted on

How to Fix Docker Container CrashLoopBackOff – Step-by-Step Guide for DevOps

How to Fix Docker Container CrashLoopBackOff – Step‑by‑Step Guide for DevOps

Docker containers entering a CrashLoopBackOff state can stall your CI/CD pipelines and cause production outages. This article walks you through the most common reasons and provides concrete commands to diagnose and resolve the issue.

Table of Contents

  1. Understanding CrashLoopBackOff
  2. Quick Health‑Check Commands
  3. Common Root Causes & Fixes
  4. Automating the Fix with a Script
  5. When to Open a Support Ticket

Understanding CrashLoopBackOff

CrashLoopBackOff is a Kubernetes status that indicates a container repeatedly starts, crashes, and is then restarted by the kubelet. The platform backs off exponentially to avoid a tight restart loop.

Typical triggers include:

  • Mis‑configured entrypoint or command.
  • Missing environment variables or secrets.
  • Application runtime errors (e.g., uncaught exceptions).
  • Resource limits that cause OOM kills.

Quick Health‑Check Commands

Open a terminal and run the following commands against the problematic pod (replace my‑pod and my‑ns with your values):

# Get pod status and recent events
kubectl describe pod my-pod -n my-ns | grep -A5 "State:" 

# Show the last 20 lines of the container log
kubectl logs my-pod -n my-ns --tail=20

# If the container has multiple containers, specify the name
kubectl logs my-pod -c my-container -n my-ns --tail=20
Enter fullscreen mode Exit fullscreen mode

If the logs show a permission denied or missing file error, you are likely dealing with a bad image or an incorrect command/args definition.


Common Root Causes & Fixes

1. Incorrect command or args

Kubernetes overrides the Docker image's ENTRYPOINT when you specify command or args in the pod spec. A typo can crash the process instantly.

Fix:

# Before – problematic spec
spec:
  containers:
  - name: app
    image: myrepo/app:latest
    command: ["/usr/loca/bin/app"]   # typo: missing "l"
Enter fullscreen mode Exit fullscreen mode
# After – corrected spec
spec:
  containers:
  - name: app
    image: myrepo/app:latest
    command: ["/usr/local/bin/app"]
Enter fullscreen mode Exit fullscreen mode

2. Missing Environment Variables or Secrets

If your code expects DB_PASSWORD but the variable is not defined, the process may exit with status 1.

Fix:

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-secret
      key: password
Enter fullscreen mode Exit fullscreen mode

3. Resource Limits Causing OOM Kill

When a container exceeds its memory limit, the kernel kills it, and Kubernetes reports CrashLoopBackOff.

Fix: Increase the limit or optimise memory usage.

resources:
  limits:
    memory: "512Mi"
  requests:
    memory: "256Mi"
Enter fullscreen mode Exit fullscreen mode

4. Application‑Level Exceptions

Sometimes the binary starts but throws an exception after a few seconds. The log will usually contain a stack trace.

Fix: Rebuild the image with proper error handling or add a health‑check script that exits gracefully.


Automating the Fix with a Script

Below is a Bash helper that checks the most common failure points and patches the deployment on the fly. Save it as fix-crashloop.sh and run it against the affected namespace.

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE=${1:-default}
DEPLOYMENT=${2:-my-app}

# 1. Pull current pod spec
kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o yaml > /tmp/${DEPLOYMENT}.yaml

# 2. Detect missing command entry
if ! grep -q "command:" /tmp/${DEPLOYMENT}.yaml; then
  echo "Adding default command..."
  yq eval '.spec.template.spec.containers[0].command = ["/usr/local/bin/start.sh"]' -i /tmp/${DEPLOYMENT}.yaml
fi

# 3. Ensure memory limits are present
if ! grep -q "limits:" /tmp/${DEPLOYMENT}.yaml; then
  echo "Setting memory limits..."
  yq eval '.spec.template.spec.containers[0].resources.limits.memory = "512Mi"' -i /tmp/${DEPLOYMENT}.yaml
fi

# 4. Apply the patched manifest
kubectl apply -f /tmp/${DEPLOYMENT}.yaml

echo "Patch applied. Monitor the pod with: kubectl get pods -n $NAMESPACE -w"
Enter fullscreen mode Exit fullscreen mode

You can Download the pre‑configured script here, or Get the complete patch tool if you prefer a ready‑made binary. For a full repository of fixes, Access the full repository fix.


When to Open a Support Ticket

If after applying the above steps the pod still cycles, collect the following artefacts before contacting support:

  1. Full kubectl describe pod output.
  2. Container logs (kubectl logs --previous).
  3. The current Deployment YAML (kubectl get deployment -o yaml).
  4. Node resource usage (kubectl top node).

Providing these logs speeds up triage and helps the engineering team pinpoint obscure kernel‑level issues.


Conclusion

CrashLoopBackOff is rarely a mystical Kubernetes bug; it almost always stems from a mis‑configuration or a runtime exception. By systematically checking the command, environment, resources, and logs, you can resolve the majority of cases in under ten minutes. Automate repetitive patches with the script above, and keep the outbound resources handy for rapid onboarding.

Top comments (0)