How to Fix Docker Container CrashLoopBackOff – Step‑by‑Step Guide for DevOps
Docker containers entering a CrashLoopBackOff state can stall your CI/CD pipelines and cause production outages. This article walks you through the most common reasons and provides concrete commands to diagnose and resolve the issue.
Table of Contents
- Understanding CrashLoopBackOff
- Quick Health‑Check Commands
- Common Root Causes & Fixes
- Automating the Fix with a Script
- When to Open a Support Ticket
Understanding CrashLoopBackOff
CrashLoopBackOff is a Kubernetes status that indicates a container repeatedly starts, crashes, and is then restarted by the kubelet. The platform backs off exponentially to avoid a tight restart loop.
Typical triggers include:
- Mis‑configured entrypoint or command.
- Missing environment variables or secrets.
- Application runtime errors (e.g., uncaught exceptions).
- Resource limits that cause OOM kills.
Quick Health‑Check Commands
Open a terminal and run the following commands against the problematic pod (replace my‑pod and my‑ns with your values):
# Get pod status and recent events
kubectl describe pod my-pod -n my-ns | grep -A5 "State:"
# Show the last 20 lines of the container log
kubectl logs my-pod -n my-ns --tail=20
# If the container has multiple containers, specify the name
kubectl logs my-pod -c my-container -n my-ns --tail=20
If the logs show a permission denied or missing file error, you are likely dealing with a bad image or an incorrect command/args definition.
Common Root Causes & Fixes
1. Incorrect command or args
Kubernetes overrides the Docker image's ENTRYPOINT when you specify command or args in the pod spec. A typo can crash the process instantly.
Fix:
# Before – problematic spec
spec:
containers:
- name: app
image: myrepo/app:latest
command: ["/usr/loca/bin/app"] # typo: missing "l"
# After – corrected spec
spec:
containers:
- name: app
image: myrepo/app:latest
command: ["/usr/local/bin/app"]
2. Missing Environment Variables or Secrets
If your code expects DB_PASSWORD but the variable is not defined, the process may exit with status 1.
Fix:
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
3. Resource Limits Causing OOM Kill
When a container exceeds its memory limit, the kernel kills it, and Kubernetes reports CrashLoopBackOff.
Fix: Increase the limit or optimise memory usage.
resources:
limits:
memory: "512Mi"
requests:
memory: "256Mi"
4. Application‑Level Exceptions
Sometimes the binary starts but throws an exception after a few seconds. The log will usually contain a stack trace.
Fix: Rebuild the image with proper error handling or add a health‑check script that exits gracefully.
Automating the Fix with a Script
Below is a Bash helper that checks the most common failure points and patches the deployment on the fly. Save it as fix-crashloop.sh and run it against the affected namespace.
#!/usr/bin/env bash
set -euo pipefail
NAMESPACE=${1:-default}
DEPLOYMENT=${2:-my-app}
# 1. Pull current pod spec
kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o yaml > /tmp/${DEPLOYMENT}.yaml
# 2. Detect missing command entry
if ! grep -q "command:" /tmp/${DEPLOYMENT}.yaml; then
echo "Adding default command..."
yq eval '.spec.template.spec.containers[0].command = ["/usr/local/bin/start.sh"]' -i /tmp/${DEPLOYMENT}.yaml
fi
# 3. Ensure memory limits are present
if ! grep -q "limits:" /tmp/${DEPLOYMENT}.yaml; then
echo "Setting memory limits..."
yq eval '.spec.template.spec.containers[0].resources.limits.memory = "512Mi"' -i /tmp/${DEPLOYMENT}.yaml
fi
# 4. Apply the patched manifest
kubectl apply -f /tmp/${DEPLOYMENT}.yaml
echo "Patch applied. Monitor the pod with: kubectl get pods -n $NAMESPACE -w"
You can Download the pre‑configured script here, or Get the complete patch tool if you prefer a ready‑made binary. For a full repository of fixes, Access the full repository fix.
When to Open a Support Ticket
If after applying the above steps the pod still cycles, collect the following artefacts before contacting support:
- Full
kubectl describe podoutput. - Container logs (
kubectl logs --previous). - The current Deployment YAML (
kubectl get deployment -o yaml). - Node resource usage (
kubectl top node).
Providing these logs speeds up triage and helps the engineering team pinpoint obscure kernel‑level issues.
Conclusion
CrashLoopBackOff is rarely a mystical Kubernetes bug; it almost always stems from a mis‑configuration or a runtime exception. By systematically checking the command, environment, resources, and logs, you can resolve the majority of cases in under ten minutes. Automate repetitive patches with the script above, and keep the outbound resources handy for rapid onboarding.
Top comments (0)