Over the last while I've been cataloguing production DevOps errors — the exact strings that show up in logs at 2 a.m. — and writing a fix for each one. A pattern jumps out fast: a small number of errors account for a huge share of the pages. Here are the twelve that come up most, with the one-thing-to-check-first for each.
None of these are exotic. That's the point. The stuff that actually pages you is rarely exotic — it's the same dozen failure modes wearing different hats.
1. CrashLoopBackOff
The pod started, died, and Kubernetes is now backing off between restarts. CrashLoopBackOff is a symptom, never a cause. Go straight to kubectl logs <pod> --previous — the logs from the crashed container are where the real error lives. Nine times out of ten it's a bad config value, a missing env var, or a failed migration on startup.
2. ImagePullBackOff / ErrImagePull
Kubernetes can't pull the image. Don't guess — kubectl describe pod spells it out in Events. It's almost always a typo in the tag, a missing imagePullSecret, or a registry rate limit.
3. OOMKilled (exit code 137)
This is not "the node ran out of memory." It's "this container hit its own cgroup memory limit and the kernel killed it." Different problem, different fix. Compare the pod's resources.limits.memory against what it actually uses (kubectl top pod) before you touch anything at the node level.
4. No space left on device
The classic — and the trap is when df -h shows free space anyway. Then it's one of two things: you're out of inodes (df -i), or a process is holding a deleted-but-still-open file (lsof +L1). rm won't reclaim that space until you restart the process holding the file descriptor.
5. DNS timeouts inside pods
An external lookup that works from the node but intermittently times out inside a pod is almost always the ndots:5 search-domain cascade colliding with a conntrack UDP race — you get a flat 5-second stall that blows your client timeout. Overriding ndots on the pod spec and running NodeLocal DNSCache is the fix.
6. FATAL: sorry, too many clients already (Postgres)
Bumping max_connections is the trap, not the fix — each connection costs real memory. You need a pooler (PgBouncer), not 500 backend processes.
7. Connection refused
Something reached the host and nothing was listening on that port. It's rarely DNS or the network — it's the service being down, bound to 127.0.0.1 instead of 0.0.0.0, or a firewall. ss -tlnp on the target tells you in one line.
8. TLS handshake timeout
Usually not a cert problem at all — it's a network path problem (MTU, a proxy, or a firewall silently dropping the handshake) masquerading as TLS. Test raw connectivity first with openssl s_client -connect host:443.
9. Read-only file system
A filesystem that was mounted read-write and is suddenly read-only almost always means the kernel remounted it ro after detecting I/O errors. Check dmesg — you may be looking at a failing disk, not a permissions issue.
10. Multi-Attach error for volume
A ReadWriteOnce volume can attach to exactly one node at a time — not one pod, one node. If a node goes NotReady with the volume still attached, a pod rescheduled elsewhere gets this error. Kubernetes waits ~6 minutes before force-detaching on purpose — to protect your data from being written by two hosts at once.
11. 502 Bad Gateway
502 means your proxy reached the upstream and the upstream said no (or died). It's rarely the proxy. connect() failed (111: Connection refused) in the NGINX error log → your app isn't listening where the proxy thinks it is.
12. exec format error
You built an image for one CPU architecture and ran it on another (hello, Apple Silicon → x86 clusters). Build multi-arch, or match your --platform.
The pattern
Every one of these has the same shape: the error message describes the symptom the system noticed, not the cause you need to fix. CrashLoopBackOff isn't why your pod is dying. OOMKilled isn't the node. The skill isn't memorizing fixes — it's knowing which single command turns the symptom back into a cause.
I keep a full, searchable library of these — every error above has a complete guide with the diagnostic workflow, an example root-cause analysis, and the prevention checklist. If you want the deeper version of any of them:
- The Kubernetes ones live in the Kubernetes troubleshooting toolkit
- Everything else is in the full error-guide library (Linux, Postgres, Docker, NGINX, and more)
What's the error that pages your team most? Curious whether it's on this list or something I should go write up next.
Top comments (0)