CrashLoopBackOff is the one everyone watches. Here are 20 Kubernetes failures — in logs and events — that actually deserve an alert.
It's 3am and the dashboard is green. Checkout is timing out anyway, because CoreDNS quietly fell over twenty minutes ago and nothing you alert on noticed. Most teams running Kubernetes alert on CrashLoopBackOff and OOMKilled and call it covered — that's maybe a third of the failure modes that take down real clusters. The other two-thirds live in pod events and kubelet logs, and they stay invisible until the first incident teaches you they exist.
Here are 20 of them. Each has paged someone. Each shows up in your cluster's logs or events before it shows up on a status page — if you're looking.
Pod lifecycle failures
1. CrashLoopBackOff
The container starts, dies, restarts, dies again, and Kubernetes backs off exponentially between attempts so the restarts get further apart over time. The pod is "running" the whole time and never serves a request. Everyone monitors this one. The trap is treating it as a single failure mode — half the time the real cause is item #4 or #2 wearing a CrashLoop costume.
2. OOMKilled
The container blew past its memory limit and the kernel killed -9 it mid-execution. There's often no application error to find — the process just stops, no stack trace, no goodbye. Don't go looking in app logs. Watch the pod's last-state reason in events, and watch memory working set climb toward the limit before it happens. This is the cleanest case for correlating a metric (memory) with an event (the kill).
3. ImagePullBackOff
The node can't pull the image. A rotated registry credential, a tag someone garbage-collected, a typo in image:. It almost always rides in on a deploy, and if your rollout strategy tolerates partial failure it can sit there half-applied while you assume the deploy "went out."
4.CreateContainerConfigError
A ConfigMap or Secret the pod references doesn't exist. Classic ordering bug: the Deployment shipped before the config did, or the Secret lives in a different namespace than the one you kubectl apply'd into. The pod can't even start, so there are no app logs to read — only the event.
5.RunContainerError
The runtime accepted the container and then failed to actually run it. Bad entrypoint, a command that isn't executable, a volume mounted where the binary expects to write. The pod parks in a not-ready state and stays there. Rare enough that nobody has an alert for it, annoying enough to eat an afternoon.
Resource and scheduling failures
6.Insufficient CPU / Insufficient Memory
The scheduler can't find a node big enough, so pods sit Pending. This one creeps. Requests accumulate as services get added, headroom shrinks, and one Tuesday a perfectly normal deploy has nowhere to land. The fix is usually obvious; noticing in time is the hard part.
7.FailedScheduling
Broader than running out of resources — node affinity that matches nothing, a taint with no matching toleration, a topology spread constraint that can't be satisfied. The event message tells you exactly what failed, in plain words. Nobody reads it, because nobody is watching scheduling events at 6pm on a Friday.
8.Evicted pods
The kubelet evicts pods when a node runs low on disk, memory, or PIDs. A Deployment will reschedule the replacement; a bare pod won't, and the timing isn't always instant. One eviction is noise. A burst of them is a node telling you it's about to become item #9.
9.Node NotReady
A node stops answering the control plane. Every pod on it is now suspect, but they aren't rescheduled until the node has been NotReady past the eviction timeout — and during that grace window, traffic is still being sent to workloads that may already be dead. The dashboard looks fine. The node-condition event and the gap in that node's metrics are where the truth is.
Storage failures
10.FailedMount / FailedAttachVolume
A persistent volume won't attach. The usual cause with cloud disks (EBS, GCE PD) is that the volume is still bound to a node that just got terminated, and the detach hasn't completed. The pod hangs in ContainerCreating, looking patient, going nowhere.
11.ProvisioningFailed
Dynamic provisioning couldn't create the volume at all — out of quota, missing IAM permission, a StorageClass pointed at the wrong provisioner. The PVC stays Pending and anything that depends on it is stuck behind it. Stateful workloads feel this first.
Networking failures
12.DNS resolution failures
CoreDNS gets overloaded, misconfigured, or knocked over, and suddenly nothing can resolve a service name. This is the worst kind of failure because it isn't one failure — it's every service-to-service call failing at once, so your alerts (if they fire) point everywhere and nowhere. Grep for NXDOMAIN and connection-refused to port 53. When DNS is the culprit, it's usually the quiet thing underneath a hundred loud things — exactly the silent failure pattern.
13.Connection refused to a ClusterIP service
The Service object exists and looks healthy. It has zero ready endpoints. Every pod behind it failed readiness, so the Service is a front door to an empty room — connections get refused, and the Service's own status gives you no hint. Check endpoints, not the Service.
14.NetworkPolicy blocking legitimate traffic
Someone tightened a NetworkPolicy and accidentally fenced off traffic that's supposed to flow. There's no error — the connection just hangs and times out, which sends everyone debugging the wrong service. The tell is timing: sudden timeouts that start right after a policy changed. Correlate the symptom with the change and this goes from a two-hour hunt to a one-line answer.
Deployment and rollout failures
15.ProgressDeadlineExceeded
A rollout has been grinding longer than its progress deadline because the new pods won't go ready. Your deploy is stuck, and depending on maxUnavailable you may be limping along on reduced capacity while you assume it shipped. This one should page the person who hit "deploy," not the on-call.
16.FailedRollingUpdate
The rolling update can't advance — new pods keep failing readiness. With maxUnavailable: 0 you're safe but frozen on the old version. Set it higher and you're shedding capacity to make room for pods that aren't coming up. Either way you're now running a version you didn't intend to be running.
17.Readiness probe failures
Pods are up but failing readiness, so Kubernetes pulls them from the Service endpoints and they take no traffic. A few failures during startup is normal and you should not alert on it. Sustained failures mean the pod is quietly out of rotation while looking, in kubectl get pods, almost fine.
18.Liveness probe failures
The container is running but not answering liveness checks, so Kubernetes restarts it — over and over. Frequent liveness restarts are a symptom, not a cause: a deadlock, a slow memory leak, a thread pool that's wedged. If you only watch the restart count you'll keep restarting your way around a bug that needs a real fix.
Other critical patterns
19.RBAC permission denied
A ServiceAccount tries to talk to the Kubernetes API and gets forbidden. Almost always a new namespace or a new workload that shipped without the Role/RoleBinding it needs. It surfaces in pod logs, not events, so it hides from anyone watching only the cluster's event stream.
20.HPA can't scale
The Horizontal Pod Autoscaler wants more pods and can't get them — the metrics server is down so it's flying blind, or there's no room on any node, or it hit maxReplicas right as traffic spiked. The cruel part is the timing: the autoscaler fails exactly when you need it, during the surge, and a stale-metrics HPA fails silently. Watch the HPA's own conditions, not just the replica count.
How to monitor them
You can write 20 alert rules. Each needs a query, a threshold, a notification target, and an owner who keeps it honest as the cluster changes shape — and a few of these (DNS, NetworkPolicy, HPA-on-stale-metrics) are genuinely hard to express as a single threshold at all. Plenty of teams do exactly this in Prometheus, and it works until it's twenty rules nobody remembers writing.
The other option is to point your log and event shipper — FluentBit, Vector, whatever you run — at Epok and ship kubelet logs plus Kubernetes events. Epok's Kubernetes rule pack covers these failure modes from the first log line or event: no queries to write, no thresholds to tune. When a pod starts CrashLooping or a node goes NotReady, you get the pod, the node, the error string, and how long it's been going — and the alert cites the exact log line or event it fired on, so you're not reverse-engineering why it paged.
Kubernetes is one corner of what Epok watches. It's detection-first observability across logs, metrics, traces, infrastructure, and RUM — it finds what broke and drafts the likely root cause, with every claim cited to the exact line, span, or metric behind it. That matters more than it sounds: half the list above (OOMKilled, Node NotReady, HPA) only makes sense when you read the event and the metric together, which is the whole point of watching more than one signal.
FAQ
What are the most common Kubernetes failures to alert on? CrashLoopBackOff, OOMKilled, and ImagePullBackOff are the three nearly every team watches. The ones that cause the longest incidents are usually the quieter ones: DNS resolution failures, a Service with no ready endpoints, NetworkPolicy blocking traffic, and nodes going NotReady. They don't trip a simple threshold, so they tend to go unmonitored until the first outage.
Why doesn't OOMKilled show up in my application logs? Because the kernel kills the process with SIGKILL when it exceeds its memory limit, the application never gets a chance to log anything. The signal lives in the pod's last-state reason in Kubernetes events, alongside the memory metric climbing toward the limit just before the kill. Alert on the event and the metric together, not on app logs.
Can I alert on Kubernetes events without writing custom rules? Yes. Tools that ingest kubelet logs and Kubernetes events can pattern-match the well-known failure reasons (FailedScheduling, ProvisioningFailed, Evicted, and so on) out of the box. Epok's Kubernetes rule pack covers the failure modes in this list from the first log line or event, without per-rule queries or thresholds to maintain.
How is detection-first monitoring different from writing Prometheus alert rules? With rule-based monitoring you predefine every condition — a query, a threshold, a target — and maintain them as the cluster changes. Detection-first monitoring watches the signals directly and tells you when something breaks, including failures that are hard to express as a single threshold (DNS storms, cascading timeouts, an autoscaler flying blind on stale metrics). Both approaches can coexist; detection-first reduces the count of rules you have to author and babysit.
Try Epok free. First alerts in minutes.
No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.
Start 14-day trial
https://getepok.dev/blog/kubernetes-failures-you-should-alert-on
Top comments (0)