DEV Community: Ashitha Ravindran

20 Kubernetes Failures You Should Be Alerting On

Ashitha Ravindran — Tue, 07 Jul 2026 15:24:46 +0000

CrashLoopBackOff is the one everyone watches. Here are 20 Kubernetes failures — in logs and events — that actually deserve an alert.

It's 3am and the dashboard is green. Checkout is timing out anyway, because CoreDNS quietly fell over twenty minutes ago and nothing you alert on noticed. Most teams running Kubernetes alert on CrashLoopBackOff and OOMKilled and call it covered — that's maybe a third of the failure modes that take down real clusters. The other two-thirds live in pod events and kubelet logs, and they stay invisible until the first incident teaches you they exist.

Here are 20 of them. Each has paged someone. Each shows up in your cluster's logs or events before it shows up on a status page — if you're looking.

Pod lifecycle failures

1. CrashLoopBackOff
The container starts, dies, restarts, dies again, and Kubernetes backs off exponentially between attempts so the restarts get further apart over time. The pod is "running" the whole time and never serves a request. Everyone monitors this one. The trap is treating it as a single failure mode — half the time the real cause is item #4 or #2 wearing a CrashLoop costume.

2. OOMKilled
The container blew past its memory limit and the kernel killed -9 it mid-execution. There's often no application error to find — the process just stops, no stack trace, no goodbye. Don't go looking in app logs. Watch the pod's last-state reason in events, and watch memory working set climb toward the limit before it happens. This is the cleanest case for correlating a metric (memory) with an event (the kill).

3. ImagePullBackOff
The node can't pull the image. A rotated registry credential, a tag someone garbage-collected, a typo in image:. It almost always rides in on a deploy, and if your rollout strategy tolerates partial failure it can sit there half-applied while you assume the deploy "went out."

4.CreateContainerConfigError
A ConfigMap or Secret the pod references doesn't exist. Classic ordering bug: the Deployment shipped before the config did, or the Secret lives in a different namespace than the one you kubectl apply'd into. The pod can't even start, so there are no app logs to read — only the event.

5.RunContainerError
The runtime accepted the container and then failed to actually run it. Bad entrypoint, a command that isn't executable, a volume mounted where the binary expects to write. The pod parks in a not-ready state and stays there. Rare enough that nobody has an alert for it, annoying enough to eat an afternoon.

Resource and scheduling failures

6.Insufficient CPU / Insufficient Memory
The scheduler can't find a node big enough, so pods sit Pending. This one creeps. Requests accumulate as services get added, headroom shrinks, and one Tuesday a perfectly normal deploy has nowhere to land. The fix is usually obvious; noticing in time is the hard part.

7.FailedScheduling
Broader than running out of resources — node affinity that matches nothing, a taint with no matching toleration, a topology spread constraint that can't be satisfied. The event message tells you exactly what failed, in plain words. Nobody reads it, because nobody is watching scheduling events at 6pm on a Friday.

8.Evicted pods
The kubelet evicts pods when a node runs low on disk, memory, or PIDs. A Deployment will reschedule the replacement; a bare pod won't, and the timing isn't always instant. One eviction is noise. A burst of them is a node telling you it's about to become item #9.

9.Node NotReady
A node stops answering the control plane. Every pod on it is now suspect, but they aren't rescheduled until the node has been NotReady past the eviction timeout — and during that grace window, traffic is still being sent to workloads that may already be dead. The dashboard looks fine. The node-condition event and the gap in that node's metrics are where the truth is.

Storage failures

10.FailedMount / FailedAttachVolume
A persistent volume won't attach. The usual cause with cloud disks (EBS, GCE PD) is that the volume is still bound to a node that just got terminated, and the detach hasn't completed. The pod hangs in ContainerCreating, looking patient, going nowhere.

11.ProvisioningFailed
Dynamic provisioning couldn't create the volume at all — out of quota, missing IAM permission, a StorageClass pointed at the wrong provisioner. The PVC stays Pending and anything that depends on it is stuck behind it. Stateful workloads feel this first.

Networking failures

12.DNS resolution failures
CoreDNS gets overloaded, misconfigured, or knocked over, and suddenly nothing can resolve a service name. This is the worst kind of failure because it isn't one failure — it's every service-to-service call failing at once, so your alerts (if they fire) point everywhere and nowhere. Grep for NXDOMAIN and connection-refused to port 53. When DNS is the culprit, it's usually the quiet thing underneath a hundred loud things — exactly the silent failure pattern.

13.Connection refused to a ClusterIP service
The Service object exists and looks healthy. It has zero ready endpoints. Every pod behind it failed readiness, so the Service is a front door to an empty room — connections get refused, and the Service's own status gives you no hint. Check endpoints, not the Service.

14.NetworkPolicy blocking legitimate traffic
Someone tightened a NetworkPolicy and accidentally fenced off traffic that's supposed to flow. There's no error — the connection just hangs and times out, which sends everyone debugging the wrong service. The tell is timing: sudden timeouts that start right after a policy changed. Correlate the symptom with the change and this goes from a two-hour hunt to a one-line answer.

Deployment and rollout failures
15.ProgressDeadlineExceeded
A rollout has been grinding longer than its progress deadline because the new pods won't go ready. Your deploy is stuck, and depending on maxUnavailable you may be limping along on reduced capacity while you assume it shipped. This one should page the person who hit "deploy," not the on-call.

16.FailedRollingUpdate
The rolling update can't advance — new pods keep failing readiness. With maxUnavailable: 0 you're safe but frozen on the old version. Set it higher and you're shedding capacity to make room for pods that aren't coming up. Either way you're now running a version you didn't intend to be running.

17.Readiness probe failures
Pods are up but failing readiness, so Kubernetes pulls them from the Service endpoints and they take no traffic. A few failures during startup is normal and you should not alert on it. Sustained failures mean the pod is quietly out of rotation while looking, in kubectl get pods, almost fine.

18.Liveness probe failures
The container is running but not answering liveness checks, so Kubernetes restarts it — over and over. Frequent liveness restarts are a symptom, not a cause: a deadlock, a slow memory leak, a thread pool that's wedged. If you only watch the restart count you'll keep restarting your way around a bug that needs a real fix.

Other critical patterns
19.RBAC permission denied
A ServiceAccount tries to talk to the Kubernetes API and gets forbidden. Almost always a new namespace or a new workload that shipped without the Role/RoleBinding it needs. It surfaces in pod logs, not events, so it hides from anyone watching only the cluster's event stream.

20.HPA can't scale
The Horizontal Pod Autoscaler wants more pods and can't get them — the metrics server is down so it's flying blind, or there's no room on any node, or it hit maxReplicas right as traffic spiked. The cruel part is the timing: the autoscaler fails exactly when you need it, during the surge, and a stale-metrics HPA fails silently. Watch the HPA's own conditions, not just the replica count.

How to monitor them
You can write 20 alert rules. Each needs a query, a threshold, a notification target, and an owner who keeps it honest as the cluster changes shape — and a few of these (DNS, NetworkPolicy, HPA-on-stale-metrics) are genuinely hard to express as a single threshold at all. Plenty of teams do exactly this in Prometheus, and it works until it's twenty rules nobody remembers writing.

The other option is to point your log and event shipper — FluentBit, Vector, whatever you run — at Epok and ship kubelet logs plus Kubernetes events. Epok's Kubernetes rule pack covers these failure modes from the first log line or event: no queries to write, no thresholds to tune. When a pod starts CrashLooping or a node goes NotReady, you get the pod, the node, the error string, and how long it's been going — and the alert cites the exact log line or event it fired on, so you're not reverse-engineering why it paged.

Kubernetes is one corner of what Epok watches. It's detection-first observability across logs, metrics, traces, infrastructure, and RUM — it finds what broke and drafts the likely root cause, with every claim cited to the exact line, span, or metric behind it. That matters more than it sounds: half the list above (OOMKilled, Node NotReady, HPA) only makes sense when you read the event and the metric together, which is the whole point of watching more than one signal.

FAQ
What are the most common Kubernetes failures to alert on? CrashLoopBackOff, OOMKilled, and ImagePullBackOff are the three nearly every team watches. The ones that cause the longest incidents are usually the quieter ones: DNS resolution failures, a Service with no ready endpoints, NetworkPolicy blocking traffic, and nodes going NotReady. They don't trip a simple threshold, so they tend to go unmonitored until the first outage.

Why doesn't OOMKilled show up in my application logs? Because the kernel kills the process with SIGKILL when it exceeds its memory limit, the application never gets a chance to log anything. The signal lives in the pod's last-state reason in Kubernetes events, alongside the memory metric climbing toward the limit just before the kill. Alert on the event and the metric together, not on app logs.

Can I alert on Kubernetes events without writing custom rules? Yes. Tools that ingest kubelet logs and Kubernetes events can pattern-match the well-known failure reasons (FailedScheduling, ProvisioningFailed, Evicted, and so on) out of the box. Epok's Kubernetes rule pack covers the failure modes in this list from the first log line or event, without per-rule queries or thresholds to maintain.

How is detection-first monitoring different from writing Prometheus alert rules? With rule-based monitoring you predefine every condition — a query, a threshold, a target — and maintain them as the cluster changes. Detection-first monitoring watches the signals directly and tells you when something breaks, including failures that are hard to express as a single threshold (DNS storms, cascading timeouts, an autoscaler flying blind on stale metrics). Both approaches can coexist; detection-first reduces the count of rules you have to author and babysit.

Try Epok free. First alerts in minutes.

No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.

Start 14-day trial

https://getepok.dev/blog/kubernetes-failures-you-should-alert-on

Silent Failures: The Bug That Won't Page You

Ashitha Ravindran — Tue, 07 Jul 2026 15:06:29 +0000

A silent failure is when a service stops logging and no alert fires. Here's how absence detection catches the bug that never throws an error.

The webhook consumer died at 2:41am. OOM kill — the kernel reaped it mid-batch, no exception, no stack trace, no final log line. The process was just gone. Its queue kept filling. Stripe events stopped getting processed. And every alert rule you own stayed green, because zero errors is comfortably under every threshold ever written.

That's a silent failure: a service stops doing its job and the only evidence is the absence of evidence. No error fires, so nothing pages. The signal flatlines — logs go quiet, traces thin out, the metric exporter freezes on its last value — and your monitoring reads that flatline as health. Somebody finds out six hours later when a customer asks why their payment never went through.

This is the deadliest class of production bug, and almost nobody monitors for it. Here's why, and what to do instead.

Why error-based alerting can't see a silent failure
Every alerting system you've used works the same way: define a condition, fire when it's true. CPU over 90%. Error rate over 5%. p99 over 500ms. Status code 500. Every one of those needs something to happen — they evaluate against data. When a service dies quietly there is no data to evaluate, no errors, no slow spans, no spike. The rule sits there content because zero is below the threshold. The thing that should fire the alert is the same thing that erased its input.

Heartbeats and synthetic checks patch part of this. Ping a health endpoint every 30 seconds, page if it stops answering. Fine — for things that have an endpoint. But the workloads that fail silently are usually the ones that don't: background workers, cron jobs, queue consumers, batch processors, the metrics sidecar nobody remembers deploying. There's no /healthz to poll on a job that wakes up every fifteen minutes.

Absence is a signal — across logs, traces, and metrics
The core idea is almost embarrassingly simple: if something that reliably produces a signal stops, that's the alert. The hard part is that "signal" isn't just logs, and a real outage usually breaks more than one at once. A silent failure shows up in three places:

—
Logs go quiet A service that writes a line every few seconds goes dark for minutes.
—
Trace throughput drops. A service that handles a steady request stream falls out of your traces, or its span volume falls off a cliff.
—
Metrics stop reporting. An exporter dies, a scrape target goes dark, a gauge stops emitting new points — and the dashboard keeps rendering the last value forever, green and lying.
That third one is the trap. A frozen gauge looks healthier than a falling one. Your API server does 200 requests a minute and logs each; when that goes to zero for three minutes, the service is down or something fundamental changed — and the same holds when its trace throughput collapses or its metrics stop arriving. You want to know in all three cases. Most tools only watch for the wrong number, never the missing one.

You can't set a static threshold for "too quiet"
Here's where naive absence checks fall apart: silence is often normal. A batch job that runs hourly is supposed to be quiet between runs. Your API server at 3am Sunday logs a fraction of what it does Monday at noon. A metric that only updates on deploy isn't dead between deploys — it's idle. Alert on raw silence and you'll page yourself constantly, which is its own way of going blind.

So absence detection has to learn what normal looks like before it can flag what isn't. It needs a per-service baseline — by hour of day, by day of week — across whichever signals that service emits. Then it watches the live stream against that baseline. Log volume, trace throughput, or metric cadence drops to zero while the baseline says "this should be busy right now"? Alert. The baseline for this slot is already near zero, like an idle batch job or a deploy-only gauge? Stay quiet. The judgment isn't "is it silent" — it's "is it silent when it shouldn't be."

This is what Epok's absence detection does. It starts protecting a new service within about an hour using short-term cadence analysis, then fills in full weekly baselines over seven days. The same logic runs across logs, trace throughput, and metric reporting — so a dead exporter or a service that vanished from traces gets caught the same way a silent log stream does. And because Epok watches every signal together, the alert points at the exact stream, span, or metric that went dark, not a vague "something's wrong."

Four silent failures, four absences
The OOM'd worker. A consumer chewing through Stripe webhook events gets OOM-killed. No error log — the kernel did the killing, not the app. Its log stream goes flat; Epok flags the gap against the worker's normal cadence instead of waiting for an error that will never come.

The deleted cron. A job that runs every fifteen minutes stops, because a deploy quietly dropped its crontab entry. Nothing errors anywhere. The next scheduled window arrives, the expected output doesn't, and the absence is the tell.

The dead exporter. A metrics sidecar dies after a node restart. The app is fine. But the gauges it published stop advancing, and the dashboard keeps showing the last good number. Epok reads the metric against its baseline cadence, sees it stopped reporting, and alerts — rather than trusting a frozen value.

The stalled replica. A read replica falls behind and stops serving queries. The app fails over to the primary and keeps working, so there are zero application errors — but the replica's logs go quiet and its query traces vanish. Absence detection catches it before the primary buckles under the redirected load.

None of these throws an exception. None trips a threshold. Every one is invisible to a tool that only knows how to watch for things getting louder.

The takeaway
Most monitoring is tuned for noise — the spike, the error storm, the red graph. The failures that cost you a Sunday are the quiet ones, and they need the opposite instinct: a baseline of what each service sounds like healthy, and an alarm for the silence that breaks it. Silent failures are a subset of the incidents that hide between alerts — the ones that never trip a rule. Their close cousin is the brand-new error a deploy introduces that no rule was written to catch yet.

Absence detection is included in Epok's 14-day trial — every feature (verify current pricing at getepok.dev/pricing). Point your logs, traces, and metrics at Epok, give it about an hour to learn your cadences, and it pages you when something goes quiet that shouldn't.

The scariest bug isn't the one that fills your logs with errors. It's the one that leaves them empty.

FAQ
What is a silent failure in production?
A silent failure is when a service stops doing its job without throwing an error — an OOM kill, a crashed worker, a deleted cron, a dead exporter. There's no exception or status code to alert on, so threshold-based monitoring stays green while the work quietly stops. The only evidence is the absence of the signal the service normally produces.

Why doesn't my alerting catch when a service stops logging?
Standard alert rules fire when a condition becomes true — high error rate, high latency, a 500. A silent failure produces none of those; it produces nothing, and zero reads as below threshold. Catching it requires watching for the absence of expected logs, traces, or metrics, which most tools don't do by default.

How do you detect a silent service without false-paging on idle ones?
Baseline each service's normal cadence — per hour of day, per day of week — instead of using a static threshold. An hourly batch job is supposed to be quiet between runs; an API server at 3am Sunday is naturally slower than Monday noon. Absence detection alerts only when a service goes quiet relative to its own learned baseline, not on raw silence.

Is silence only a logs problem?
No. The same failure shows up as logs going quiet, trace throughput collapsing, or a metric exporter that dies and freezes its last value on the dashboard. Watching all three together catches more outages and avoids trusting a frozen gauge that looks healthy because nothing is updating it.

Try Epok free. First alerts in minutes.

No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.

Start 14-day trial

https://getepok.dev/blog/silent-failures-the-bug-that-wont-page-you

Catch New Errors in Production Before Your Users Do

Ashitha Ravindran — Tue, 07 Jul 2026 14:42:56 +0000

Catch New Errors in Production Before Your Users Do
The errors that take you down are the ones you've never seen before. Here's how automatic fingerprinting catches new errors in production on the first occurrence.

A new exception in production almost never trips an alarm, because you didn't write a rule for an error you'd never seen. To catch new errors in production you have to detect the absence of history — a log line whose pattern has never appeared before — not breach a threshold. That's a different job than counting 500s, and it's the one most monitoring setups quietly skip.

Picture the Friday deploy. Tests green, staging clean, laptop shut. Forty minutes later a customer emails support: "I can't log in." Your auth service has been throwing NullPointerException at SessionResolver.resolve since 4:51pm. Nobody got paged. There was no rule for it — there couldn't have been — and the 500 rate barely moved because the bad path only hits users with a saved OAuth token. The error was real, it was new, and it sailed straight past everything you'd set up.

Why threshold monitoring misses brand-new errors
Most monitoring is built around failures you've already met. You write an alert for database connection errors after the night they took you down. You build a dashboard for HTTP 500 rate. You wire a Slack notification for p99 latency. Every one of those watches for a problem you already had the imagination to anticipate.

The expensive outages are the ones you didn't. A library upgrade surfaces a serialization bug. A feature flag flips on a code path that's been dead for six months. A dependency starts returning a shape your parser never expected. None of these match a rule, because the whole point is that they're new. They live in the gap between your alerts — the same gap where silent failures and slow cascades hide.

A new error is, by definition, the one thing your threshold monitoring can't have a threshold for.

What actually makes an error "new"
Here's the part that sounds trivial and isn't. Connection timeout after 30s on port 5432 and Connection timeout after 45s on port 5432 are not two errors. They're one error with two numbers in it. The thing you care about is the shape: Connection timeout after s on port .

So you normalize. Strip the volatile bits — numbers, IP addresses, UUIDs, hex strings, timestamps, request IDs — and what's left is the error's fingerprint. Hash that, and you've got a stable identity that survives every reboot, retry, and rescheduled pod. If a fingerprint has never appeared in your history, the error is new. If it has, it isn't, no matter how many fresh parameter values it shows up with.

The craft is all in the normalization. A naive version is worse than useless:

—
error from 10.0.1.55:8080 and error from 10.0.2.30:9090 must collapse to one fingerprint, or every host churn reads as a novel error.
—
request abc-123-def failed and request xyz-789-ghi failed are the same failure with different request IDs.
—
disk 87% full and disk 92% full are the same warning crossing different marks.
—
But connection refused and connection reset are genuinely different errors, even though they're one word apart — and a normalizer that's too aggressive will fold them together and hide a real regression.
Get the normalization wrong in either direction and the detector is dead. Too loose and you drown in false novelty. Too tight and you mask the thing you built it to find.

Why grep and a regex won't get you there
The first instinct is to grep for ERROR or FATAL and pipe it somewhere. That holds up until you're doing a few million lines an hour across twenty services. Then you've got thousands of matches that are really the same three errors wearing different parameters, and no way to tell which one started today.

Two pieces are missing, and they're the hard ones. You need fingerprinting to group the noise into distinct error identities. And you need a baseline — a memory of what's normal — to know which identities are actually new. Skip the baseline and everything looks new on Monday, because you've never seen a Monday morning before. Skip the fingerprinting and you alert on every distinct string, which is alert fatigue with extra steps.

Build it yourself and you're now maintaining a fingerprint store, a normalization ruleset with a long tail of edge cases, a rolling historical baseline per service, and an alerting pipeline on top. It's a real project. Most teams either never start it or ship a brittle version and stop trusting it.

How Epok catches new errors in production
Epok runs new-error detection automatically on every log stream — no rule, no regex, no config. It normalizes each error message so superficial differences don't read as novelty, fingerprints the underlying pattern, and checks that fingerprint against your service's history. When one shows up that the baseline has never seen, it flags it on the first occurrence.

The alert carries the things you'd otherwise go digging for: the normalized pattern, how many times it's fired and how fast, which services it touched, and when it first appeared relative to your last deploy. Severity is graded automatically — a new error firing 100+ times in five minutes is critical; ten times is a warning; a known error that vanished for a day and came back gets flagged lower, because that's a regression, not a debut.

New-error detection is log-native, but it isn't the whole product, and it isn't logs-only. The same detection-first engine watches your metrics, traces, infrastructure, and RUM for the same flavor of "we've never seen this before" — a latency distribution that just grew a second hump, an endpoint whose error rate is creeping, a service that went quiet when it should be talking. And because every verdict is cited to the exact line, span, or metric it came from, you check the evidence instead of trusting a summary.

What it looks like at 4:55pm on a Friday
You deploy. Four minutes later, Slack:

New error in api-server — FATAL: connection pool exhausted · 47 occurrences in 5 min · CRITICAL · first seen 4 min after deploy v2.4.1

You didn't write a rule for connection-pool exhaustion. You didn't know your pool sizing was a problem until the deploy made it one. There's no dashboard for it because nobody knew to build one. Epok caught it for the only reason that matters: it had never happened before, and a new error in production that's firing forty-seven times in five minutes is worth your attention before a customer makes it your problem.

That's the difference between counting errors and noticing them. Counting needs you to already know what to look for. This doesn't.

Try it
New-error detection is on every tier, including the 14-day free trial. Point your logs at Epok and it works immediately — no rules, and no warmup for this detector specifically: fingerprinting catches new errors from the very first log line. (The volume baselines that power rate-anomaly detection take about seven days to settle; new-error detection doesn't wait on them.)

Pricing is flat and legible: a 14-day free trial with every feature (~1 TB of ingest), then Team at $199/mo for 1 TB/month, Growth at $599/mo for 4 TB/month, and Custom from $5,000/mo for higher volume. Overage is $0.20/GB, and there's no free-forever tier. Verify current pricing at getepok.dev/pricing.

FAQ
How do you detect a new error you've never seen before?
By normalizing each error message into a stable fingerprint and comparing it against your service's history. Numbers, IPs, UUIDs, hex strings, and timestamps get stripped out so cosmetic differences don't count as new; the remaining pattern is hashed. If that fingerprint has never appeared in the baseline, the error is genuinely new and gets flagged on its first occurrence — no rule required.

How is new-error detection different from error rate monitoring?
Error-rate monitoring tells you how many errors you have and fires when a known type crosses a threshold. New-error detection tells you which errors you've never seen before, which is the category a threshold can't cover — there's no historical rate to breach. The two are complementary: one catches spikes in familiar failures, the other catches novel ones the moment they appear.

Why can't I just grep my logs for ERROR or FATAL?
Grep gives you matches, not identities. Across millions of lines and dozens of services you get thousands of hits that are the same handful of errors with different parameters, and no way to tell which one started today. You'd still have to build the fingerprinting that groups them and the baseline that distinguishes new from familiar — which is most of the work.

Does new-error detection need a warmup period before it works?
The new-error detector itself does not — fingerprinting catches genuinely new patterns from the first log line, because "new" means "not in the history we've recorded so far." Rate-anomaly detection, which learns what a normal volume looks like by hour and day, takes about seven days to build a useful baseline. The two are separate.

Try Epok free. First alerts in minutes.

No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.

Start 14-day trial

https://getepok.dev/blog/catch-new-errors-before-users-report-them

Datadog Alternatives for Small Teams (2026)

Ashitha Ravindran — Tue, 07 Jul 2026 10:33:33 +0000

Datadog alternatives for small teams in 2026: an honest look at Grafana, Axiom, Better Stack, and Epok — by billing model, not just per-GB rate.

The best Datadog alternative for a small team depends on one question most comparison posts skip: do you want a cheaper place to store telemetry, or something that watches it for you? Grafana Cloud, Axiom, and Better Stack are the strongest cheaper-storage options. If you want the detection and root cause already running, that's a different purchase, and where Epok fits.

Here's the trap that sends teams looking in the first place. The POC is cheap because traffic is low. You wire up Datadog, the dashboards look great, the bill is a rounding error. Then production traffic shows up. Someone logs user_id and trace_id as indexed fields, cardinality explodes, and three weeks later the observability bill has quietly passed the compute bill it was supposed to be watching. We wrote about why your AWS logging bill is out of control; the same metering trap applies here.

Datadog is a genuinely good product. The problem isn't the product. The pricing assumes a dedicated platform team and a five-figure observability budget, and most teams of 5-20 engineers have neither.

Why Datadog gets expensive for a team of five

The pricing page reads simple: a per-GB ingestion rate. The bill is not. Indexing is charged separately per million events. Retention is its own line. APM is per host. Custom metrics are metered by cardinality, so every unique tag value you emit is something you're paying to keep.

None of that is hidden, exactly. It's just spread across enough SKUs that you can't predict your bill until you're already living with it, and forecasting spend becomes its own part-time job. That's the real cost. Not the rate, the unpredictability.

What a small team actually needs

Strip it back. A team of 5-20 engineers needs to know when something breaks without writing the alert rule first. To search logs, metrics, and traces fast when debugging. New errors surfaced and grouped, not buried in a query nobody thought to run. A Slack or PagerDuty ping, not a dashboard nobody has open at 2am. And a bill they can predict.

What they don't need: 400 integrations, a dashboard marketplace, or custom metrics with fifteen tag dimensions. Most of Datadog's surface area is built for the org that has a team to run Datadog.

The honest options

Grafana Cloud + Loki

The most popular alternative, and for good reason. Loki stores logs, Grafana visualizes, the free tier is real, and paid is usage-metered per GB (competitor pricing reflects public pricing pages as of Q1 2026 — vendor pricing changes; verify current rates).

The catch is that Grafana hands you a blank canvas. You build the dashboards, the alert rules, the recording rules. LogQL has a learning curve. If you want anomaly detection you're writing PromQL and hoping it catches the right thing. With a dedicated SRE this is great: flexible and cheap. On a team where everyone wears five hats, "build your own monitoring" becomes the project that's perpetually 80% done.

Axiom

A clean, fast log tool with a generous trial (around 500 GB/month ingest, 30-day retention) (public pricing pages as of Q1 2026 — verify current rates). If you want a cheaper, Datadog-shaped log experience, Axiom is a good pick. The query speed is genuinely impressive.

Same structural limit as Grafana, though: Axiom stores and searches your logs, it doesn't watch them. You still create the monitors, write a query per failure mode, and tune the thresholds yourself. The anomaly detection is basic and needs configuration to be useful.

Better Stack (formerly Logtail)

Uptime monitoring plus log management in one tool, billed per GB (public pricing pages as of Q1 2026 — verify current rates). Polished UI, decent alerting, and the bundled uptime checks are a real convenience if that's a box you also need to tick.

The log analysis is thinner than the dedicated tools — no automatic anomaly detection, no pattern clustering, no root cause. It's a log database with a nice front end and alerting bolted on. For some teams that's exactly enough.

Epok

Epok is built on a different premise: detection-first observability. The tools above are search-first. They store everything, give you a query bar, and finding the problem is your job. Epok inverts that. Send your logs, metrics, traces, infrastructure, and RUM over OpenTelemetry or common shippers, and it watches them together: new errors, volume anomalies, silent services, latency regressions, pattern shifts. When something breaks, it drafts the root cause (which service, what changed, who's affected) with every claim cited to the exact log line, span, or metric behind it. No dashboards to build, no alert rules to write.

Pricing is one number. A 14-day free trial with every feature (~1 TB), no credit card. Then Team at $199/month (1 TB/month, 30-day retention, AI root cause, 10 users, unlimited notifications). Growth at $599/month for 4 TB/month. Custom from $5,000/month with SSO and self-hosted options. Overage is $0.20/GB. No per-query fees, no cardinality charges. Log any field you want; high-cardinality fields don't change the price. Always check current numbers at getepok.dev/pricing.

Don't compare the per-GB rate — compare the purchase

Most comparisons line up per-GB rates and pick the lowest. But storage tools and a detection engine aren't the same product, so the rate isn't a fair axis. At roughly 1 TB/month of telemetry:

—
Datadog lands in the four-figures-per-month range once ingestion, indexing, retention, and per-host fees stack up (public pricing pages as of Q1 2026 — verify current rates).
—
Grafana Cloud, Axiom, Better Stack are usage-metered per GB and typically a much smaller line item (public pricing pages as of Q1 2026 — verify current rates), with the detection, dashboards, and alerts left for you to build and maintain.
—
Epok is $199/month flat for Team (1 TB/month), with detection and root-cause analysis already running. Overage $0.20/GB.
The storage tools win on raw per-GB cost, because that's all they sell you. The watching, the alerting, and the root-cause work are your time, billed in nights and weekends instead of dollars. The honest comparison is total cost of ownership: per-GB rate plus the engineering hours to turn raw telemetry into something that pages you correctly.

So which one

If you want a full APM platform and have the budget plus a person to run it, Datadog is genuinely good. Paying for it is a reasonable call.

If you want cheap, fast storage and the time to build dashboards and alerts on top, Grafana Cloud or Axiom are the strong picks. Better Stack if bundled uptime monitoring matters to you.

If you want something that watches your telemetry and tells you when it breaks, root cause cited to the exact line, span, or metric and nothing to configure, that's the gap we built Epok to fill. The trial is full-feature, no credit card, every detector on every tier.

FAQ

What's the cheapest Datadog alternative for a small team?

By raw per-GB rate, usage-metered storage tools like Grafana Cloud, Axiom, and Better Stack are typically the cheapest line item. The lowest rate isn't the lowest total cost, though: with those tools you build and maintain the detection, dashboards, and alerts yourself, so factor in engineering time. A flat-price detection engine like Epok ($199/month for 1 TB on the Team plan) trades a higher per-GB rate for predictability and built-in analysis.

Why is Datadog so expensive?

Datadog bills across multiple metered SKUs: per-GB ingestion, separate per-million-event indexing, retention, per-host APM, and cardinality-based custom-metric charges. The rates are public, but the total is hard to forecast until production traffic and high-cardinality fields land, which is why bills often spike weeks after a cheap POC. (Competitor pricing reflects public pricing pages as of Q1 2026 — vendor pricing changes; verify current rates.)

Do cheaper observability tools do automatic anomaly detection and root cause?

Mostly no. Grafana, Axiom, and Better Stack store and search telemetry; detection and root cause are things you configure or build with queries and alert rules. Epok runs detection across logs, metrics, traces, infrastructure, and RUM automatically and drafts a cited root cause. That's the difference between buying storage and buying intelligence.

How much does Epok cost?

A 14-day free trial with every feature (~1 TB), no credit card. Team is $199/month for 1 TB/month, Growth is $599/month for 4 TB/month, and Custom starts at $5,000/month with SSO and self-hosted options. Overage is $0.20/GB, with no per-query or cardinality charges. There is no free-forever tier; verify current pricing at getepok.dev/pricing.

Try Epok free. First alerts in minutes.
No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.

Start 14-day trial

https://getepok.dev/blog/datadog-alternatives-for-small-teams

Root Cause Analysis Across Every Signal, On One Screen

Ashitha Ravindran — Fri, 26 Jun 2026 08:52:24 +0000

Automated root cause analysis that reads logs, metrics, and traces together and cites every claim to its source, so you cut MTTR instead of hopping tabs

It's 2:11am. Checkout is throwing 500s and compose-post-service is screaming, seven times its normal error volume, top of every dashboard you own. So you start there. Twenty-five minutes in, someone asks the question that ends the incident: "is the auth service even up?" It isn't. user-service let a TLS cert expire and logged almost nothing on the way down. The loudest service was the victim. The quiet one was the cause.

That gap, between the service with the most errors and the service that actually broke, is where most of your mean-time-to-resolution goes. Real root cause analysis isn't "find the noisiest service." It's reading logs, metrics, and traces together to find who everyone else is pointing at, then proving it. Epok does that automatically and cites every claim back to the exact log line, span, or metric, so you cut MTTR instead of assembling the answer across six tabs at 2am.

Why the loudest service is rarely the root cause

We ran this as a controlled experiment because the pattern is so consistent. Twenty services, steady traffic, mature baselines. Then we injected a cascade.

user-service starts failing TLS handshakes. It barely logs: a cert error is a few lines, then silence. Downstream, compose-post-service can't authenticate anyone, so it throws on every request. Thousands of errors. The loud victim. home-timeline-service falls over too, also pointing back upstream.

Sort by error count and you investigate compose-post-service first. It has 7x the logs. It's wrong. Error volume measures distance from the failure, not proximity to it: the services furthest downstream shout the most, because every request they handle now fails. Pick by loudness and the cascade hands you a decoy.

The causal information is sitting right there in the text. When service A can't reach service B, A's error says so:

compose-post-service: "User authentication failed: user-service returned 500" compose-post-service: "Cannot validate user token: connection to user-service timed out" home-timeline-service: "Timeline generation failed: user-service unavailable"

Three different services, all naming the same upstream. user-service is referenced in errors from three other services. compose-post-service is referenced by zero. The thing everyone points at is the cause. A human gets there eventually, usually after the 25-minute detour through the victim. The machine should get there in thirty seconds, because counting "who points at whom" is exactly the bookkeeping software is good at and tired engineers are not.

What automated RCA actually has to do

No single signal hands you the answer; the evidence is scattered on purpose. The error text names the culprit, but only if you read across services instead of within one. The traces, where you have them, draw the dependency edge from victim to culprit directly. The metrics catch the inflection earliest: user-service's handshake-failure rate bends before the downstream flood starts, which tells you the order of events.

Each is a partial view, and the work that eats your night is correlation: logs in one tab, APM in another, a dashboard in a third, your eyes lining up timestamps across all of them. Worse than tedious, it's biased. Whichever signal you open first anchors you. Start in the log-volume chart and you anchor on the loud victim.

Epok reads the signals together and ranks the probable cause from all of them at once. It pulls service names out of error text with word-boundary matching, counts cross-service references, checks which signal moved first, weighs the error category (auth, connection, and timeout errors point upstream), and folds in trace dependency edges and severity. In our run, user-service ranked first despite 7x fewer error logs than the victim, because the ranking weighs who-points-at-whom and what-moved-first, not raw volume. None of those steps is clever on its own. Doing all of them in the seconds after an alert, automatically, on every incident, is the product.

Cited root cause beats a confident summary

Plenty of tools now generate an incident narrative. The question that matters is whether you can check it. A summary that says "likely caused by user-service" and shows nothing is a coin flip you're asked to trust at 2am. We've been on the receiving end of enough wrong-but-confident AI summaries to refuse to ship one.

Every line of Epok's root cause draft links to its evidence: the error string and which service emitted it, the span where the latency went, the metric and timestamp where it bent. Click "user-service is referenced by three other services" and you land on those three log lines. The verdict is falsifiable in one click. It also lets Epok be honest about uncertainty: when the evidence is thin, it says so and ranks lower instead of laundering a guess into a headline. A confident wrong answer costs you more than no answer.

Where each signal earns its keep

This cascade was error-shaped, so the text carried most of the load. That won't always be true. Error text wins when the culprit is un-instrumented: databases, caches, queues, cron jobs, and network gear rarely carry a trace agent but show up by name in downstream errors, and a cert expiry or DNS change produces connection errors naming the failing endpoint. Traces win for latency degradation, where nothing errors but time is vanishing in the call chain. Metrics win for the early inflection and slow drift no single log line reveals. Same ranking engine, different evidence. The point of multi-signal RCA isn't that one signal beats another. It's that Epok uses whichever is present instead of failing because your culprit skipped the one your tool depends on. That's also why "silent" failures are so dangerous, and we go deep on those in the incidents that hide between alerts.

The honest comparison with APM

Datadog's Watchdog and New Relic's Decisions are genuinely good at correlating alerts into one incident, and when APM is instrumented across the call chain, both can surface the origin. We won't pretend otherwise. The catch is where that answer lives and how much you assemble. The "which service" verdict sits in the trace product, so you leave the alert, open APM, read the call graph, and correlate it back yourself. When the culprit is un-instrumented, the trace-based origin signal goes quiet and you're grepping logs in a separate tab.

The assembly is the real cost: not the license, the twenty-five minutes of tab-hopping per incident, multiplied across every page and a tired on-call rotation. Epok collapses that to one screen, detection, ranked cause, what changed, and blast radius, with the evidence one click deep. The tool finds it and proves it instead of handing you a search bar. We make that argument in full in detection-first observability.

What it costs

Most platforms meter the signals separately: a per-GB log rate, a per-host or per-instrumented-service APM charge, and a cardinality penalty on top. The bill grows with how many signals you turn on. For ~1 TB/month of logs across 20 services, the per-host APM line item alone typically runs into the high hundreds of dollars per month on the major platforms, on top of ingestion. (Competitor pricing reflects public pricing pages as of Q1 2026 — vendor pricing changes; verify current rates.)

Epok is one flat price with the signals included, not metered. The 14-day trial opens every feature on roughly 1 TB of volume. Team is $199/month (1 TB), Growth is $599/month (4 TB), Custom starts at $5,000/month; overage is $0.20/GB. There's no free-forever tier; verify current pricing at getepok.dev/pricing. Detection and cross-signal root cause aren't a separate SKU, they're the product. The point isn't "skip APM to save money." It's that your signals land in one tool that reads them together and shows its work, instead of three meters and three tabs you stitch by hand.

What the experiment showed, and what it didn't

One synthetic, controlled run. Not a published benchmark, not an accuracy claim about your stack. With that caveat: the cascade collapsed into a focused set of alerts instead of one page per failing service, and the ranked cause pointed at user-service, the quiet culprit, with the top "what changed" entries all referencing it. Your results will vary with your services, your instrumentation, and how your errors are worded.

We built Epok because the information needed to find root cause is almost always already in your telemetry. Collecting it was never the hard part. Reading it together, fast, at 2am, and showing the work so you can trust the answer, that's the part everyone else left to you.

Point your logs, metrics, traces, and RUM at the 14-day trial, break something on purpose, and watch it name the culprit before you've opened a second tab.

FAQ

What is automated root cause analysis?

Automated root cause analysis is when a tool, not an engineer, identifies the service or change that caused an incident and shows the evidence. Epok does it by reading logs, metrics, and traces together: extracting service names from error text, counting which services reference which, checking which signal moved first, and ranking the probable cause with every claim cited to a specific log line, span, or metric.

Why isn't the service with the most errors usually the root cause?

Because error volume measures distance from the failure, not proximity to it. In a cascade, downstream services fail on every request and produce far more errors than the upstream culprit, which may log only a handful of lines before going quiet. Ranking by error count points you at the loud victim. Ranking by who-references-whom and what-moved-first points you at the actual cause.

How does multi-signal RCA reduce MTTR?

It removes the manual correlation step. Instead of opening logs, APM, and dashboards in separate tabs and lining up timestamps by eye, you get a ranked, cited root cause on one screen seconds after the alert. The minutes normally spent assembling that answer, and the bias of anchoring on whichever tab you opened first, are where the MTTR savings come from.

Do I need traces or APM for Epok to find the root cause?

No. Epok uses whatever signal is present. When a service is un-instrumented, common for databases, caches, queues, and network infrastructure, it still appears by name in downstream error text, and Epok reads that. Where traces exist, the dependency edge corroborates the conclusion. Where metrics exist, an early inflection helps order the events.

Can I verify Epok's root cause conclusion?

Yes. Every claim in the root cause draft links to its source: the exact log line, span, or metric. A statement like "user-service is referenced in errors from three other services" links directly to those three lines, so you can confirm the verdict in one click instead of trusting a summary.

Try Epok free. First alerts in minutes.
No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.

Start 14-day trial
https://getepok.dev/blog/multi-signal-root-cause