A silent failure is when a service stops logging and no alert fires. Here's how absence detection catches the bug that never throws an error.
The webhook consumer died at 2:41am. OOM kill — the kernel reaped it mid-batch, no exception, no stack trace, no final log line. The process was just gone. Its queue kept filling. Stripe events stopped getting processed. And every alert rule you own stayed green, because zero errors is comfortably under every threshold ever written.
That's a silent failure: a service stops doing its job and the only evidence is the absence of evidence. No error fires, so nothing pages. The signal flatlines — logs go quiet, traces thin out, the metric exporter freezes on its last value — and your monitoring reads that flatline as health. Somebody finds out six hours later when a customer asks why their payment never went through.
This is the deadliest class of production bug, and almost nobody monitors for it. Here's why, and what to do instead.
Why error-based alerting can't see a silent failure
Every alerting system you've used works the same way: define a condition, fire when it's true. CPU over 90%. Error rate over 5%. p99 over 500ms. Status code 500. Every one of those needs something to happen — they evaluate against data. When a service dies quietly there is no data to evaluate, no errors, no slow spans, no spike. The rule sits there content because zero is below the threshold. The thing that should fire the alert is the same thing that erased its input.
Heartbeats and synthetic checks patch part of this. Ping a health endpoint every 30 seconds, page if it stops answering. Fine — for things that have an endpoint. But the workloads that fail silently are usually the ones that don't: background workers, cron jobs, queue consumers, batch processors, the metrics sidecar nobody remembers deploying. There's no /healthz to poll on a job that wakes up every fifteen minutes.
Absence is a signal — across logs, traces, and metrics
The core idea is almost embarrassingly simple: if something that reliably produces a signal stops, that's the alert. The hard part is that "signal" isn't just logs, and a real outage usually breaks more than one at once. A silent failure shows up in three places:
—
Logs go quiet A service that writes a line every few seconds goes dark for minutes.
—
Trace throughput drops. A service that handles a steady request stream falls out of your traces, or its span volume falls off a cliff.
—
Metrics stop reporting. An exporter dies, a scrape target goes dark, a gauge stops emitting new points — and the dashboard keeps rendering the last value forever, green and lying.
That third one is the trap. A frozen gauge looks healthier than a falling one. Your API server does 200 requests a minute and logs each; when that goes to zero for three minutes, the service is down or something fundamental changed — and the same holds when its trace throughput collapses or its metrics stop arriving. You want to know in all three cases. Most tools only watch for the wrong number, never the missing one.
You can't set a static threshold for "too quiet"
Here's where naive absence checks fall apart: silence is often normal. A batch job that runs hourly is supposed to be quiet between runs. Your API server at 3am Sunday logs a fraction of what it does Monday at noon. A metric that only updates on deploy isn't dead between deploys — it's idle. Alert on raw silence and you'll page yourself constantly, which is its own way of going blind.
So absence detection has to learn what normal looks like before it can flag what isn't. It needs a per-service baseline — by hour of day, by day of week — across whichever signals that service emits. Then it watches the live stream against that baseline. Log volume, trace throughput, or metric cadence drops to zero while the baseline says "this should be busy right now"? Alert. The baseline for this slot is already near zero, like an idle batch job or a deploy-only gauge? Stay quiet. The judgment isn't "is it silent" — it's "is it silent when it shouldn't be."
This is what Epok's absence detection does. It starts protecting a new service within about an hour using short-term cadence analysis, then fills in full weekly baselines over seven days. The same logic runs across logs, trace throughput, and metric reporting — so a dead exporter or a service that vanished from traces gets caught the same way a silent log stream does. And because Epok watches every signal together, the alert points at the exact stream, span, or metric that went dark, not a vague "something's wrong."
Four silent failures, four absences
The OOM'd worker. A consumer chewing through Stripe webhook events gets OOM-killed. No error log — the kernel did the killing, not the app. Its log stream goes flat; Epok flags the gap against the worker's normal cadence instead of waiting for an error that will never come.
The deleted cron. A job that runs every fifteen minutes stops, because a deploy quietly dropped its crontab entry. Nothing errors anywhere. The next scheduled window arrives, the expected output doesn't, and the absence is the tell.
The dead exporter. A metrics sidecar dies after a node restart. The app is fine. But the gauges it published stop advancing, and the dashboard keeps showing the last good number. Epok reads the metric against its baseline cadence, sees it stopped reporting, and alerts — rather than trusting a frozen value.
The stalled replica. A read replica falls behind and stops serving queries. The app fails over to the primary and keeps working, so there are zero application errors — but the replica's logs go quiet and its query traces vanish. Absence detection catches it before the primary buckles under the redirected load.
None of these throws an exception. None trips a threshold. Every one is invisible to a tool that only knows how to watch for things getting louder.
The takeaway
Most monitoring is tuned for noise — the spike, the error storm, the red graph. The failures that cost you a Sunday are the quiet ones, and they need the opposite instinct: a baseline of what each service sounds like healthy, and an alarm for the silence that breaks it. Silent failures are a subset of the incidents that hide between alerts — the ones that never trip a rule. Their close cousin is the brand-new error a deploy introduces that no rule was written to catch yet.
Absence detection is included in Epok's 14-day trial — every feature (verify current pricing at getepok.dev/pricing). Point your logs, traces, and metrics at Epok, give it about an hour to learn your cadences, and it pages you when something goes quiet that shouldn't.
The scariest bug isn't the one that fills your logs with errors. It's the one that leaves them empty.
FAQ
What is a silent failure in production?
A silent failure is when a service stops doing its job without throwing an error — an OOM kill, a crashed worker, a deleted cron, a dead exporter. There's no exception or status code to alert on, so threshold-based monitoring stays green while the work quietly stops. The only evidence is the absence of the signal the service normally produces.
Why doesn't my alerting catch when a service stops logging?
Standard alert rules fire when a condition becomes true — high error rate, high latency, a 500. A silent failure produces none of those; it produces nothing, and zero reads as below threshold. Catching it requires watching for the absence of expected logs, traces, or metrics, which most tools don't do by default.
How do you detect a silent service without false-paging on idle ones?
Baseline each service's normal cadence — per hour of day, per day of week — instead of using a static threshold. An hourly batch job is supposed to be quiet between runs; an API server at 3am Sunday is naturally slower than Monday noon. Absence detection alerts only when a service goes quiet relative to its own learned baseline, not on raw silence.
Is silence only a logs problem?
No. The same failure shows up as logs going quiet, trace throughput collapsing, or a metric exporter that dies and freezes its last value on the dashboard. Watching all three together catches more outages and avoids trusting a frozen gauge that looks healthy because nothing is updating it.
Try Epok free. First alerts in minutes.
No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.
Start 14-day trial
https://getepok.dev/blog/silent-failures-the-bug-that-wont-page-you
Top comments (0)