Catch New Errors in Production Before Your Users Do
The errors that take you down are the ones you've never seen before. Here's how automatic fingerprinting catches new errors in production on the first occurrence.
A new exception in production almost never trips an alarm, because you didn't write a rule for an error you'd never seen. To catch new errors in production you have to detect the absence of history — a log line whose pattern has never appeared before — not breach a threshold. That's a different job than counting 500s, and it's the one most monitoring setups quietly skip.
Picture the Friday deploy. Tests green, staging clean, laptop shut. Forty minutes later a customer emails support: "I can't log in." Your auth service has been throwing NullPointerException at SessionResolver.resolve since 4:51pm. Nobody got paged. There was no rule for it — there couldn't have been — and the 500 rate barely moved because the bad path only hits users with a saved OAuth token. The error was real, it was new, and it sailed straight past everything you'd set up.
Why threshold monitoring misses brand-new errors
Most monitoring is built around failures you've already met. You write an alert for database connection errors after the night they took you down. You build a dashboard for HTTP 500 rate. You wire a Slack notification for p99 latency. Every one of those watches for a problem you already had the imagination to anticipate.
The expensive outages are the ones you didn't. A library upgrade surfaces a serialization bug. A feature flag flips on a code path that's been dead for six months. A dependency starts returning a shape your parser never expected. None of these match a rule, because the whole point is that they're new. They live in the gap between your alerts — the same gap where silent failures and slow cascades hide.
A new error is, by definition, the one thing your threshold monitoring can't have a threshold for.
What actually makes an error "new"
Here's the part that sounds trivial and isn't. Connection timeout after 30s on port 5432 and Connection timeout after 45s on port 5432 are not two errors. They're one error with two numbers in it. The thing you care about is the shape: Connection timeout after s on port .
So you normalize. Strip the volatile bits — numbers, IP addresses, UUIDs, hex strings, timestamps, request IDs — and what's left is the error's fingerprint. Hash that, and you've got a stable identity that survives every reboot, retry, and rescheduled pod. If a fingerprint has never appeared in your history, the error is new. If it has, it isn't, no matter how many fresh parameter values it shows up with.
The craft is all in the normalization. A naive version is worse than useless:
—
error from 10.0.1.55:8080 and error from 10.0.2.30:9090 must collapse to one fingerprint, or every host churn reads as a novel error.
—
request abc-123-def failed and request xyz-789-ghi failed are the same failure with different request IDs.
—
disk 87% full and disk 92% full are the same warning crossing different marks.
—
But connection refused and connection reset are genuinely different errors, even though they're one word apart — and a normalizer that's too aggressive will fold them together and hide a real regression.
Get the normalization wrong in either direction and the detector is dead. Too loose and you drown in false novelty. Too tight and you mask the thing you built it to find.
Why grep and a regex won't get you there
The first instinct is to grep for ERROR or FATAL and pipe it somewhere. That holds up until you're doing a few million lines an hour across twenty services. Then you've got thousands of matches that are really the same three errors wearing different parameters, and no way to tell which one started today.
Two pieces are missing, and they're the hard ones. You need fingerprinting to group the noise into distinct error identities. And you need a baseline — a memory of what's normal — to know which identities are actually new. Skip the baseline and everything looks new on Monday, because you've never seen a Monday morning before. Skip the fingerprinting and you alert on every distinct string, which is alert fatigue with extra steps.
Build it yourself and you're now maintaining a fingerprint store, a normalization ruleset with a long tail of edge cases, a rolling historical baseline per service, and an alerting pipeline on top. It's a real project. Most teams either never start it or ship a brittle version and stop trusting it.
How Epok catches new errors in production
Epok runs new-error detection automatically on every log stream — no rule, no regex, no config. It normalizes each error message so superficial differences don't read as novelty, fingerprints the underlying pattern, and checks that fingerprint against your service's history. When one shows up that the baseline has never seen, it flags it on the first occurrence.
The alert carries the things you'd otherwise go digging for: the normalized pattern, how many times it's fired and how fast, which services it touched, and when it first appeared relative to your last deploy. Severity is graded automatically — a new error firing 100+ times in five minutes is critical; ten times is a warning; a known error that vanished for a day and came back gets flagged lower, because that's a regression, not a debut.
New-error detection is log-native, but it isn't the whole product, and it isn't logs-only. The same detection-first engine watches your metrics, traces, infrastructure, and RUM for the same flavor of "we've never seen this before" — a latency distribution that just grew a second hump, an endpoint whose error rate is creeping, a service that went quiet when it should be talking. And because every verdict is cited to the exact line, span, or metric it came from, you check the evidence instead of trusting a summary.
What it looks like at 4:55pm on a Friday
You deploy. Four minutes later, Slack:
New error in api-server — FATAL: connection pool exhausted · 47 occurrences in 5 min · CRITICAL · first seen 4 min after deploy v2.4.1
You didn't write a rule for connection-pool exhaustion. You didn't know your pool sizing was a problem until the deploy made it one. There's no dashboard for it because nobody knew to build one. Epok caught it for the only reason that matters: it had never happened before, and a new error in production that's firing forty-seven times in five minutes is worth your attention before a customer makes it your problem.
That's the difference between counting errors and noticing them. Counting needs you to already know what to look for. This doesn't.
Try it
New-error detection is on every tier, including the 14-day free trial. Point your logs at Epok and it works immediately — no rules, and no warmup for this detector specifically: fingerprinting catches new errors from the very first log line. (The volume baselines that power rate-anomaly detection take about seven days to settle; new-error detection doesn't wait on them.)
Pricing is flat and legible: a 14-day free trial with every feature (~1 TB of ingest), then Team at $199/mo for 1 TB/month, Growth at $599/mo for 4 TB/month, and Custom from $5,000/mo for higher volume. Overage is $0.20/GB, and there's no free-forever tier. Verify current pricing at getepok.dev/pricing.
FAQ
How do you detect a new error you've never seen before?
By normalizing each error message into a stable fingerprint and comparing it against your service's history. Numbers, IPs, UUIDs, hex strings, and timestamps get stripped out so cosmetic differences don't count as new; the remaining pattern is hashed. If that fingerprint has never appeared in the baseline, the error is genuinely new and gets flagged on its first occurrence — no rule required.
How is new-error detection different from error rate monitoring?
Error-rate monitoring tells you how many errors you have and fires when a known type crosses a threshold. New-error detection tells you which errors you've never seen before, which is the category a threshold can't cover — there's no historical rate to breach. The two are complementary: one catches spikes in familiar failures, the other catches novel ones the moment they appear.
Why can't I just grep my logs for ERROR or FATAL?
Grep gives you matches, not identities. Across millions of lines and dozens of services you get thousands of hits that are the same handful of errors with different parameters, and no way to tell which one started today. You'd still have to build the fingerprinting that groups them and the baseline that distinguishes new from familiar — which is most of the work.
Does new-error detection need a warmup period before it works?
The new-error detector itself does not — fingerprinting catches genuinely new patterns from the first log line, because "new" means "not in the history we've recorded so far." Rate-anomaly detection, which learns what a normal volume looks like by hour and day, takes about seven days to build a useful baseline. The two are separate.
Try Epok free. First alerts in minutes.
No credit card. Every detector included, root cause on every incident. Full baseline coverage at 7 days.
Start 14-day trial
https://getepok.dev/blog/catch-new-errors-before-users-report-them
Top comments (0)