Monitoring has a property that makes it uniquely dangerous: it is the code that only runs during a disaster. Every other part of your system runs constantly, in front of users, throwing errors you notice. Your alerting runs the day the database fills up, and not before, and if it is broken you find out at the exact moment you were relying on it and in the exact way that guarantees nobody was watching. A silent alert is worse than no alert, because no alert you know you do not have, and a silent one you believe you do.
So we stopped trusting that our alerts worked because we had written them, and built a harness to prove it. For every alert rule we deploy, the harness drives a real signal that should trip it, and then it does the thing that actually matters: it waits for the notification to physically arrive in a capture relay, a small service standing in for the phone that would otherwise buzz. Firing in the monitoring tool's own UI does not count. Delivery counts. The full path from signal to a message in hand, or it is a failure.
The first real run failed most of the rules. Here is what was hiding behind green dashboards.
The flagship alert was dead
The most important rule we own is the one that fires when the error rate climbs: too many HTTP 500s, page somebody. The harness drove four hundred and fifty real server errors at it, verified that at least ninety percent of them genuinely returned 500, and waited. Nothing arrived. The single alert we would most want to trust had never once fired, and would not have fired in a real incident.
The cause was a name. Our metrics come out of OpenTelemetry, where names are dotted, like http.server.request.duration. Somewhere between reading a tutorial and writing the query, we had assumed the monitoring backend would normalise those dots to underscores, the way a lot of Prometheus tooling does, and written the alert query against the underscore form. The backend does not normalise. It keeps the dotted names exactly as sent. So the alert queried a metric that did not exist, matched nothing, computed an error rate of zero forever, and sat there calm and green while the real metric, under its real dotted name, recorded every one of those five hundreds. The query was not wrong in a way that errors. It was wrong in a way that silently returns the empty set, which arithmetic then turns into a perfectly plausible zero.
The one that took every log alert down with it
That was the bad one. The next one was worse, and the harness only caught it because it insists on delivery rather than on the rule looking correct.
A whole category of our alerts is log-based: match a line in the logs, fire. Every single one of them was evaluating against nothing, and the reason had nothing to do with the alert rules. It was a retention setting.
We reconcile our log retention through the backend's settings API. That API had moved to a new version with a new request shape, and we were still sending the old shape. The old body was not rejected. The decoder on the other end silently ignores fields it does not recognise, so it read our now-unknown fields as absent, took the default for the retention it could not find, and that default was zero. The call returned a cheerful 200. And a retention of zero days means instant expiry: newly written log data was being stamped to expire immediately, so the store the log alerts query was, for practical purposes, always empty. Ingestion looked completely healthy the whole time, because logs were arriving. They were just aging out the instant they landed.
Two systems each did something defensible. The settings API accepted a request it partly did not understand rather than failing, which is a common and often reasonable choice. Our deploy sent a body that had gone stale against a version bump. Neither raised anything. The result was that the single most consequential number in our log pipeline, how long a log lives, had been quietly set to zero, and the only visible symptom was every log alert being permanently, healthily silent. If we had been testing that the rules looked right, they looked perfect. Only testing that a real logged event produces a delivered page could surface it, because only that test actually reads back out of the store that had been emptied.
The alerts that fired forever, which is the same as never
A third failure ran in the opposite direction, and it is the subtle one. A handful of audit alerts were firing constantly. That sounds like the opposite of a silent alert, but it has the identical effect, and understanding why is worth the detour.
When one of these alerts fired, the monitoring backend wrote a log line about the evaluation, a breadcrumb, and that line contained the alert's own filter expression as text. That breadcrumb was itself ingested as a log, like anything else. So the next time the rule evaluated, it found a matching line: its own footprint from last time. The rule had become self-sustaining. It matched its own shadow, every evaluation, forever.
An alert that is always firing never transitions from not-firing to firing, and that transition is the thing that sends a notification. So the alerting layer, seeing a condition that was already true and stayed true, suppressed further notifications, exactly as designed, to spare you a page every minute about a thing you have already been told. The consequence is that a real event, a genuine audit action worth paging about, arrived into a rule that was already stuck on and therefore said nothing. An alert pinned permanently on is as mute as one pinned permanently off. The fix was to scope those rules to log lines emitted by our own services, so they stop matching the backend's chatter about themselves, with one deliberate exception whose real signal genuinely comes from outside our services.
The rule that treated a pod's first breath as normal
The last one is a small, sharp lesson about rate calculations. An alert built on the rate of change of a metric cannot compute a rate from a series' very first sample, because there is nothing before it to compare against. So when a fresh pod emitted its first burst of errors, that burst became the baseline rather than a spike, and the rule that should have caught it saw only a starting point. We fixed it in the harness by driving such signals in two waves, a small one to establish the series and a real one after it, which is also a fair description of what production traffic does on its own and what a synthetic test has to imitate deliberately.
Every failure hid in a seam you can only find by exercising it
Every one of these was invisible to inspection. The queries read correctly. The rules were deployed. The dashboards were green. Ingestion was healthy. Each failure lived in a seam between two things that were each individually fine: a metric name and an assumption about normalisation, a request body and a version bump, an alert and the log its own firing produces, a rate function and a series that has just begun. You cannot read a seam. You have to exercise it.
So treat monitoring like what it is, which is code, and specifically code whose only production run is the emergency. Code that only runs in the emergency needs its tests to run every other day, because there is no gentle first failure to warn you. Drive a real signal end to end, and assert on the artifact at the very end of the chain, the delivered notification, not on the rule looking healthy in the middle. Anything short of that is testing that you wrote an alert, which you already knew, rather than testing that it fires, which is the only thing you actually wanted.
If you would rather your identity provider was already watching itself this way, Authagonal runs its own alerting through exactly this harness, so the day something breaks, the page that should fire actually does.
Top comments (0)