Monitoring has two ways to be wrong, and we treat them as if they were
symmetrical. They are not, and the asymmetry should drive how you build checks.
A false alarm costs you an interruption. Someone looks, finds nothing, mutters,
goes back to what they were doing. Annoying, self-correcting, and — this is the
important part — it tells you the monitoring is alive.
A false all-clear costs you the entire outage. Nobody looks, because nothing
asked them to. The first signal is a customer, and by then you have lost both the
time and the claim that you knew before they did.
We spend most of our tuning effort on the first failure mode. Almost all of the
damage comes from the second.
Three ways a check reports green on a broken service
1. The check is narrower than the thing it claims to cover
GET /health returns 200. The endpoint returns a literal {"status":"ok"} from a
handler that touches nothing. It confirms that the process is running and the HTTP
listener is bound. It says nothing about the database connection pool, the expired
credential to the payment provider, or the disk that filled up twenty minutes ago.
This is not laziness; it is drift. The check was written when the service had two
dependencies and was honest about both. Four years and eleven dependencies later,
the check still passes because it still does exactly what it did in 2022.
The useful question is not "is the check passing" but "what would have to break
for this check to fail?" If the answer is "the process would have to be dead",
you have a liveness probe wearing a health check's name tag. Both are legitimate,
but only one of them is allowed on the dashboard the on-call engineer looks at.
2. The check runs somewhere that cannot fail the way users do
Checks that run inside the cluster share fate with the thing they measure. They use
internal DNS, skip the load balancer, bypass the CDN, sit inside the network policy,
and often talk to the pod directly rather than through the service. Every one of
those is a component that can break for users while leaving the check perfectly
happy.
Certificate monitoring makes this concrete. Probe public HTTPS endpoints from inside
an environment whose egress
terminates and re-signs TLS
and every certificate that comes back is issued by the gateway, not by the site you
think you are watching. Build a certificate monitor there, point it at production,
and it reports healthy forever — because it is measuring the appliance one hop away
rather than the service on the other side of the internet. The check is fine. The
vantage point makes it meaningless.
The rule that falls out: at least one check has to traverse the same path a user
does, including the parts you do not own. Internal checks tell you which component
broke. Only an external check tells you whether anyone is actually affected.
3. The check stopped running
This is the one that produces the longest outages, and it is structurally
invisible. A cron job that no longer fires, a worker that died at 03:00, a pipeline
whose credential silently expired — none of these emit anything. And the absence of
an alert is indistinguishable from the absence of a problem.
Think about what your dashboard actually renders when a checker is dead. Most of
them show the last known state, which was green, with a timestamp nobody reads. The
system is not lying to you. You just asked it a question it has no way to answer.
The heartbeat inverts the logic
The fix for the third case is to stop asking "did something report a failure" and
start asking "did something report at all".
A dead man's switch is an external service expecting a ping on a schedule. Your
checker pings it after every successful run. If the ping stops, the external
service alerts. The alert now fires on silence, which means the failure of your
monitoring is itself a monitored event.
#!/bin/sh
set -e
run_all_checks # exits non-zero if anything is wrong
# only reached when the checks ran AND passed
curl -fsS -m 10 --retry 3 "https://hc-ping.example/$CHECK_UUID" > /dev/null
Three things about that snippet are load-bearing.
set -e and the ordering. The ping is last. If the checks fail, or crash, or
the box runs out of memory halfway through, the ping never happens, and you get an
alert from the outside. Ping first and you have built a system that reports success
before knowing whether there is any.
The pinged service must be external. A heartbeat monitored by the same
infrastructure that runs the checker shares fate with it. Hosted dead man's switch
services exist and are cheap; the point is the independence, not the vendor.
The expected period should be roughly twice the run interval. One missed run on
a five-minute job is usually a blip. Two is a pattern. Alert on the pattern, or you
will train yourself to ignore the alert — which returns you to a false all-clear by
a slower route.
Designing checks that fail loudly
A few habits that follow from all of this.
Make "unknown" a distinct state from "healthy". If your checker cannot reach a
target, that is not a pass and it is not necessarily a fail — it is stale data, and
it should render differently. Most dashboards have two colours where they need
three. A check that has not reported in an hour should look visibly wrong even if
its last result was green.
Test the failure path, not just the success path. A check nobody has ever seen
fail is a check nobody has verified. Break the dependency deliberately in a staging
environment and confirm the alert actually arrives, at the actual destination, in
the actual channel. Alert routing rots quietly: people leave, channels get archived,
webhooks expire. The routing is part of the system and needs the same treatment as
the code.
Write down what each check does not cover. One line next to the definition.
"Does not verify database connectivity." It costs nothing to write and it is the
sentence that saves you during the post-mortem, because it converts an unknown
unknown into a known gap that somebody can choose to close or accept.
Prefer a check that occasionally cries wolf over one that never speaks. Given
the asymmetry at the top of this post, a slightly noisy check is a trade you should
be willing to make on purpose rather than one you drift into.
The uncomfortable summary
Green means one of two things: the system is healthy, or the system is not being
measured. Most monitoring setups cannot distinguish between the two, and the
dashboard renders both identically.
The work is not making the checks more sensitive. It is making sure that when the
measurement itself fails, something notices — because that is the failure mode that
costs you the whole outage, and it is the only one that gets quieter the worse it
gets.
Top comments (1)
Strong framing with the asymmetry. One thing I would add from running this at scale: most teams already own a dead man's switch and do not realize it. In Grafana Alerting, for example, an alert rule's NoData handling can be set to Alerting instead of OK, which is exactly your external heartbeat for any check that emits metrics. The common failure is leaving it on the default, which makes missing data resolve to green and recreates your third failure mode silently. The ping still matters for the cron job that dies at 03:00, but for checks that report metrics, watch the stream rather than the result.