The Health Check That Was Red and Green at the Same Time

A business-critical health check was showing red. Red on the dashboard, red in the alert history, red enough that someone was about to get paged. But when I opened the failing runs, the service underneath was returning HTTP/2 200. The check was failing, and the thing it was checking was fine. Red and green at the same time.

That contradiction turned into one of the more useful debugging lessons I've had in monitoring, and the thing that cracked it wasn't a better tool or a deeper log dive. It was a single habit: read the failure duration before anything else.

Read the duration first

When a synthetic monitor fails, the first number I look at isn't the error message. It's how long the check ran before it failed.

• Fast fail (under roughly 150 milliseconds, with no response body read): the check died before it ever got a real answer from the application. That points at the environment: network, DNS, routing, egress, a blocked runner. The application was never actually reached.
• Full duration, then an assertion failure: the check ran the whole transaction, got a response, and then decided that response was wrong. That points at the script: a selector, an SSO step, or the assertion logic itself.

This one split does most of your triage before you read a single log line. The duration tells you which half of the world to look in: the environment the test runs in, or the test itself.

Back to the red-and-green health check. It was failing in about 24 milliseconds. No real round trip completes in 24 milliseconds, and the response headers showed HTTP/2 200 underneath. Fast fail plus a healthy status: by the heuristic, this wasn't the application failing. It was the assertion.

The assertion that lied

The check had exactly one assertion: pass only if the status code equals 200.

That looks sensible until you notice how much it depends on. The status code has to survive the whole path from the service, through whatever proxies and gateways sit in between, to the bit of code inside the check that reads it and compares it. Anywhere along that path it can arrive in a form the check doesn't recognise, and when that happens the check reports a failure even though the service answered correctly. That's what was happening here. The response headers said HTTP/2 200. The assertion still called it red.

A status-code-only assertion is weak in both directions, and the direction nobody notices is the dangerous one. The false red is loud. Someone gets paged, someone investigates, someone eventually finds out the application was healthy the whole time. The false green is silent. Gateways, CDNs, and friendly error pages will happily hand you an HTTP 200 wrapped around an empty body or an error message, and a check that only reads the status code will call that a pass. Your dashboard stays green through a real outage.

The fix was to make the assertion ask a question worth asking. Instead of checking the status code, check that the response body actually contains what a healthy response contains, a specific content marker that only a genuinely working service returns, and pair that with a duration sanity window. The duration window catches both ends: a hang, and a suspiciously instant response that couldn't possibly have done any real work. Status code stays in as a supporting signal, never as the sole criterion.

The rewritten check went green immediately and stayed there. Three hundred consecutive passes, median around 23 milliseconds.

The tooling copied the mistake faithfully

That bad assertion didn't come from nowhere. It came from an old test template, and the conversion tooling that migrated the legacy scripts had reproduced it exactly, everywhere the template had been used. This is worth sitting with if you're using AI-assisted conversion for any kind of migration. The tooling did precisely what it was asked to do. It was faithful. It just had no opinion about whether the thing it was faithfully reproducing was any good, so a single bad pattern from years ago got copied forward at machine speed into a brand new platform.

Assertions are the part of a migrated test that most needs a human to read it, because an assertion is the only place where the test states what "working" means. Everything else is mechanics.

But going green was never the point. The point was making sure it couldn't come back. I generalized the fix into a platform rule, no status-code-only assertions anywhere, and replicated the corrected pattern to the sibling monitors that had inherited the same template. The goal was to make the mistake structurally unrepeatable, not just fixed in one spot.

The discipline that matters most

There's a version of this job where every red monitor is an annoyance to be silenced, and the fastest way to silence one is to loosen its assertion until it passes. Resist that completely.

Not long after, two monitors started failing together while their sibling checks in other environments passed the same set of test cases. The pattern was suspicious in a specific way, same environment, same data lookups, and it would have been trivial to relax the assertions and make the red go away. I didn't, because the evidence pointed at a genuine data-layer problem in that environment. Loosening the assertion wouldn't have fixed anything. It would have hidden a real "data not found" behind a green light. The right move was to keep the assertion honest and escalate the underlying cause.

That's the rule I'd hand a new monitoring engineer on day one: if a check fails, diagnose the root cause, and never adjust the assertion just to make the test pass. The moment you loosen an assertion to force green, you've converted your monitor from something that tells you the truth into something that tells you what you want to hear.

What it all comes down to

A synthetic monitor has exactly one job: tell you the truth about what a real user would experience. Everything here is in service of that one job.

• Read the failure duration first. It tells you whether to suspect the environment or the test.
• Assert on meaning, not on a status code. Verify the response actually contains what a healthy response contains.
• Never loosen an assertion to force a green. Diagnose the cause instead.
• Review anything your automation generates. It reproduces your bad assumptions as faithfully as your good ones.

The monitor that was red and green at the same time wasn't broken hardware or a flaky network. It was a monitor asking the wrong question. Once it asked the right one, it started telling the truth, and a monitor that tells the truth is the only kind worth having.