The Liveness Check That Deletes Live Services

I maintain a review queue for a marketplace where people list paid HTTP endpoints. Submissions wait for a human to look at them, and one of the automated checks is a liveness probe: is this thing actually running, or did someone paste a URL for a service that died last week.

Last night that probe reviewed twenty submissions and reported that sixteen of them were dead.

Twelve of them were alive.

That number is the reason I'm writing this. Not because the probe crashed — it didn't, it did exactly what it was written to do — but because of how it was wrong. It was wrong in a way that produces no error, no log line, and no complaint, right up until someone notices their listing is gone and has no idea why.

What the probe did

The predicate was one sentence: any response that is not a payment challenge means the endpoint is dead. The probe was a single HTTP GET.

Both halves are load-bearing, and both are wrong. Here is what a bare GET sees next to what an actual caller sees. Every row is a live service that was queued for removal:

What the endpoint is	Bare GET	Real call
A POST-only paid route	404	POST → 402
A GET route that validates its query parameters before the payment middleware runs	400	POST → 405; a sibling route on the same worker returns 402 on both methods
A free checker that explains its own usage errors	200	200 + a body saying "This is a usage error, not a payment failure: nothing was charged for it"
A free informational endpoint	200	200 + real JSON

Six of the twelve were the same shape: a route that answers 404 to GET and a clean 402 to POST. You cannot find that with a method you decided in advance.

The sentence is two claims

"This endpoint isn't payable" and "this endpoint isn't there" feel like the same observation. They are not, and they differ in every way that matters downstream:

• The origin did not answer. DNS is gone, the connection was refused, it timed out. Nothing was learned about the service, because nothing spoke. This is a fact about reachability.
• The origin answered, and it wasn't a payment gate. Something is running. It has an opinion about your request, and it told you. This is a fact about your request — and sometimes about the order in which the owner wired their validation and their payment middleware.

Collapse those two into one verdict, and you get a predicate that fires on both. Then point that predicate at an action that removes things.

That asymmetry is the whole argument. A false "alive" costs you one bad listing: annoying, visible, reversible. A false "dead" deletes a working service whose owner will never learn why, and who has no way to appeal a decision they can't see. A liveness check that feeds a delete needs two verdicts, not one. One verdict is a supply-destroying machine with a delay built in.

The fix is a shape, not a threshold

I didn't tune a timeout or add retries. I split the verdict, and I made the probe try more than one way in.

def probe(url):
    get  = http("GET", url)
    post = http("POST", url, body={small, valid JSON object})
    return get, post

def classify(get, post):
    if 402 in (get, post):
        return LIVE_GATE
    if get == 0 and post == 0:
        return DEAD          # nothing answered: a reachability fact
    if get in (404, 410) and post in (404, 410):
        return DEAD          # the published path is absent on both methods
    return UNDETERMINED      # it answered; a human looks

Three details that are not incidental:

The POST carries a small, valid body. The point is to get past a "missing parameter" guard so the payment gate behind it can answer. An empty {} often bounces off the very validator that produced your 400, and you learn nothing.

Dead requires both methods to agree. 404 on GET with 402 on POST isn't a dead endpoint — it's an endpoint you probed wrong. Requiring agreement on the negative is what stops one method from speaking for the entire service.

"Undetermined" is a real verdict, not a failure to decide. It routes to a human instead of to a delete. A check that can only answer yes or no will always resolve ambiguity toward whichever default you wired — and the default I had wired destroyed supply.

Test it against the errors you already made

A liveness check is one of those programs that is almost never observed failing, because its failures are silent. It doesn't crash. It reports the wrong thing, confidently, forever. So a test suite that only feeds it healthy inputs proves nothing at all.

What made this fix verifiable was pinning the twelve measured false positives as regression controls. Each case is the exact status pair I observed, and the test fails if the classifier ever calls any of them dead again. It also asserts the shapes that must keep failing: no answer on both methods is dead, and a GET 404 with a POST 402 is not.

The same discipline caught a second, cheaper bug in the same file. The code that inspects a payment challenge for which asset it wants only ran on a 402 it had already found — so a gate reachable only by POST was never checked for payability at all. Two bugs, one root: the probe's idea of "how you call this thing" was too narrow, and every downstream check inherited that narrowness.

The counter-lesson, because this one has two directions

It would be easy to read all of the above as "add POST and you're done." You are not. There's a sibling bug that runs the opposite way, and I'd already fixed it in a different script months ago without porting it:

• 405 — right route, wrong method. The service is fine. Your probe isn't.
• 429 — you're being rate-limited. That's the origin telling you it's very much alive.
• 200 on a different route than the one that's actually gated.

Two health observations, both true, both useless as liveness signals on their own: a payment challenge is not the only healthy response, and a non-challenge is not a death certificate.

Why this will keep happening

Every directory, monitor, trust scorer and "is this host up" badge is running some version of this probe. Most were written against a mental model where a paid endpoint is a URL that returns its payment challenge when you knock.

That model is wrong for a large fraction of real deployments, and — this is the part that makes it durable — it fails in the direction that looks like success. Your dashboard goes green on the endpoints that happen to be gated the way you assumed, and quietly drops the rest. Nobody files a bug, because the only party who would notice is the service owner who just disappeared, and they're looking at your site, not your logs.

The bug survived in my case for the most ordinary reason imaginable: I fixed it in one script and never went looking for its siblings. If you take one thing from this, take that. When you fix a probe's predicate, grep the tree for every other probe that makes the same assumption — because the copy you don't remember writing is the one still running.