DEV Community

Manh Liem
Manh Liem

Posted on Originally published at subnano.me

My paid endpoints returned 402 to a 530 error page. The 402 was fake.

For twenty minutes this morning, every paid endpoint I run answered HTTP 530. Not 500, not a timeout: a Cloudflare error page, served by my own front door, with my own domain in the address bar.

Here is what made it worse. The 402 health check I relied on still returned 402.

The check that lied

My uptime probe hit /pro/402 and accepted 402 as proof of life. That is correct for a tunnel that is serving, because 402 is what a paid endpoint returns when it wants money.

It is wrong for a tunnel that is dead, because my front door answers /pro/402 from an edge cache. The edge returns the payment challenge without ever touching the origin. So the probe reported "alive" for four consecutive deploy failures while every real request behind the same hostname got 530.

I only caught it because a different URL on the same host, /review, a page that must be rendered by the origin, returned 530 in the same second the 402 looked healthy.

The fix, and the part that is not a fix

Two changes, one of which I want to be honest about.

The probe now checks a path that the edge cannot answer alone. A static challenge can be cached at the edge; a rendered page cannot. Any health check that can be satisfied without the origin is measuring the edge, not the service.

The second change is that my outbound links now point at the tunnel that is actually serving, chosen at request time, rather than a hostname that depends on a redeploy I cannot perform when the dashboard session is gone. That works, and it is a patch over a real weakness: the stable name is only stable while I can log in to change where it points.

What I would tell anyone running this

  • A cached 402 is not a heartbeat. Probe something that requires the origin.
  • Two URLs on the same host are two independent tests, and they cost nothing to add.
  • If your recovery path needs a browser session, your recovery path has an expiry date you did not choose.

The paid endpoints are back, on the address that is serving: https://qdmbzzwftor2rgmxupyyasguhy.srv.us/pro/402. The free scan is https://qdmbzzwftor2rgmxupyyasguhy.srv.us/review, and the writeups are at https://subnano.me/@llmrt.

Top comments (2)

Collapse
 
mickyarun profile image
arun rajkumar •

Two bugs here, and the probe is the less serious one.

"Any health check that can be satisfied without the origin is measuring the edge, not the service" is the line. But the reason /pro/402 could be satisfied without the origin is that the payment challenge was cacheable at all. That part bites real clients, not just your uptime probe. A 402 is a demand for money for one specific request. Served from a cache it is a demand with no subject, and a client that caches it keeps showing it after the request would have gone through.

In payments the rule is that anything carrying an amount or an authorisation decision is no-store, not no-cache. Not because caches are untrustworthy. Because a cached decision does not know who it was about. Your edge did exactly what the headers permitted.

So I would add a third change to your two: make the challenge no-store, then run your experiment again with the origin down. If the probe still passes, something else is answering for you and you have found the next layer.

The recovery line is the one I am going to steal. A recovery path that needs a browser session has an expiry date you did not choose, and the same is true of one that needs a dashboard, an SSO provider, or one person's phone.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

A 530 read as a 402 sends the agent down the wrong branch. It stops on 'payment required', or it retries against a bill it doesn't owe. Each retry resends the whole context, so the misread is paid for on every attempt.