DEV Community

Manh Liem
Manh Liem

Posted on

My payment endpoint returned 503 for nine hours and three separate bugs were hiding it

#webdev #ai #python #devops

At 20:31 UTC my public payment endpoint stopped serving. The health check kept reporting OK for another five hours.

When I finally traced it, there were three independent bugs stacked on top of each other. Any one of them alone would have kept the endpoint down. This is the post-mortem, because the stacking is the interesting part.

The symptom

Buyers hitting /review got 503 No tunnel available. The same URL had returned 200 forty minutes earlier. Local service was fine the whole time:

$ curl -o /dev/null -w '%{http_code}' http://127.0.0.1:8139/review
200
$ curl -o /dev/null -w '%{http_code}' https://my-tunnel-host/review
503
Enter fullscreen mode Exit fullscreen mode

So the origin was healthy and the public path was not. That splits the problem into three layers: the tunnel client, the tunnel hostname, and the health check that decides whether to repair either.

Bug 1: TLS to the tunnel edge was being cut, TCP was not

The tunnel client logs were unambiguous:

ERR Unable to establish connection with Cloudflare edge
    error="TLS handshake with edge error: read tcp 10.149.67.50:36150->198.41.200.63:7844: i/o timeout"
Enter fullscreen mode Exit fullscreen mode

An i/o timeout on a read during the TLS handshake looks like a dead network. It was not. The TCP handshake to the same IP completed in 0.2 seconds:

socket.create_connection(('198.41.200.63', 7844), timeout=8)  # OK, 0.2s
Enter fullscreen mode Exit fullscreen mode

TCP connects, TLS never completes. That pattern means something between me and the edge is inspecting the handshake and dropping it, not that the host is unreachable. QUIC did not help either, it failed earlier with context deadline exceeded while requesting the tunnel.

What worked was routing the tunnel client through an HTTP proxy. Registration went from never to 40 seconds:

2026-10-30T01:51:43Z INF precheck component="TCP Connectivity" status=pass
2026-10-30T01:51:43Z INF precheck component="Cloudflare API" details="API is reachable" status=pass
Enter fullscreen mode Exit fullscreen mode

The lesson is not "use a proxy". The lesson is that a TCP-level reachability test told me nothing about whether the TLS path worked, and I had been treating them as the same signal.

Bug 2: the recovery loop only recognized one tunnel provider

I run two tunnels as fallbacks. The supervisor looked for the live hostname by regex:

for m in re.findall(r'https://[a-z0-9-]+\.trycloudflare\.com', txt):
    doms.append(m)
Enter fullscreen mode Exit fullscreen mode

That pattern only matches one provider. The hostname that was actually serving traffic came from a different one, so the supervisor never saw it. It returned None, logged no live tunnel domain found, and waited.

The failure mode is worth naming: the recovery code could not see the thing it was supposed to recover. It did not crash, it did not error, it printed a message that sounded like a legitimate waiting state and slept for five hours.

Bug 3: the health check preferred a directory listing over a working endpoint

Once the supervisor could see both hostnames, it still chose the wrong one. It asked the index API which hostnames were registered and picked the first hit:

if d.get('routerDispatchEligible') is True:
    out.append(u)
Enter fullscreen mode Exit fullscreen mode

Being listed is not the same as being reachable. The listed hostname returned 503 on ten consecutive requests. The unlisted one returned 200 on ten consecutive requests. The supervisor swapped a working endpoint for a dead one because the dead one had better metadata.

Worse, the single-request probe used to test reachability returned True for both. Here are ten requests each, back to back:

listed host      503 503 503 503 503 503 503 503 503 503
unlisted host    200 200 200 200 200 200 200 200 200 200
Enter fullscreen mode Exit fullscreen mode

A single probe hit an intermittent window and reported the dead host as healthy. The fix was to require a majority of probes to pass, and to stop treating listing as a reason to override a host that is currently serving.

What I changed

Three changes, each one removing a way for the system to lie to itself:

  1. Test the path the user takes, not a proxy for it. I was probing POST /mcp because it proved the backend was alive. But it also returned 200 intermittently during the outage, so it certified a dead endpoint. Probing GET /review, the page a buyer actually opens, matched reality.
  2. Majority vote instead of a single probe. One success is not health. Requiring three of four removed the intermittent-window false positive.
  3. Metadata is not availability. A directory listing is a claim. Reachability is a measurement. When they disagree, the measurement wins.

The part I did not expect

After fixing all three, the endpoint worked, and I ran a real paid request through it end to end to confirm: 402 challenge, signed payment, settlement, delivery with paid: true, and a confirmed on-chain transaction.

That was the first time in nine hours that a buyer path had been proven to work. The nine hours of downtime had not produced a single error I could act on, because every layer reported success.

If your health check cannot distinguish "the thing is up" from "the thing can serve a request", you do not have a health check. You have a status page.

Top comments (0)