Cloudflare's bot challenges silently break navigation POSTs (and how an Astro forms package survived them)

#astro #cloudflare #debugging #webdev

Every payment attempt on the live site failed. Every test I ran passed.

That sentence cost me a full day, ten hypotheses, and three patch releases. What I found at the bottom is something I now check on every site I touch, because the failure is invisible from every place developers normally look: Cloudflare's bot challenges can make navigation POST requests structurally impossible to complete, for real users only, while your tests sail through.

Here is the whole arc, including the wrong turns, because the wrong turns are where the lesson lives.

The setup

I maintain an open source forms package for Astro called cool-astro-forms. One of its features is a payment page: a visitor lands on a branded page with an amount, passes a Turnstile check, clicks Pay, and the server creates a Stripe Checkout session and sends them there.

The flow was a classic HTML form submit. Navigation POST to the create-session endpoint, server answers with a 303 redirect to Stripe. Boring, dependable, works without JavaScript. The kind of thing you write once and trust.

It ran perfectly in development. It ran perfectly in my test browser. It ran perfectly under curl. Then the site owner tried to make a real payment in his own Chrome and got this: first click does nothing, second click shows a security check error.

Normal window, private window, same result. Every attempt, all day.

The hypothesis graveyard

I want to be honest about how long the wrong path was, because during it every single fix was real. They just were not the fix.

Hypothesis 1: expired Turnstile tokens. Real bug. The token silently expires after about five minutes and the widget still shows success. A visitor who reads the page slowly submits a dead token. I shipped a recovery flow: dead token posts now bounce back to the page with the amount preserved and a fresh widget. Failures continued.

Hypothesis 2: the remoteip parameter. Real bug. I was passing the visitor's IP to Turnstile's verify endpoint, and dual-stack visitors can present different addresses to the page and the verify call. Verification failed for people whose networks behaved differently from mine, which is the purest form of works-from-my-machine. I dropped remoteip. Failures continued.

Hypotheses 3 through 9. Consumed tokens. Browser extensions. Service workers. Cookies. Stale caches. Each one plausible, each one testable, each one innocent. Meanwhile the error codes I was surfacing kept saying timeout-or-duplicate, which pointed back at tokens, which kept me circling the token theory long after it stopped explaining anything.

The trap of a day like this: every fix improves the system, so it always feels like progress, and none of it touches the actual fault.

The breakthrough: instrument the browser that fails, not the one that works

The turning point was giving up on reproducing the failure in my environment and instrumenting the failing browser itself, live, during a real attempt.

What the network log showed changed everything. The first click's navigation POST to the create-session endpoint returned a 503 with the tab wedged mid-navigation. Then, from the same page, in the same browser session, with the same cookies, a fetch() to the same endpoint went straight through and reached the application.

Same origin. Same credentials. Same moment in time. The only difference was the transport.

A third data point sealed it: a byte-identical request sent through curl, with the browser's user agent and Sec-Fetch-Mode: navigate headers faithfully copied, passed in 1.5 seconds.

So the request was fine, the browser was fine, and the server was fine. What was not fine was the combination: a navigation POST, from a real browser, whose TLS fingerprint the edge had decided not to trust.

The mechanism

Cloudflare's bot defenses can challenge requests they find suspicious. For a normal GET navigation, that works: the visitor sees an interstitial, passes, and the request is replayed.

A challenged navigation POST cannot work that way. The interstitial cannot replay the POST body. The request dies. The visitor sees either a wedge or an opaque 503, depending on timing. And because my page had already consumed the Turnstile token on that first doomed attempt, the second click carried a stale token and produced the perfectly misleading timeout-or-duplicate error that kept me chasing token bugs.

The cruelest part is the selection effect. The challenge fires based on client fingerprint trust. Automated test browsers and curl often present fingerprints the edge has no opinion about. The site owner's real, aging, extension-laden, cookie-rich Chrome was exactly the client that drew the challenge. My tests passed because they were tests. His payments failed because he was real.

I never found this in any Cloudflare error log I had access to, and nothing in the application ever saw the request. The failure lived entirely in the space between the visitor and the origin.

The observation that cracked it open

The site had previously run on WordPress with a paid payments plugin, and payments there never failed this way. Same domain, same Cloudflare zone, same protective settings.

The owner asked the obvious question: why did WordPress work?

The answer took one look at the plugin's source: it submits payments over AJAX. Not because its authors foresaw any of this, presumably, but because that is just how modern WordPress plugins are built. The transport was the entire difference. XHR and fetch requests do not get interstitials. There is no page to interrupt.

The architecture was never the problem. The transport was.

The fix: change the transport, not the security

The patch that ended the saga did three things:

// Before: native form submit (navigation POST, challengeable)
// After: fetch first, native submit kept as the no-JS fallback

const res = await fetch(form.action, {
  method: "POST",
  body: new FormData(form),
  headers: { Accept: "application/json" },
});
const data = await res.json();
if (data.ok) {
  location.assign(data.url); // the hop to Stripe is a plain GET
}

The payment page submits over fetch. The endpoint recognizes fetch clients by the Accept header and answers 200 {ok, url} with JSON instead of a 303.
The hop to Stripe happens as location.assign(url), a plain GET navigation, which survives any challenge normally.
The native form submit stays in place as the no-JS fallback, and failures render inline with the amount preserved and the edge's error code visible, so the next debugging day starts with data instead of guesses.

Just as important is what the fix did not do: no Cloudflare exception rules, no allowlists, no turning anything off. The zone's bot protections stayed fully active. Security settings that punish your checkout are not something to disable; they are something to route around with a transport that was never in their blast radius.

The same night, the first click in the owner's own Chrome, the browser that had failed all day, rendered a Stripe checkout page.

What I take from it

"Works from my machine" has an evil twin. Fails only from real machines. When your tests pass and real users fail, stop trying to reproduce it in your environment and instrument theirs. The failing browser is the only honest witness.

The edge is part of your application. My code never saw the failing requests. No server log, no application error, nothing. If your mental model of the request path ends at your origin, an entire class of failures is invisible to you.

Transport is an architectural decision. A navigation POST and a fetch POST are not two flavors of the same thing. They pass through different machinery at the edge, they fail differently, and one of them cannot survive an interstitial. I picked navigation submits for their simplicity and paid for it. The fetch-first, GET-hop pattern is now baked into the package so nobody using it has to relearn this.

Surface error codes all the way to the visitor's URL. Half my wasted day came from errors that said nothing. The package now carries the edge's own error code into the recovery URL, which turns a visitor's "it did not work" into a diagnosis you can read off a screenshot.

Real fixes can still be the wrong fix. The token expiry bug was real. The dual-stack bug was real. Fixing them made the product better and the investigation longer, because visible progress is the best camouflage a root cause ever gets.

The package that survived all this is cool-astro-forms, MIT licensed, on npm. The payment transport, the token-gated buttons, and the error surfacing described here all shipped as releases 0.1.5 through 0.1.10, each one from a finding on a live production site. If you run forms or payments behind Cloudflare, I hope this saves you the day it cost me.