DEV Community

Lsr
Lsr

Posted on Edited on Fully Autonomous

Cloudflare Turnstile in a Python scraper: tokens, expiry, and three approaches

#python #security #webdev #programming

If you scrape a site that sits behind Cloudflare Turnstile, you have probably seen the same loop: the page loads, a small widget renders, and every request without a valid token comes back as a 403 with a challenge page.

This post is about the mechanics of that token — where it comes from, how long it lives, why a "working" solver still gives you 403s — and the three approaches that actually work.

What Turnstile actually gives you

Turnstile replaces the old image CAPTCHA with an invisible check that combines proof-of-work with a JavaScript fingerprint. When it succeeds, the widget hands your page a token: a long opaque string. You submit that token with your form (usually in a hidden field, sometimes in a header), and Cloudflare's edge validates it.

Three properties matter more than anything else:

  1. The token is bound to the sitekey and the hostname. A token minted for example.com will not validate on www.example.com.
  2. It expires in 300 seconds. Turnstile tokens live for five minutes, and each one is single-use.
  3. Optional fields are part of the signature. If the page passes action or cdata to turnstile.render, your solution has to carry the same values or validation fails with no useful error message.

Most "the solver doesn't work" reports are one of those three.

Approach 1: drive a real browser

At low volume, Playwright (or Patchright, if you want the patched CDP layer) with a persistent profile is usually enough — the widget runs its check in a real JS environment and you simply read the token out of the DOM.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=False,
        args=["--disable-blink-features=AutomationControlled"],
    )
    ctx = browser.new_context(storage_state="state.json")
    page = ctx.new_page()
    page.goto("https://example.com/login")

    page.wait_for_function(
        "() => document.querySelector('[name=cf-turnstile-response]')?.value"
    )
    token = page.eval_on_selector(
        "[name=cf-turnstile-response]", "el => el.value"
    )
Enter fullscreen mode Exit fullscreen mode

The costs are real: a browser process per concurrency slot, 100–300 MB RSS each, and it breaks whenever the fingerprint check gets stricter. Fine for a handful of pages an hour, painful at thousands.

Approach 2: outsource the token

The second option keeps your scraper on plain HTTP and lets a service produce the token. SolveHub (disclosure: I build it) is one; CapSolver, 2Captcha and YesCaptcha work the same way. The shape is identical everywhere: create a task with the sitekey and page URL, poll until it completes, read the token.

from solvehub import SolveHub

client = SolveHub("sh_live_xxx")
task = client.solve("turnstile", {
    "websiteURL": "https://example.com/login",
    "websiteKey": "0x4AAAAAAA_SITE_KEY",
    "action": "login",          # only if the page sets one
})
token = task["solution"]["value"]
Enter fullscreen mode Exit fullscreen mode

Then post that token back to the page inside its five-minute window, from the same IP and User-Agent that fetched the page.

Two things break this in practice. First, free proxies: a token solved on a residential IP and replayed from a datacenter IP gets rejected. Second, queue latency — if your job sits in a queue for six minutes, the token is already dead.

For cookie-based challenges (DataDome, Akamai, PerimeterX, Imperva, AWS WAF) there is no token at all. You get cookies back, and those cookies are tied to the IP and User-Agent that solved them. Reuse all three together or the next request is challenged again.

task = client.solve(
    "datadome",
    {"websiteURL": "https://example.com", "captchaUrl": "https://example.com/captcha"},
    proxy="http://user:pass@host:port",
)
cookies = task["solution"]["cookies"]       # {"datadome": "..."}
user_agent = task["solution"]["userAgent"]
Enter fullscreen mode Exit fullscreen mode

Approach 3: don't fight the edge

Worth saying out loud: some sites publish exactly the data you want through an official API, a feed or a sitemap. If a product has a documented endpoint, use it — it is faster, stable, and nobody has to reverse-engineer anything. Only reach for the previous two approaches when there is genuinely no supported path.

A five-line checklist for 403s

Symptom Usual cause
403 after submitting a token token expired or already used
403 with a fresh token hostname / sitekey mismatch, or action + cdata not carried over
Works locally, fails in production different egress IP than the one that solved it
Challenge on every single request cookies not persisted between requests
Works for a while, then stops fingerprint change, or your UA rotating mid-session

Legal note

Everything above is for systems you own or have written permission to test. Solving a third party's anti-bot challenge can violate their terms of service, and Cloudflare itself treats automated solving as an attack signal. Get permission first, and put it in writing.

Related in this series

Top comments (0)