DEV Community: Lsr

Eight checks before you buy a CAPTCHA solver: the 403 debugging ladder

Lsr — Sun, 08 Nov 2026 07:20:52 +0000

You have a 403. The internet has two answers ready: buy a proxy, or buy a CAPTCHA solver. Both are right maybe half the time, and the wrong one costs you a month of engineering before you notice.

This is the ladder I run before spending money on either. Each step is cheaper than the one below it, and most 403s stop at step 2.

1. Read the response, do not guess what blocked you

A 403 from a WAF and a 403 from an application login check look identical in your exception handler and completely different in the headers:

curl -sS -D - -o /dev/null 'https://target.example/page' | head -40

cf-ray plus cf-mitigated: challenge — Cloudflare is challenging you, not the app.
x-datadome or a datadome cookie — DataDome.
_abck or bm_sz — Akamai Bot Manager.
_px3 or _pxhd — PerimeterX / HUMAN.
A 200 whose body is 2 KB of JavaScript and no content — that is a challenge page, not a success.

That single distinction decides the next seven steps. If the body is the real page and the app says no, no amount of solving will help you.

2. Is it the environment or the IP?

Run the same request from a different network — your laptop on home broadband is enough.

Works from home, fails from the datacenter: it is IP reputation, and a proxy is the right purchase.
Fails from both: it is you (fingerprint, headers, cookies) or it is the account. A proxy will only make it worse and more expensive.

3. Real browser or headless?

If a normal browser loads the page and Playwright does not, the difference is TLS/JA4, HTTP/2 settings or header order — not your IP, and not something a solver fixes. Compare your client fingerprint against the browser with any JA3/JA4 checker before spending on either.

Two reliable fixes here: run a real browser build with matching versions, or use a client that reproduces the browser fingerprint. Rotating proxies at this stage is the classic mistake — five hundred IPs behind one headless fingerprint is one user with five hundred hats.

4. If a token is required, which token?

Solver APIs are not interchangeable. A Turnstile token is useless on an Akamai page, and a datadome cookie is not a token at all — it is a session credential bound to the IP and User-Agent that received it. Identify the challenge from step 1, then ask for that specific thing.

Challenge	How you recognise it	Price per 1,000 solved
Cloudflare Turnstile	`cf-turnstile` widget, `0x4AAA...` sitekey	from $0.20
Cloudflare Interstitial	full-page check, no widget to click	$0.40
Akamai Bot Manager	`_abck`, `bm_sz` cookies	$0.80
PerimeterX / HUMAN	`_px3` cookie, `px-captcha` frame	$1.00
AWS WAF	`aws-waf-token`	$1.00
DataDome	`x-datadome` header, `datadome` cookie	$1.20
Imperva	`incap_ses_`, `visid_incap_`	$1.60
Arkose FunCaptcha	`arkoselabs` script, `funcaptcha` frame	$2.00

Prices above are from one vendor — mine, see the note at the bottom — and they are there to make the point that the spread is 10x. Paying FunCaptcha money for a Turnstile page is a silent 10x overrun.

5. Are you submitting the token correctly?

This is the most common self-inflicted 403, and it looks exactly like a solver that does not work:

sitekey and page must match. A token minted for example.com/login will not validate on example.com/checkout.
action and cdata must be carried over. If the page passes them, the token has to be minted with the same values.
Tokens are single-use and short-lived. Turnstile tokens are valid for about 300 seconds and one submission.
Post the field name the page expects. cf-turnstile-response, g-recaptcha-response, h-captcha-response — three different names, one wrong guess sends you back to the challenge.

6. For cookie challenges, identity must not move

DataDome, Akamai, PerimeterX, Imperva and AWS WAF hand you a cookie that is bound to the IP and User-Agent that earned it. Reusing it from a different proxy exit fails instantly, and the failure looks like the solver produced a bad value.

Rule: one identity per session — same proxy, same User-Agent, same cookie jar. Rotate identities between sessions, never inside one.

7. Check your own traffic shape before blaming the target

A scraper that opens 40 parallel requests against one host gets challenged regardless of how clean the requests are. Concurrency of 4 to 8 per host, a small delay, and reusing a session for a whole crawl will outperform any fingerprint trick. If your logs show a challenge on the first request every time, this is not a capacity problem — go back to step 3.

8. Do the unit economics on accepted answers, not attempts

This is the step almost nobody does, and it changes which vendor wins. If you pay per solved task, failed solves are free, so your real cost is:

cost per accepted answer = price per solved task / acceptance rate

A $0.20 solve that the site rejects adds a retry, a new session and possibly a raised risk score. Two retries at $0.20 plus a session reset can cost more than one $1.20 solve that lands first time. Price per 1,000 is a starting point; cost per accepted answer is the number that shows up in your invoice.

Where a solver actually fits

If steps 1 to 7 leave you with a page that is blocked purely because it wants a token or cookie you cannot produce headlessly, that is the moment to buy solving capacity — and to buy it by challenge type, not by volume. I run SolveHub, one REST endpoint for the eight challenge types in the table above, billed only on completed tasks, with the cookie challenges accepting your own proxy so the credential comes back usable.

Client code, if you want it: solvehub-python (stdlib only) and solvehub-node.

The two earlier posts in this series go deeper on the two most common cases: Turnstile tokens and expiry and why rotating proxies makes DataDome worse.

Disclosure: I build the solver API linked above. Everything in steps 1 to 7 applies whether you buy from me or from nobody.

Use only on systems you own or are authorised to test.

DataDome in Playwright: why rotating your proxy makes blocking worse

Lsr — Sun, 08 Nov 2026 07:10:18 +0000

You add a proxy pool. You rotate it every request. The blocks get more frequent, not less. Then you read that DataDome needs "better residential IPs", you buy a bigger pool, and the block rate climbs again.

I have spent a lot of time on this failure mode. The short version: rotation is not a mitigation for DataDome, it is an accelerator. Here is the mechanism, and the two approaches that actually hold up.

What DataDome is actually scoring

A CAPTCHA from DataDome is the visible output of a score computed before it, from signals that mostly have nothing to do with your IP:

TLS fingerprint (JA3/JA4) and HTTP/2 fingerprint — cipher order, extensions, SETTINGS frame, header order. Your proxy changes none of these. A real Chrome does.
Navigator and environment consistency — userAgent, platform, hardwareConcurrency, deviceMemory, WebGL renderer string, canvas hash, fonts, audio context.
Coherence between identity and network — a Europe/Paris timezone and fr-FR locale on an IP geolocated to a datacenter in Virginia is a signal on its own.
Behaviour before the challenge — mouse movement, scroll, keystroke timing, request cadence.

Now consider a pool of 500 IPs all driven by one headless browser build. From DataDome's side, that is not 500 users. It is one fingerprint wearing 500 hats, and it gets scored accordingly. A stable IP with a consistent, plausible fingerprint is usually safer than a fresh IP every request.

The cookie is not portable

When DataDome does let you through, it drops a datadome cookie. That cookie is bound to the identity that earned it — the IP it was issued to, the User-Agent sent with it, the TLS fingerprint of the client that used it.

So:

Symptom	What it usually means
Cookie works, then dies when you rotate	Cookie is bound to the original egress IP
Cookie dies only after you restart the browser	Bound to the fingerprint / session, not just IP
Every request returns the interstitial, instantly	Score already high from fingerprint reuse across many IPs
Works in curl, blocked in Playwright	Header order, HTTP/2 fingerprint, or missing browser-only headers
Works locally, blocked on the server	Timezone/locale vs IP mismatch, or a datacenter ASN

If you fix only the IP, you fix the symptom in row 1 and make rows 3 and 5 worse.

Approach 1 — one stable identity per worker

Instead of rotating, give each worker its own long-lived identity and never rotate mid-session:

One persistent browser context per worker (launch_persistent_context), so cookies and storage survive between pages.
One sticky proxy per worker for the life of that context — not per request.
Warm the session on a benign page first so there is real navigation history and interaction telemetry before you hit the protected URL.
Keep the browser build patched and consistent with the UA string you report.

This is slower to set up and uses more memory, which is exactly why rotation is popular — but it is the only shape that survives a persistent score.

Approach 2 — solve the interstitial out of band, reuse the cookie

If you do not need a full browser on every request, you can let a solver earn the cookie once and then replay it from your normal HTTP client. You submit the protected URL and the challenge URL, and get back the cookie plus the User-Agent that was used to obtain it.

from solvehub import SolveHub

client = SolveHub("sh_live_xxx")

task = client.solve(
    "datadome",
    {
        "websiteURL": "https://example.com",
        "captchaUrl": "https://example.com/captcha",
    },
    proxy="http://user:pass@host:port",   # the egress IP the cookie will be bound to
)

cookies    = task["solution"]["cookies"]      # {"datadome": "..."}
user_agent = task["solution"]["userAgent"]

Then the part everyone gets wrong:

session = requests.Session()
session.headers["User-Agent"] = user_agent        # must match, verbatim
session.cookies.update(cookies)                   # same cookie domain
r = session.get("https://example.com/target")     # from the SAME egress IP

Three rules that decide whether this works:

Same egress IP. The cookie was issued to one IP. Sending it from another is what makes people conclude "the solver does not work".
Byte-identical User-Agent. Not the same family — the same string, including the Chrome build suffix.
Re-solve when it expires. These cookies are short-lived by design. Cache per identity with a TTL, not forever.

Which one to choose

Full browser automation with JS-heavy pages and interactive flows: Approach 1.
High-volume read paths, or when Playwright is too expensive per request: Approach 2.
Both together: persist the context for navigation, refresh the cookie out of band when the score degrades.

The one change that helps in every case is to stop treating IP rotation as the answer. Fix the fingerprint, keep identities stable, and rotate identities — not IPs on a single identity.

Disclosure: I build SolveHub, a CAPTCHA API that handles DataDome, Cloudflare Turnstile and Interstitial, Akamai, PerimeterX, Imperva, AWS WAF, Arkose FunCaptcha and image OCR through a single endpoint. Use it only against systems you own or are authorized to test.

Related in this series

Turnstile tokens and expiry in Python: seencxy/cloudflare-turnstile-in-a-python-scraper-tokens-expiry-and-three-approaches-5592
Runnable client code: solvehub-python · solvehub-node

Cloudflare Turnstile in a Python scraper: tokens, expiry, and three approaches

Lsr — Sun, 08 Nov 2026 07:03:56 +0000

If you scrape a site that sits behind Cloudflare Turnstile, you have probably seen the same loop: the page loads, a small widget renders, and every request without a valid token comes back as a 403 with a challenge page.

This post is about the mechanics of that token — where it comes from, how long it lives, why a "working" solver still gives you 403s — and the three approaches that actually work.

What Turnstile actually gives you

Turnstile replaces the old image CAPTCHA with an invisible check that combines proof-of-work with a JavaScript fingerprint. When it succeeds, the widget hands your page a token: a long opaque string. You submit that token with your form (usually in a hidden field, sometimes in a header), and Cloudflare's edge validates it.

Three properties matter more than anything else:

The token is bound to the sitekey and the hostname. A token minted for example.com will not validate on www.example.com.
It expires in 300 seconds. Turnstile tokens live for five minutes, and each one is single-use.
Optional fields are part of the signature. If the page passes action or cdata to turnstile.render, your solution has to carry the same values or validation fails with no useful error message.

Most "the solver doesn't work" reports are one of those three.

Approach 1: drive a real browser

At low volume, Playwright (or Patchright, if you want the patched CDP layer) with a persistent profile is usually enough — the widget runs its check in a real JS environment and you simply read the token out of the DOM.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=False,
        args=["--disable-blink-features=AutomationControlled"],
    )
    ctx = browser.new_context(storage_state="state.json")
    page = ctx.new_page()
    page.goto("https://example.com/login")

    page.wait_for_function(
        "() => document.querySelector('[name=cf-turnstile-response]')?.value"
    )
    token = page.eval_on_selector(
        "[name=cf-turnstile-response]", "el => el.value"
    )

The costs are real: a browser process per concurrency slot, 100–300 MB RSS each, and it breaks whenever the fingerprint check gets stricter. Fine for a handful of pages an hour, painful at thousands.

Approach 2: outsource the token

The second option keeps your scraper on plain HTTP and lets a service produce the token. SolveHub (disclosure: I build it) is one; CapSolver, 2Captcha and YesCaptcha work the same way. The shape is identical everywhere: create a task with the sitekey and page URL, poll until it completes, read the token.

from solvehub import SolveHub

client = SolveHub("sh_live_xxx")
task = client.solve("turnstile", {
    "websiteURL": "https://example.com/login",
    "websiteKey": "0x4AAAAAAA_SITE_KEY",
    "action": "login",          # only if the page sets one
})
token = task["solution"]["value"]

Then post that token back to the page inside its five-minute window, from the same IP and User-Agent that fetched the page.

Two things break this in practice. First, free proxies: a token solved on a residential IP and replayed from a datacenter IP gets rejected. Second, queue latency — if your job sits in a queue for six minutes, the token is already dead.

For cookie-based challenges (DataDome, Akamai, PerimeterX, Imperva, AWS WAF) there is no token at all. You get cookies back, and those cookies are tied to the IP and User-Agent that solved them. Reuse all three together or the next request is challenged again.

task = client.solve(
    "datadome",
    {"websiteURL": "https://example.com", "captchaUrl": "https://example.com/captcha"},
    proxy="http://user:pass@host:port",
)
cookies = task["solution"]["cookies"]       # {"datadome": "..."}
user_agent = task["solution"]["userAgent"]

Approach 3: don't fight the edge

Worth saying out loud: some sites publish exactly the data you want through an official API, a feed or a sitemap. If a product has a documented endpoint, use it — it is faster, stable, and nobody has to reverse-engineer anything. Only reach for the previous two approaches when there is genuinely no supported path.

A five-line checklist for 403s

Symptom	Usual cause
403 after submitting a token	token expired or already used
403 with a fresh token	hostname / sitekey mismatch, or `action` + `cdata` not carried over
Works locally, fails in production	different egress IP than the one that solved it
Challenge on every single request	cookies not persisted between requests
Works for a while, then stops	fingerprint change, or your UA rotating mid-session

Legal note

Everything above is for systems you own or have written permission to test. Solving a third party's anti-bot challenge can violate their terms of service, and Cloudflare itself treats automated solving as an attack signal. Get permission first, and put it in writing.

Related in this series

Why rotating proxies makes DataDome blocking worse: seencxy/datadome-in-playwright-why-rotating-your-proxy-makes-blocking-worse-2259
Runnable client code: solvehub-python · solvehub-node