How to tell which anti-bot a website uses (and fetch the page anyway)

If you scrape the web, you know the wall: a 403, a 429, or a "Just a moment…" page that never resolves. Before you sink a day into building a scraper, two questions matter:

1. Which anti-bot is protecting this site? (Cloudflare, DataDome, PerimeterX/HUMAN, Akamai, Imperva…)
2. Can I get the page anyway?

Here's a way to answer both in one call.

Detecting the anti-bot vendor

Each vendor leaves fingerprints in the response headers and body:

Vendor	Tell-tale signs
Cloudflare	cf-ray header, Server: cloudflare, "Just a moment", challenge-platform
DataDome	x-datadome header, datadome cookie, captcha-delivery.com
PerimeterX / HUMAN	_px* cookies, px-captcha
Akamai Bot Manager	_abck / ak_bmsc cookies
Imperva / Incapsula	incap_ses cookie, x-iinfo header

If you just want the answer without wiring this up yourself, the Web Unblocker actor returns a protectionDetected field for any URL.

Fetching the page anyway

The strategy that works from cloud IPs:

1. Fast path — an HTTP request with real Chrome TLS/JA3 impersonation (curl_cffi). Cheap, instant for unprotected pages.
2. Browser render — if that's blocked, render in a hardened browser (Camoufox — a fingerprint-hardened Firefox) which clears most JavaScript challenges.
3. Retry on a fresh IP — if it's still blocked, rotate to a new residential IP and try again.

One call, both answers

{
  "startUrls": [{ "url": "https://example.com/protected" }],
  "renderJs": "auto",
  "outputFormat": "html",
  "maxRetries": 2,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

You get back the final HTML (or clean Markdown, or a screenshot), the HTTP status, the anti-bot vendor detected, and a bypassed flag telling you whether the real page came through:

{
  "url": "https://example.com/protected",
  "success": true,
  "statusCode": 200,
  "protectionDetected": "Cloudflare",
  "bypassed": true,
  "method": "http"
}

When to use it

• You keep getting 403/429 on a target and want the HTML to parse yourself.
• You want to know what a site runs before committing to a scraper build.
• You need protected pages as clean Markdown for a RAG/LLM pipeline.

Try it here: Web Unblocker on Apify. Only public pages, no logins — respect each site's terms.