Originally published at https://lurkapi.com/blog/scraper-403-forbidden
If Chrome opens a page and your Python script gets 403 Forbidden, it’s usually a bot wall in front of the site that refused you: Cloudflare, DataDome, Akamai, PerimeterX or AWS WAF. Walls judge how your client connects (the TLS handshake, the HTTP version and its settings, the order of your headers) and compare that with the name in your User-Agent. requests gets every one of those wrong for a browser, which is why pasting in Chrome’s User-Agent so rarely helps.
The 403 also says which wall sent it and whether it’s a block or a challenge, and each has a different fix.
Send Chrome’s handshake, not just its User-Agent
requests introduces itself as python-requests/ plus a version number, so swapping the User-Agent feels like the obvious fix. Put Chrome’s User-Agent on a plain HTTP client, though, and Kickstarter and Priceline still answer 403 with the same block page as before. The rest of the request still says “script”: requests negotiates TLS through OpenSSL with its own ciphers and extensions, and it only speaks HTTP/1.1, where Chrome talks to these sites over HTTP/2. Cloudflare calls the result a JA3 or JA4 fingerprint, and a Chrome name over a non-Chrome fingerprint is an easy catch.
The same request three ways. Chrome’s User-Agent alone still gets a 403 from Kickstarter and Priceline; Chrome’s handshake gets both pages and six more, and IMDb answers it with a JavaScript challenge (a 202).
Send the same request with Chrome’s handshake, HTTP/2 settings and header order, and both pages load, and so do six more walled sites. In Python, curl_cffi gives you that handshake with one argument:
curl_cffi · Python
from curl_cffi import requests
r = requests.get(url,
impersonate="chrome")
Read the 403: each wall signs its response
Each vendor marks its response:
| Wall | Look for | What it is |
|---|---|---|
| Cloudflare |
cf-mitigated: challenge and a “Just a moment...” page. Without that header, server: cloudflare and cf-ray over a 403 page (error 1020, or the site’s own) |
A challenge with the header, a block without it |
| DataDome |
x-datadome-cid header, datadome cookie, a page of about 780 bytes that loads captcha-delivery.com
|
A block, often sent as a 401 |
| Akamai |
server: AkamaiGHost, an “Access Denied” page with “Reference #18…” |
A block, sometimes sent with a 200 |
| PerimeterX (HUMAN) |
_pxhd cookie, window._pxAppId and px-captcha in the page, “Press & Hold” |
A block page with a press-and-hold challenge |
| AWS WAF |
x-amzn-waf-action: challenge on a 202. On CloudFront, a 403 that says “Request blocked.” |
A challenge (202), captcha (405) or block (403) |
| No vendor marks | A bare “403 Forbidden” with no vendor headers or cookies | An IP or User-Agent rule, or the site itself |
Check the headers first, then the cookies, then the page. The first match names the wall.
Don’t read server: cloudflare as a Cloudflare block: Priceline’s 403 travels through Cloudflare, but the page is PerimeterX’s. And Akamai’s _abck and bm_sz cookies show up on pages that load fine; they mean Akamai is there, not that you’re blocked.
To check in code, pass the status, headers and body to this function. It looks for the challenge headers and Akamai’s page before it looks at the status, because those can arrive with a 202 or even a 200.
which_wall() · Python
import re
import requests
AKAMAI_PAGE = re.compile(
r"errors\.edgesuite\.net|Reference #18")
def which_wall(status, headers, body):
h = {k.lower(): v for k, v in headers.items()}
via = h.get("via", "").lower()
if h.get("cf-mitigated") == "challenge":
return "Cloudflare challenge"
if h.get("x-amzn-waf-action"):
return f"AWS WAF {h['x-amzn-waf-action']}"
if AKAMAI_PAGE.search(body):
return "Akamai block"
if status not in (401, 403, 429, 503):
return None
if "x-datadome-cid" in h or "captcha-delivery.com" in body:
return "DataDome block"
if "px-captcha" in body or "_pxAppId" in body:
return "PerimeterX block"
if h.get("server", "").lower() == "akamaighost":
return "Akamai block"
if "cf-ray" in h:
return f"Cloudflare {status}"
if "Request blocked" in body and "cloudfront" in via:
return "AWS WAF block"
return f"Unbranded {status}"
r = requests.get(url) # your request
print(r.status_code, which_wall(r.status_code, r.headers, r.text))
In axios, a 403 throws AxiosError: Request failed with status code 403, and the same three inputs are err.response.status, err.response.headers and err.response.data.
What each wall’s block page looks like
kickstarter.com · Cloudflare challenge
HTTP/2 403
server: cloudflare
cf-mitigated: challenge
cf-ray: a480d8d37e259aac-IAD
set-cookie: __cf_bm=…
<title>Just a moment...</title>
<noscript>… Enable JavaScript and cookies to continue …</noscript>
<script>(function(){window._cf_chl_opt = {… cType: 'managed', …};
… a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/…'; …
marketwatch.com · DataDome, as a 401
HTTP/2 401
server: CloudFront
x-datadome: protected
x-datadome-cid: …
x-dd-b: 2
set-cookie: datadome=…; Max-Age=31536000; Domain=.marketwatch.com; …
<title>marketwatch.com</title>
<p id="cmsg">Please enable JS and disable any ad blocker</p>
<script>var dd={'rt':'c','cid':'…','hsh':'…','t':'bv',…,
'host':'geo.captcha-delivery.com','cookie':'…'}</script>
<script src="https://ct.captcha-delivery.com/c.js"></script>
sephora.com · Akamai
HTTP/2 403
server: AkamaiGHost
akamai-grn: 0.c424c317.1791585316.3a9f823
<TITLE>Access Denied</TITLE>
<H1>Access Denied</H1>
You don't have permission to access "http://www.sephora.com/"
on this server.<P>
Reference #18.c424c317.1791585316.3a9f823
<P>https://errors.edgesuite.net/18.c424c317.1791585316.3a9f823</P>
priceline.com · PerimeterX through Cloudflare
HTTP/2 403
server: cloudflare
cf-cache-status: DYNAMIC
set-cookie: _pxhd=…
<meta name="description" content="px-captcha">
<title>Access to this page has been denied</title>
window._pxAppId = 'PX9aTjSd0n';
script.src = 'https://captcha.px-cloud.net/PX9aTjSd0n/captcha.js?…';
Press & Hold to confirm you are<br>a human (and not a bot).
imdb.com · AWS WAF challenge, as a 202
HTTP/2 202
x-amzn-waf-action: challenge
via: 1.1 ….cloudfront.net (CloudFront)
<title></title>
window.gokuProps = { "key":"…", "iv":"…", "context":"…" };
<script src="https://….token.awswaf.com/…/challenge.js"></script>
AwsWafIntegration.getToken().then(() => { window.location.reload(true); });
Some blocks don’t look like errors
raise_for_status() only fires on a 4xx or 5xx, and axios only throws outside 2xx. Walls don’t always answer that way:
- Lowe’s sends Akamai’s Access Denied page with a 200. A scraper that trusts the status saves the block page as data.
- IMDb’s AWS WAF challenge, which it sends once your handshake looks like Chrome’s, comes back as a 202. Both libraries treat that as success.
- MarketWatch’s DataDome block is a 401, which reads like a login problem.
- Best Buy doesn’t answer at all. It drops the connection, so you get a connection error instead of a 403.
- Kickstarter gives Chrome’s handshake a 200, but its homepage is a JavaScript app: the content appears only once a browser runs it.
So run the check on every response before you parse it, 200s included, and always set a timeout.
A block needs a better client, a challenge needs a browser
A block refuses your client or your IP. Fix the client first with Chrome’s handshake. If the block survives that, the wall is judging your IP (DataDome scores IP reputation alongside the client), and the fix is a different network.
A challenge is a page that runs JavaScript to check that a browser is there. A good handshake often avoids it: Kickstarter challenges a plain client and serves Chrome’s handshake the page. When the challenge comes anyway, as IMDb’s does, only a browser that runs the script gets through, which means Playwright or Puppeteer and a Chrome to keep running.
Or let one call climb for you
LurkAPI’s scrape endpoint climbs that ladder for you: a quick direct request, then Chrome’s handshake from a static ISP IP, then a browser, then other networks for walls that judge the IP. It stops at the first step that gets the page and charges for that step: 1 credit up to the browser, and more only for the other networks. A call that every step fails costs nothing.
Three URLs, one endpoint. Sephora opens on the first, cheapest step and Priceline on Chrome’s handshake (1 credit each); Chrome’s handshake gets only Kickstarter’s JavaScript shell, so it goes on to the browser (1 credit). 3 credits for the three pages.
Priceline needed one step up:
Scrape Priceline’s homepage · 1 credit
curl -G "https://api.lurkapi.com/v1/web/scrape" \
--data-urlencode "url=https://www.priceline.com/" \
-d format=markdown \
-H "x-api-key: $LURK_KEY"
Response · trimmed
{
"success": true,
"credits_charged": 1,
"status": 200,
"url": "https://www.priceline.com/",
"content": "Skip to main content\n…\n- [Hotels](https://www.priceline.com/hotels)\n- [Cars](…",
"via": "static",
"attempts": [
{ "via": "direct", "ms": 116, "status": 403,
"wall": "perimeterx challenge" },
{ "via": "static", "ms": 565, "status": 200,
"wall": null }
],
"billing": { "mode": "auto", "credits": 1,
"breakdown": { "base": 1 } }
}
attempts lists every step and the wall it met, in plain labels like perimeterx challenge or datadome block, so you can see why a page cost what it did.
Kickstarter went one step further: Chrome’s handshake got a 200, the endpoint saw a page that JavaScript still had to build, and the browser came back with the full page in about 7 seconds for 1 credit. Most pages take 1 to 3 seconds and browser renders 5 to 15, so give the call a 90-second timeout for the slow ones. The web scraping guide lists every step and its price.
Paste the URL that’s giving you the 403 and see which step gets it:
Run GET /v1/web/scrape in the live playground
Questions
Why do I get 403 Forbidden with Python requests but not in the browser?
A bot wall in front of the site saw a client that isn’t a browser. The User-Agent says python-requests, and the TLS handshake and HTTP version don’t match Chrome’s. The response’s headers, cookies or page say which wall it was.
AxiosError: Request failed with status code 403
Axios throws on any status outside 2xx. Read err.response.headers and err.response.data to see who sent it. On a page Chrome opens fine, it’s usually a bot wall reacting to the axios/ User-Agent and Node’s non-browser handshake; on an API URL, usually a missing or wrong token.
Does changing the User-Agent fix a 403?
Rarely, on a site behind a bot wall. Kickstarter and Priceline still answer 403 to a plain client sending Chrome’s User-Agent, and serve the page once the request also has Chrome’s TLS handshake and HTTP/2 settings.
What is cf-mitigated: challenge?
A Cloudflare response header that marks a challenge page. The page clears only in a browser that runs Cloudflare’s JavaScript, which then gets a cf_clearance cookie.
How do I tell if a site uses PerimeterX?
Look for a _pxhd or _px3 cookie, window._pxAppId in the HTML, or a “Press & Hold” page that loads from px-cloud.net. PerimeterX is now called HUMAN.
Top comments (1)
quick add from the proxy side, since it trips people up. a 403 can come from your proxy too, not just the wall. curl -v shows you: if the CONNECT itself gets the 403, that's the proxy refusing the port or host and no fingerprint change fixes it. if you see 200 Connection established first, the 403 is the site and everything in your post applies.
(we run a proxy service, so we're biased toward checking this first)