Originally published on the Devil Scrapes blog.
Quick answer: Why does my scraper get blocked? Usually not because your parser is wrong — it's because the request itself doesn't look like it came from a browser. A plain HTTP client sends a different handshake than Chrome, reuses an IP with no browsing history behind it, drops session continuity between pages, and ignores the target's own rate-limit signals. Any one of those is enough to get flagged; most scrapers hit two or three at once.
It's rarely the parser
When a scraper "stops working," the instinct is to check the CSS selector or the JSON path first. That's usually not it. If a run comes back with a splash page, a stripped-down HTML shell, or a response that doesn't match what a browser would see, the request never made it past the target's front door as a trusted client. The parser was never given real data to parse. Here's what actually decides whether a request gets treated as a browser or a bot.
TLS and HTTP fingerprints
Every HTTP client — Python's requests, Node's fetch, curl — negotiates its TLS handshake and orders its HTTP headers in a way that's specific to that library, not to the browser it's pretending to be. Real Chrome, Firefox, and Safari each send a distinctive cipher-suite order, extension list, and header casing. A target that inspects the handshake (plenty do, quietly, without ever showing a challenge page) can tell "Python script" from "Chrome tab" before your request body is even read, independent of your User-Agent string.
We rotate through real Chrome, Firefox, and Safari handshake profiles per request, so the target's edge sees a browser-shaped connection rather than a library's default fingerprint.
IP reputation: residential vs. datacenter
The exit IP carries its own trust score, built from a history of prior traffic. A datacenter IP block that's been used by a thousand scrapers before yours starts with a reputation deficit, no matter how clean your request looks. A residential IP — an exit that looks like an ordinary home connection — usually starts closer to neutral. Neither is a silver bullet: datacenter exits are faster and cheaper, and they clear plenty of targets fine, so the right call is target-by-target, not a blanket "always residential" rule. We rotate through both residential and datacenter proxy pools and pick per target based on what actually clears.
Session and cookie continuity
Pagination on modern sites is usually bound to a session — a cursor, a cookie, a token issued on page one that page two expects back. Rotate your exit IP mid-session without carrying that session forward, and the target doesn't see "a new visitor," it sees a broken client and invalidates the cursor. We hold a sticky session — the same exit IP and cookie jar — for the length of a paginated crawl, and only rotate to a fresh session when a page actually needs one.
Rate limits: Retry-After is not a suggestion
Most targets that rate-limit tell you exactly how long to wait, in the Retry-After header on a 429 or 503 response. Ignoring it and retrying immediately is one of the fastest ways to escalate a soft rate limit into a hard IP ban. We back off exponentially on 408, 429, and 5xx responses and honor Retry-After when the target sends one, capped at a handful of attempts per page so a stuck request doesn't run forever.
The trap that doesn't look like a block at all
The failure mode that cost us the most debugging time wasn't a 403 — it was a 200 OK. On a geo-sensitive target, a residential exit in the wrong country doesn't get rejected. It gets served a normal-looking page: prices in the wrong currency, a country-selector splash screen instead of the product page, or a locale variant that silently doesn't match what was requested. There's no error, no challenge, nothing a naive status-code check would ever flag — just data that's confidently wrong sitting next to rows that are correct.
We pin the exit country on every geo-sensitive target instead of letting the proxy pick one at random, and we added a guard that treats an unexpected currency symbol or a country-selector marker as a failed fetch to retry — not a result to parse and ship.
What this looks like for you
Every one of the mechanics above is a maintenance burden if you own it yourself: fingerprint drift as browsers update, proxy pool churn, session bugs that only show up at scale, rate-limit tuning per target, and geo bugs that pass every test except the one that matters. We absorb all of it inside the Actor, so what lands in your dataset is rows, not a debugging session at 3 a.m.
Run it on Apify:
FAQ
Why does my scraper get blocked even with a correct User-Agent header?
Because the User-Agent string is just one header among many. The TLS handshake and the full HTTP header order and casing are usually what a target's edge actually inspects, and a plain HTTP client sends a library-specific fingerprint there regardless of what User-Agent you set on top of it.
Is a residential proxy always better than a datacenter proxy?
No — it depends on the target. Residential exits generally carry a better starting trust score, but datacenter exits are faster, cheaper, and clear plenty of targets without issue. The right choice is per-target, decided by what actually gets through, not a fixed rule.
Why did my scraper get a 200 response with wrong data instead of an error?
Geo-sensitive targets often don't reject a foreign exit outright — they serve a normal-looking response in the wrong locale: wrong currency, a country-selector page, or a mismatched variant. It looks like success unless you specifically check the content against what's expected for the country you asked for.
What does honoring Retry-After actually prevent?
It prevents a soft, temporary rate limit from escalating into a hard IP ban. Retrying immediately after a 429 tells the target your client isn't respecting its limits, which is exactly the pattern that gets an IP or session blacklisted rather than just throttled.
Can session continuity really break a scrape on its own?
Yes. If a target's pagination cursor is bound to a session cookie and you rotate the exit IP between pages without carrying the session forward, the target sees an inconsistent client and drops the cursor — the crawl stalls or restarts from page one, not because the parser broke, but because the session did.
Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.
Actors mentioned in this post:
Top comments (0)