The web still returns a 200 for almost everything. That is not the same as being reachable reliably, on a schedule, at scale. Here are the six barriers that sit between a request and the data in 2026, why they are multiplicative rather than additive, and how to work out which tier of infrastructure a given site is actually forcing on you.
If you have shipped a crawler this year, you already know the feeling: the code that worked in a notebook against ten URLs falls over the moment it runs continuously against ten thousand. Nothing in your parser changed. What changed is that the modern web is built and defended for a world where more than half of traffic is automated, and the defences do not distinguish your legitimate data pipeline from the bad bots they were bought to stop.
That "more than half" is not a figure of speech. PromptCloud's State of Web Access 2026 report puts automated traffic at 53% of the total, with bad bots alone accounting for 40% (source: Thales's 2026 Bad Bot Report). Roughly a fifth of all websites sit behind a single vendor's infrastructure, which means access policy for a large slice of the web is now set in a handful of places. For anyone building data collection, the practical question has shifted. Reach is mostly solved. Reliable, repeatable reach is the engineering problem, and it is defined by six barriers.
The six barriers, from most to least common
Web application firewalls (WAF). The most common barrier by far, and the one you rarely chose to fight, because it is bundled into the CDN or hosting the site already uses. A WAF inspects requests at the edge and drops the ones that look wrong before your code ever touches application logic. It is always-on and it is everywhere.
Anti-bot / bot management. A layer above the WAF that scores behaviour rather than individual requests. It watches request cadence, header consistency, session shape, and the small tells that separate a browser driven by a human from one driven by a script. Its defining property is that it learns: it builds a signature for a recurring scrape and gets better at recognising yours specifically over time. A pattern that works today can degrade on its own next week with no change on your side.
CAPTCHA. Selective rather than blanket, concentrated on logins, checkouts, and forms, which is to say on exactly the high-value actions. It is best thought of as a continuous line item rather than a one-off obstacle, because it reappears wherever the value is highest.
JavaScript rendering. Not a defence at all, just how a large share of the web is now built. The initial HTML is a shell and the content assembles in the browser, which forces you to run browser infrastructure to see what a user sees. It costs you regardless of anyone's intent to block you, and it is the barrier most likely to quietly dominate your compute bill.
Rate limiting. The bluntest instrument in the set. Cross a request threshold and you get a 403 or a 429, often on the very next request. It is trivial to implement, which is exactly why it is common, and it turns "fetch everything quickly" into a paced, scheduled problem.
TLS fingerprinting. The rarest and the hardest to diagnose, because it rejects you during the TLS handshake, before a single HTTP request is logged. Your client's TLS signature does not match a real browser's, and the connection dies before you get a status code to debug. When a target "works in curl but not in your stack, and there is no error you can read," this is the usual suspect.
Why they stack, and why that is the whole problem
The trap is treating these as a checklist you clear once. In practice they combine, and the effect is multiplicative, not additive. The report frames access difficulty as tiers of required infrastructure, and the model is worth internalising because it tells you what a site is actually costing you:
- Tier 1 is a plain HTTP request. A requests.get and a parser is enough.
- Tier 2 is a headless browser behind a firewall: you need to render JavaScript and present as a real client to get past edge filtering.
- Tier 3 adds rotating IPs, paced scheduling, and TLS parity: you are now managing identity and timing, not just fetching.
- Tier 4 is specialist territory, where dedicated anti-bot systems require dedicated countermeasures and constant upkeep.
Two things make this hard to plan around. First, each barrier you add is an independent way for the pipeline to break and a separate thing to monitor, so a Tier 3 target is not "a bit harder" than Tier 1, it is a qualitatively different operational commitment. Second, any site can move up a tier at any time, without warning, because the barrier is usually a configuration change on infrastructure the site owner does not even operate themselves. You can go to bed at Tier 2 and wake up at Tier 3.
And there is a pattern to which sites sit where. The data that matters most and the data that is hardest to reach reliably tend to be the same data. Competitor pricing, live inventory, high-demand listings: these are precisely the pages their owners have the most commercial incentive to defend, so difficulty tracks value almost by definition. The easy targets are easy because nobody minds.
Diagnosing your tier
Before you reach for heavier infrastructure, work out what you are actually up against, because the wrong diagnosis is expensive in both directions. A rough field procedure:
1. Plain GET with an honest client.
200 with your data in the body? -> Tier 1. Stop. Do not add machinery.
200 but the body is an empty shell? -> JS rendering. Move toward Tier 2.
403 / 429 immediately? -> rate limiting or WAF.
Connection fails before any HTTP status? -> suspect TLS fingerprinting (Tier 3+).
2. Add a real headless browser (proper TLS, real headers, JS execution).
Works now? -> Tier 2. Budget for browser infra.
Still blocked after N clean requests,
then blocked on a pattern? -> behavioural bot management is learning you. Tier 3+.
3. Watch it over days, not minutes.
Success rate decays on a fixed schedule/volume? -> you are being profiled. Plan for IP rotation,
pacing, and continuous monitoring, not a one-off fix.
The point of the exercise is restraint as much as escalation. Plenty of teams run Tier 3 machinery against Tier 1 sites because a single 403 spooked them, and plenty run Tier 1 code against Tier 3 sites and call the source "flaky." Match the infrastructure to the tier, and re-check the tier periodically, because it moves.
What this means for how you build
The honest takeaway is that a crawler is not a script you finish, it is a system you keep alive. The status code lies: a 200 tells you the request completed, not that the data behind it is complete, current, or reachable again tomorrow at volume. Budget for the barriers you will actually hit, instrument every one of them so a silent tier change surfaces as an alert rather than a gap in your data, and be ruthless about not over-building against sites that never needed it. The web has not closed. It has just made reliable access an engineering discipline with a running cost, and the first step to managing that cost is naming the six things that create it.
FAQ
Why does my scraper work locally but fail at scale?
Because the barriers that matter are triggered by volume and pattern, not by a single request. A handful of requests from your laptop looks like a human; thousands of paced, structurally identical requests from the same identity looks like exactly what it is, and trips rate limits and behavioural bot management that a small test never reaches. Anti-bot systems specifically learn the signature of a recurring scrape over time, so a pattern that passes today can degrade on its own later with no change to your code.
What is TLS fingerprinting and why is it so hard to debug?
TLS fingerprinting identifies your client from the characteristics of its TLS handshake, the way it negotiates the encrypted connection, and rejects it if that signature does not match a real browser. It is hard to diagnose because the rejection happens before any HTTP request is processed, so you never get a status code or an error body to inspect: the connection simply fails. The classic tell is a target that behaves in one client but silently dies in another with no readable error, which usually means you are being blocked at the handshake rather than at the application layer.
Top comments (0)