Zyte, where I work, has just published The State of Web Access, an audit of 24,898 popular landing pages that measures which technical barriers sites put in front of automated visitors. It is free, it is 55 pages, and the underlying data is browsable in a data explorer broken down by barrier, industry and country. I want to share it here because most of the writing about "the web is closing" is anecdote, and this is the first dataset I have seen that is big enough to argue with.
This post is my read of it from the point of view of someone who writes and debugs scrapers. I will go through the method first, then the six barriers with the numbers, and along the way I will add context from other sources, because the report measures adoption and does not always explain the mechanism underneath. Where I disagree with the report's framing, or think a number needs a caveat, I say so.
How it was measured
Numbers without a method are marketing, so here is the method.
The sample comes from Semrush's top website lists. The global list gives 11,100 landing page records, which is the base for the global percentages. A regional list adds 22,137 unique landing pages, snapshotted in October 2025, sampled across countries and categories. Deduplicated, that is 24,898 sites across 230 countries, mapped into 110 industries.
For each landing page the audit checked six barriers, each detected a different way:
- Web Application Firewall (WAF). HTTP responses and server headers matched against the fingerprint database from wafw00f 2.3.2.
- Antibot services. Vendor fingerprints in scripts, request patterns and endpoints that appear when a commercial bot mitigation product is active.
- CAPTCHA. HTML snippets, script loads and network calls tied to known providers. This detects the infrastructure, not whether a challenge was actually served to the scanner.
- JavaScript dependence. A plain HTTP fetch compared against a fully rendered page. If rendering grew the meaningful HTML by more than 50%, the page counts as JavaScript-dependent.
- Rate limiting. A short burst of requests at mixed pacing, recording whether and how fast the site started rejecting them.
- TLS fingerprinting. Inspecting how the site responds to the TLS handshake, to see whether it checks that the client looks like a standard browser before any HTTP request is processed.
robots.txt was fetched and parsed separately as a policy signal rather than a barrier. Scanning ran on Scrapy Cloud with a modified version of caniscrape 0.1.8 and Zyte's internal tier API.
The report is upfront about the limits, and they matter. Everything is landing pages, and plenty of sites leave the homepage open for SEO while defending search results and product pages far harder, so treat every figure here as a floor. "Present" means "detectable via the fingerprints we scanned for", so custom or niche systems are undercounted. And the regional dataset is the pages most visited from a country, not the pages of sites based there.
The headline numbers
| Barrier | Share of 11,100 landing pages |
|---|---|
| WAF (named or generic) | 92.4% |
| JavaScript-dependent | 40.6% |
| CAPTCHA infrastructure | 22.5% |
| Rate limiting | 22.1% |
| Dedicated antibot | 18.5% |
| TLS fingerprinting | 13.8% |
| No detectable barrier | 18.5% |
Half of all pages run two or more barriers. The distribution by count is 18.5% none, 30.7% one, 25% two, 12.1% three, 5.4% four, 7.4% five and 0.8% all six. That 0.8% sits almost entirely in high-value commercial categories.
The "18.5% open" figure is the one that will get quoted, and I think the more useful one is the shape of that distribution. Most of the defended web is not hardened, it is layered, and each layer is cheap for the site and adds a different failure mode for you.
WAF: the default nobody chose
92.4% of pages sit behind a WAF, but only 63.5% run one that fingerprints to a named vendor, and the remaining 29% match generic WAF signatures. Cloudflare is 34.6% of all pages, generic or unidentified 31.6%, Amazon CloudFront 9.6%, Akamai Kona 4.9%, AWS Elastic Load Balancer 4.3%, and then a long tail of 13 more. The top three vendors hold 71.7%.
The detail I find more interesting is that 61.9% of sites run a single WAF vendor in standard mode, and only 1.4% run multiple vendors or behavioural analysis. The WAF arrives bundled with the CDN and most operators never touch the defaults. 18% of all URLs in the audit rely on a WAF and nothing else.
For context, W3Techs as of 3 September 2026 puts Cloudflare in front of 25.0% of all websites in its 10 million site crawl, which is 84.6% of sites with an identifiable reverse proxy. So among popular sites the market is actually less concentrated than across the whole web, but the same vendor is at the top either way, and the antibot market below is even more lopsided.
Antibot: one vendor decides what a browser is
18.5% of landing pages run a dedicated bot management product, and these are the systems that actually look at behaviour, mouse movement, keyboard events, canvas rendering, and browser attribute consistency, rather than just headers and IP reputation.
Cloudflare Bot Management is 75.9% of those deployments. Imperva is 9.1%, Akamai Bot Manager 7.7%, PerimeterX (now HUMAN) 5%, DataDome 2%. The report calls this a one-horse race, and it is, and I would put the implication more bluntly than the report does: for three quarters of the behaviourally defended web, one company's model of what a real browser looks like is the definition of a real browser. When that model changes, everyone's scrapers break on the same day.
Adoption by industry is what you would guess. Adult content 36%, furniture 33%, gambling 31%, beauty, computer hardware and tobacco at 30%. These are sectors with checkout flows, inventory and account fraud, so the antibot is protecting revenue, not content.
CAPTCHA: two vendors, and it skews small
22.5% of pages have CAPTCHA infrastructure. reCAPTCHA is 49.4% of that, Cloudflare Turnstile 39.4%, GeeTest 7.7%, hCaptcha 1.9%, Arkose FunCaptcha 1.5%. Two vendors cover 88.8% of CAPTCHA-enabled sites.
Two things worth knowing. First, a lot of this is invisible, reCAPTCHA v3 and Turnstile score you in the background and only escalate when the score is bad, so a "CAPTCHA present" site may never show you a puzzle if the rest of your fingerprint holds up. Second, CAPTCHA adoption is inversely correlated with traffic. Lower-traffic sites are more likely to use one than high-traffic sites, because a CAPTCHA is the cheapest defence to bolt on. Tobacco leads adoption at 52%, which the report attributes to age verification requirements rather than anti-scraping.
JavaScript: 141 KB of page that HTTP never sees
40.6% of landing pages need JavaScript to produce meaningful content, by the 50% growth definition above. On those pages, rendering adds 140,923 bytes on average, which is 75% of the final HTML payload. Restaurants and airlines lead at 66%, travel at 65%, pharmaceuticals and sports at 64%.
The report is careful to say this is mostly architecture, not defence, and I agree. Single page application frameworks won, so the HTML you get over plain HTTP is a shell. Do not confuse this figure with the HTTP Archive Web Almanac number of roughly 697 KB of JavaScript on the median home page, that is script bytes downloaded, this is DOM added by executing them, they measure different things and both point the same way.
There is a defensive correlation though. Sites that require JavaScript are more likely to also run a WAF and rate limiting, and high-traffic sites are more likely to require it. Your three options are still the same as they were five years ago: render in a browser, find the internal API the page calls and hit that instead, or accept partial data. The second is still the best when it works.
Rate limiting: it is really blocking, and it is really 403
22.1% of pages throttle bursts. What surprised me is how they do it. Of the sites that rate limit, 94% block after a single request in the burst pattern and 5% after a burst of eight. That is not rate limiting in any RFC sense, it is bot detection that happens to fire on request cadence.
The status codes confirm it. The first blocking response is 403 Forbidden on 92.6% of rate-limiting sites, 429 Too Many Requests on 5.7%, 503 on 1.2%, 401 on 0.4%. RFC 6585 has defined 429 since 2012, along with a Retry-After header telling you how long to back off, and almost nobody uses it.
This has a direct consequence for how you write retry logic. If your client backs off on 429 and gives up on 403, you are giving up on the majority of rate limits. A 403 from a WAF rule, a 403 from a rate limiter, and a 403 from an antibot verdict are indistinguishable from the status code alone, so you need to classify by body, headers and timing, and log which layer you think produced it. High-traffic sites are 66% more likely to rate limit than low-traffic ones. Apparel and fashion leads at 54%, sporting goods 49%, jewellery, photography and restaurants at around 42%.
TLS fingerprinting: the barrier before the request
13.8% of landing pages check the TLS handshake, and this is the one I think engineers understand least, because it fails before any HTTP happens and the error looks like a network problem.
The mechanism is that your client announces its supported TLS versions, cipher suites and extensions in the ClientHello, and different TLS libraries do this differently. Python's requests on OpenSSL, Go's crypto/tls and Chrome's BoringSSL all produce distinguishable handshakes, so a site can tell "Chrome user agent, OpenSSL handshake" is lying before it reads a single header. JA3, from Salesforce in 2017, turned this into an MD5 hash of those fields in order. Chrome then started randomising the order of its ClientHello extensions in January 2023, which changed the JA3 hash on every connection, and JA4 from FoxIO replaced it by sorting the fields before hashing and producing a human-readable string. Clients like curl-impersonate exist specifically to reproduce a browser's handshake from a non-browser client.
The report's co-occurrence data is the useful part. TLS-fingerprinting sites are 5.9x more likely to rate limit (78.1% of them do, against 13.2% of other sites), 3.2x more likely to run antibot and 1.8x more likely to run CAPTCHA. They are also less likely to require JavaScript, 27% against 42.8%, which the report reads as performance-focused, server-rendered sites. So a TLS check is a strong predictor that everything else is switched on too, which makes it a cheap first probe when you size up a new target.
How the barriers stack
The WAF is the only barrier that regularly stands alone. A third of WAF sites run nothing else. TLS fingerprinting appears alone on under 8% of TLS sites, JavaScript alone on 0.6%, rate limiting alone on 0.2%, CAPTCHA alone on 0.1%, and antibot alone on 0%. Two stacks dominate, WAF only at 31% of sites and WAF plus JavaScript at 20%. The report gives 3.3% for its "full-stack combination" of active defences, against 0.8% running all six measured barriers, and both cluster in the highest-value commercial categories.
The report's advice out of this is to plan for multi-path retrieval, instrument failures by layer, and treat sessions as an asset with reputation and state. That matches my experience, and I would add that "instrument by layer" is the one most teams skip, because a generic "blocked" counter hides whether you are losing at TLS, at the WAF, at rendering or at a challenge, and those have completely different fixes.
The cost view, with a disclosure
Zyte mapped every domain to its own five-tier access model, testing each once over plain HTTP and once through a headless browser and taking the higher of the two. The result is 59.3% Simple, 28.6% Easy, 7.9% Moderate, 3% Complex and 1.2% Advanced, so 88% of landing pages sit in the two cheapest tiers. Apparel and fashion has the highest mean tier at 2.86, retail 2.60, sporting goods 2.50, real estate 2.37.
This is Zyte's pricing model applied to Zyte's own data, so read it as "how hard is this for one particular unblocking stack" rather than an independent difficulty score. The relative ordering between industries is the part I would trust, the absolute "88% is cheap" framing is the part I would want to see reproduced by someone else.
robots.txt is turning into a negotiating position
75.8% of sites publish a valid robots.txt. Only 0.5% rely on it with no technical defence behind it, and 48.3% pair it with two or more technical barriers, so the honour system is now backed by enforcement almost everywhere.
The AI section is where the data gets interesting. 13.7% of sites name AI crawlers, GPTBot is disallowed on 8.4% of sites, CCBot 7.3%, ClaudeBot 6.3%, Google-Extended 5.6%, Bytespider 5.5%, meta-externalagent 4.4%, ChatGPT-User 4.3%, PerplexityBot 4.2%. Newspapers name AI agents on 69% of sites, publishing on 51%, mass media on 44%. Two thirds of newspapers name five or more crawlers.
The counter-intuitive finding is that SEO and search-indexing bots are named in more robots.txt files (19.2%) than AI crawlers (13.7%). The report reads this as publishers valuing search as a traffic partner and AI as a content taker, which fits Cloudflare's own data from July 2026 that 52% of crawler requests on its network are now for AI training, up from 22% in spring 2025, and Imperva's finding that automated traffic passed 51% of all web traffic in 2024.
The report predicts that "access will become more explicitly negotiated". I would say that has already started, and it is happening in the protocol layer, which the report does not cover. RFC 9309 finally standardised robots.txt in September 2022. Cloudflare switched new domains to blocking AI crawlers by default in July 2025 and started a pay-per-crawl beta. In September 2025 it proposed a Content Signals Policy that adds a line like Content-Signal: search=yes, ai-train=no to robots.txt to separate indexing from training from inference. And Web Bot Auth, now at the IETF as draft-meunier-web-bot-auth-architecture, lets a crawler sign its requests using RFC 9421 HTTP Message Signatures with a Signature-Agent header pointing at its public key directory, so a site can verify who a bot is instead of guessing from IP ranges. Put the report's numbers next to those standards and the direction is clear: the advisory layer is becoming an authenticated, priced access layer, and "good bot" is about to mean "bot with a key".
By industry
The 20-group heatmap in the report is worth a look on its own, a few rows stood out to me:
- Media and publishing has the highest WAF adoption at 97% and by far the most AI blocking in robots.txt at 30%, but low antibot (14%) and rate limiting (16%). Publishers defend with paywalls and registration walls, not infrastructure.
- Real estate and automotive stack rate limiting (39% and 38%) and JavaScript (48% and 44%) on top of the baseline WAF, because listing and pricing data is the business.
- Retail and e-commerce is the contested middle, 93% WAF, 23% antibot, 27% CAPTCHA, 44% JavaScript, 31% rate limiting, needing to be visible to search and customers while opaque to competitors.
- Public sector, agriculture and energy stop at the CDN-bundled WAF with single-digit AI blocking.
What I would do differently after reading it
- Probe TLS first when sizing a new target. It costs one handshake and predicts most of the rest of the stack.
- Stop treating 403 as terminal. Classify it. Most rate limits and most WAF blocks arrive as 403 with no Retry-After.
- Count failures per layer, TLS, WAF, render, challenge, antibot verdict, not per site.
- Look for the internal API before reaching for a browser on the 40% of pages that need JavaScript.
- Read robots.txt as a statement of what the operator will tolerate, and expect that file to start carrying content signals and to be paired with signed-request verification.
What I would like to see next
A landing-page audit is a floor, and the interesting fights are on search and detail pages, so a deep-page follow-up would be the most valuable extension. The snapshot is October 2025 with no earlier baseline, so the "web is getting harder" claim is inferred rather than measured, and a second run in a year would turn it into a trend line. And the cost tiers need an outside reproduction.
Those are real gaps, but the dataset is still the biggest public measurement of access controls I know of, and the explorer lets you check any of the claims above against your own industry or country. The PDF is free with no form to fill in. If you scrape for a living, or you run a site and want to know what your peers are doing, it is worth an hour.
Top comments (0)