Every agent demo you've seen runs on a friendly web. Mine went outside.
We benchmark agents on static pages and local corpora because the real web is a hostile place to measure: walls move, challenges rotate, results depend on your IP's reputation that day. So almost nobody publishes the numbers. I wanted them, so I built the measurement: a vanilla agent — no challenge solving, no retries, no logins — opening 200 real URLs across five categories (docs, news, e-commerce, dev-tools, social) and trying to read each one.
The instrument is navette, my 659 KB browser-for-agents — but this post is about the web, not the tool. Any headless engine would produce the same shape of numbers; the dataset is linked at the end either way.
A measurement, not a verdict. One exit IP (residential), one day, zero challenge solving, zero retries, zero logins. A site that walls itself off from this probe has not been judged — it has been measured once, from one vantage point.
What a vanilla agent actually gets
| Category | URLs | readable | challenges | JS-gates | blocks all AI bots in robots.txt | llms.txt |
|---|---|---|---|---|---|---|
| docs | 40 | 39 (97%) | 0 | 1 | 0 | 13 |
| news | 40 | 34 (85%) | 3 | 3 | 35 | 5 |
| e-commerce | 40 | 33 (82%) | 2 | 5 | 8 | 9 |
| dev-tools | 40 | 33 (82%) | 5 | 2 | 4 | 14 |
| social | 40 | 26 (65%) | 2 | 12 | 25 | 14 |
Aggregate: 165/200 — 82.5% — readable end-to-end by a vanilla agent. Which sounds generous until you invert it: more than one page in six is closed, and the closed sixth is not randomly distributed. It concentrates exactly where agents add economic value — news, shopping, social.
The short version of the whole table: documentation is free, and everything else negotiates.
-
Docs: 39/40. The single miss in the entire corpus is
go.dev/tour— an interactive playground that builds itself in JS, not a documentation page. Every actual doc page — MDN, python.org, React, Rust, Kubernetes, AWS — served the agent exactly what it serves you. The web's public library is genuinely public. - News: 85% readable — but the wall is declared, not armed. Three challenges, three JS-gates. The striking number sits in robots.txt: 35 of 40 news sites block every tracked AI crawler. The robots file says no while the front door still opens. That gap is the interesting object: publishers are positioned to close it whenever the economics settle.
- Social: 65%, and 12 of 40 are JS-gates. The category that simply declines to render for strangers.
The three walls, in order of arrival
1. The declared wall (robots.txt). The corpus tracks ten AI crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot, Bytespider…). 72 of 200 hosts block them all; four more partially. The most-blocked bots: GPTBot (69 hosts), Google-Extended (67), Bytespider (67), ClaudeBot (66), anthropic-ai (66). A robots clause is free to declare and free to ignore — it's a social contract, not a fence. The measurement says the social contract is already signed everywhere the content is expensive to make.
2. The enforced wall (challenges and JS-gates). Turnstile, "Just a moment", DataDome, PerimeterX — 12 challenges across the corpus, plus 23 empty/JS-gate pages that render nothing without executing the gate. This is the fence. It's also the wall an agent could defeat and mostly shouldn't — defeating it is the compliance question, not the engineering question.
3. The coming tollbooth (x402 and friends). This is why I ran the bench twice. The x402 protocol family — HTTP 402 plus X-Payment headers, machine-payable per request — is moving from proposal to deployments. My probe watches for it: HTTP 402 status, payment headers, and the declared-paywall markup publishers already ship (isAccessibleForFree:false in schema.org — Google's paywall markup, readable by any agent that bothers to look).
The result: zero. Zero 402s, zero X-Payment headers, across the 164 probes that answered — the honest qualifier is that 36 of 200 passive probes were themselves blocked by an edge (403), so absence is measured on what answered. The tollbooth is signed but not yet installed.
But I can name the shape of it, because one site in the corpus is already running all three layers at once:
The Intercept:
isAccessibleForFree: falsein its markup (a declared paywall), every tracked AI crawler blocked in robots.txt (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Amazonbot) — and its front page still fully readable, right now, by a vanilla agent.
That is a metered parking lot with the gate up. The prediction this dataset lets you test: the enforced wall (2) exists to protect the tollbooth (3) while it's being built. The markup is already telling you where the gate will come down.
And the walls move — that's the point of a dated, reproducible dataset. Between my October 6 and October 8 runs: one more news publisher flipped to block-all (34 → 35), and overall readability moved from 84.5% to 82.5%. Nobody notices one publisher flipping a robots line. The dataset does.
llms.txt, honestly
55 of 200 sites ship an llms.txt. Adoption concentrates in docs and dev-tools — the categories with zero walls. The sites that most need a machine-negotiation channel (news, social) are the least interested in one: five news sites total. llms.txt is a courtesy lane on a road that's busy installing toll booths.
The dataset
200 rows × 5 categories: outcome classification, wall type, robots.txt AI-clause stance per bot, llms.txt presence, x402/paywall signals, timings. CC BY 4.0, committed to the repo (bench/tollbooth-results.{json,csv} + bench/DATASET.md for the schema), reproducible with one command against any navette daemon:
NAVETTE_PORT=8766 python3 bench/tollbooth.py
An open ask: if you run agents against real sites daily — I'm looking at the crawler and extraction folks (pulpie-mcp and neighbors) — send me your failure cases. The next run of this bench should include the URLs where your agents hit walls. That's how this becomes the reference dataset instead of my weekend run.
The instrument, for the curious: navette — a single Rust binary driving the WebView your OS already ships (WebKit/WebView2/WebKitGTK), 659 KB, MCP-native, MIT. The bench harness is bench/tollbooth.py in the same repo; the daemon it drives is the same one my agent uses to read this very web.
Top comments (7)
The gap between the declared wall in robots.txt and actual edge enforcement is essentially an unhedged legal option. Putting a crawler disallow clause into robots.txt or flagging schema.org metadata costs nothing in latency, and it establishes a clean evidentiary trail for future litigation or licensing negotiations. Dropping an active gate immediately risks subscriber friction and edge compute overhead, so publishers leave the gate open while securing their right to sue or settle downstream.
The absence of x402 headers reflects the clearing friction of per-request micro-settlements rather than publisher indifference. When the transaction cost of metering, verifying, and clearing a fraction of a cent exceeds the value of the scrap of text, point-to-point HTTP tollbooths rarely clear. Content owners do not wait for browser-level payment headers; they monetize scraping by signing bilateral licensing deals with the frontier model labs and leaving passive crawler traffic to edge defenses.
The unhedged-option frame matches the measured gap almost exactly: in the news category, 35 of 40 hosts declare block-everything in robots.txt — and 85% of the pages still read end-to-end, three challenges and three JS-gates across the whole category. That is precisely your cheap-evidentiary position: declarations priced at zero latency, held for the negotiation table rather than the request path.
Social is the inverse cell — 25 declarations and the heaviest actual enforcement in the corpus (12 JS-gates, identity walls), which fits your frame too: where the audience is the asset, the gate gets armed. And your clearing-friction reading of zero x402 across 200 URLs is the half I under-weighted going in: a per-request settlement that costs more to meter than the scraped text is worth will never clear at the point-to-point layer — consistent with the one site in the corpus running a declared paywall whose gate was up. The declaration exists for the licensing conversation, not for the vanilla agent.
The part your comment adds to the dataset's read: the interesting walls aren't at the HTTP layer at all — they're in bilateral deals the crawler never sees. Which is why the measurement reads as "docs are free, everything else negotiates" even where robots.txt says nothing is being negotiated.
hmm, for sites that have an API behind them, would you count the agent calling it instead of reading the page as a pass or a cheat? From our experience at ApyHub (I co-founded it), a lot of agent tasks don't need the page at all, just the data behind it, which is why our MCP puts 1,500+ endpoints and counting behind 4 tools.
Neither — it's a different instrument, and the distinction carries the whole claim the bench makes. It measures the front door: what a vanilla agent can read as a third party with no agreement. Calling the API behind the page is the negotiated lane — built for partners, often keyed, governed by ToS. If the API is open, that's a fact about the site's exposure worth its own column: "walls the HTML, serves the data to anyone with a GET." If it's keyed, that's neither pass nor cheat — it's the tollbooth thesis working as designed: access by agreement instead of access by scraping.
Which is the polite way to say your MCP is on the same side of the trend as the walls: 1,500 endpoints behind 4 tools is permissioned access with clear terms — the deliberate version of what paywalls do by accident. For bench v2, "HTML closed but a documented API exists" is a third state worth counting, because it separates sites that close by neglect from sites that close by contract — and the second group is the one building the tollbooth on purpose.
The gap runs in both directions, and the direction I got hit from was the opposite one.
As someone who runs a handful of small static tool sites, I never wrote a single line of crawler policy — no robots.txt rules about AI, no declarations, nothing. And that turned out to be exactly the problem. The platform default did the declaring for me: since mid-2025 the CDN in front of those sites blocks AI crawlers by default on new domains. So I was silently excluded from AI search results — not because I asked to be, but because I never asked not to be. Nobody emails you about that. You find out by noticing you're absent.
Fixing it meant explicitly allowlisting the retrieval crawlers by name — OAI-SearchBot, ClaudeBot, PerplexityBot — which is a strange thing to have to do in order to be findable. The uncomfortable part is that the effective policy was never my declaration. It was a default someone else chose, applied to me while I wasn't looking.
One caveat I'd keep separate from the article's framing: training crawlers and retrieval crawlers are different classes with different costs. Allowing a bot that fetches a page because a human asked for it isn't the same as opening the door to bulk training. Treating them as one bucket gets you a policy that's either too broad or too blunt to survive contact with reality.
Your case is the row the bench table was missing: the tollbooth that arrives before the owner knows they're on the road. Every other wall in the dataset was declared somewhere; yours was armed and undeclared — the mirror image of the news category, where 35 hosts declared a wall that wasn't armed. Declared-but-open and open-but-blocked are the same measurement gap pointing in opposite directions, and yours is the worse direction: nobody emails you, because the policy was never yours.
Which is "defaults are policy" again, one layer down from where that sentence usually lives — at the CDN, a default someone else chose is the policy that ships, and the fix you landed (allowlisting the retrieval bots by name to become findable) might be the strangest config sentence of the year. Your training/retrieval caveat should shape bench v2's columns, by the way: they're two classes with two costs, and the per-bot split — SearchBot-class discoverability versus GPTBot-class bulk — is exactly what the next table should measure. A site that wants the first without the second needs the granularity, and a bench that lumps them measures the wrong wall.
Số liệu 72 host block hẳn AI crawler khá thú vị — mình cũng đang thấy xu hướng này tăng nhanh trong 6 tháng qua, đặc biệt là các site lớn có nội dung độc quyền (news, documentation, knowledge base). Con số 82.5% readable nghe "tốt" nhưng thực tế các URL đọc được thường là public docs, blog cá nhân, hay content marketing — cái agent thực sự cần (pricing page, gated API docs, internal wiki) thường nằm trong 17.5% còn lại.
Về x402: chưa thấy cũng dễ hiểu, spec mới finalize gần đây và adoption curve của payment protocol thường chậm hơn blocking mechanism. Dự đoán Q3-Q4 năm nay sẽ bắt đầu thấy x402 headers xuất hiện trên các API marketplace trước, rồi mới đến content site.
Một góc nhìn thiếu: bài test "vanilla agent" — có thử rotate user-agent / residential IP / headless fingerprint không? Vì nhiều site block ở tầng WAF (Cloudflare, Akamai, DataDome) dựa trên TLS fingerprint + behavioral analysis chứ không chỉ user-agent string. Nếu chưa test các vector này thì con số block rate thực tế có thể thấp hơn khi deploy production agent.
Curious: trong 72 host block, có phân tích được bao nhiêu % dùng robots.txt vs 403/429 vs challenge page (turnstile, hCaptcha)? Giúp Prioritize giải pháp sau này (site: labagent .tech)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.