Last week I published a smaller version of this audit — 30 sites, four AI crawlers. The most common reply was "sample too small." Fair.
So I rebuilt it: 111 sites, 9 AI crawlers, 4 site-level assets, plus two external signals (Tranco traffic rank, Hacker News mention count). English and Chinese sites. And I split the sample by business model instead of grouping everything into "big sites vs small sites."
That last split is where the data stops being a curiosity and starts being useful.
Because the top of the web is not doing what the GEO industry says it's doing. Of the 19 content sites ranked inside the global top 1,000, exactly zero publish an llms.txt file. Not one. Meanwhile 7 of their top-1,000 neighbours do — and all 7 sell software or website builders.
Here's the full dataset.
What I measured
Site-level (fetched with a normal desktop Chrome user agent, no special treatment):
| Signal | How it was judged |
|---|---|
| 9 crawler rules | Parsed robots.txt groups properly (wildcard fallback, Allow: / override), for GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent |
llms.txt |
Must return 200, start with #, and be >40 chars. A SPA catch-all page does not count |
sitemap.xml |
Must contain <urlset or <sitemapindex
|
| JSON-LD |
"@type" present in server-rendered homepage HTML. FAQPage / Organization recorded separately |
External:
- Tranco rank — a research-grade top-1M list, used as a proxy for authority. Sites are bucketed ≤1k / 1k–10k / >10k.
-
Hacker News mentions —
nbHitsfor"domain.com"via the public Algolia API. This is a proxy for third-party written coverage, not for AI citations. I'm explicit about that because it matters for the conclusion.
Sample: 111 sites (68 English, 43 Chinese). 91 of them returned a usable robots.txt; the other 20 (Wikipedia, Stack Overflow, Reddit, Medium, Quora, Forbes…) were unreachable from this network and are excluded from all robots.txt percentages rather than guessed at. They stay in the external-signal analysis.
Finding 1: the top 1,000 publish almost no GEO assets — and get mentioned the most
Top-1,000 sites (Tranco ≤ 1,000), split by what they sell:
| Type | Sites | Have llms.txt
|
Have JSON-LD | Median HN mentions |
|---|---|---|---|---|
| Content / media / community | 19 | 0 (0%) | 3 (16%) | 12,963 |
| Software / SaaS / dev tools | 13 | 4 (31%) | 6 (46%) | 991 |
| Commerce / marketplaces | 8 | 3 (38%) | 3 (38%) | 242 |
Read that again. The 19 highest-authority content sites — Wikipedia, Stack Overflow, Reddit, Medium, Quora, TechCrunch, The Verge, Wired, CNET, Forbes, HBR, McKinsey, Baidu, Sina, Sohu, 163, QQ, Xiaohongshu, Bilibili — have zero llms.txt files between them. And they are mentioned on Hacker News 13× more often than the top-1,000 software companies that do publish one.
The 7 sites in the top 1,000 that do have an llms.txt: HubSpot, Atlassian, Slack, Mailchimp, Shopify, Squarespace, Wix. Every one of them is selling you a tool, a store, or a website. Not one is in the business of owning content.
This is the cleanest version of the argument I can put in a table: llms.txt correlates with wanting to be found. It does not correlate with being cited. If it did, Wikipedia would be invisible and Wix would dominate AI answers.
Finding 2: blocking AI crawlers is a business-model decision, not a technical one
This is the part that surprised me. Everyone frames robots.txt AI-blocking as a capability question — "do you have the technical maturity to control crawlers?" The data says it's a revenue question.
Sites blocking at least one AI crawler, by type:
| Type | Blocking ≥1 crawler | Blocking GPTBot |
|---|---|---|
| Content / media / community | 12/36 — 33% | 9/36 — 25% |
| Software / SaaS / dev tools | 4/42 — 10% | 2/42 — 5% |
| Commerce / marketplaces | 0/13 — 0% | 0/13 — 0% |
And it holds across all nine crawlers, not just OpenAI's:
| Crawler | Content sites | Software sites | Gap |
|---|---|---|---|
| GPTBot | 25% | 5% | +20 pts |
| ClaudeBot | 28% | 2% | +25 pts |
| PerplexityBot | 25% | 2% | +23 pts |
| CCBot | 28% | 2% | +25 pts |
| Google-Extended | 25% | 2% | +23 pts |
| Applebot-Extended | 25% | 0% | +25 pts |
| meta-externalagent | 22% | 0% | +22 pts |
A consistent 20–25 point gap on every single crawler is not noise. It's a policy.
The logic is embarrassingly simple once you see it. If your content is the product, an AI reading it for free is a competitor. If your product sits behind a signup, an AI quoting your docs is a sales channel. Nobody at Figma or Wix is worried that ChatGPT will clone their pricing page — they want ChatGPT to name them when someone asks for a design tool.
The sites blocking hardest are the ones with something to lose: Baidu (9/9 crawlers blocked), Xiaohongshu (9/9), Sohu (8/9), Zhihu (9/9), Wired (8/9), CNET (8/9), The Verge (7/9), TechCrunch (6/9).
Finding 3: the middle tier works hardest, the top tier doesn't bother
llms.txt adoption by authority tier:
| Tier | Sites | llms.txt |
JSON-LD |
|---|---|---|---|
| ≤ 1k | 40 | 18% | 30% |
| 1k–10k | 45 | 44% | 49% |
| > 10k | 16 | 6% | 25% |
The 1k–10k band publishes llms.txt at 2.4× the rate of the top 1,000. That band is exactly where companies are climbing and can feel the traffic moving — Vercel, Supabase, Linear, Asana, ClickUp, Zapier, Intercom, Product Hunt, Webflow, Semrush.
The top 1,000 don't need the file. The bottom doesn't know it exists. The middle is where the anxiety lives, and anxiety produces checklists.
If you're a mid-size SaaS reading this: your competitors have probably already shipped it. 44% is not an early-adopter number anymore. That's the entry ticket, not the advantage.
Finding 4: the Chinese web is a five-year-old gap, not a one-year one
| Signal | English sites (68) | Chinese sites (43) |
|---|---|---|
llms.txt |
26 — 38% | 3 — 7% |
| JSON-LD on homepage | 33 — 49% | 7 — 16% |
| Median HN mentions | 1,045 | 10 |
| Blocking ≥1 AI crawler | 21% | 13% |
The whole Chinese-language sample produced three llms.txt files: ifeng.com, gitee.com, ctrip.com. Three, out of 43, including every major tech media brand and every major cloud provider. Alibaba Cloud, Tencent Cloud, Huawei Cloud, Volcengine, Juejin, CSDN, 36Kr — none.
Structural markup is barely better: 16% versus 49%.
Chinese teams block AI crawlers less than English teams (13% vs 21%) — so the access door is open, but almost nothing is published in a form an engine can parse. That's an unusual combination: permissive robots, empty markup. The crawl budget is being spent on pages that give the model nothing structured to lift.
For anyone working in the Chinese market this is the actual finding. It's not that competition is fierce. It's that the floor is empty.
The uncomfortable aggregate
-
50 of 111 sites have none of the three assets (
llms.txt,sitemap.xml, JSON-LD) in place. - 15 of 111 have all three.
- Median HN mentions across the whole sample: 310. Mean: 50,505. The mean is 160× the median, because four domains (GitHub, Medium, Reddit, TechCrunch) sit above 500,000 and drag it there. Any "average mentions" figure in the GEO space is almost certainly built on a mean and is therefore worthless.
What I'd actually do, by type
If you sell software — publish the assets, but don't expect them to move citations. 44% of your tier already has llms.txt; skipping it is a self-inflicted wound, shipping it is table stakes. Your real leverage is being written about somewhere a model already trusts, which is a distribution problem, not a markup problem.
If you sell content or media — decide the crawler policy on purpose instead of by default. Blocking all 9 crawlers keeps your archive intact and removes you from AI answers entirely. Allowing them gives away the archive and buys brand recall. There's no free option; pick one and write it down. Right now most publishers have chosen "block" without running the numbers.
If you're in the Chinese market — the 7% is real, and it's the cheapest gap in this entire dataset. Structured data plus an llms.txt costs an afternoon. Almost nobody on your side of the web has done it.
Method limits, stated plainly
-
robots.txtblocking ≠ exclusion from AI answers. An engine can cite a page it crawled earlier, or via a licensed feed. Treat this as policy intent, not outcome. - HN mentions are a tech-community proxy. It over-weights developer tools and under-weights consumer and Chinese sites (median 10 for CN). It is not an AI citation count. Measuring actual citations across 111 domains × 5 engines is a different, much more expensive study — that's what my product does, and I didn't want to launder my own tool's output into "independent research."
-
20 sites were unreachable from this network and are excluded from
robots.txtpercentages. The exclusion skews toward very large English sites — Wikipedia, Reddit, Stack Overflow, Medium, Quora, Forbes — so the top-1,000 content numbers are computed on the 19 I could reach, not on an ideal sample. - Type classification is mine, based on how each site makes money. Reasonable people could file McKinsey as consulting rather than content; the direction of every finding survives those reclassifications, which is why I kept the buckets coarse.
Raw data: one CSV row per domain, 9 crawler verdicts and 4 asset flags each. The whole collection script is 120 lines of Python with nothing beyond the standard library — drop your own domain list in and re-run it. That beats trusting my numbers.
Disclosure: I build Bidvix, a tool that checks how often a brand gets cited by ChatGPT, Gemini, Claude, Perplexity and DeepSeek. The audit above was collected independently of the product, and the product's own output is deliberately not part of it. If you want the underlying checklist rather than the tool, here's the version with no signup.
Top comments (0)