DEV Community

Ke Gao
Ke Gao

Posted on

I audited 111 sites for AI-search readiness. The 19 biggest content sites scored zero.

Last week I published a smaller version of this audit — 30 sites, four AI crawlers. The most common reply was "sample too small." Fair.

So I rebuilt it: 111 sites, 9 AI crawlers, 4 site-level assets, plus two external signals (Tranco traffic rank, Hacker News mention count). English and Chinese sites. And I split the sample by business model instead of grouping everything into "big sites vs small sites."

That last split is where the data stops being a curiosity and starts being useful.

Because the top of the web is not doing what the GEO industry says it's doing. Of the 19 content sites ranked inside the global top 1,000, exactly zero publish an llms.txt file. Not one. Meanwhile 7 of their top-1,000 neighbours do — and all 7 sell software or website builders.

Here's the full dataset.

What I measured

Site-level (fetched with a normal desktop Chrome user agent, no special treatment):

Signal How it was judged
9 crawler rules Parsed robots.txt groups properly (wildcard fallback, Allow: / override), for GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent
llms.txt Must return 200, start with #, and be >40 chars. A SPA catch-all page does not count
sitemap.xml Must contain <urlset or <sitemapindex
JSON-LD "@type" present in server-rendered homepage HTML. FAQPage / Organization recorded separately

External:

  • Tranco rank — a research-grade top-1M list, used as a proxy for authority. Sites are bucketed ≤1k / 1k–10k / >10k.
  • Hacker News mentions — nbHits for "domain.com" via the public Algolia API. This is a proxy for third-party written coverage, not for AI citations. I'm explicit about that because it matters for the conclusion.

Sample: 111 sites (68 English, 43 Chinese). 91 of them returned a usable robots.txt; the other 20 (Wikipedia, Stack Overflow, Reddit, Medium, Quora, Forbes…) were unreachable from this network and are excluded from all robots.txt percentages rather than guessed at. They stay in the external-signal analysis.

Finding 1: the top 1,000 publish almost no GEO assets — and get mentioned the most

Top-1,000 sites (Tranco ≤ 1,000), split by what they sell:

Type Sites Have llms.txt Have JSON-LD Median HN mentions
Content / media / community 19 0 (0%) 3 (16%) 12,963
Software / SaaS / dev tools 13 4 (31%) 6 (46%) 991
Commerce / marketplaces 8 3 (38%) 3 (38%) 242

Read that again. The 19 highest-authority content sites — Wikipedia, Stack Overflow, Reddit, Medium, Quora, TechCrunch, The Verge, Wired, CNET, Forbes, HBR, McKinsey, Baidu, Sina, Sohu, 163, QQ, Xiaohongshu, Bilibili — have zero llms.txt files between them. And they are mentioned on Hacker News 13× more often than the top-1,000 software companies that do publish one.

The 7 sites in the top 1,000 that do have an llms.txt: HubSpot, Atlassian, Slack, Mailchimp, Shopify, Squarespace, Wix. Every one of them is selling you a tool, a store, or a website. Not one is in the business of owning content.

This is the cleanest version of the argument I can put in a table: llms.txt correlates with wanting to be found. It does not correlate with being cited. If it did, Wikipedia would be invisible and Wix would dominate AI answers.

Finding 2: blocking AI crawlers is a business-model decision, not a technical one

This is the part that surprised me. Everyone frames robots.txt AI-blocking as a capability question — "do you have the technical maturity to control crawlers?" The data says it's a revenue question.

Sites blocking at least one AI crawler, by type:

Type Blocking ≥1 crawler Blocking GPTBot
Content / media / community 12/36 — 33% 9/36 — 25%
Software / SaaS / dev tools 4/42 — 10% 2/42 — 5%
Commerce / marketplaces 0/13 — 0% 0/13 — 0%

And it holds across all nine crawlers, not just OpenAI's:

Crawler Content sites Software sites Gap
GPTBot 25% 5% +20 pts
ClaudeBot 28% 2% +25 pts
PerplexityBot 25% 2% +23 pts
CCBot 28% 2% +25 pts
Google-Extended 25% 2% +23 pts
Applebot-Extended 25% 0% +25 pts
meta-externalagent 22% 0% +22 pts

A consistent 20–25 point gap on every single crawler is not noise. It's a policy.

The logic is embarrassingly simple once you see it. If your content is the product, an AI reading it for free is a competitor. If your product sits behind a signup, an AI quoting your docs is a sales channel. Nobody at Figma or Wix is worried that ChatGPT will clone their pricing page — they want ChatGPT to name them when someone asks for a design tool.

The sites blocking hardest are the ones with something to lose: Baidu (9/9 crawlers blocked), Xiaohongshu (9/9), Sohu (8/9), Zhihu (9/9), Wired (8/9), CNET (8/9), The Verge (7/9), TechCrunch (6/9).

Finding 3: the middle tier works hardest, the top tier doesn't bother

llms.txt adoption by authority tier:

Tier Sites llms.txt JSON-LD
≤ 1k 40 18% 30%
1k–10k 45 44% 49%
> 10k 16 6% 25%

The 1k–10k band publishes llms.txt at 2.4× the rate of the top 1,000. That band is exactly where companies are climbing and can feel the traffic moving — Vercel, Supabase, Linear, Asana, ClickUp, Zapier, Intercom, Product Hunt, Webflow, Semrush.

The top 1,000 don't need the file. The bottom doesn't know it exists. The middle is where the anxiety lives, and anxiety produces checklists.

If you're a mid-size SaaS reading this: your competitors have probably already shipped it. 44% is not an early-adopter number anymore. That's the entry ticket, not the advantage.

Finding 4: the Chinese web is a five-year-old gap, not a one-year one

Signal English sites (68) Chinese sites (43)
llms.txt 26 — 38% 3 — 7%
JSON-LD on homepage 33 — 49% 7 — 16%
Median HN mentions 1,045 10
Blocking ≥1 AI crawler 21% 13%

The whole Chinese-language sample produced three llms.txt files: ifeng.com, gitee.com, ctrip.com. Three, out of 43, including every major tech media brand and every major cloud provider. Alibaba Cloud, Tencent Cloud, Huawei Cloud, Volcengine, Juejin, CSDN, 36Kr — none.

Structural markup is barely better: 16% versus 49%.

Chinese teams block AI crawlers less than English teams (13% vs 21%) — so the access door is open, but almost nothing is published in a form an engine can parse. That's an unusual combination: permissive robots, empty markup. The crawl budget is being spent on pages that give the model nothing structured to lift.

For anyone working in the Chinese market this is the actual finding. It's not that competition is fierce. It's that the floor is empty.

The uncomfortable aggregate

  • 50 of 111 sites have none of the three assets (llms.txt, sitemap.xml, JSON-LD) in place.
  • 15 of 111 have all three.
  • Median HN mentions across the whole sample: 310. Mean: 50,505. The mean is 160× the median, because four domains (GitHub, Medium, Reddit, TechCrunch) sit above 500,000 and drag it there. Any "average mentions" figure in the GEO space is almost certainly built on a mean and is therefore worthless.

What I'd actually do, by type

If you sell software — publish the assets, but don't expect them to move citations. 44% of your tier already has llms.txt; skipping it is a self-inflicted wound, shipping it is table stakes. Your real leverage is being written about somewhere a model already trusts, which is a distribution problem, not a markup problem.

If you sell content or media — decide the crawler policy on purpose instead of by default. Blocking all 9 crawlers keeps your archive intact and removes you from AI answers entirely. Allowing them gives away the archive and buys brand recall. There's no free option; pick one and write it down. Right now most publishers have chosen "block" without running the numbers.

If you're in the Chinese market — the 7% is real, and it's the cheapest gap in this entire dataset. Structured data plus an llms.txt costs an afternoon. Almost nobody on your side of the web has done it.

Method limits, stated plainly

  1. robots.txt blocking ≠ exclusion from AI answers. An engine can cite a page it crawled earlier, or via a licensed feed. Treat this as policy intent, not outcome.
  2. HN mentions are a tech-community proxy. It over-weights developer tools and under-weights consumer and Chinese sites (median 10 for CN). It is not an AI citation count. Measuring actual citations across 111 domains × 5 engines is a different, much more expensive study — that's what my product does, and I didn't want to launder my own tool's output into "independent research."
  3. 20 sites were unreachable from this network and are excluded from robots.txt percentages. The exclusion skews toward very large English sites — Wikipedia, Reddit, Stack Overflow, Medium, Quora, Forbes — so the top-1,000 content numbers are computed on the 19 I could reach, not on an ideal sample.
  4. Type classification is mine, based on how each site makes money. Reasonable people could file McKinsey as consulting rather than content; the direction of every finding survives those reclassifications, which is why I kept the buckets coarse.

Raw data: one CSV row per domain, 9 crawler verdicts and 4 asset flags each. The whole collection script is 120 lines of Python with nothing beyond the standard library — drop your own domain list in and re-run it. That beats trusting my numbers.


Disclosure: I build Bidvix, a tool that checks how often a brand gets cited by ChatGPT, Gemini, Claude, Perplexity and DeepSeek. The audit above was collected independently of the product, and the product's own output is deliberately not part of it. If you want the underlying checklist rather than the tool, here's the version with no signup.

Top comments (0)