Every SEO tool and checklist assumes a stack of files matters — robots.txt, canonical tags, Open Graph, structured data, and now llms.txt. But assumption isn't evidence. So we scanned the Tranco top 1,000 sites and just looked at what they actually ship.
First surprise: of the 1,000, only 554 served a browsable HTML homepage at all. The rest are CDN endpoints, DNS infrastructure and API hosts with nothing to crawl. Every percentage below is over those 554 real homepages, not the raw list.
The basics are table stakes
Among sites that have a homepage, the fundamentals are close to universal:
- meta description — 82%
- robots.txt — 82%
- Open Graph — 73%
- canonical — 69%
If your site is missing any of these, you're behind the top of the web. They're the floor, not an edge — which is worth knowing before you spend a sprint "optimising" them.
Then it falls off a cliff
- Structured data (JSON-LD) — 41%. Fewer than half of the biggest sites ship any.
- FAQPage schema — just 2%.
That 2% is the headline. Rich results and FAQ schema get endless attention in SEO content, and almost none of the largest sites actually implement FAQPage. Whatever advantage it confers is wide open, because the field is nearly empty.
The 2026 shift is on the AI side
The genuinely new movement isn't in classic SEO — it's in how sites treat AI crawlers:
- 27% now name at least one AI crawler in robots.txt to block it — GPTBot on 20%, ClaudeBot 19%, Google-Extended 17%. Blocking the AI bots has quietly become a mainstream default at the top of the web.
- 16% already publish an llms.txt — the proposed "here's how to read my site" file for LLMs. Adoption is concentrated in developer-tool and infrastructure companies: Cloudflare, GitHub, Stripe, Shopify. The audience that would build against an LLM is the audience shipping the file for one.
The honest caveat
The Tranco top 1,000 skews heavily technical — SaaS, infra, developer platforms — so these numbers run ahead of the general web, not level with it. Read them as "what the most sophisticated sites do," not "what the average site does." The llms.txt figure especially is a leading indicator, not a mainstream one.
Reproduce it
The scan is a public domain list plus a short fetch-and-parse script — no private data, no paywalled crawler. Point it at any list and you'll get comparable numbers. And to check your own site's files, the SEO tools cover each signal directly: a robots.txt tester, an Open Graph and schema generator, an llms.txt generator and an AI-crawler control builder — all in the browser.
Originally published on LK Forge.
Top comments (4)
Nice work turning this into an actual scan instead of another checklist post. One premise worth pushing on: the 2% FAQPage number reads like an opportunity, but I'd bet it's mostly sites skipping it on purpose since Google demoted FAQ rich results for most non-authoritative domains last year. Viewfy is built for finding buyer threads, but its citation checks show the same pattern, llms.txt adoption tracking dev-tool sites almost exactly. Worth checking whether your own FAQ markup still earns anything before adding it back.
Good addition — and I think we actually agree. The post's read on the 2% was that the "advanced SEO" layer guides keep recommending is mostly empty space, not that it's low-hanging fruit. What your point adds is the why: Google's 2023 rollback of FAQ rich results to authoritative health/gov domains means, for most sites, the markup stopped paying rent. So 2% probably isn't neglect — it's sites correctly dropping schema that no longer earns SERP real estate.
That's a sharper framing than "underused," so I appreciate it. And yeah — anyone still shipping FAQPage should confirm it earns FAQ appearances in Search Console before treating it as a win, rather than adding it on faith because a checklist says to.
Followed up on my own site to put a number on it: lkforge ships FAQPage JSON-LD on ~1,297 pages, and GSC shows 0 FAQ rich-result appearances over the last 180 days — the only search-appearance types earning impressions are Video and Translated Result. So the markup is completely inert for SERP here, exactly as you'd predict post-2023. Keeping it now only really makes sense as structured Q&A for AI/answer engines to lift, not for classic rich results.
That’s a really useful real-world datapoint. ~1,297 pages with zero FAQ rich-result appearances over 180 days makes the distinction pretty clear: FAQPage JSON-LD is no longer something you should keep around expecting a Google SERP enhancement. I’d separate that from the broader question of whether the structured Q&A has any value. If the content is genuinely useful and accurately reflects the page, the markup can still provide machine-readable context for systems consuming structured data — including AI/answer systems — even if Google never turns it into a visible rich result.