Crypto spent a decade learning to rank on Google. The next decade of discovery happens inside ChatGPT, Claude, Perplexity and AI Overviews, where one generated answer replaces ten blue links. Those systems only quote what their crawlers can fetch, and what robots.txt permits them to read.
On 17 November 2026 we fetched three things from 21 prominent crypto domains: /llms.txt, /ai.txt, and the AI-crawler directives in robots.txt.
Result: 18 of 21 domains (85.7%) are missing at least one piece of that basic access layer. Only three have all of it.
What we measured
For each apex domain we issued a plain HTTP GET to /llms.txt and /ai.txt, recorded the status code, and separately counted AI-crawler directives declared in robots.txt. Zero means no directive at all.
| Domain | llms.txt | ai.txt | AI directives |
|---|---|---|---|
| base.org | 200 | 404 | 0 |
| coinbase.com | 200 | 404 | 0 |
| uniswap.org | 200 | 200 | 0 |
| aave.com | 200 | 404 | 8 |
| chain.link | 200 | 404 | 0 |
| thirdweb.com | 404 | 404 | 0 |
| optimism.io | 200 | 404 | 4 |
| arbitrum.io | 404 | 404 | 0 |
| solana.com | 429 | 429 | 0 |
| ens.domains | 404 | 404 | 0 |
Five domains have no llms.txt at all: thirdweb.com, arbitrum.io, ens.domains, gitcoin.co and zora.co. Twelve publish one but declare zero AI-crawler rules. solana.com returned 429 to automated fetches, meaning even a legitimate crawler is throttled before it reads a page.
Why it matters commercially
When someone asks an assistant which L2 to build on, or which wallet to trust, the answer is assembled from pages the model can actually read. A project that blocks or ignores crawlers is not penalised in a ranking. It is absent from the answer.
The pattern is not neglect
Most teams heard about llms.txt and shipped one. What almost nobody did is decide who may crawl them and write it down. That is a policy gap, not a content gap, and it takes minutes to close.
A workable baseline:
- Publish an llms.txt describing what the project is and what matters on the site.
- Add explicit Allow rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended to robots.txt.
- Make sure AI crawlers are not silently throttled at the edge.
- Re-check quarterly, because the crawler landscape changes.
Full methodology and the complete 21-domain table: https://ai-index-optimizer.com
If you run a crypto project and want the audit for your own domain, it takes about a minute and needs no account.
Top comments (0)