DEV Community

AI Index Optimizer
AI Index Optimizer

Posted on

85% of top crypto projects are invisible to AI search

Crypto spent a decade learning to rank on Google. The next decade of discovery happens inside ChatGPT, Claude, Perplexity and AI Overviews, where one generated answer replaces ten blue links. Those systems only quote what their crawlers can fetch, and what robots.txt permits them to read.

On 17 November 2026 we fetched three things from 21 prominent crypto domains: /llms.txt, /ai.txt, and the AI-crawler directives in robots.txt.

Result: 18 of 21 domains (85.7%) are missing at least one piece of that basic access layer. Only three have all of it.

What we measured

For each apex domain we issued a plain HTTP GET to /llms.txt and /ai.txt, recorded the status code, and separately counted AI-crawler directives declared in robots.txt. Zero means no directive at all.

Domain llms.txt ai.txt AI directives
base.org 200 404 0
coinbase.com 200 404 0
uniswap.org 200 200 0
aave.com 200 404 8
chain.link 200 404 0
thirdweb.com 404 404 0
optimism.io 200 404 4
arbitrum.io 404 404 0
solana.com 429 429 0
ens.domains 404 404 0

Five domains have no llms.txt at all: thirdweb.com, arbitrum.io, ens.domains, gitcoin.co and zora.co. Twelve publish one but declare zero AI-crawler rules. solana.com returned 429 to automated fetches, meaning even a legitimate crawler is throttled before it reads a page.

Why it matters commercially

When someone asks an assistant which L2 to build on, or which wallet to trust, the answer is assembled from pages the model can actually read. A project that blocks or ignores crawlers is not penalised in a ranking. It is absent from the answer.

The pattern is not neglect

Most teams heard about llms.txt and shipped one. What almost nobody did is decide who may crawl them and write it down. That is a policy gap, not a content gap, and it takes minutes to close.

A workable baseline:

  1. Publish an llms.txt describing what the project is and what matters on the site.
  2. Add explicit Allow rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended to robots.txt.
  3. Make sure AI crawlers are not silently throttled at the edge.
  4. Re-check quarterly, because the crawler landscape changes.

Full methodology and the complete 21-domain table: https://ai-index-optimizer.com

If you run a crypto project and want the audit for your own domain, it takes about a minute and needs no account.

Top comments (0)