A 137,000-domain Ahrefs study published in June 2026 found that 97% of llms.txt files received zero requests from any bot or human. GPTBot, ClaudeBot, and their peers crawl HTML directly and ignore the file entirely. This article breaks down what the data actually shows, why llms.txt became a speculative SEO ritual, and which signals genuinely move AI citation frequency.
This article was originally published on byfaro.ai/blog/ai-crawlers-ignore-llmstxt-97-of-the-time-so-stop-making-one. Read the full guide there — it includes step-by-step examples and Faro audit tool links.
What you'll learn
This guide covers the full picture on ai crawlers ignore llms.txt 97% of the time so stop making one, including:
- Why this matters for AI agent discoverability
- What to check on your own site
- The specific fixes that move your score
Faro audits whether AI agents can find, understand, and act on your business — like GTMetrix but for AI discoverability. Free scan at byfaro.ai.
Top comments (0)