Ahrefs measured how often AI crawlers fetch llms.txt files and found 97% get zero requests. The study is methodologically sound but the conclusion misses the most important use case: inference-time context injection, not pre-crawl indexing. llms.txt was never primarily a crawler instruction file.
This article was originally published on byfaro.ai/blog/llms-txt-ahrefs-study. Read the full guide there — it includes step-by-step examples and Faro audit tool links.
What you'll learn
This guide covers the full picture on ahrefs says 97% of llms.txt files get zero requests. they missed the point., including:
- Why this matters for AI agent discoverability
- What to check on your own site
- The specific fixes that move your score
Faro audits whether AI agents can find, understand, and act on your business — like GTMetrix but for AI discoverability. Free scan at byfaro.ai.
Top comments (1)
ngl ahrefs is just looking at the wrong metric here. the point of llms.txt is more about providing a standard for the future than current traffic numbers.