Originally published on pagelens.io.
Every GEO guide now says "add an llms.txt". I wanted to know if any AI system actually fetches it, so I checked the server logs of a new site that had one from day one.
Setup: one small site launched 7 Oct 2026, every request logged on the server (so non-JS bots count), user agents matched to AI companies, IPs checked against published ranges where they exist. Window: 8–15 Oct.
Numbers
| File | AI bot requests | Share |
|---|---|---|
| All pages | 567 | 100% |
/robots.txt |
73 | 13% |
/sitemap.xml |
50 | 9% |
/llms.txt |
6 | 1% |
The six llms.txt requests: ClaudeBot, GPTBot, meta-externalagent, ChatGPT-User, PerplexityBot, OAI-SearchBot. Once each. The four OpenAI/Perplexity ones were IP-verified.
What I take from it
- It does get fetched. Every major AI company's bot grabbed it within a week.
- It's not treated like robots.txt. ClaudeBot asked for robots.txt 46 times that week and llms.txt once.
- A fetch isn't usage. Can't tell from logs whether anything followed its links. The most-read pages were linked from everywhere anyway.
- ChatGPT-User fetched it once. That agent acts for a person mid-conversation, so the only hint of answer-time use is a single event.
So: add one, it's ten minutes and free. Don't expect it to get you into AI answers. Robots.txt access, CDN not blocking bots, content in the HTML and links from other sites still do the work.
Check your own logs:
grep -i "llms.txt" access.log | grep -Ei "GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|ChatGPT-User"
Curious if people with bigger sites see the same pattern.
Top comments (0)