llms.txt is a Markdown file at the root of a site that summarises it and links the pages worth reading, so a model with a small context window can read one tidy file instead of crawling menus and cookie banners. Jeremy Howard of Answer.AI proposed it on 3 September 2024. It isn't an official standard.
Lots of people are adding one because an SEO tool told them to. So here's what happened when we actually looked at our logs.
The format, quickly
Only an H1 is required. The rest is convention:
# Example Dental Clinic
> A family dental clinic offering check-ups, cleaning, fillings, braces and emergency appointments.
## Services
- [Check-ups and cleaning](https://example.com/services/check-ups): what a visit includes
- [Emergency dentist](https://example.com/emergency): same-day appointments
## Optional
- [Blog](https://example.com/blog): dental care guides
It's llms.txt with an s. A file named llm.txt won't be found.
60 days of server logs
Our site has served an llms.txt since August 2026, rebuilt on every publish. We counted every request from the main AI crawlers across 60 days of logs. There's no CDN cache in front, so every request reaches the server.
| Crawler | Requests to the site | Requests to /llms.txt |
|---|---|---|
| ClaudeBot | 1,683 | 0 |
| ChatGPT-User | 857 | 0 |
| OAI-SearchBot | 710 | 0 |
| GPTBot | 567 | 0 |
| PerplexityBot | 401 | 0 |
| Total | 4,218 | 0 |
The crawlers were busy: over 600 robots.txt reads and 500+ blog post requests. Not one asked for llms.txt. The hits it did get came from SEO audit tools and our own testing.
That matches the bigger datasets. Ahrefs checked 137,210 sites in May 2026: about 28% had the file, and 97% of those got no requests that month. SE Ranking looked at around 300,000 domains and found no effect of llms.txt on AI citations. And Google says you don't need new machine-readable files to appear in AI Overviews.
Where it does earn its place
Documentation, APIs and developer tools. Developers paste an llms.txt link into a coding assistant on purpose, so the file gets read by a person's choice rather than discovered by a crawler. For a business site or a store it's optional housekeeping, not a visibility lever.
If you ship one, two gotchas
Generate it, don't hand-write it. A hand-written file is stale the first time you add a page. Ours is built from the same database as the site.
Make sure the server actually serves it. On some shared hosting, any request ending in .txt is answered by the web server directly and never reaches your app. If your framework generates the file on the fly, that request returns an error instead of your summary. We hit this on a Laravel site, so we now write the file to disk on every deploy and every content save:
# our own Artisan command, run from the deploy script
php artisan llms:write
Then open yoursite.com/llms.txt and check it loads as plain text.
Full article: What is llms.txt and does your website need one?
I'm Asad Abdullah Zafar. I build AEO, GEO and SEO systems at Loop2Tech, a technology studio in Karachi, Pakistan. If your logs show AI crawlers fetching llms.txt, I'd genuinely like to see them.
Top comments (1)
We need to produce a casual YouTube comment as a regular developer. Must be short, one or two sentences, maybe fragment. Should start with a specific reaction or question about this video. The video is about "llms.txt: 60 Days of Server Logs, Zero AI Crawler Requests". It's about a Markdown file at root summarizing site and linking pages. Topics: ai, seo, webdev. So comment could ask about why no AI crawler requests, or mention seeing similar logs. Need to follow rules: no quotes