llms.txt is deceptively simple — it's just Markdown — so most people write one in five minutes and move on. The problem: a file can look perfectly valid and still be broken in ways that quietly waste an AI crawler's time or get it ignored. Here are the seven mistakes I run into most, and the quick fix for each.
1. More than one H1
The spec wants exactly one top-level # Title. Extra # headings scramble the structure a model relies on. Fix: keep a single #, demote every other heading to ##.
2. No blockquote summary
The one- or two-sentence > blockquote right under the title is the first context a model reads. Skipping it is the single most common miss. Fix: add one line.
# Acme
> Acme turns CSV files into charts. These links cover setup, the API, and pricing.
3. Sections with no links (or no sections at all)
Empty ## sections add nothing; a wall of links with no structure is just as bad. Fix: group your key pages under real headings (## Docs, ## Guides) with a Markdown bullet list of links under each.
4. Broken links
This is the big one. A file can be flawless Markdown and still point at 404s — which burns crawl budget and erodes trust. Most linters check syntax and never actually fetch the URLs, so broken links sail straight through. Fix: verify every link resolves. (This is the check I built my validator around — it fetches each one.)
5. Relative or http:// links
Models may not resolve relative paths like /docs/start, and plain http:// links look untrustworthy and can break under HTTPS-only crawling. Fix: use absolute https:// URLs everywhere.
6. The wrong content type
Serve the file as text/plain; charset=utf-8. A surprising number of servers return text/html for it (a default misconfiguration), and some consumers then won't treat it as llms.txt at all. Fix: set the content type explicitly.
7. A bloated, uncurated dump
llms.txt is a curated index, not a sitemap. Hundreds of undescribed links bury the signal you're trying to send. Fix: keep the best pages, add a short : description after each, and push secondary links to an ## Optional section.
Catch all seven at once
You don't have to eyeball these by hand. Paste your domain into an llms.txt validator — it scores structure, fetches every linked URL, and tells you exactly which of the above to fix, in plain language.
A clean report: 100/100 with every link fetched. Broken links drag the score down even when the Markdown is perfect — which is exactly the failure mode syntax-only checkers miss.
Fix these seven and you're ahead of most llms.txt files on the web today. The bar is genuinely low right now — which is the opportunity.
Top comments (0)