llms.txt is a plain-text file at your domain root that gives AI crawlers a curated map of your site — what it is, what matters, where the canonical pages live. Think robots.txt, but for helping a model understand your site rather than restricting what it can fetch. No schema to validate against, no build step required. Readability by a model is the whole specification.
That simplicity is also the trap. Because there's no schema and no build step, it's tempting to hand-write it once and forget it exists. On any site whose content changes — prices, product names, which pages are live — a hand-written llms.txt starts lying to crawlers the first time you ship an update and forget the file.
What actually goes wrong
Picture an ecommerce or SaaS site with a llms.txt listing products and prices. Someone changes a price in the CMS or database. The rendered site is correct. The llms.txt, written by hand three months ago, still says the old number. A crawler reads it, an AI answer quotes it, and now a prospective buyer is told a price you don't charge — which is worse for trust than not being quoted at all.
The same failure mode hits anything else you'd put in the file: a page that got renamed or removed, a section that got restructured, a product that got discontinued. None of it throws an error. The file just quietly stops matching reality, and nothing in your build pipeline notices, because llms.txt isn't code — it's prose nobody re-reads.
The fix: generate it, don't author it
Treat llms.txt as a build artifact, not a hand-maintained document. If your site already has a single source of truth for its content — a CMS, a data file, a database — write a small script that renders llms.txt from that same source at build time, the same way you'd render a sitemap.xml. Then structural drift becomes impossible by construction: the file can't say something the site doesn't, because they're generated from the same data in the same step.
Concretely:
- Identify the source of truth for whatever you'd list — a products array, a CMS collection, a content directory. Not a separate list someone maintains by hand.
- Write the generator alongside your other build-time generation (sitemap, RSS, OG images) rather than as a one-off script that's easy to forget exists.
- Run it on every build, not on a schedule or "when someone remembers." If it's not wired into CI, it will go stale exactly like the hand-written version did.
- Keep the format dead simple: an H1 with the site name, a one-line blockquote summary, then linked sections with a sentence each. Resist the urge to make it comprehensive — a curated map beats an exhaustive dump.
Is it worth the effort?
Worth being honest here: support for llms.txt is uneven, and no major AI provider has committed to reading it publicly. Nobody can promise you a ranking or citation lift from shipping one. But the cost calculus is lopsided — it's one generator function, no maintenance burden once it's wired into the build, and the downside of skipping it is nil while the downside of a stale hand-written one is an AI confidently repeating something false about you. If any part of your traffic strategy touches AI search, generate one from your real data and never touch it by hand again.
Full answer, with the robots.txt comparison and what to include for different site types: https://agentkitworks.com/answers/what-is-llms-txt
Top comments (1)
tbh i was wondering how to handle the sync for this. makes way more sense to just script it into the build process.