There is a small file you can put at the root of your website that does something no amount of good copywriting can. It tells an AI model, directly and unambiguously, who you are, what your most authoritative pages are, and which parts of your site it should ignore.
It's called llms.txt. It takes minutes to create. Almost nobody has one. And in a web where assistants read your site to decide whether to recommend you, that's a strange gap to leave open.
What is llms.txt? (Short answer)
An llms.txt file is a Markdown file placed at the root of your domain, at https://yourdomain.com/llms.txt, that gives Large Language Models a concise, structured overview of your site. It states your identity, points to your canonical sources, and tells models which sections carry authority and which don't.
It's the same idea as robots.txt, aimed at a different reader. robots.txt controls permission: which crawlers may access which paths. llms.txt provides context: for the models that do crawl you, here is what your site means and where the truth lives. They're complementary, not competing. You want both.
Key takeaways
- llms.txt is a context file, not a permission file. It doesn't block anyone. It tells AI models what your site is, what's authoritative, and what to cite.
- It lives at your domain root, in plain Markdown, exactly like robots.txt.
- The value is in curation. Pointing a model at your five canonical pages beats letting it reconstruct your business from a hundred scattered ones.
- Almost nobody has published one. That makes it a cheap, uncontested advantage right now, before it becomes table stakes.
Why a model needs this file
Think about what an assistant actually does when it wants to answer a question about your company. It fetches pages, reads text, guesses which of them is authoritative, and assembles an answer. Nothing in that process tells it that your documentation is the source of truth and your old landing page from 2023 is not. It infers. It gets it wrong. It quotes your pricing page as though it were your product philosophy.
llms.txt collapses that guesswork. Instead of forcing a model to reconstruct your business from fragments, you hand it a map: this is who we are, this is our category, these three to five URLs are canonical, these directories are authoritative, these ones aren't.
That has four practical effects, which map neatly onto why the file exists at all:
Standardized context. One structured source of truth every model can parse the same way, rather than a different impression per crawl.
Better citations. When an assistant knows which of your pages is canonical, it cites that one. Clear citation paths mean you get quoted from the page you'd have chosen.
Crawler guidance. You steer agents toward your most important content and away from noise, so limited attention gets spent on what represents you.
Future proofing. The agent web is arriving. A machine-readable description of your site is going to matter more each year, not less.
What actually belongs in a good llms.txt
Most llms.txt examples online are a title and a list of links. That's a start, but it leaves most of the value on the table. A file that genuinely earns trust signals carries more:
- Identity metadata. Your brand name, a clean description, and your industry, stated plainly rather than inferred from a homepage headline.
- Authoritative content links. Your real documentation, guides, examples, changelog, and research, organized by section rather than dumped as a flat list.
- Explicit allow and canonical directives. Which directories carry authority, and which three to five URLs are the canonical sources you want cited.
- Explicit disallow directives. Which paths are not authoritative. Pricing pages and legal boilerplate are the classic examples: real pages, but a terrible basis for an AI's description of what you do.
- Supporting references. Useful URLs that give context without being canonical.
- Content governance and attribution. How often the content updates, and how you'd like to be credited.
That last group is what separates a link dump from a document a model can rely on. You are not just listing pages. You are declaring which of them speak for you.
How our free generator builds it for you
Writing all of that by hand for a real site is tedious, which is exactly why most teams never do it. So we built the AI LLMs.txt Generator to do the tedious part.
You give it your URL. From there:
It crawls your site. By default it reads about 20 pages at a link depth of 2, and you can push it to 100 pages and depth 5 for larger sites. It deliberately prioritizes /docs and /api paths, because that's usually where a site's real substance lives.
It classifies and scores what it finds. Every discovered page is sorted into a section, whether that's Documentation, Examples, Blog, Changelog, Integrations, Research, Support, or About, and scored for authoritativeness. That ranking is what decides which pages are worth surfacing to a model and which are noise.
It curates with an LLM pass. A single model call handles the judgment calls a crawler can't: classifying your industry, cleaning up your brand name, estimating how often your content changes, and selecting the three to five canonical sources that best represent you.
It applies sensible defaults. Pricing and legal pages are always treated as non-authoritative and pushed to the disallow list. They're real pages, they're just not what you want an AI quoting when someone asks what your company does.
It renders the file. Out comes a complete llms.txt with identity metadata, authoritative content organized by section, explicit allow, canonical, and disallow directives, supporting references, content governance, and attribution.
Then you preview it, edit anything you disagree with in real time, download it, and drop it at your domain root. The whole loop is minutes, and the tool is free.
Deploying it, and what to do next
Put the file at https://yourdomain.com/llms.txt. That's it. No configuration, no build step, no plugin. If you can serve robots.txt, you can serve this.
Two things worth doing afterwards. First, read the file before you ship it. The generator makes good judgment calls, but you know your site better than a crawler does. If it marked something canonical that shouldn't be, fix it. The point of the preview step is that you stay the editor.
Second, check the rest of your foundation. llms.txt is one signal in a larger picture called agent-readiness: whether AI systems can find your pages, read them, are permitted to crawl them, and can discover what your site offers. Publishing an llms.txt while quietly blocking GPTBot at the CDN is a wasted afternoon. If you want to see where you stand across all of it, our agent-readiness scanner will score your site and hand you the specific fixes.
The honest case for doing this today
llms.txt is not yet a universal, enforced standard, and anyone who tells you it guarantees citations is overselling. What it is: a cheap, explicit, machine-readable statement of what your site means, published in the place a model would look for one, at a moment when almost none of your competitors have bothered.
That's the whole argument. Low cost, real upside, no incumbent to fight. The brands that publish one now are simply making themselves easier to understand than the brands that don't, and being easy to understand is most of what it takes to be recommended.
Generate your llms.txt file for free
FAQ
What is an llms.txt file?
A Markdown file that gives Large Language Models a concise, structured overview of your project: your identity, your authoritative pages, your canonical sources, and which sections to ignore. It helps AI assistants understand your content and cite it correctly.
Where should I place the file?
In the root directory of your website, at https://yourdomain.com/llms.txt, exactly like robots.txt.
What format does it use?
Standard Markdown, with clear headers and lists so models can parse it reliably.
Does llms.txt replace robots.txt?
No. robots.txt controls crawling permissions, deciding who may access what. llms.txt is complementary: it provides context and guidance to the models that do crawl you, so they give better answers about your content. Keep both.
Will publishing an llms.txt guarantee AI cites me?
No, and be skeptical of anyone who claims otherwise. It removes ambiguity about what your site means and which pages are canonical, which makes correct citation far more likely. It's one strong signal among several, not a switch.
How many pages will the generator crawl?
About 20 by default at a link depth of 2, configurable up to 100 pages and depth 5. It prioritizes /docs and /api paths, where the substantive content usually lives.
Top comments (0)