DEV Community: CiteScore

How to write an llms.txt file (with a free generator)

CiteScore — Sat, 10 Oct 2026 17:37:05 +0000

Large language models increasingly read websites to answer questions, but a typical page is full of navigation menus, scripts, cookie banners, and footers. An llms.txt file is a short Markdown file that gives those models a clean map of your site: what it is, and which pages matter most. This guide covers what the file is, where it goes, how to format it, what to avoid, and how it differs from robots.txt.

What llms.txt is

llms.txt is a proposed convention, described at llmstxt.org since 2024. It is not an official standard from a standards body, and no AI provider has promised to read it in every case. Treat it as low-cost documentation for tools that choose to use it. Its main appeal is that it is easy to write and easy to maintain.

Where it goes

Place the file at the root of your domain, so it is available at https://example.com/llms.txt. A file in a subdirectory such as /docs/llms.txt will not be found by convention. Serve it as plain text with a 200 status code, and make sure it is not behind a login or a redirect chain.

The Markdown format

The format is deliberately simple. It has four parts, in this order:

An H1 with the name of the site or project. This is the only required element.
A blockquote summary. One or two sentences that say what the site is and who it is for.
Optional paragraphs of context. Plain text that explains anything the links cannot, such as scope or how the documentation is organized.
H2 sections containing link lists. Each entry is a Markdown link, a colon, and a short description.

The skeleton looks like this:

# Site Name

> One or two sentences that say what the site is and who it is for.

Optional context paragraph.

## Section name

- [Page title](https://example.com/page): Short description of what the page covers.

A complete example

Here is a file for a fictional note-taking app called Quillpad:

# Quillpad

> Quillpad is a note-taking app for teams that stores notes as plain Markdown and syncs them across devices. Setup guides, API docs, and pricing are linked below.

Quillpad saves every note as a standard .md file. The API is versioned, and the current version is v2.

## Documentation

- [Getting started](https://example.com/docs/getting-started): Install Quillpad and create your first workspace.
- [Markdown syntax](https://example.com/docs/markdown): Supported formatting, including tables and task lists.
- [API reference](https://example.com/docs/api): Endpoints, authentication, and rate limits for v2.

## Product

- [Pricing](https://example.com/pricing): Plans, seat limits, and what each tier includes.
- [Security](https://example.com/security): Encryption, data residency, and audit reports.

## Optional

- [Changelog](https://example.com/changelog): Release notes by date.
- [Careers](https://example.com/careers): Open roles.

The Optional section has a specific meaning in the convention. It marks secondary content that a tool can skip when it has limited context space.

Common mistakes

A missing or wrong H1. The H1 anchors the file. Without it, the file does not follow the format.
Links with no descriptions. A bare list of URLs tells a model very little. Each link needs a sentence about what is there.
Serving HTML instead of Markdown. A catch-all route can return your HTML error page at /llms.txt. Check the response with curl or your browser's network tab.
Dead or redirecting links. Use final URLs, and check them whenever you restructure the site.
Listing everything. A 2,000-line file is less useful than a focused 30-line one. Prioritize the pages you most want people and tools to find.
Marketing copy in the summary. The blockquote should describe the site, not sell it. Phrases like "the best" or "revolutionary" add nothing.
Letting it go stale. A file that points to removed pages is worse than no file at all.
Blocking it. Do not disallow /llms.txt in robots.txt, and do not require authentication to read it.

How it differs from robots.txt

These two files do different jobs, and most sites benefit from having both.

robots.txt tells crawlers what they may request. It holds allow and disallow rules for each user agent and follows the Robots Exclusion Protocol (RFC 9309). It controls crawling, and compliance is voluntary, though major crawlers generally honor it.

llms.txt does not restrict access. It is a curated map that points a model to the content you consider most useful and briefly explains each part. If you want to keep a crawler away from pages, use robots.txt or real access controls. If you want to point a model toward your best documentation, write an llms.txt.

A short checklist

Put the file at /llms.txt on your root domain.
Start with an H1 and a one- or two-sentence blockquote.
Use H2 sections with full URLs and one-line descriptions.
Move secondary material under "Optional."
Confirm that every link returns a 200 status.
Review the file whenever you make major changes.

If you would rather not write the file by hand, the free llms.txt generator can produce a starting draft. Edit it, then check the result against the rules above before you publish.

Which AI crawlers should you allow in robots.txt? (GPTBot, PerplexityBot, Google-Extended and more)

CiteScore — Sat, 10 Oct 2026 14:44:15 +0000

AI assistants find and cite websites in two different ways. Some crawlers collect pages to train future models. Others fetch pages so an assistant can search the web and link to your site in an answer. Your robots.txt file can treat these differently, but only if you know what each user-agent does.

One caveat first: robots.txt is a request, not a lock. Reputable crawlers honor it. Anything else needs server-level or firewall blocking.

The major AI crawlers

User-agent	Owner	What it does
GPTBot	OpenAI	Collects content for training OpenAI models
OAI-SearchBot	OpenAI	Indexes pages for ChatGPT search
ChatGPT-User	OpenAI	Fetches a page when a ChatGPT user asks for it
ClaudeBot	Anthropic	Collects content for training Anthropic models
Claude-SearchBot	Anthropic	Indexes pages for Claude's search features
Claude-User	Anthropic	Fetches a page when a Claude user asks for it
PerplexityBot	Perplexity	Indexes pages for Perplexity answers and citations
Perplexity-User	Perplexity	Fetches a page when a Perplexity user asks for it
CCBot	Common Crawl	Builds a public web archive that many AI teams use for training
Google-Extended	Google	A control token, not a crawler. Governs use of content for Gemini training and grounding
Applebot-Extended	Apple	A control token. Governs use of Applebot-crawled content for Apple's AI training

Google-Extended and Applebot-Extended are not separate crawlers. They are opt-out tokens that apply to content already crawled by Googlebot or Applebot. Blocking them does not remove you from Google Search, and AI Overviews follow Googlebot and standard directives such as nosnippet.

A recommended robots.txt block

If you have no AI rules today, your site is already open to all of these bots. The block below opts out of training while staying visible in AI search:

# Training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Search and answer crawlers: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

Keep your existing rules for Googlebot and * in their own groups.

Training vs. answer crawlers: the trade-off

Training crawlers feed future model versions. Blocking them signals that you do not want your content used for future training. It does not remove anything a model has already learned, and it is hard to measure its effect on your traffic.

Answer crawlers decide whether you appear in live answers. If you block PerplexityBot or OAI-SearchBot, your pages are less likely to be cited when people ask a relevant question. For most sites that want visibility, that is the bigger cost.

The block above is a reasonable default. Adjust it if you have licensing concerns, paywalled content or private material. Vendors also note that user-triggered fetchers such as ChatGPT-User may not apply robots.txt rules, so treat those rules as best effort.

How to verify your rules

Open https://yoursite.com/robots.txt in a browser. It should return 200 and plain text. A 404 means no rules, so everything is allowed. A 5xx error can stop crawlers entirely.
Test the rules with Python's standard library:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://yoursite.com/robots.txt")
rp.read()
for bot in ["GPTBot", "OAI-SearchBot", "PerplexityBot"]:
    print(bot, rp.can_fetch(bot, "https://yoursite.com/blog/post"))

Search your server logs or analytics for these user-agent strings. Confirm that the bots you allowed reach your pages, and that the bots you blocked request only robots.txt.
Check your CDN or WAF. A bot-protection rule that returns 403 blocks crawlers regardless of robots.txt, and it can block the bots you meant to allow.

Python's parser does not handle every wildcard the way Google does, so test important paths with more than one tool.

Common mistakes

Blocking Googlebot by accident. Google-Extended is not Googlebot. Blocking Googlebot removes you from Google Search.
Assuming a * rule covers AI bots. A crawler follows the most specific group that matches its name and ignores *. If GPTBot has its own group, that group must contain every rule you want for GPTBot.
Using robots.txt for privacy. The file is public and it is only a request. Protect private pages with authentication.
Mixing up Disallow and Allow. Disallow: / blocks the whole site, while an empty Disallow: allows everything. A misplaced slash can hide your entire site from search.
Ignoring the CDN. Bot-protection rules can override what your robots.txt says.
Expecting instant results. Crawlers cache robots.txt, and removal from AI answers takes time.

Test your robots.txt AI-crawler rules for free at https://citescore.vercel.app/free-check

How to write an llms.txt file (with a copy-paste template)

CiteScore — Sat, 10 Oct 2026 14:42:06 +0000

How to write an llms.txt file (with a copy-paste template)

If you want AI assistants and large language models to understand your site accurately, you can give them a short, curated map. That map is an llms.txt file. This guide explains what it is, where it goes, how to format it, and how to test it before you rely on it.

What llms.txt is

llms.txt is a proposed standard, introduced by Jeremy Howard in 2024 and documented at llmstxt.org. It gives language models a plain Markdown file that summarises your website: what it is, what it offers, and which pages matter most.

It works alongside two files you probably already have. robots.txt tells crawlers what they may access. A sitemap.xml lists every URL. llms.txt does something different: it selects the most important content and explains it in a format a model can read within a limited context window.

Be realistic about adoption. The standard is still informal, and not every AI company has confirmed that it reads llms.txt. Treat it as low-cost documentation that helps the tools that do use it, not as a guaranteed ranking lever.

Where it goes

The file must live at the root of your domain:

https://example.com/llms.txt

Serve it with a 200 status as plain text or Markdown. It should not redirect to a login page or depend on JavaScript to render. If you have subdomains such as docs.example.com and want them covered, publish a separate llms.txt on each one.

The format

The structure is deliberately simple. Include these parts, in this order:

An H1 title. The name of the project or site. This is the only required element.
A blockquote summary. One or two sentences that explain what the site is.
Optional paragraphs. Extra context in plain prose, such as who the site is for or what is out of scope.
H2 sections. Groups of links, each written as a Markdown list item: - [Link title](URL): short note.
An Optional section. Links a model can skip when context is tight.

Use absolute URLs and point them at clean, public pages that load directly.

Full copy-paste template

Replace the placeholders and delete any section you do not need:

# Example Co

> Example Co makes invoicing software for freelancers. The product covers quotes, recurring billing and tax exports for the US and UK.

Pricing changes from time to time, so check the pricing page for current numbers. Documentation covers the public REST API and the Zapier integration.

## Docs

- [Getting started](https://example.com/docs/getting-started): Create an account and send your first invoice
- [API reference](https://example.com/docs/api): Endpoints, authentication and rate limits
- [Webhooks](https://example.com/docs/webhooks): Event types and payload examples

## Product

- [Features](https://example.com/features): Full list of invoicing and tax features
- [Pricing](https://example.com/pricing): Plans and per-seat costs

## Optional

- [Blog](https://example.com/blog): Product updates and tutorials
- [Careers](https://example.com/careers): Open roles

Common mistakes

Putting the file in the wrong place. /docs/llms.txt is not the root. Crawlers look for /llms.txt.
Dead or redirected links. Every URL should return 200. A list full of 404s signals an unmaintained file.
Listing everything. Pasting 300 URLs defeats the purpose. Pick the 10 to 30 pages that answer the most common questions.
Vague descriptions. "Our amazing solutions" tells a model nothing. Say what the page actually contains.
Marketing copy in the summary. Keep the blockquote factual. Plain statements are easier to use than slogans.
Forgetting the H1. Without a title the file does not follow the format, and parsers may fail on it.
Blocking the file. If a firewall, CDN rule or robots.txt rule blocks /llms.txt, nothing else you write matters.
Letting it go stale. Update the file when you launch, rename or remove a major page.

How to test it

Check the status and redirects. Run curl -I https://example.com/llms.txt and confirm a 200 response with no redirect chain.
Check the content type. It should be text/plain or text/markdown, not text/html.
Read it as a stranger would. Open the file and ask whether someone who has never seen the site can tell what it offers from the first 20 lines.
Check every link. Extract the URLs from the file and request each one with curl. Flag anything that is not a 200.
Confirm the structure. The first line should be a single H1, followed by a blockquote, then H2 sections that contain list items.
Check robots.txt. Make sure the pages you link are not disallowed for the AI user agents you want to reach.

Repeat these checks after every deploy that touches the file.

Summary

A good llms.txt is short, accurate and maintained. Start with ten strong links, a factual summary and a clean root URL, then expand as your documentation grows.

You can check your site's AI-crawler setup for free at https://citescore.vercel.app/free-check