llms.txt vs robots.txt for AI search: an indie site decision guide

Quick answer

Do not ship llms.txt because someone promised it will boost Google AI Overviews. Google says its Search systems do not use llms.txt or other special AI files for Search visibility. Treat normal SEO, crawlable HTML, useful content, internal links, canonical URLs, and sitemaps as the base layer.

Still, llms.txt can be worth shipping for a different reason: agent usability. If your site has developer docs, tools, product pages, changelogs, pricing notes, or practical guides that an AI agent may need to summarize or act on, a small llms.txt file gives that agent a clean map of what matters.

Use this default:

Need Google Search / AI Overviews visibility?
-> keep normal SEO strong; do not rely on llms.txt.

Need to control training, search indexing, or user-requested AI fetches?
-> use robots.txt tokens for each crawler family.

Need agents to understand your site faster?
-> publish a concise llms.txt plus clean markdown or simple HTML pages.

For IndieSeek, this means llms.txt is a low-cost agent-readiness file, while robots.txt remains the policy file for crawler access.

Who this is for

This guide is for indie developers, solo SaaS builders, SEO operators, and technical founders who are seeing conflicting advice about llms.txt, AI crawler blocking, ChatGPT search, Claude search, Perplexity citations, and Google AI Mode.

It is especially relevant if you publish practical content, documentation, product pages, or tools and want two things at the same time:

• keep your content discoverable through search and answer engines;
• avoid accidentally allowing every training crawler just because you want search visibility.

If you need the broader content-quality checklist, start with the AI search SEO checklist for indie sites. If your issue is a broken preview after sharing a page, use the Slack OG image debugging checklist or the free meta tag checker.

What changed and why now

The confusion comes from three changes landing at the same time.

First, Google expanded AI features in Search and published explicit guidance for generative AI Search optimization. The key line for this topic is conservative: Google says you do not need llms.txt, special AI files, AI text files, or extra machine-readable markup to appear in Google Search generative features.

Second, AI companies have split crawlers by purpose. OpenAI documents independent controls for OAI-SearchBot and GPTBot: one is about surfacing sites in ChatGPT search results, while the other signals whether content should be used for model training. Anthropic documents ClaudeBot, Claude-User, and Claude-SearchBot. Perplexity documents PerplexityBot for search results and Perplexity-User for user-triggered page access.

Third, Cloudflare and the wider developer ecosystem are pushing "agent-ready" sites: markdown versions, llms.txt, llms-full.txt, and clear instructions that help coding agents and browsing agents use a site without wasting context on navigation, scripts, and layout.

Those are not the same job. Mixing them is how indie sites end up with either false hope or over-blocked content.

The decision tree

Are you trying to rank in Google Search, AI Overviews, or AI Mode?
  -> Prioritize normal SEO. Keep pages crawlable, useful, canonical, and linked.
  -> Do not count llms.txt as a Google Search ranking or inclusion lever.

Are you trying to appear in ChatGPT, Claude, or Perplexity search answers?
  -> Check the crawler that powers that surface.
  -> Allow search/index crawlers you actually want.
  -> Do not block the search crawler while expecting citations from that platform.

Are you trying to opt out of model training?
  -> Use crawler-specific robots.txt tokens such as GPTBot, Google-Extended, or ClaudeBot.
  -> Understand that user-triggered fetchers may have separate rules.

Are you trying to help agents complete a task on your site?
  -> Add a short llms.txt file.
  -> Link only the pages an agent should read first.
  -> Prefer stable URLs, concise descriptions, and markdown-friendly pages.

Practical setup for an indie site

1. Keep the normal SEO layer boring

Before touching AI-specific files, verify the basics:

curl -I https://example.com/
curl https://example.com/sitemap.xml
curl https://example.com/robots.txt

Your public pages should return 200, your canonical URLs should match the production route, and your sitemap should include the important pages. If the page is blocked from normal crawling, hidden behind JavaScript-only rendering, duplicated across routes, or missing internal links, llms.txt will not fix the traffic problem.

2. Separate search crawlers from training crawlers

Use a small policy table before editing robots.txt:

Surface	Usually allow when you want visibility	Usually block when you want training opt-out
Google Search and AI Search features	Googlebot and normal Google Search crawling	Google-Extended controls Gemini training/grounding use and does not affect Google Search inclusion
ChatGPT search	OAI-SearchBot	GPTBot
Claude search and user fetches	Claude-SearchBot, maybe Claude-User	ClaudeBot
Perplexity answers	PerplexityBot, maybe Perplexity-User	Perplexity says its listed crawlers are not training crawlers, but user fetch behavior is separate

The exact policy depends on your business. A product marketing site usually wants search and answer-engine visibility. A paid content site may prefer stricter blocking. A developer-tool site may want agents to read docs while still opting out of model training.

3. Add llms.txt as a curated map, not a dump

A good llms.txt file is short. It should tell an agent what the site is, which pages matter, and why those pages exist.

# Example Product

> A small Mac utility for reviewing AI-generated Markdown and HTML documents.

## Product

- [Overview](https://example.com/): What the app does, who it is for, and current limitations.
- [Pricing](https://example.com/pricing/): Free and paid plan boundaries.
- [Download](https://example.com/download/): Current stable Mac download.

## Guides

- [How to review AI-generated docs](https://example.com/blogs/review-ai-docs/): A practical workflow for checking generated Markdown and HTML.
- [Launch checklist](https://example.com/blogs/launch-checklist/): SEO, metadata, and distribution checks for indie launches.

Do not include every tag page, archive page, legal page, or thin marketing variant. Agents have context budgets. A noisy llms.txt file teaches the agent that your site is noisy.

4. Make linked pages easy to parse

The real gain is not the file name. The gain is that an agent can retrieve useful content with less cleanup.

For a small site, that means:

• server-render the main content;
• keep page titles and headings descriptive;
• avoid hiding the important answer behind interactive widgets;
• include product limits, pricing boundaries, and support routes in plain text;
• keep URLs stable;
• consider markdown copies for docs or long technical guides.

Cloudflare's own agent-readiness guidance points agents to markdown versions and llms.txt so they do not waste context on HTML chrome. That is the right mental model: reduce agent friction, not chase a ranking shortcut.

Common mistakes

Mistake 1: blocking the crawler you need for visibility

If you block a search/index crawler and then expect that same platform to cite you, you are working against yourself. OpenAI and Perplexity both document search-oriented bots that should be allowed if you want visibility in their search answers.

Mistake 2: assuming one AI bot equals one purpose

Crawler names now encode different jobs: training, search indexing, user-requested browsing, testing tools, and cloud-agent fetches. Review the official documentation before copying a generic "block all AI bots" snippet.

Mistake 3: treating llms.txt as access control

llms.txt is a content map. It is not a training opt-out, not a paywall, not a crawler permission system, and not a replacement for robots.txt, auth, WAF rules, or rate limits.

Mistake 4: publishing an auto-generated dump

An automatically generated list of every URL is usually worse than a curated list. If a sitemap already does that job, llms.txt should add editorial judgment: which pages should an agent read first?

IndieSeek publishing checklist

Use this before shipping an AI-ready content file:

[ ] Important pages return anonymous HTTP 200.
[ ] Canonical tags match production URLs.
[ ] Sitemap includes the page.
[ ] robots.txt allows normal search crawlers.
[ ] robots.txt separates search/index bots from training bots.
[ ] llms.txt is under a practical context size and links only important pages.
[ ] Each llms.txt link has a one-line description.
[ ] Linked pages expose the answer in plain HTML or markdown-friendly content.
[ ] Internal links connect the new page to a real topic cluster.
[ ] Search submission runs after production deploy.

For IndieSeek's current site, the build already generates a concise /llms.txt from app, tool, and blog metadata. The next useful step is not more special files; it is keeping each new article specific enough that both search crawlers and agents can understand the task it solves.

FAQ

Does llms.txt help Google SEO?

Not directly. Google says Search does not use llms.txt as a special file for generative AI Search visibility. If a file is crawlable, Google may discover it like other text files, but that is not the same as treating it as a ranking or inclusion mechanism.

Should an indie site still add it?

Yes, if the site has docs, tools, product pages, or guides that agents may need to read. The cost is low when the file is curated and generated from real metadata. The risk is wasting time on it before fixing basic SEO.

Should I block all AI crawlers?

Usually not for a public product site that wants discovery. Split the question by purpose: allow search/index crawlers you want, block training crawlers if that matches your content policy, and handle abusive traffic with WAF or bot controls.

Should llms.txt be in the sitemap?

It does not need to be. If you want agents and humans to find it, link it from docs or expose it at the root. The sitemap should still focus on indexable pages that users search for.

Sources

• Google Search Central: optimizing for generative AI features
• Google crawler documentation for Google-Extended
• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• Cloudflare managed robots.txt documentation
• Cloudflare Agent Readiness announcement
• Answer.AI llms.txt proposal