llms.txt, robots.txt and AI crawlers: how to make your website readable by AI

GetVizible — Sun, 25 Oct 2026 01:42:25 +0000

Short answer: allow the AI crawlers in robots.txt, serve your key facts as plain HTML text, add schema markup, and publish an llms.txt file: a short Markdown summary of who you are with links to your most important pages.

A lot of "why doesn't ChatGPT mention us?" problems start with something boring: the bots that fetch pages for AI answers can't read the site. Here's the checklist.

The AI crawlers you should know

Company	User agent	Used for
OpenAI	`GPTBot`	Training data
OpenAI	`OAI-SearchBot`	ChatGPT search results
OpenAI	`ChatGPT-User`	Pages fetched when a user asks
Anthropic	`ClaudeBot`	Training data
Anthropic	`Claude-SearchBot`, `Claude-User`	Search and user-requested fetches
Perplexity	`PerplexityBot`, `Perplexity-User`	Search index and user-requested fetches
Google	`Google-Extended`	Use in Gemini (a control token; crawling is done by Googlebot)
Apple	`Applebot-Extended`	Use in Apple's AI features
Common Crawl	`CCBot`	Open dataset used by many models

The important distinction: blocking a search or user agent removes you from that engine's live answers. Blocking a training agent only affects future model versions. A lot of sites added a blanket AI block in 2024–2025 to opt out of training and accidentally took themselves out of ChatGPT search and Perplexity too.

Decide per agent. For most brands that want to be recommended, allowing all of them is the simplest choice.

A robots.txt that allows AI crawlers

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

Also check your CDN or WAF. Bot-protection settings can silently block these agents even when robots.txt allows them. Your server logs will tell you whether they're getting 200s or 403s.

What llms.txt is (and isn't)

llms.txt is a proposed convention: a Markdown file at /llms.txt that gives language models a short, clean summary of your site. It typically has:

a title and one-paragraph description
key facts (what you do, who it's for, pricing, contact)
links to the most useful pages, one line each

# Example Co

> Example Co makes invoicing software for freelancers in the EU.

## Key facts
- Pricing: free plan; Pro is €12/month
- Founded 2024, based in Lisbon

## Docs
- [Getting started](https://example.com/docs/start): set up in 5 minutes
- [Pricing](https://example.com/pricing): plans and limits

Some sites also publish llms-full.txt with the full text of their key pages. Support differs by engine, so treat llms.txt as a cheap, helpful extra rather than a guarantee. Crawler access and clear page text matter more.

Other technical checks

Server-rendered text. Prices, features and FAQs should be in the HTML, not loaded only by JavaScript. Several AI fetchers don't execute JS.
Schema markup. Organization, Product or SoftwareApplication, FAQPage, Article and BreadcrumbList, matching the visible text.
A sitemap listing every public page with accurate lastmod dates.
Fast, stable pages without interstitials that hide content.
One canonical URL per page.

FAQ

Does blocking GPTBot remove me from ChatGPT?
Blocking GPTBot stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot, and user-triggered fetches use ChatGPT-User, so block or allow those separately.

Is llms.txt required?
No. It's a voluntary convention. It's quick to add and gives AI tools a clean summary of your site, but crawler access, clear page text and third-party mentions matter more.

Originally published on the Get AI Vizible blog. Disclosure: I'm the founder of Get AI Vizible, which checks AI crawler access, llms.txt and schema as part of tracking how ChatGPT, Claude, Gemini and Perplexity describe your brand.

DEV Community: GetVizible

llms.txt, robots.txt and AI crawlers: how to make your website readable by AI

The AI crawlers you should know

A robots.txt that allows AI crawlers

What llms.txt is (and isn't)

Other technical checks

FAQ