DEV Community

faisal hassan
faisal hassan

Posted on Edited on

How to Add llms.txt and AI Search Bot Rules (OAI-SearchBot, PerplexityBot, ClaudeBot) to Any Website

Traditional sitemap.xml files list every URL on your domain for search engine crawlers, whereas llms.txt is a lightweight Markdown map placed at your site root (yourdomain.com/llms.txt) that helps AI assistants and coding agents read your core documentation without HTML navigation clutter.

1. Separating AI Search Citations from Model Training in robots.txt

If you want your technical guides cited in ChatGPT Search, Perplexity, or Claude answers while controlling model pre-training access, split your user-agent rules:

text
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

  1. Free Browser Generator for llms.txt & AI robots.txt To save time formatting Markdown sections and bot rules manually, I built a free client-side llms.txt & AI Crawler Policy Generator that outputs a downloadable llms.txt file and platform-specific robots.txt rules in one click.

If you also run local open-weight models via Ollama or llama.cpp, you can test hardware memory footprints with the Local LLM & GPU VRAM Compatibility Calculator.

Top comments (0)