<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: faisal hassan</title>
    <description>The latest articles on DEV Community by faisal hassan (@faixu).</description>
    <link>https://dev.to/faixu</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4256051%2F71f84781-cd82-4dfc-a971-a16f52c9b797.jpg</url>
      <title>DEV Community: faisal hassan</title>
      <link>https://dev.to/faixu</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/faixu"/>
    <language>en</language>
    <item>
      <title>How to Add llms.txt and AI Search Bot Rules (OAI-SearchBot, PerplexityBot, ClaudeBot) to Any Website</title>
      <dc:creator>faisal hassan</dc:creator>
      <pubDate>Fri, 20 Nov 2026 16:25:46 +0000</pubDate>
      <link>https://dev.to/faixu/how-to-add-llmstxt-and-ai-search-bot-rules-oai-searchbot-perplexitybot-claudebot-to-any-website-14ol</link>
      <guid>https://dev.to/faixu/how-to-add-llmstxt-and-ai-search-bot-rules-oai-searchbot-perplexitybot-claudebot-to-any-website-14ol</guid>
      <description>&lt;p&gt;Traditional &lt;code&gt;sitemap.xml&lt;/code&gt; files list every URL on your domain for search engine crawlers, whereas &lt;strong&gt;&lt;code&gt;llms.txt&lt;/code&gt;&lt;/strong&gt; is a lightweight Markdown map placed at your site root (&lt;code&gt;yourdomain.com/llms.txt&lt;/code&gt;) that helps AI assistants and coding agents read your core documentation without HTML navigation clutter.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Separating AI Search Citations from Model Training in &lt;code&gt;robots.txt&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;If you want your technical guides cited in ChatGPT Search, Perplexity, or Claude answers while controlling model pre-training access, split your user-agent rules:&lt;/p&gt;

&lt;p&gt;text&lt;br&gt;
User-agent: OAI-SearchBot&lt;br&gt;
Allow: /&lt;/p&gt;

&lt;p&gt;User-agent: PerplexityBot&lt;br&gt;
Allow: /&lt;/p&gt;

&lt;p&gt;User-agent: Claude-SearchBot&lt;br&gt;
Allow: /&lt;/p&gt;

&lt;p&gt;User-agent: GPTBot&lt;br&gt;
Disallow: /&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Free Browser Generator for &lt;code&gt;llms.txt&lt;/code&gt; &amp;amp; AI &lt;code&gt;robots.txt&lt;/code&gt;
To save time formatting Markdown sections and bot rules manually, I built a free client-side &lt;strong&gt;&lt;a href="https://technolayman.blogspot.com/p/free-llmstxt-ai-crawler-policy.html" rel="noopener noreferrer"&gt;llms.txt &amp;amp; AI Crawler Policy Generator&lt;/a&gt;&lt;/strong&gt; that outputs a downloadable &lt;code&gt;llms.txt&lt;/code&gt; file and platform-specific &lt;code&gt;robots.txt&lt;/code&gt; rules in one click.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you also run local open-weight models via Ollama or llama.cpp, you can test hardware memory footprints with the &lt;strong&gt;&lt;a href="https://technolayman.blogspot.com/p/local-llm-gpu-vram-compatibility.html" rel="noopener noreferrer"&gt;Local LLM &amp;amp; GPU VRAM Compatibility Calculator&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>ai</category>
      <category>webdev</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
