<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ansvra</title>
    <description>The latest articles on DEV Community by Ansvra (@ansvrahq).</description>
    <link>https://dev.to/ansvrahq</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4251873%2F536d8ff4-034e-4832-acd7-cb81c974f513.png</url>
      <title>DEV Community: Ansvra</title>
      <link>https://dev.to/ansvrahq</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ansvrahq"/>
    <language>en</language>
    <item>
      <title>Allow AI search crawling without allowing GPTBot collection</title>
      <dc:creator>Ansvra</dc:creator>
      <pubDate>Wed, 18 Nov 2026 12:32:01 +0000</pubDate>
      <link>https://dev.to/ansvra/allow-ai-search-crawling-without-allowing-gptbot-collection-49fj</link>
      <guid>https://dev.to/ansvra/allow-ai-search-crawling-without-allowing-gptbot-collection-49fj</guid>
      <description>&lt;p&gt;A website can permit discovery for AI search while declining a separate training-related crawl. The useful starting point is a decision about each purpose, followed by a check that the deployed configuration expresses it.&lt;/p&gt;

&lt;p&gt;This guide is adapted from &lt;a href="https://ansvra.com/articles/oai-searchbot-gptbot-and-chatgpt-user-explained/" rel="noopener noreferrer"&gt;Ansvra’s full OpenAI crawler guide&lt;/a&gt;. Operator references were checked on 18 November 2026. Recheck them before changing production rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  Choose the purpose before the rule
&lt;/h2&gt;

&lt;p&gt;OpenAI documents independent controls for OAI-SearchBot and GPTBot. OAI-SearchBot supports its search features; GPTBot collects material that may be used for foundation-model training. ChatGPT-User serves certain user-initiated actions, for which robots.txt may not apply. OAI-AdsBot checks submitted advertising pages. These are different access questions. &lt;a href="https://developers.openai.com/api/docs/bots" rel="noopener noreferrer"&gt;OpenAI crawler reference&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Write down the intended policy before opening the configuration file. For this example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Public pages may be crawled for search discovery.&lt;/li&gt;
&lt;li&gt;GPTBot collection is declined.&lt;/li&gt;
&lt;li&gt;Private content continues to require authentication.&lt;/li&gt;
&lt;li&gt;No firewall or access-control exception is introduced.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Express the two independent preferences
&lt;/h2&gt;

&lt;p&gt;The following fragment illustrates that policy. Merge it with your existing rules after reviewing them; do not replace the whole file with it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fragment expresses crawler preferences. It does not protect a private document, remove material collected earlier, or demonstrate that a page will be selected for an answer.&lt;/p&gt;

&lt;p&gt;If your existing wildcard group restricts other public paths, review those restrictions before adding a specific bot group. Under the &lt;a href="https://www.rfc-editor.org/rfc/rfc9309.html" rel="noopener noreferrer"&gt;Robots Exclusion Protocol&lt;/a&gt;, a matching specific group is not simply an extension of the wildcard group. Groups matching the same product token are combined. Within the applicable rules, the longest matching path governs; an equally specific allow/disallow tie favors allow.&lt;/p&gt;

&lt;p&gt;For example, a search policy that excludes a members section needs its own applicable restriction:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User-agent: OAI-SearchBot
Allow: /
Disallow: /members/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That last line is still a crawler preference. Keep authentication on member content and verify it independently.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test what is actually deployed
&lt;/h2&gt;

&lt;p&gt;Keep a copy of the earlier file and record the change date. Test the public robots.txt response and representative URLs after deployment, including a public article, an excluded path and a private resource.&lt;/p&gt;

&lt;p&gt;Inspect the final response after redirects. A successful status can still deliver a login page or a challenge rather than the expected content. Conversely, a rule allowing a crawler does not override a separate block elsewhere in the request path.&lt;/p&gt;

&lt;p&gt;Review the origin and any delivery or security layer together. Separate these outcomes in your notes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;What it establishes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Applicable robots rule&lt;/td&gt;
&lt;td&gt;The published crawler preference for that path&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Observed request and response&lt;/td&gt;
&lt;td&gt;What the website served for that request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Saved AI answer&lt;/td&gt;
&lt;td&gt;Whether the inspected answer mentioned a brand or cited a page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Website traffic evidence&lt;/td&gt;
&lt;td&gt;Whether a recorded visit occurred&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Do not collapse the four checks into a single “AI visibility passed” result.&lt;/p&gt;

&lt;h2&gt;
  
  
  Treat a user-agent name as a claim
&lt;/h2&gt;

&lt;p&gt;A declared bot name is useful for investigation, but it is not authentication. If you are considering special access for a crawler, corroborate the request against the operator’s current published verification information. Avoid hardcoding an address list copied from an old article.&lt;/p&gt;

&lt;p&gt;For ordinary troubleshooting, retain the timestamp, requested path, response status and relevant access decision. Keep private request data out of public reports. A request record alone does not establish what happened inside a model or why an answer selected a source.&lt;/p&gt;

&lt;h2&gt;
  
  
  Review policy, access and visibility separately
&lt;/h2&gt;

&lt;p&gt;After a change, confirm the new file is live, investigate later request evidence, and review answer observations on their own terms. Keep failures and unavailable information visible rather than recording them as measured absence.&lt;/p&gt;

&lt;p&gt;The goal is an access decision you can explain and verify. A permissive rule is not a citation guarantee; a citation is not a website visit.&lt;/p&gt;

&lt;p&gt;For the fuller treatment of user-requested retrieval, ad validation and request verification, see the &lt;a href="https://ansvra.com/articles/oai-searchbot-gptbot-and-chatgpt-user-explained/" rel="noopener noreferrer"&gt;original Ansvra guide&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>security</category>
      <category>seo</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
