<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: amay</title>
    <description>The latest articles on DEV Community by amay (@amaydev).</description>
    <link>https://dev.to/amaydev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4239404%2F180a0f99-f148-44a5-8f85-ea5313e6b45e.jpg</url>
      <title>DEV Community: amay</title>
      <link>https://dev.to/amaydev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/amaydev"/>
    <language>en</language>
    <item>
      <title>cloudflare now sorts ai bots into 3 buckets: search, agent, training. since sept 15, new domains block agent + training bots by default on pages with ads.</title>
      <dc:creator>amay</dc:creator>
      <pubDate>Wed, 18 Nov 2026 17:39:22 +0000</pubDate>
      <link>https://dev.to/amaydev/cloudflare-now-sorts-ai-bots-into-3-buckets-search-agent-training-since-sept-15-new-domains-2dfm</link>
      <guid>https://dev.to/amaydev/cloudflare-now-sorts-ai-bots-into-3-buckets-search-agent-training-since-sept-15-new-domains-2dfm</guid>
      <description>&lt;p&gt;if you build agents that fetch live pages:&lt;/p&gt;

&lt;p&gt;you can get a challenge page instead of the real content&lt;br&gt;
the code often doesn’t error, it just stores “verify you’re human” and your llm runs with that&lt;br&gt;
so check the content looks right, not just the status code&lt;br&gt;
and identify your bot honestly instead of spoofing a browser&lt;/p&gt;

&lt;p&gt;site owners, watch out too. googlebot does search and training in one bot, so the old “block ai bots” toggle can catch it on ad pages.&lt;/p&gt;

&lt;p&gt;how are you handling web fetching for agents rn?&lt;/p&gt;

&lt;p&gt;that’s about 100 words, and the reddit version works as is&lt;br&gt;
for x it’s still over 280 characters. cut everything after “so check the content looks right” and end with the question, or make it a 2 post thread&lt;br&gt;
i dropped the markdown for agents bit to save space. if you want it back, add one line: “if the site enables it, you can ask for accept: text/markdown”&lt;/p&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
