DEV Community

amay
amay

Posted on

cloudflare now sorts ai bots into 3 buckets: search, agent, training. since sept 15, new domains block agent + training bots by default on pages with ads.

if you build agents that fetch live pages:

you can get a challenge page instead of the real content
the code often doesn’t error, it just stores “verify you’re human” and your llm runs with that
so check the content looks right, not just the status code
and identify your bot honestly instead of spoofing a browser

site owners, watch out too. googlebot does search and training in one bot, so the old “block ai bots” toggle can catch it on ad pages.

how are you handling web fetching for agents rn?

that’s about 100 words, and the reddit version works as is
for x it’s still over 280 characters. cut everything after “so check the content looks right” and end with the question, or make it a 2 post thread
i dropped the markdown for agents bit to save space. if you want it back, add one line: “if the site enables it, you can ask for accept: text/markdown”

Top comments (0)