From 15 September, Cloudflare blocks "Agent" traffic by default on pages that carry ads. Agent means bots browsing live on someone's behalf, which includes the crawlers AI assistants send out to check a product before recommending it. Search crawlers stay allowed.
The default lands on new domains onboarding to Cloudflare, new sites added by existing customers, and every existing free-tier account. A configured paid zone is untouched on the day.
What Cloudflare counts as an agent
Cloudflare now sorts AI traffic into Search, Agent and Training. Search indexes your content to answer questions about it later. Training absorbs it into a model. Agent is the new one: automated activity acting in real time on a person's behalf, with a human waiting on the other end. Cloudflare's own examples are chat fetch bots such as ChatGPT-User, and browser-use agents driving Chrome.
That is the fetch that runs when someone asks an assistant about your product and it goes to look at your pricing page before answering. Block it and the assistant still answers. It just answers from whatever it already had.
robots.txt does not cover this
The block happens at Cloudflare's edge, before the request reaches your origin. robots.txt is a request you make of a crawler that chooses to honour it, and it has no bearing on what the edge does with the connection. Allowing has to be configured per zone, in Cloudflare's AI traffic controls or as a WAF rule matching the user agent.
The agents worth allowing
ChatGPT-User # ChatGPT (also ChatGPT-User/2.0)
OAI-SearchBot # ChatGPT
Claude-User # Claude
Claude-SearchBot # Claude
Perplexity-User # Perplexity
PerplexityBot # Perplexity
Google-Agent # Gemini
GoogleAgent-Mariner # Gemini
Googlebot # Gemini
GrokBot # Grok
xAI-Grok # Grok
Grok-DeepSearch # Grok
The training crawlers, if you still want those blocked
GPTBot
ClaudeBot
anthropic-ai
meta-externalagent
Bytespider
Google-Extended
Watch the multi-purpose crawlers
Cloudflare applies the most restrictive matching rule to a crawler that does more than one job, and names Googlebot, Applebot and Bingbot as ones that get caught by a Training block. A blanket "block training" can therefore cost you ordinary search indexing. Set the categories separately and check what each named agent falls under before you save.
Decide it rather than inherit it
If you sell ad space, the block may be exactly what you want, and Cloudflare is handing it to you for free. The bad outcome is having the default applied to you on the fifteenth without a decision, then working out months later why assistants describe your product from a stale snapshot while your search traffic looks fine.
Check the zone before the fifteenth. It is a five-minute job now and an invisible problem afterwards.
Top comments (0)