Cloudflare will happily block AI bots for you. The question is which ones. "AI bot" covers three different jobs: crawlers that collect text to train future models, crawlers that build an AI search index, and fetchers that open one page because a person just asked ChatGPT or Claude about it. Block all three and you're out of the training sets, but you've also vanished from the AI answers that now send real visitors.
I ran the same request against two sites behind Cloudflare, changing only the bot name, to see what each setting actually does. Below are the results, the current dashboard path (it moved in 2026), and what I'd pick for a business site or a blog.
The short answer
- Block training bots (GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended) if you don't want your content in future models. They send you no visitors.
- Allow AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). OpenAI's docs say sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers".
- Allow user agents (ChatGPT-User, Claude-User, Perplexity-User). Block them and "summarise this link" fails for anyone who pastes your URL.
- In Cloudflare that's Training: Block, Search: Allow, Agent: Allow. New domains start with all three on Allow, so nothing is blocked until you change it.
Three kinds of AI bots, and why they're not the same

Each company runs separate crawlers for separate jobs. Blocking one does not block the others.
Every big AI company now publishes separate user agents, and their documentation spells out what each one is for:
-
OpenAI.
GPTBotcollects training data.OAI-SearchBotindexes pages for ChatGPT search.ChatGPT-Userfetches a page when a user's request needs it, and because a person started it, OpenAI notes that robots.txt rules "may not apply". Changes to robots.txt take about 24 hours to be picked up. -
Anthropic.
ClaudeBotis training,Claude-SearchBotimproves search results,Claude-Userfetches for a person using Claude. Anthropic says its bots respect robots.txt, includingCrawl-delay. -
Perplexity.
PerplexityBotbuilds its search index and, per Perplexity, isn't used to train foundation models.Perplexity-User"generally ignores robots.txt rules", because a user asked for the page. -
Google.
Google-Extendedisn't a crawler at all. It's a robots.txt token that tells Google not to use your pages for Gemini training. It "does not impact a site's inclusion in Google Search". Normal Googlebot crawling is unaffected either way. -
Others.
CCBot(Common Crawl, a dataset many models train on),Bytespider(ByteDance),Amazonbotandmeta-externalagent. All four are on Cloudflare's own managed robots.txt block list.
The practical takeaway: "should I block AI bots?" is really three questions, and most sites answer them differently.
What Cloudflare actually returned to each bot
To see the difference, I sent the same request with curl to two sites behind Cloudflare and changed only the User-Agent. Site A is this blog, with no AI blocking. Site B is a company site that has AI training blocked.

18 November 2026. Same request, only the User-Agent changed. Training names are blocked on Site B; search and user agents pass.
On Site B the four training crawlers got a 403; everything else, including OAI-SearchBot, ChatGPT-User, Claude-User and Googlebot, got a 200. Site B's robots.txt has no AI rules at all, so the block happens at Cloudflare's edge, before the request reaches the server. That's the split you want: the training bots are refused, and the AI search engines can still read and cite the site.
One caveat that matters if you repeat this. My requests came from an ordinary connection, not from OpenAI's or Anthropic's IP ranges. Cloudflare identifies verified bots by where they come from, and its policies cover "Verified bots with that behavior and matching unverified bots". So a spoofed name tells you what the rule does with that name. It doesn't prove how the real crawler is treated. For that, look at the real traffic (next sections). You can run the same test on your own site:
# One bot name
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
https://example.com/
# Several at once
for ua in GPTBot ClaudeBot CCBot Bytespider OAI-SearchBot ChatGPT-User \
Claude-SearchBot Claude-User PerplexityBot Perplexity-User Googlebot; do
printf "%-18s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/
done
Where the setting is now: AI Crawler and Agent access policies

The three policies and a sensible starting point for a site that wants to be cited by AI search.
Older guides tell you to flip a single Block AI bots switch. That switch has been replaced. In the dashboard today, pick the domain and go to Security › Settings, filter by Bot traffic, and open AI Crawler and Agent access policies. You set one action per behaviour:
- Training. Allow, Block, Block on pages with ads, or Disallow (which only adds rules to your robots.txt and relies on the bot obeying them). Cloudflare puts mixed-purpose crawlers that do both training and search in this group, so blocking Training can hit them too.
- Search. Allow, Block, or Block on pages with ads. Blocking here removes you from AI search answers, not just from scrapers.
- Agent. Same options. Cloudflare itself recommends leaving personal agents allowed.
"Block on pages with ads" is for sites that earn from display ads and don't want AI tools reading those pages without showing the ads. For a business site with no ads it behaves like Allow.
If you remember Cloudflare announcing that it blocks AI crawlers by default: that applied to the earlier rollout. The current docs say new domains start with Search, Agent and Training all on "Allow (do not block)". Don't assume you're protected. Check the setting, or run the curl test above.
Managed robots.txt and Content Signals
Blocking at the edge and robots.txt are two separate layers. Cloudflare can also manage robots.txt for you (all plans). With it on, Cloudflare puts its own block in front of your file: a Content-signal line saying search is fine and AI training isn't (search=yes, ai-train=no), followed by Disallow rules for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent.
Content Signals are a newer, voluntary format with three values: search, ai-input (using your page to answer a question live) and ai-train. Nothing forces a crawler to honour them, the same as the rest of robots.txt, which is why the edge block exists. If you'd rather write robots.txt by hand, this is the equivalent:
# Block AI training, keep search and user-requested fetches
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
# Everyone else, including OAI-SearchBot, Claude-SearchBot and PerplexityBot
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Check it with a robots.txt validator before you deploy: a stray Disallow: / under User-agent: * blocks Googlebot as well.
Per-bot control with AI Crawl Control
If the three groups are too coarse, AI Crawl Control (it used to be called AI Audit) lists each crawler that visited your site, with its request count and how often it ignored robots.txt. Each row has an action: Allow, Block, or Charge (pay per crawl, still a closed beta). Block creates a WAF custom rule for you. It's available on every plan, with two limits on Free: detection is by User-Agent only, and you can't customise the block response.
This is the screen to use for the verified-bot question above. Look at what really arrives, not what a spoofed curl request shows. It's also where you'd allow one mixed-purpose crawler that the Training policy caught.
Why can't ChatGPT or Claude read my website?
This is the other half of the search traffic: owners who never meant to block anything and find that ChatGPT says it can't open their page. Work through it in this order:
- The AI policies. Search or Agent set to Block refuses OAI-SearchBot, ChatGPT-User, Claude-User and the rest. Set them to Allow.
- Bot Fight Mode or a WAF rule. A custom rule matching "bot" in the User-Agent, a country block or a strict rate limit catches AI fetchers too. Check Security › Events for the user agent and the rule that fired.
- A JavaScript challenge. "Under Attack" mode or a managed challenge stops fetchers that don't run JavaScript. The page loads fine in your browser, because you passed the challenge long ago.
-
robots.txt. A
Disallowfor OAI-SearchBot or Claude-SearchBot, or a blanket one under*. -
Content that only exists after JavaScript runs. If the HTML your server sends is an empty shell, a crawler that doesn't render scripts sees nothing to quote. Check with
curl: if the text you care about isn't in the response, the bot can't see it either.
Run the curl loop from earlier with your own domain. A 403 or a challenge page for OAI-SearchBot or ChatGPT-User is the answer. A 200 with your content in it means the block is somewhere else, or the AI tool simply hasn't crawled you yet.
What I'd choose, by site type
- Freelancer, agency or local business site. Block Training, allow Search and Agent. You want to be the business ChatGPT names when someone asks for a recommendation. Getting cited depends on more than access; how to get cited by ChatGPT and AI Overviews covers the rest.
- Blog or content site. The same, and measure it. AI referrals show up in GA4 if you set up the channel: see measuring AI traffic in GA4. If AI search sends you nothing after a few months, you've lost nothing by allowing it.
- Paywalled, licensed or ad-funded content. Consider "Block on pages with ads", or block Search as well. You give up AI citations in exchange for keeping the content to your own pages.
- Documentation or support sites. Allow everything except perhaps Training. You want people's AI assistants to give correct answers about your product.
Whatever you choose, it's a setting, not a one-off decision. Look at AI Crawl Control after a month and adjust. If Cloudflare also caches your HTML, why cf-cache-status says DYNAMIC and does Cloudflare cache 404 responses cover the caching side of the same dashboard.
FAQ
Should I block AI bots on Cloudflare?
Block AI training crawlers if you don't want your content used to train future models, but allow AI search crawlers and user-requested fetchers if you want to appear in ChatGPT, Claude and Perplexity answers. In Cloudflare's AI Crawler and Agent access policies that means Training: Block, Search: Allow, Agent: Allow.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot indexes pages for ChatGPT search. They are controlled separately: blocking GPTBot keeps you out of training, while blocking OAI-SearchBot removes your site from ChatGPT search answers. ChatGPT-User is a third agent that fetches a page when a user asks for it.
Does blocking AI bots hurt my Google rankings?
No. Blocking Google-Extended, GPTBot or ClaudeBot does not affect normal Google Search; Google states that Google-Extended is not a ranking signal and does not change inclusion in Search. Only blocking Googlebot would. Check that a robots.txt rule meant for AI bots does not sit under User-agent: *.
Does Cloudflare block AI bots by default?
Not on new domains, according to Cloudflare's current docs: Search, Agent and Training all start on Allow (do not block). Check Security, Settings, filter Bot traffic, AI Crawler and Agent access policies, or send a test request with curl using an AI bot's User-Agent.
Why can't ChatGPT access my website?
Common causes are Cloudflare's Search or Agent policy set to Block, Bot Fight Mode or a WAF rule catching the fetcher, a JavaScript challenge, a robots.txt Disallow for OAI-SearchBot, or content that only appears after JavaScript runs. Security Events in Cloudflare shows which rule refused the request.
Is Cloudflare AI Crawl Control free?
Yes, AI Crawl Control is available on all Cloudflare plans. It lists crawlers with their request counts and lets you allow or block each one. On the Free plan detection is by User-Agent only and the block response cannot be customised; pay per crawl is a closed beta.
Do AI bots respect robots.txt?
The major training and search crawlers say they do: OpenAI, Anthropic and Perplexity all document robots.txt support for GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. User-initiated fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it. To enforce a block, use Cloudflare's edge policies.
Originally published on knowledgemarkg.com.
Top comments (1)
We need to produce a short YouTube comment, 1-2 sentences, casual, start lower case, reaction or question about this video. Maybe ask about specific bots: "does this block GPT-4?" Or comment about false positives. Keep lowercase start. No punctuation at end maybe. No quotes. No hashtags. No URLs. Make sure no em dash. Provide plain text. Let's craft: "i tried the Cloudflare rule they suggested and it started blocking my own search crawler, any tips to whitelist it?" That's