Moved to Cloudflare? Check It Isn't Quietly Blocking AI Crawlers

You moved your site onto Cloudflare. DNS resolved, the padlock is green, every page loads in your browser. Job done?

Not quite. Cloudflare now decides, at the edge, what each client gets, and several of its features treat AI crawlers differently from people. If you never opened those settings, you don't actually know whether ChatGPT search, Claude or Perplexity can still reach your pages.

This post walks through the settings that matter (as Cloudflare documents them in October 2026), a curl test you can run in a few minutes, and how to read what comes back.

First, know which bot does what

"AI bots" is not one thing. The big operators run separate user agents for training, for search, and for fetching a page because a person asked a question. Blocking one does not block the others.

Operator	Training	Search / indexing	User-triggered fetch
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Perplexity	(none listed)	PerplexityBot	Perplexity-User

Sources: OpenAI crawlers, Anthropic's crawler FAQ, Perplexity crawlers.

Two details from those pages matter here. OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers. And both OpenAI and Perplexity say their user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, because a person asked for the page. So "should I let AI bots in?" is really three questions: training, search, and live fetches.

The Cloudflare settings that can affect AI crawlers

There are five places to look. They overlap, so check all of them.

1. AI bot policies: Search, Agent, Training

Cloudflare now groups AI traffic by behavior (Bots: AI bots):

• Search: collects or indexes your content so it can answer questions later.
• Agent: acts in real time on a person's behalf, "such as chat fetch bots and browser-use agents."
• Training: crawls content to train or fine-tune a model.

You'll find them under Security Settings > Configure AI bot policies. For each one you can choose Block, Block on pages with ads, or Allow (Block AI Bots docs). Since September 15, 2026, Training also has a Disallow AI Training option. According to Cloudflare's announcement, it publishes a no-training preference in robots.txt, keeps crawlers Cloudflare calls "Accountable" mixed-use crawlers (Applebot, Bingbot, Googlebot) allowed for search, and blocks other training crawlers.

Two things in that announcement deserve attention:

• Block now reaches search engines too. Block and Block on pages with ads for Training now also apply to mixed-use crawlers, "including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training."
• The old switch is gone. The legacy "Block AI bots" toggle is deprecated, and existing settings were migrated to the new controls. If you had the old toggle on, check what it turned into.

For new domains, Cloudflare proposes presets during onboarding. Per the same post, a site that doesn't monetize with ads gets Search, Training and Agent all set to Allow. A site that monetizes with ads gets Search Allow, Training Disallow AI Training and Agent Block on pages with ads. You can change any of them later. Don't assume. Open the page and read what's actually set.

2. Bot Preference Sync (formerly managed robots.txt)

Bot Preference Sync writes your policy choices into robots.txt. When it's on and at least one policy is set to Block, Block on pages with ads or (for Training) Disallow, Cloudflare prepends managed directives and its Content Signals Policy to your existing file. If you have no robots.txt, Cloudflare creates one. Cloudflare's blog says it's on by default for new customers.

Content Signals are three machine-readable categories: search, ai-input and ai-train. Cloudflare's docs are clear that robots.txt compliance is voluntary. It states a preference, and enforcement happens elsewhere.

So your live robots.txt may no longer be the file you deployed.

3. AI Crawl Control

AI Crawl Control (formerly AI Audit, available on all plans) shows which AI crawlers request your content, and lets you Allow or Block each one (Manage AI crawlers). Blocking a crawler there creates or updates a WAF custom rule. On paid plans you can choose whether blocked crawlers get 403 Forbidden or 402 Payment Required.

One detail matters for testing. On the Free plan, AI Crawl Control identifies AI crawlers "based on their user agent strings." Paid plans can use Bot Management detection instead.

4. Bot Fight Mode

Bot Fight Mode is a free toggle that identifies traffic matching known bot patterns and issues computationally expensive challenges. Cloudflare notes it "may challenge API or mobile app traffic". Also, you can't bypass it with WAF custom rules or Page Rules: Skip, Bypass and Allow have no effect on it. If it gets in your way, the documented options are turning it off or moving to Super Bot Fight Mode, which supports Skip rules.

5. Your own WAF custom rules

Country blocks or a user-agent blocklist copied from a forum can catch crawlers too. The Block action returns 403 for most security features (or 429 for rate limiting). Challenge actions require the client to pass a Cloudflare challenge first. Anthropic, for example, says its bots won't try to bypass CAPTCHAs.

If you write allow rules, Cloudflare exposes cf.client.bot (a request from a known good bot) and cf.verified_bot_category. That's safer than trusting the User-Agent header, which anyone can fake. Perplexity's own docs recommend combining user-agent matching with its published IP ranges.

The curl test

Send the same request with different user agents and compare. The strings below come from the vendors' docs; version numbers may change. Anthropic doesn't publish a full UA string on that page, so the ClaudeBot line is a minimal test string containing the product token.

URL="https://example.com/pricing"

declare -A UA=(
  [browser]="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
  [GPTBot]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
  [OAI-SearchBot]="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
  [ChatGPT-User]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
  [PerplexityBot]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
  [Perplexity-User]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"
  [ClaudeBot]="Mozilla/5.0 (compatible; ClaudeBot/1.0)"
)

for name in "${!UA[@]}"; do
  curl -s -o "out_$name.html" -D "hdr_$name.txt" -A "${UA[$name]}" \
    -w "$name\t%{http_code}\t%{size_download} bytes\n" "$URL"
  grep -i -E "^(cf-mitigated|cf-ray):" "hdr_$name.txt" | sed 's/^/    /'
done

Then check what robots.txt says now:

curl -s https://example.com/robots.txt | sed -n '1,80p'

Look for a BEGIN Cloudflare block, Content-signal: lines, and Disallow: / under any user agent you care about.

Reading the result

What you see	What it means
Same status and similar size as the browser	No user-agent-based block on this URL
403 with a cf-ray header (Cloudflare adds it to responses, see HTTP headers)	Blocked at the edge: AI Crawl Control, an AI bot policy, or a custom rule
402	AI Crawl Control block configured as "Payment Required"
cf-mitigated: challenge header	A Cloudflare challenge page; per the docs this header is set on every challenge page type
429	Rate limiting
200 but robots.txt disallows the bot	No enforcement, just a stated preference that compliant crawlers follow

An important limit: your curl comes from your IP with a borrowed user agent. Cloudflare verifies real bots by other means: Web Bot Auth signatures, published IP lists, or reverse DNS (Verified bots). So this test catches user-agent-based rules (including AI Crawl Control on the Free plan), but it can't perfectly reproduce how verified bot traffic is treated, in either direction. Confirm in the dashboard: Security > Analytics > Events shows which service acted on a request, and the AI Crawl Control Crawlers tab shows requests and unsuccessful requests per crawler.

Choosing deliberately

A reasonable starting point for a business that wants to be found in AI answers:

• Search: Allow. This is how OAI-SearchBot, Claude-SearchBot and PerplexityBot build what they cite.
• Agent: think before blocking. Cloudflare's Agent category includes chat fetch bots, the kind that open your page when someone asks about you.
• Training: your call. Disallow AI Training keeps Accountable mixed-use crawlers for search. Plain Block now affects Googlebot and Bingbot as well.

Common mistakes

• Testing only in a browser. Humans and crawlers can get different answers from the same URL.
• Treating robots.txt as a firewall. It's a preference. Cloudflare's own docs say so.
• Choosing "Block" for Training to "just stop AI". After September 15, that also stops Googlebot and Bingbot.
• Forgetting Bot Fight Mode. Your WAF allow rule won't override it.
• Allowing by user agent alone. Use cf.client.bot or IP ranges, or anyone can claim to be GPTBot.

If you'd rather have this checked continuously, AURA GEO audits what your server actually returns and keeps synthetic user-agent tests separate from real crawler evidence. There's a related explainer on checking CDN and WAF rules that block AI bots.

Comments

[Dmitry Sergeev]

We need to produce a short comment, casual, about the video: "Moved to Cloudflare? Check It Isn't Quietly Blocking AI Crawlers". Should ask a specific reaction or question about this video: e.g., "anyway, did you notice the robots.txt changes?" Something like that. Must follow style: start with lowercase letter, casual voice. No punctuation at end or minimal. No quotes. No URLs. Not formal. Should be one sentence, up to two. No double hyphen. Avoid em dash. Use