Eight user agents, and the one that decides whether ChatGPT cites you is not among them.
An r/SEO post from April, 53 points and 40 comments, says Cloudflare quietly cut the author's site off from ChatGPT, from Perplexity and from Google's AI Overviews. I went and read the block. It names ChatGPT's training crawler and not its search crawler, and it never mentions Perplexity at all. The Google agent it does name is one Google says does not affect inclusion or ranking in Search.
Take just two of the eight agents. GPTBot governs inclusion in OpenAI's training data, and Google-Extended governs grounding in Gemini Apps. Those are real things to give up. They are not the three things the warning names.
What is actually in the file
Cloudflare prints the whole block on its managed robots.txt page:
# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed Content
Read that against Cloudflare's own crawler reference table and a pattern falls out. GPTBot is in, OAI-SearchBot is out. ClaudeBot is in, Claude-SearchBot is out. For both of those pairs the table calls the blocked agent an AI Crawler and the one left alone AI Search. Applebot-Extended is in and plain Applebot is not, the same split again. The table has no Applebot-Extended row, so it cannot tell you what Cloudflare calls that one. The block runs along the training and search seam, and that looks deliberate to me.
OpenAI's side of it is one line: "Each setting is independent of the others". I worked through the three OpenAI bots and which one governs search visibility in an earlier post, so I will point there rather than do it twice.
Google-Extended is the entry that causes the trouble, because it sounds like it governs AI answers in Search. Google's crawler documentation scopes it to training and grounding future Gemini models, then says flatly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
So what does control AI Overviews? Nothing, in the sense people mean. From Google's AI features page: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search. To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls." Those are the ordinary snippet controls, so opting out of the AI answer means opting out of the snippet. Google states the trade in one paragraph and almost nobody repeats it.
One more from the crawler page, which kills a piece of advice I keep seeing: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings..." Anyone telling you to check your access logs for Google-Extended is sending you after a string that does not exist.
Did anyone turn this on, or did it turn itself on
Cloudflare's documentation describes the managed file as a switch, in the same grammatical mood every time. "When you turn on the managed robots.txt setting, Cloudflare generates and maintains a robots.txt file that instructs known AI crawlers to stay away from your content." I searched the docs, the changelog, four announcement posts and the July 2025 press release for a case where it got enabled on an existing domain without the owner acting. I did not find one. That is a failure to find rather than a proof of absence.
What did change is adjacent and gets mistaken for it. The press release from 1 July 2025 says "every new domain will now be asked if they want to allow AI crawlers..." A question at onboarding, for new domains, governing the traffic block rather than the file.
One Cloudflare default does appear in a file you never touched, and it is not a block: "Domains on the Free plan that do not have their own robots.txt file and do not use the managed robots.txt feature will display the Content Signals Policy when a crawler requests the robots.txt file for your domain." Cloudflare adds that the policy "does not express any specific preferences about your content".
Check it over the network, not on disk
The managed file is assembled at the edge. Cloudflare says it "will prepend our managed robots.txt before your existing robots.txt, combining both into a single response", so the file in your web root and the file the world receives are two different objects. Your hosting file manager, FTP and your SEO plugin's robots editor can none of them see the difference, because none of them sits downstream of the edge.
curl -s https://example.com/robots.txt
Opening the same URL in a browser tab does the same job.
If the block is on, you get the # BEGIN Cloudflare Managed content marker above whatever your own site serves, the eight Disallow groups, then # END Cloudflare Managed Content. Note the capitalisation: lowercase c on the opening marker, capital C on the closing one. Either marker is conclusive on its own, and so is User-agent: GPTBot sitting in a file you never edited.
The false positive to know about is the Content Signals preamble. It runs to a screenful, opens with "As a condition of accessing this website, you agree to abide by the following content signals:", and defines a vocabulary in lettered clauses. Every line of it starts with a #, and a robots.txt comment does nothing. So: if every line starts with a hash, nothing is blocked. A block is a Disallow: line with no hash in front of it.
If you find AI crawler rules and no Cloudflare markers, the source is inside your own stack. WordPress core's complete default output, from the do_robots() source, is three lines:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Core adds no AI rules of any kind. The function ends by passing that output through a robots_txt filter, which is the hook a plugin would use to append its own. So the rules came from a physical file in the web root, a plugin on that filter, an SEO plugin's own robots editor, or the edge. Those are the only four places I can find that they could come from, so checking in that order settles it.
If you do change something, OpenAI is the only operator I found that publishes a lag: "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Nobody else gives a figure, so do not read that one as the industry's.
The date is 15 September 2026, and it is a different toggle again
Most of the argument is about a default Cloudflare changed in 2025. There is a second change dated 15 September 2026, and that one has an action attached. From the Block AI bots documentation:
"On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed. Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy 'Block AI bots' option. Before September 15, all customers can opt out of these new defaults."
Two things happen there and they need keeping apart. The ads-page default is for new domains only, so if your site is already on Cloudflare that half does not reach you at all.
The other half does. From Cloudflare's announcement on 1 July 2026: "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)." The legacy service there is the "AI Scrapers and Crawlers" toggle from July 2024.
My reading, and Cloudflare does not put it in these words: if you clicked that toggle in 2024 and do nothing before 15 September 2026, you are opting into blocking a crawler Cloudflare itself names as Googlebot. I am not going to model what that does to any particular site. The opt-out sits in Security Settings and it closes on 15 September 2026.
The category is not marginal either. Cloudflare Radar's AI insights page splits crawl traffic by declared purpose, and for the seven days to 18 August 2026 it put crawlers declaring both training and search at 33.3%, training alone at 40.2% and search alone at 18.5%.
Bot Fight Mode is not the same feature
In the same thread, somebody undid all of Cloudflare's bot blocking and found Claude, Grok and Perplexity still blocked. Another commenter named Bot Fight Mode as the cause, on the grounds that those three are not verified bots. The mechanism is right. The list is not.
Bot Fight Mode is a separate product with a separate toggle. It does not produce a file, it issues challenges, and it is rigid by design: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules. This is because Bot Fight Mode does not run on the Ruleset Engine..." Cloudflare's Free-plan table gives the type of bots it detects as "Simple bots from cloud hosting providers and headless browsers" and the control as "Applied to all traffic across a domain". A crawler on cloud IPs with no exemption and no rule you can write around it is the shape of a block that survives every AI setting you turn off. That is my reading of four documented properties, not a sentence Cloudflare writes.
The list is where it comes apart. I went through all 693 entries in Cloudflare's own bots directory on 18 August 2026. ClaudeBot is in there, labelled Verified, operator Anthropic, and so is Claude-SearchBot. Perplexity has no entry, and Cloudflare has published why: on 4 August 2025 it wrote that it had "de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling". Perplexity disputed that publicly and I am not going to adjudicate it. Grok and xAI have no entry, and I found nothing Cloudflare has published about them either way. Absence carries no single meaning: Bytespider and CCBot are missing too, and both sit on Cloudflare's own block list.
One thing I would like to state and cannot. Cloudflare's verified bots page says "Historically, Verified bots have been excluded in default bot configurations across all plans." The Bot Fight Mode page says nothing about verified bots either way, and I read all of it. Configurable allowlisting is a Super Bot Fight Mode feature, Pro and above.
The Content-signal line does not do what its wording implies
Content-signal: search=yes, ai-train=no, use=reference is the first non-comment line of the managed block, and it is the part people read as the actual instruction to AI companies.
It is Cloudflare's convention. contentsignals.org is Cloudflare's own site, calls itself "Cloudflare's implementation", and now sits alongside the IETF's AI Preferences working group. That group is real and chartered, neither of its two standards-track documents is a published RFC, and its charter puts "Technical enforcement of preferences" out of scope in writing.
The vocabulary has a sharp edge in it. Cloudflare's policy text defines search as "building a search index and providing search results" and then adds: "Search does not include providing AI-generated search summaries." So search=yes is not permission to be summarised in an AI answer. The signal for that is ai-input, the managed block omits it entirely, and under the policy's own terms omitting a signal expresses no preference either way. A quick reader gets this backwards.
use=reference is newer. Cloudflare calls content-use a test, and added the field to existing customers' files on 1 July 2026, so anyone who turned the setting on before that got a fourth field without doing anything.
Whether anybody acts on any of it, I could not establish. OpenAI's crawler documentation does not mention content signals. Google is worse than silent: its robots.txt specification publishes a closed list, "Google supports the following fields (other fields such as crawl-delay aren't supported)", and that list is user-agent, allow, disallow and sitemap. The same page says "Google ignores invalid lines in robots.txt files". Anthropic, Perplexity, Meta, Apple, Amazon and ByteDance say nothing either way, and Cloudflare makes no adoption claim of its own.
Nobody else is doing this to you quietly
I went looking for a second host with a silent AI default and did not find one. WP Engine documents two and neither is AI related: a block on sites still running on a .wpengine.com subdomain, and a platform-wide crawl delay of 10 seconds nobody asked for. Pressable's live-site default allows crawling. For Kinsta, SiteGround and cPanel I found no primary documentation either way.
The one that surprised me sits lower down. hosting.com documents that on shared and reseller servers, "if a robots.txt file does not exist in the document root (public_html) directory, the server automatically creates a new robots.txt file at midnight". The article never says what goes in that file, so I cannot tell you whether it mentions AI crawlers.
So should you block them
I do not have an answer for you, and I want to be exact about why.
Nothing I read measures the outcome. No study in any of it says what blocking training crawlers does to your citations or your traffic. The argument on both sides is people reasoning from first principles about a system nobody has measured. Fine as reasoning, not evidence.
What I can tell you is narrower. The seam in that block list runs between training and search rather than across AI as a category, so what you give up by leaving it on is not what the loudest warning says you give up. And if you clicked an AI toggle in 2024 and have not opened the screen since, Cloudflare has put a date on it in writing. The opt-out window closes on 15 September 2026, and the screen it closes on is Cloudflare's Security Settings.
Top comments (0)