Article image
Protecting AI Crawler Access: Auditing Cloudflare Rules and CDN Licenses
For agencies that run SEO and technical infrastructure for clients, visibility no longer means only ranking on Google. Clients also want to be cited in answers from ChatGPT, Claude, Perplexity and Gemini. When a client seems to drop out of those answers, the cause is often not a penalty or thin content. It is access: a robots.txt rule, a CDN bot setting, a rate limit, or a security plan that quietly changed.
Cloudflare matters here because of its reach. The company says more than 20% of web domains sit behind it, and in 2026 it rebuilt how its customers control AI crawlers. This guide covers what changed, how to set a sensible policy per client, how to check what crawlers actually experience, what llms.txt does and does not do, and how a lapsed CDN subscription can change the picture. It ends with a record-keeping routine you can run across a whole client portfolio. Details are current as of late September 2026, and Cloudflare's dashboard labels and defaults are still moving, so confirm against the linked docs before you change a live zone.
- What Cloudflare changed in 2026 July 1: one toggle becomes three categories Until this summer, Cloudflare's main AI control was a single "Block AI Bots" switch. On July 1, 2026, it introduced separate controls for three behaviors, available to all customers including the Free plan:
Category What Cloudflare says it covers
Search Crawling that collects or indexes your content so it can answer questions about it later. Site owners are expected to get referral traffic or other compensation in return.
Agent Automated behavior acting in real time on a person's behalf, such as chat fetch bots (ChatGPT-User is Cloudflare's example) and browser-use agents.
Training Crawling that collects content to train or fine-tune a model.
Cloudflare also tracks other behaviors (ad verification, feed fetching, SEO auditing, transactions and more), and a single bot can carry more than one label. The same update changed what "Verified" means: a verified bot is no longer treated as allowed by default. Verification confirms identity, and what a verified bot can do depends on its category and your policy.
September 15: Disallow AI Training and the "Accountable" designation
Bots such as Googlebot, Bingbot and Applebot serve both search and AI training under one identity. Cloudflare's July announcement said such crawlers would be judged on all of their behaviors, which meant that blocking "Training" would also block them. On September 15, 2026, it shipped a refinement that changes how you should configure clients.
New Training setting: Disallow AI Training. It publishes a no-training preference in robots.txt, keeps "Accountable" mixed-use crawlers (Apple, Google, Microsoft) allowed for search, and blocks the training-only crawlers run by companies such as Amazon, Anthropic, Meta and OpenAI. Blocking those does not affect search.
"Block" now means more. Block and "Block on pages with ads" now apply to mixed-use crawlers too. Choosing either for Training will stop Googlebot, Bingbot and Applebot, search included. To stop training and keep search, Cloudflare points site owners to Disallow AI Training.
Legacy settings were migrated. "Block AI Bots" is deprecated, and Managed Robots.txt is replaced by a feature called Bot Preference Sync.
Agent has no "Disallow" option. Cloudflare says the web has no well-established directive for expressing that preference to agents, so Agent is Allow, Block on pages with ads, or Block.
Bing is the exception. Microsoft is building robots.txt support for a no-training preference, targeted for early 2027. Until then, Disallow AI Training does not convey that preference to Bing through robots.txt.
How existing zones were migrated depends on what the site had before:
Zone's previous state Search Training Agent
"Block AI Bots" was off Allow Allow Allow
"Block AI Bots" was on (either mode) Allow Disallow AI Training Block on pages with ads
Domains that had already configured the three granular controls keep the practical effect of their choices, with any Training "Block" becoming Disallow AI Training.
New domains onboarded from September 15 are offered one of two presets:
Setting Site doesn't monetize with ads Site monetizes with ads
Preference Sync Enabled Enabled
Search Allow Allow
Training Allow Disallow AI Training
Agent Allow Block on pages with ads
Coverage from July described the September defaults as also reaching existing free-plan zones that never changed their settings, while Cloudflare's September 15 post describes migration based on each zone's prior configuration. Because the descriptions differ, read each client's actual settings rather than assuming.
What this means for agency clients
A client that once flipped "Block AI Bots" may now have Agent set to Block on pages with ads. That can affect assistants that fetch a page live for a user, so check whether that matches what the client wants.
Ad-monetized client sites (publishers, affiliate and lead-gen sites running ad scripts) are the ones the ad-related presets target. A service business with no ads mostly sees "Allow" across the board.
Choosing Block on Training is a search decision as well as an AI decision. Treat it as one.
- Know which crawler does what The major AI companies now separate their crawlers by purpose, which is what lets you allow citation while refusing training.
Company Training Search / indexing User-triggered fetch
OpenAI GPTBot OAI-SearchBot ChatGPT-User
Anthropic ClaudeBot Claude-SearchBot Claude-User
Perplexity (not separately listed) PerplexityBot Perplexity-User
Details worth knowing:
OpenAI says its settings are independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and ChatGPT-User is not what decides search eligibility. Search-related changes can take roughly a day to take effect after a robots.txt update.
Anthropic documents ClaudeBot as training collection, Claude-User as fetching pages when a Claude user asks a question, and Claude-SearchBot as crawling to improve search result quality. It warns that blocking Claude-SearchBot may reduce your visibility in Claude search answers. Anthropic also says IP blocking may not work reliably, because its bots use public cloud IP addresses and it does not publish IP ranges.
User-triggered fetchers behave differently. Reporting from early 2026 notes that Anthropic says all three of its bots honor robots.txt, while OpenAI and Perplexity warn that robots.txt rules may not apply to ChatGPT-User and generally do not apply to Perplexity-User. Vendor policies change, so re-read the current docs before promising a client anything.
A robots.txt that allows citation but declines training looks like this (adapt it to the client, and align it with whatever Cloudflare's Bot Preference Sync publishes so the two don't contradict each other):
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
Remember what this file is. Robots.txt is a preference that well-behaved crawlers honor, not access control. Cloudflare's own documentation says compliance is voluntary and points to AI Crawl Control for actually enforcing a block. That works in both directions: a robots.txt that says "Allow" does nothing if a WAF rule, rate limit or bot setting is turning the crawler away at the edge.
- Test what crawlers really experience A common audit step is to request a page while pretending to be a crawler:
curl -I -A "OAI-SearchBot" https://clientdomain.com/
This is a useful smoke test, but it cannot prove that the real crawler gets through. Cloudflare verifies bots using cryptographic Web Bot Auth signatures, published IP lists paired with a stable user-agent, or reverse DNS, and its Managed Ruleset includes rules that flag requests impersonating well-known bots such as Googlebot and Bingbot. A spoofed user-agent from your laptop is not a verified request, so the result may not match what the real crawler sees.
A more reliable routine:
Review Security Events in the client's Cloudflare zone for blocks or challenges against crawler traffic, and look specifically for search crawlers you meant to allow.
Compare against origin server logs to see whether requests from the crawlers you care about reach the site at all and which status codes they receive (403, 429 and 503 are the ones to chase).
Where a vendor publishes IP ranges (OpenAI does), validate suspicious traffic against them rather than trusting the user-agent string.
Check that custom WAF rules, rate limits and any origin-side security plugin are not turning crawlers away. Robots.txt and Cloudflare's AI settings can both say "Allow" while another layer says no.
- A reality check on llms.txt Many agencies publish an /llms.txt file, a Markdown index of a site's key pages intended for language models. It is worth being precise about what it does.
Google says it doesn't use it. In an update to its AI optimization guidance on June 15, 2026, Google clarified that llms.txt files are not needed for Search, including its generative AI features, and have no positive or negative effect on rankings.
Server-log data shows almost nobody reads it. Ahrefs analyzed 137,000 domains and found that about 28% publish an llms.txt file (a figure it flags as an upper bound, because its customers skew technical). Of the valid files, 97% received zero requests in May 2026. Among the few that were fetched, AI retrieval bots such as OAI-SearchBot and PerplexityBot accounted for only about 1.1% of requests, while SEO audit tools accounted for more than a fifth.
It has a place. Documentation-heavy sites and developer tools use it, and coding agents look for it. Cloudflare's own developer docs point agents to an llms.txt index.
The practical takeaway for client work: keep an llms.txt if you have a genuine use for it, but do not sell it as an AI-visibility fix. What decides whether an assistant can cite a page is whether the crawlers you want can fetch the page itself, consistently, with a normal 200 response. Audit that first. If you do serve llms.txt, confirm it returns 200 and is not behind a challenge, but treat that as housekeeping.
- The license lapse risk: what happens when a CDN plan drops A second cause of sudden changes in crawler access is plan drift. Agencies typically manage a mix of arrangements:
Contract (Enterprise) accounts negotiated directly with Cloudflare. Cloudflare doesn't publish a price for this tier.
Self-managed Pro or Business plans, currently $25 per month ($20 per month billed annually) and $250 per month ($200 per month billed annually) respectively, often tied to a client's card.
Host-provisioned Cloudflare Enterprise. Some hosts include it with their plans. According to xCloud's own comparison page, Kinsta, Rocket.net, Levamo, 10Web, Convesio and BigScoots bundle it at no line-item cost for customers who host with them, while xCloud sells it as a per-domain add-on delivered through Cloudflare for SaaS, where the customer needs no Cloudflare account of their own. Treat that page as a vendor's description, not a neutral source.
What Cloudflare says happens on a failed payment
Cloudflare retries a failed charge automatically, up to five times, during a five-day grace period.
If payment isn't resolved, the account is automatically downgraded to the Free plan. The websites stay active, but paid features and add-ons are lost.
To restore them you must pay the outstanding balance and re-subscribe to each product individually.
Cloudflare emails the billing contact about a failed renewal. The practical risk is that the notice lands in an inbox nobody checks, such as a former employee's, a client's old address or a card owner who isn't watching.
Why the plan tier matters for bot control
Cloudflare's three AI controls (Search, Agent, Training) are available on every plan, so they aren't tied to a paid tier. What differs by plan is the layer of bot protection around them:
Free includes Bot Fight Mode, a simple on/off toggle. Cloudflare says it can't be customized, and that custom rule Skip actions can't bypass it.
Pro and Business include Super Bot Fight Mode, which runs on the Ruleset Engine, offers separate actions for definitely automated, likely automated (Business) and verified bot traffic, and can be skipped with custom rules.
Enterprise with the Bot Management add-on adds a bot score from 1 to 99 that you can build rules around. Custom rules are available on all plans, with limits that rise on higher plans.
So if a client zone relied on paid bot features for exceptions and then dropped to Free, the exceptions built on those features would no longer work as designed. Cloudflare's billing docs say you lose paid features and add-ons on downgrade but don't itemize what that means for each zone's rules, so the safe assumption is to re-audit the zone after any plan change instead of assuming settings carried over.
- The audit checklist Run this across your client portfolio, and repeat it after any plan change, host move or major Cloudflare policy update.
[ ] Read the AI bot policy. In the client's zone, go to Security > Settings > Configure AI bot policies. Record Search, Training and Agent. Confirm Search is Allow. If Training is set to Block or Block on pages with ads, confirm the client accepts that Googlebot, Bingbot and Applebot are affected too.
[ ] Check the Agent setting against what the client wants, especially on zones that previously used "Block AI Bots".
[ ] Read the live robots.txt. Fetch /robots.txt, confirm it reflects the intended posture, and check that Bot Preference Sync and any hand-written rules aren't contradicting each other.
[ ] Look at Security Events and origin logs for 403, 429 and 503 responses to the crawlers you want.
[ ] Review custom rules and rate limits, including anything on the origin side.
[ ] Confirm the plan and who provides it: direct Cloudflare account, a hosting provider's bundled tier, or a reseller add-on.
[ ] Check the billing side: renewal date, payment method and its expiry, and whether the billing email reaches someone who will act on it.
[ ] Note when you last verified, so the next review has a date to work from.
- Keeping the records straight with InstaRenewal Most of the failures above come down to information nobody wrote down: which Cloudflare account a zone lives under, who pays, when the plan renews, and what policy the client agreed to. That is the part of the problem a renewal and ownership record can solve.
InstaRenewal is a manual renewal-date and ownership record-keeping platform for web agencies, covering domains, SSL/TLS certificates, hosting accounts and plugin or software licenses. It does not connect to Cloudflare, read your WAF or bot settings, or store credentials. What it gives you is a central place where your team records the facts and the dates, so a review happens before a date passes instead of after something breaks.
For CDN and crawler-access work, record these against each client:
Client (example) CDN provided by Plan Renews Who pays Intended AI posture Last reviewed
client-a.com Direct Cloudflare account Business Nov 14 Agency card Search Allow, Training Disallow, Agent Allow Sep 24
client-b.com Host-bundled Enterprise Via hosting plan Hosting renewal Client Search Allow, Training Allow, Agent Allow Sep 24
client-c.org Reseller add-on Per-domain add-on Jan 3 Agency Not yet confirmed Never
A few habits make this useful:
Log which account or provider the CDN tier comes from, so a host change or a departing staff member doesn't orphan the zone.
Record the payment method's expiry next to the plan renewal, since a lapsed card is the most common trigger for the five-day-grace downgrade described above.
Store the agreed AI posture in plain words, plus the date it was last checked in Cloudflare. The record doesn't verify anything on its own; the point is that a quarterly review date is written down and attached to the client.
Track the contract end date for Enterprise arrangements, along with any notice period in the agreement.
- Sustaining AI visibility takes infrastructure discipline Content quality still earns citations, but only crawlers that can reach the page can cite it. In 2026 that means understanding Cloudflare's three crawler categories and the September 15 changes, knowing which vendor bot does what, verifying real crawler traffic instead of relying on a spoofed test, keeping llms.txt in proportion, and treating CDN plan renewals with the same care as domain renewals. The agencies that do this consistently will spend less time explaining unexplained drops in AI visibility to clients.
Sources
Cloudflare, Your site, your rules: new AI traffic options for all customers (July 1, 2026)
Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training (September 15, 2026)
Cloudflare, Say it once: introducing Bot Preference Sync
Cloudflare, press release, September 15, 2026
Cloudflare docs: Block AI Bots / Configure AI bot policies, Verified bots, Web Bot Auth, Fake bot detection
Cloudflare docs: Bot Fight Mode, Security features interoperability, Stop malicious bots
Cloudflare billing docs: How Cloudflare billing works, Resolve a payment failure
Cloudflare, Plans and pricing
OpenAI, Overview of OpenAI crawlers
Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
Search Engine Journal, Anthropic's Claude bots make robots.txt decisions more granular
Ahrefs, We analyzed 137K sites: 97% of llms.txt files never get read
TechWyse, Google says llms.txt will not help rankings (coverage of Google's June 15, 2026 guidance update)
xCloud, Cloudflare Enterprise CDN pricing (vendor page)
Top comments (0)