Cloudflare Just Broke the AI Crawler Free Ride — What Changes September 15
The implicit deal was always: Google crawls your site, indexes it, sends you traffic. You're fine with that. Everyone wins.
That deal is now being renegotiated — by Cloudflare, for the entire internet.
On September 15, 2026, Cloudflare changes its defaults. Training and Agent bots get blocked on ad-supported pages. And if your crawler bundles search indexing with AI training — looking at you, Googlebot — Cloudflare applies the most restrictive rule. Block on ad pages. Period.
This isn't just a settings tweak. This is infrastructure-level pressure that forces every major AI lab and search engine to clean up their bot practices or lose access.
The Three-Way Split Cloudflare Is Enforcing
Cloudflare now classifies AI bots into three categories — and they want every crawler to pick one:
Search — Proactively building a database of your site to respond to later queries. Classic indexing. Cloudflare leaves this alone by default.
Agent — Real-time automation on behalf of a user. Think ChatGPT-User fetching a URL mid-conversation, or Gemini/Claude driving a browser. This is agentic AI in action.
Training — Crawling to ingest content for model training or fine-tuning. The category everyone's most pissed about.
If you run a mixed-use crawler that does all three? Cloudflare classifies you by your most aggressive use case and treats you accordingly.
Right now, Googlebot, Applebot, and BingBot are all mixed-use. They blend search indexing with training collection in the same crawler. Under the new defaults, they'll be blocked on ad-supported pages unless they split those functions.
What September 15 Actually Changes
For new Cloudflare domains and all existing free-tier customers: Training and Agent bots will be auto-blocked on any page that shows ads.
The logic is sharp: "An ad is a signal that a website owner meant for a person to land there." If you're monetizing with ads, you built the page for humans — not for training datasets.
Before September 15, Cloudflare will let existing customers opt out of these new defaults. After that, you'll need to actively change your settings if you want mixed-use crawlers back in.
What You Should Do Right Now
If you own a website:
Go to your Cloudflare dashboard → Security Settings → Configure AI bot policies.
You now have granular controls for each category:
| Category | Options |
|---|---|
| Search | Block all pages / Block on ad pages / Allow |
| Agent | Block all pages / Block on ad pages / Allow |
| Training | Block all pages / Block on ad pages / Allow |
Previously it was one toggle: "Block AI bots." Now you can let ChatGPT fetch your content in real-time for users while blocking OpenAI from training on it. That's actually a useful distinction.
If you're a Bot Management customer, you also get content use level controls:
-
immediate— No storage, no reuse, one-time access only -
reference— Indexing with attribution (the new default) -
full— Summarization and reproduction allowed
Pick immediate if you want crawlers to access your content but never store or repurpose it.
If you're building AI products that crawl the web:
Your bot needs a declared identity now. Cloudflare's BotBase (enterprise-tier) is a searchable directory of classified bots with detection IDs. The era of stealth mixed-use crawlers is ending.
If you're building a RAG pipeline that fetches live content, you're in "Agent" territory. Declare yourself. Some sites will block you. That's the deal now.
If you're training on scraped web content: you're Training. Many sites will block you by default as of September 15. Plan accordingly.
The Bigger Play: Cloudflare Is Forcing Transparency
The real move here isn't blocking. It's forcing the biggest tech companies to be honest about what their crawlers actually do.
Google, Apple, and Microsoft have run bundled crawlers for years. Search + training + agentic features, all in one bot. Cloudflare is giving them until September 15 to separate those functions — or get blocked from monetized content.
This also opens a revenue channel for publishers. Cloudflare is partnering with Ceramic.ai and You.com so site owners can opt into paid access: when your content surfaces in their AI search results, you get paid. The infrastructure for "pay for web content" is being built right now.
The Uncomfortable Truth for AI Labs
Every major AI lab has been benefiting from the assumption that web crawling is free. Cloudflare just challenged that assumption at infrastructure scale.
You can't really argue with a firewall.
Labs building agentic systems that access web content in real-time need clean bot identities, transparent declared use cases, and eventually: payment. The same pressure that's hitting Google and Microsoft is coming for everyone.
The September 15 deadline is a forcing function. If you're building anything that touches the web — as a site owner, an AI developer, or an infrastructure engineer — check your Cloudflare settings this week.
After September 15, the defaults are Cloudflare's call, not yours.
Sources: Cloudflare Blog · Cloudflare Docs · TechCrunch · NBC News
Top comments (0)