Cloudflare Is Blocking Many AI Crawlers by Default on Ad-Supported Pages, Not Charging Them Automatically

#cloudflare #ai #publishers #webscraping

Cloudflare is not automatically making AI companies pay publishers for content; starting September 15, 2026, it will default many ad-supported Cloudflare pages to block AI training crawlers, AI agents, and mixed-use bots unless the publisher opts for a different setting. The practical shift is that Cloudflare is using default-deny access to push AI companies toward separate crawler identities and paid access systems such as Pay Per Crawl and the broader Pay Per Use model reported by TechCrunch.

Cloudflare is the web infrastructure company that sits in front of a large share of sites, handling security, caching, and traffic management. That position gives its new AI Crawl Control categories unusual force: this is not a publisher trade group asking nicely, but a network gatekeeper changing the default rules for how bots reach pages.

September 15 default blocks for training and agent crawlers on ad pages

What changes on September 15 is a default setting, not a universal toll. In its July 1, 2026 announcement, Cloudflare said ad-supported pages will by default block three categories: AI Crawler bots used for training, AI Agent bots acting on behalf of users, and mixed-use crawlers that combine search/discovery with training or agent functions.

That matters because Cloudflare now separates bot purposes more explicitly in its bot reference: AI Search is distinct from AI Crawler, and publishers can treat them differently. A bot that only indexes for answer-engine search may still be allowed where a bot that also trains models is blocked.

The company says this default will apply to new domains, new sites created by existing customers, and existing free customers, not automatically to every paid Cloudflare customer. That is a narrower rollout than “the whole web now charges AI,” but it still covers a meaningful slice of sites.

The pressure point is mixed-use bots. If an AI company uses one crawler for both search visibility and model training, Cloudflare’s new defaults make that design costly. The clean workaround is to split functions into separate verified crawlers, so publishers can allow search while blocking training. That fits the broader rise of AI web-scraping barriers: the infrastructure layer is starting to demand finer-grained bot identities, not just a single “trust us” user agent.

Pay Per Crawl expands toward pay-per-use with Ceramic and You.com

Cloudflare’s payment system lets publishers choose to block, allow, or charge for bot access, but it is still partly experimental. The company’s Pay Per Crawl signup page still labels the product a private beta, and its payout documentation shows a workflow that runs through Stripe, with Cloudflare initiating charges, aggregating them, and sending publishers monthly payouts.

In practice, the model is simple enough: a publisher sets a price for an eligible crawler request, Cloudflare checks whether the crawler is allowed, and, if so, handles billing and payout plumbing. It is less like a paywall for humans than a toll booth for bots.

TechCrunch reports that Cloudflare is already stretching that idea beyond per-crawl billing toward a broader Pay Per Use system, with Ceramic and You.com named as early partners. That matters because per-request charging only solves one part of the dispute; AI companies increasingly want access for retrieval, answering, and agent workflows, not just old-style scraping.

“If they’re going to consume your content, then they should pay for your content.” Matthew Prince, Cloudflare’s CEO, put the company’s position in those terms in an Axios interview.

This is also where the policy touches licensing politics more directly. Cloudflare is not deciding copyright law, but it is building infrastructure that makes “pay or stay out” easier to enforce at the network edge. That lands in the same broader fight as cases such as the Meta copyright lawsuit, where the legal system is still working out what AI companies can ingest without permission.

Cloudflare’s referral ratios quantify publishers’ leverage problem

Cloudflare’s case for this push rests on a blunt traffic claim: AI crawlers send far less traffic back to publishers than traditional search. In an Axios interview, Chief Executive Officer Matthew Prince said Google sends a visitor for roughly every 18 crawls, while OpenAI sends one for about 1,500 crawls and Anthropic one for about 60,000 crawls.

A simple way to make that concrete: using Prince’s figures, Anthropic’s reported crawl-to-referral ratio is about 3,333 times worse than Google’s, based on 60,000 crawls per referral versus 18. That is the leverage problem in one line. AI systems can extract value from pages without sending the compensating clicks publishers built their businesses around.

Those numbers are company-reported, not an independently audited industry dataset. But they explain why Cloudflare is framing this as an existential business issue for publishers rather than a mere bot-management tweak.

The immediate winners, if the system works as Cloudflare hopes, are publishers that want to preserve ad inventory or negotiate licenses from a stronger starting position. The immediate losers are AI companies whose crawlers blur together search, training, and agent use because the cheapest operational habit — one bot for everything — now collides with default blocks.

The next milestone is operational, not rhetorical: September 15, 2026 is when the new defaults begin rolling out for the covered Cloudflare customers, while Pay Per Crawl remains in private beta and the broader pay-per-use model is still emerging.

Key Takeaways

Cloudflare is not automatically charging AI companies for content; it is defaulting many ad-supported pages to block training, agent, and mixed-use crawlers starting September 15, 2026.
The September 15 default applies to new domains, new sites set up by existing customers, and existing free customers, not automatically to every paid Cloudflare customer.
Cloudflare’s system distinguishes between AI Search bots and AI training crawlers, which pressures AI companies to split mixed-use bots into separate identities.
Pay Per Crawl lets publishers block, allow, or charge crawlers, with Stripe-based monthly payouts, but the product is still listed as a private beta.
Cloudflare’s main evidence for publisher harm is Matthew Prince’s reported crawl-to-referral ratios: 18:1 for Google, 1,500:1 for OpenAI, and 60,000:1 for Anthropic.