Nobody in scraping is talking about this and it lands in ten days.
On September 15 Cloudflare flips a default. Every bot that hits a site behind them now gets sorted into one of three buckets. Search, agent, training. Search stays allowed. Agent and training get blocked by default on any page that shows ads. That applies to every new domain, every new customer, and everyone already on the free tier. A site owner can opt out in the security settings before the 15th. Most wont, because most never open that page.
Their definition of agent is the part that matters. "Automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." Read that again. That is a fetch tool. That is an MCP server that pulls a page for a model. That is every browser agent demo from this summer.
The part that is funny if you dont run a scraper
Googlebot fetching a page and my fetch tool fetching the same page for a user is the same request. Same bytes, same page, same intent, get the content and use it. One of them has a deal with Cloudflare. One of them doesnt. After the 15th one walks in and one gets a wall.
Cloudflare says this is about publishers getting paid. Fine. But the people who can pay already have contracts. OpenAI has contracts. Google is a search bot by definition so it is exempt from its own training problem. The ones who eat this are one person shops with an agent and a fetch call. Thats me, and probably you if you are reading this.
How i noticed
i didnt read the announcement. i run scrapers for a living and on August 31 Google changed its search result links to /goto redirect wrappers. My parser didnt error. It returned zero rows for a full day and billed nobody, which is the only reason i looked. When i went digging into what else was changing this month, this was sitting there dated July 1 and nobody i follow had said a word.
Earlier in August i lost two other scrapers to DataDome on Etsy and Yelp. Residential proxies didnt help. i killed both instead of fighting it. So i already know what a silent block looks like on a revenue chart. It looks like nothing. Flat line, no error, no email.
That is what the 16th looks like for a lot of agent builders. The model calls the fetch tool, the tool gets a challenge page or a 403, and the model summarizes the challenge page like it is content. Nobody gets an alert.
What i still dont know
Cloudflare has not said what the blocked request actually gets back. Their blog does not say 403, challenge, or the pay per crawl 402. i have not been able to find a site that opted in early to test against. So this is the open question and it is the one that decides whether your guard code works.
A test you can run before the 15th
Pick a site you fetch that is on Cloudflare free tier and shows ads. Fetch it three ways today and save the status code, the response headers, and the first 500 bytes of the body.
- Plain fetch with your normal client.
- Same fetch with a user agent that says what it is, something like YourAgent/1.0.
- Same fetch through whatever your agent framework uses.
Run the same three on the 15th and 16th. Diff them. Look for the cf-mitigated header, a 403, a 402, or an HTML body with "challenge" in it. Whatever changes is your signature. Put a guard on it so your model refuses to summarize a block page.
The ask
If you run a site behind Cloudflare and you already opted in, or you already see the new behavior, paste the status code and the headers in the comments. i will post what i find on the 15th under this article, working or not. i would rather know than guess.
Source is the Cloudflare post from July 1, "Your site, your rules: new AI traffic options for all customers", on blog.cloudflare.com.
Top comments (0)