i write scrapers for a living, so read this knowing im biased. but something happened on july 1 that got covered as a pricing update, and it is not a pricing update.
Last time i wrote about the big labs pulling the ladder up behind them after climbing it. This is the same story from a different seat. Not a company fighting over data it trained on. A company that sits in the road deciding who is allowed to walk down it.
Start with what they got right
Most takes on this are lazy, so let me concede the strong part first.
Cloudflare moved from pay per crawl to pay per use. Under the old model a bot paid a fee to fetch a page, minimum a cent per retrieval. Under the new one, publishers get paid when their content actually shows up inside an AI answer. Opt in, content gets used downstream, money comes back.
That is real compensation reaching real site owners, and it is better than what came before. Anyone telling you this is pure extraction has not read it. i am not going to pretend otherwise to make the rest of this hit harder.
The payment is fine. Its everything wrapped around the payment that should bother you.
Who wrote the categories
AI crawlers now get sorted into three buckets. Search, agent, training. Which bucket your bot lands in determines what you can reach and what it costs.
Nobody outside one company defined those categories. Think about how much sits inside that definition. Is a research crawler agent or training. Is an archival bot search. Is a small team building a tool a legitimate reader or an unlicensed one. Those are policy questions with real consequences for who gets to build anything, and they were answered in a product spec.
Who made the scarcity
Starting September 15, mixed use crawlers get blocked by default on any page carrying ads.
Read that as a business move rather than a safety one. You cannot run a marketplace for access until access is scarce. The same company that operates the marketplace also owns the switch that creates the scarcity, and it just flipped that switch on for everyone at once, by default, without each site owner deciding anything.
That is not a company responding to a market. That is a company manufacturing one.
Who is on the list
The program launched with two partners.
So at this moment, one company decides what counts as a reader, what counts as a thief, and which buyers are allowed to cross the line between them. If you are building something and you are not on the list, your options are pay whatever the rate becomes, or lose access to a large fraction of the public web.
That is not a market with many sellers and many buyers. That is a licensing authority with a waiting list.
The part scrapers already knew
Everyone in this work has understood for years that bot blocking was never really about stopping bots.
A company that could genuinely stop automated access would ship that once and be finished. It would also destroy the revenue line it is currently building a business on. The incentives only ever pointed one direction. What got built is not a wall, its a meter with a wall painted on the front.
Blocking is the demo. Billing is the product. Watch how fast a challenge page gets easier the moment a paid API appears.
The site owners are the strange part
Millions of sites pointed their DNS at one company years ago because the free tier was genuinely good and the bandwidth cost nothing. That was never charity. That was acquisition cost, and it worked.
Now you get a check. You do not get a vote. Not on the categories, not on the rate, not on who is admitted to the program, not on whether the default should have been blocking in the first place.
What to actually take from this
im not telling anyone to rip out their CDN. the payments are real and some publishers will be better off.
im saying notice what quietly happened. Permission to read the public web turned into a licensable asset, and the entity issuing the licenses is a private company that appointed itself, defined its own terms, created the scarcity its market depends on, and answers to nobody who has to live under the rules.
The money is not the story. The authority is.
Top comments (0)