Cloudflare shipped something recently, and most people scrolled right past it.
It lets websites charge AI crawlers. The headlines mostly framed it as "publishers can finally bill the robots."
Framed that way, it really is just a minor news item.
But charging crawlers is only round one. What's actually happening is far bigger.
The old contract of the web is breaking
Start with the old deal.
For twenty years, you had a website, and you let search engines crawl it — because they sent you people.
Google read your page, indexed it, someone searched, your page surfaced, and a real human clicked through.
Once that person landed on your site, you monetized their attention. Ads, email capture, subscriptions, affiliate links.
The crawler took your content, your site got the visitor, the visitor got their answer. That accounting propped up half the internet.
Now the AI agent has snapped that chain.
It reads your page, extracts the useful answer, and feeds it straight to the user. The user may never visit your site at all.
The content still created value. But the site lost the visit.
Lose the visit, and the ads are gone, the email capture is gone, the affiliate click is gone.
So publishers are up in arms. But that's only round one.
The human web sells attention; the machine web sells resources
What's really changing is what gets monetized.
On the human internet, you monetize attention. You hold people, then run ads or sell them things.
On the agent internet, you monetize useful resources. The agent doesn't watch ads. It asks one thing: does this resource help me finish the job?
If yes, it pays. If no, it leaves.
Here's the counter-intuitive part.
Ask a human to pay a fraction of a cent to view a web page and they'll close the laptop on the spot. Nobody's paying a micro-fee to see the sauce recipe before the actual recipe.
But the machine doesn't care. Small amount, paid automatically, data's useful — it just pays.
Micropayments failed on the human internet for twenty years. Move them to machines, and they work by default.
That's exactly the road Cloudflare is paving. Using HTTP 402 — the "payment required" status code — the agent sends a request, the server says here's the price, the agent pays and retries.
A single request becomes a transaction.
No checkout flow, no signup first, no sales call, none of the enterprise procurement dance.
The request is the transaction. That's a big deal.
A pile of tiny resources is turning into businesses
Once a request becomes a transaction, countless tiny things on the internet can become businesses.
A dataset can charge per query. An API can charge per successful call. A research archive can charge per answer. A product catalog can charge per price comparison.
Someone will say: agents don't have wallets.
But agents are growing wallets, and growing inboxes. It's already begun.
The moment they can pay, these small resources stop being content and become assets.
So the website becomes a resource layer, and content becomes an index. Expertise becomes something scalable, data becomes something meterable, tools become interfaces an agent can call.
A new stack is forming.
At the bottom is the messy internet: PDFs, pricing pages, old blogs, support docs, comparison sites.
Then someone cleans it into structured data. Then someone turns it into an interface an agent can read. Then someone adds pricing rules — some free to pull traffic, some paid, some locked down entirely. Finally someone adds trust and analytics: is the data fresh, is the source reliable, which requests are worth money.
That whole stack will grow thousands of companies.
One question is enough: what resource does an agent urgently need, frequently need, reliably need — and is willing to pay for?
You can cut in right now — no need to wait
Someone on X said startups are harder now: traffic's down, revenue's down.
I see it the opposite way.
What you shouldn't build is the kind of little tool that can be vibe-coded in an afternoon. That stuff genuinely has no future right now.
What you should build is a business that's profitable from day one, has a moat, and is redesigned for the agent era.
And you don't have to wait for Cloudflare's whole apparatus to mature.
Do the manual version first. Sell the human-powered version first. Accumulate the data first, package the thing first.
Concrete example. Pick a niche — say medical-aesthetic clinics in one city. An owner wants to know: what are competitors charging, what procedures do they offer, what are the bad reviews complaining about, who's hiring, which promotions actually work.
That information is scattered across review sites, official pages, job postings, ad libraries — and still inside the heads of owners and staff.
Your job is to clean it into one tank of clean fuel.
At first, your first customer isn't even the clinic owner. It's the people already selling to clinics — agencies, consultants, software vendors.
Because "agent-readable competitive analysis" means nothing to the owner. But an agency gets it instantly. It closes one more client off the back of your data, and it'll happily pay you a few hundred a month.
First a report, then a dashboard, then an API, then the agent pays per call. Crawl, walk, run.
Who wins, who loses
Lay the incentive structure bare.
People who start accumulating data now win. The agent internet is still small, competition hasn't arrived. You have six to eighteen months to clean one niche's dirty data into clean fuel — and by the time competition shows up, you're already inside the door.
People who wait for the trend to mature lose. By the time Cloudflare's apparatus is common knowledge and agent wallets are everywhere, these little doors are already taken. You'll be fighting over scraps.
People who only build vibe-coded toys lose too. That stuff has no moat and gets cloned the day it ships.
This isn't "another mildly interesting piece of tech news."
Agents are becoming buyers. Websites are becoming resources.
The next batch of great internet businesses may well be these unremarkable little paid doors that agents pass through all day long.
And right now, nobody has claimed them.
Neo Teng · building a company of one
Top comments (0)