Cloudflare Now Lets You Block AI Crawlers — Should Indie Builders Care?

Cloudflare rolled out new AI traffic management controls on July 1, letting website owners separately manage Search, Agent, and Training crawlers. By default, Agent and Training bots are blocked on ad-supported pages, while Search bots are still allowed. Starting September 15, new domains will get these restrictions automatically.

For indie hackers and solo builders, this is one of those small announcements that could matter a lot.

The three types of AI crawlers

Cloudflare now distinguishes between:

• Search crawlers: bots that index your site for search engines. Generally good for traffic.
• Agent crawlers: bots that scrape your content so AI agents can answer questions about it.
• Training crawlers: bots that scrape your content to train models.

Before this update, most site owners had an all-or-nothing choice. Now you can block the crawlers that train models while still letting search engines index you.

Why this matters for solo builders

If you run a blog, a documentation site, or a small SaaS with a public-facing knowledge base, AI crawlers have probably been eating your content without attribution. That content might then show up inside ChatGPT or Claude answers, giving users the information they need without ever visiting your site.

For a solo project, that traffic matters. Fewer visitors means fewer signups, fewer newsletter subscribers, and less word of mouth.

But there is a trade-off

Blocking AI crawlers is not obviously the right move. Some AI search products do drive referral traffic. And if your content never gets indexed by AI systems, you might miss out on a new distribution channel.

A few questions I am wrestling with:

• Do AI answers help or hurt small sites in the long run?
• Should I block training crawlers but allow agent crawlers?
• Does anyone actually read robots.txt anymore, or is this just a PR move?

A related thought on open-source AI platforms

Cloudflare's move is about controlling access to your content. On the flip side, open-source AI platforms like monkeycode are about controlling the AI tools themselves. If you are building with AI and care about where your data goes, having a self-hostable option is worth knowing about.

---

What is your take? Are you blocking AI crawlers, leaving them alone, or just ignoring the whole debate?

Source: Cloudflare Blog — AI Bot Control, summarized via Crescendo AI News