Type a question into ChatGPT today and there's a decent chance the answer leans on a handful of trusted sources. Ask that same question in five years and the pattern may look nothing like it does now — not because the AI got smarter, but because half the internet quietly locked the door before the guests arrived.
We pulled the robots.txt files of 98 major websites across four industries — SaaS, e-commerce, publishers, and a mixed general category — and checked how each one treats six AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended. The goal was simple: who's letting AI read them, and who's shutting the door.
The results split almost perfectly along industry lines, and the split tells a story nobody in these newsrooms seems to be saying out loud.
The Numbers
Across the full sample, 64% of content and news publishers block at least one major AI crawler. Compare that to SaaS companies and e-commerce retailers, where the number sits at just 16% — a four-to-one gap between the industries that make money from being read and the industries that make money from being bought from.
Five publishers didn't stop at "at least one." They blocked all six crawlers, completely:
- The New York Times
- BBC
- Bloomberg
- BuzzFeed
- HuffPost
Every one of these sites has told GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended, in writing, in the one file every crawler is obligated to check first: don't come in.
Meanwhile Stripe, Shopify, Notion, Walmart, and Nike — companies with just as much to lose from misrepresentation — leave the door wide open.
ClaudeBot turned out to be the most frequently blocked crawler overall (21.4% of sites), with Applebot-Extended the least blocked (16.3%) — a small but telling detail, since Apple's crawler is newer and less publicly scrutinized than OpenAI's or Anthropic's.
Why Publishers Are Actually Doing This
The instinctive explanation — the one that fits neatly into this year's broader anxiety about AI — is that these newsrooms are worried about AI itself. That somewhere in a boardroom, an editor said "we don't trust what this technology might become" and pulled the plug out of principle.
That's not what's happening. Not really.
The real reason is far less philosophical and far more familiar: money. Publishers have watched referral traffic from Google search decline for two straight years as AI Overviews answer questions directly on the results page — a phenomenon studies have tied to click-through-rate drops of over 50% when an AI Overview appears. Every AI answer generated from a publisher's paragraph is, from a CFO's chair, a reader who never has to click through, never sees an ad, never converts to a subscription. The New York Times sued OpenAI and Microsoft in late 2023 over exactly this — unauthorized use of its journalism to train and power models that compete with it for attention. Several of the other blockers on this list have pursued or signed licensing deals of their own, treating their archives as an asset to be sold, not scraped for free.
This is a business decision wearing a safety costume. And that distinction matters, because it changes what "winning" looks like for a publisher that blocks everything.
The Reality Check Nobody's Running
Here's the part that should worry these newsrooms more than it apparently does: blocking a crawler doesn't stop an AI model from being trained on your content. It stops your site from being cited going forward, on the answers the model gives today.
Every one of these five fully-blocked publishers has decades of archived content already inside the training data of every major model released before this policy went into effect. The lockout doesn't erase the past. It just guarantees the model has no live, current reason to point back to nytimes.com the next time someone asks it a question the Times is best positioned to answer. The knowledge stays in the model. The traffic, the attribution, the brand recognition — that's what walks out the door.
Meanwhile, the panic framing — "AI could harm people, so we're protecting our journalism" — doesn't hold up against the pattern in the data. If safety were the real driver, you'd expect blocking behavior to be roughly consistent across industries handling sensitive information: healthcare content, financial advice, legal guidance. It isn't. The blocking is concentrated almost entirely in one business model: ad-and-subscription-funded media that depends on pageviews. That's not a safety stance. That's a traffic-defense stance dressed up in safety language because "we're worried about our revenue" doesn't make for a good press statement.
What Staying Open Actually Buys You
SaaS and e-commerce companies didn't arrive at their 16% blocking rate through some deeper understanding of AI ethics. They arrived there because their business model rewards being mentioned. A user who asks ChatGPT "what's a good project management tool for a five-person team" and gets Asana or Trello as an answer is a warmer lead than one who found either brand through a banner ad. Being the cited answer, in an AI-mediated search world, functions the same way ranking #1 on Google used to — except the shelf space is smaller, and once a model settles into citing certain sources for certain queries, displacing that pattern gets harder every month the incumbent stays visible and the challenger stays hidden.
That's the actual risk facing the fully-blocked publishers, and it's not a five-year problem. It's already underway. Every week an AI tool answers a news query without citing the Times, BBC, or Bloomberg, a different, more permissive source fills that gap instead — and the citation habits models build now are the ones they'll default to for years. Being locked out of AI answers isn't neutral. It's a slow transfer of authority to whichever competitor left the door open.
The Uncomfortable Trade-Off
None of this means blocking is irrational. If your core asset is exclusive reporting and your legal team believes a licensing fight is winnable, staying closed while you negotiate is a defensible position — Bloomberg and the Times both have the market weight to make that bet. What's less defensible is doing it by default, without a plan, the way plenty of smaller publishers and blogs quietly do — copying a robots.txt template they found online without understanding what they just switched off.
The uncomfortable truth sitting underneath this whole dataset: AI visibility isn't a future consideration anymore. It's a present one, and the businesses treating it that way — deliberately, strategically, industry-blind to the panic headlines — are the ones building the citation habits that will define discovery for the next decade.
Methodology: robots.txt files for 98 websites across SaaS, e-commerce, content/news, and general categories were programmatically checked for explicit Allow/Disallow rules governing GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended. A site was counted as "blocking" a crawler if its robots.txt contained a full or partial Disallow rule for that user-agent. Full dataset available on request.
Top comments (0)