Websites on Cloudflare's network refused 13.47% of AI bot requests in the third quarter of 2026, more than double the 5.98% rate a year earlier. The figures come from a TechnologyChecker.io analysis of Cloudflare Radar data, published October 1 and pushed into marketing news by a Zen Media release on October 9. If your brand is paying for AI visibility work, this number matters more than any ranking report you will see this month.
What the Q3 data actually shows
The headline stat is the 403 Forbidden rate. In Q3 2025, 5.98% of AI crawler requests hit a locked door. In Q3 2026 it was 13.47%, and September alone came in at 14.25% on provisional numbers. Add 404s (5.86%) and rate limit responses (2.22%) and at least 21.5% of AI bot requests now end in a client error.
The same report also parsed 4,242 robots.txt files in a September 28 snapshot and counted how often each bot was blocked versus allowed. OpenAI's GPTBot, which gathers training data, was disallowed 1.60 times for every allow. OAI-SearchBot, which fetches pages for ChatGPT search answers, came in at 0.91. Site owners are getting selective. They say no to training and yes to the bots that can send them traffic.
Search and user triggered crawling also jumped, from 6.51% of AI bot activity in Q2 2026 to 11.92% in Q3. A year ago it was 3.14%.
The referral side explains the mood. Per the report, Anthropic's crawlers fetched 988 pages for every visitor they referred back, OpenAI's fetched 309, and Perplexity's 1,083. Google sits at 4.9. Publishers have done that math.
What this means for marketers and business owners
A lot of companies bought "AI visibility" audits this year, rewrote their pages for ChatGPT and Perplexity, and then never checked whether those engines could reach the site at all. Zen Media's release makes the point bluntly: a robots.txt file can welcome an AI search crawler while a firewall, CDN rule, or bot protection setting quietly turns it away. Cloudflare's own bot features can do this by default.
So the first job is boring and technical. Pull your server or CDN logs and look for OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot. If they are getting 403s, no amount of content optimization will get you cited.
The second job is a policy decision. Blocking training crawlers while allowing search and user agents is now the majority pattern, and it is a reasonable default for most service businesses. You keep your content out of model training sets while staying eligible for answers that send buyers.
This is the kind of plumbing that gets missed when marketing and IT run on separate tracks. Tools like KenjiAI (kenjiai.com) are built for exactly this kind of shift where lead flow depends on systems actually talking to each other.
What to watch and do next
Do these three things this week.
- Check your CDN or WAF bot settings. On Cloudflare, review Bot Fight Mode and any "Block AI" toggles, then confirm search crawlers get a 200 response.
- Rewrite your robots.txt with separate rules per user agent. Disallow GPTBot, ClaudeBot and CCBot if you do not want training use. Explicitly allow OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot.
- Ask any AI visibility vendor to show you crawler access logs before they show you a citation score.
Then watch two numbers. The first is whether Q4 denial rates keep climbing past 14%. The second is the crawl to referral ratio for OpenAI and Anthropic. If those engines start sending real traffic back, expect publishers to open the door again. If they do not, expect the 13.47% figure to keep rising through 2027.
Source: TechnologyChecker.io, robots.txt AI Crawler Report Q3 2026, via Zen Media on GlobeNewswire, October 9, 2026.
Published by the Media Traffics | KenjiAI team. kenjiai.com
Top comments (0)