The Short Answer
If you've ever wondered whether "AI crawlers are eating my server" is a real problem or SEO folklore, here's an actual answer from an actual site: yes, they're real, they're not small, and the breakdown is not what you'd guess. In one 24-hour window on this site, AI-labeled crawlers made roughly 9,000 requests. Applebot — not any generative-AI company's bot — was 59% of that volume. Googlebot, the crawler that actually determines whether your pages show up in Google Search, was about 5%.
The Real Numbers
Pulled directly from Cloudflare's AI Crawl Control dashboard for automationscookbook.com, one 24-hour window:
| Crawler | Requests | What it actually is |
|---|---|---|
| Applebot | ~5,300 | Powers Siri, Spotlight, and Apple's own web index — not a chatbot crawler, but bucketed under "AI" |
| BingBot | ~1,280 | Microsoft's crawler — also feeds Copilot's answers |
| PetalBot | ~975 | Huawei's search/AI crawler |
| Googlebot | ~445 | The one crawler that actually drives this site's organic Search clicks |
| ChatGPT-User | tens | Not a crawler — this fires when a live user asks ChatGPT to browse a specific page in real time |
Two things stood out that we didn't expect going in.
First: "AI crawler" is a much bigger bucket than "generative AI company." Most of the volume in Cloudflare's AI Crawl Control isn't OpenAI, Anthropic, or Perplexity — it's Apple and Huawei's general-purpose crawlers, which happen to feed AI features somewhere downstream (Siri, Petal Search) but look nothing like what people picture when they say "AI is scraping my site."
Second: a meaningful chunk of crawler traffic wasn't hitting content at all. In the same window, /cdn-cgi/zaraz/s.js — a third-party tag-manager script, not a page — took 2,585 crawler requests and 15MB of transfer on its own. No crawler needs to fetch a JavaScript loader repeatedly; that's pure waste, both for the site's bandwidth bill and for whatever's left of the crawler's own budget for the pages that matter.
Why This Matters for Site Owners
- Blocking "AI bots" broadly is blunter than it sounds. A robots.txt rule aimed at generative-AI scrapers can just as easily catch Applebot or BingBot — crawlers that feed products with actual referral value (Siri suggestions, Copilot answers, Bing search) — if you're not checking which user-agents you're actually matching.
- Crawl volume and search performance are two different numbers. Googlebot being 5% of "AI crawler" traffic while still being the primary source of real organic clicks is a reminder that request count tells you about server load, not about ranking or referral value. Don't optimize for the wrong one.
-
Live agent browsing is now a measurable, separate thing from indexing.
ChatGPT-Usershows up when a person actively asks an assistant to look at a page right now — different from being crawled for training or for a search index. If you care about being citable in AI answers, this is closer to the metric that matters than raw crawl volume. -
Wasted crawl budget is fixable in minutes. Excluding non-content paths (script loaders, tracking pixels,
/cdn-cgi/*) from what crawlers bother fetching is a robots.txt or edge-rule change, not a rebuild.
FAQ
Q: Should I block AI crawlers entirely?
A: That depends on what you're optimizing for. Blocking cuts off training-data usage, but several of these same crawlers (Applebot, BingBot) also power discovery features you probably want (Siri results, Copilot answers). Check which specific user-agents you're blocking before assuming "AI bot" rules only hit generative-AI companies.
Q: Why is Googlebot such a small share of "AI crawler" traffic?
A: Because most of what gets bucketed as "AI crawler" traffic isn't Google at all — it's Apple, Huawei, Bing, and others. Googlebot's search-indexing crawl is tracked separately in Google Search Console and isn't meaningfully down; it's just a small slice of this particular combined "AI" bucket.
Q: How do I check this for my own site?
A: If you're on Cloudflare, the AI Crawl Control section (under Security in the zone dashboard) breaks down requests by crawler, shows data transfer per bot, and lists your most-crawled paths — no extra setup required for a zone already proxied through Cloudflare.
Q: What's the fix for crawlers wasting requests on non-content files?
A: Disallow the specific non-content paths (tag-manager loaders, tracking pixels, internal /cdn-cgi/* routes) in robots.txt, or add a Cloudflare rule that returns a fast 204 for known bot user-agents on those paths. Content pages should stay fully crawlable — this is about trimming the paths that were never meant to be fetched repeatedly in the first place.
Originally published on Automations Cookbook.
Top comments (0)