I run a small Home Assistant / self-hosting blog. On a normal day a few dozen humans show up. So when I finally grepped my nginx logs for AI crawlers, the number made me stop: in nine days, AI bots hit the site 18,209 times. On a blog this size, the machines reading me now outnumber the people.
Here's the full breakdown, the things that surprised me, and a few points most "should I block AI bots?" threads get wrong.
The raw numbers (9 days, one small blog)
Of 348,667 total requests, 18,209 (5.2%) came from AI/LLM user-agents:
| Bot | Requests | What it actually is |
|---|---|---|
| ChatGPT-User | 6,687 | OpenAI — live fetch when someone asks ChatGPT about a page |
| Bytespider | 3,369 | ByteDance / TikTok crawler |
| meta-externalagent | 3,274 | Meta AI |
| Amazonbot | 1,923 | Amazon |
| OAI-SearchBot | 1,211 | OpenAI search index |
| ClaudeBot | 850 | Anthropic — training / index crawler |
| PerplexityBot | 319 | Perplexity |
| DuckAssistBot | 225 | DuckDuckGo AI |
| GPTBot | 172 | OpenAI training crawler |
| CCBot | 86 | Common Crawl (feeds many models) |
| YouBot | 68 | You.com |
Surprise #1: the #1 "AI bot" isn't a crawler
The biggest source by far — ChatGPT-User, 6,687 requests — isn't crawling to train a model. It's a live fetch: someone asked ChatGPT a question, ChatGPT decided my page was relevant, and pulled it in real time to answer them. Same story with Perplexity-User and the other assistant-side fetchers.
That flips the "should I block it?" math. ChatGPT-User isn't scraping you — it's a real person, through their assistant, reading your page right now. Block it and you don't stop any training; you just stop showing up in answers and lose the visit. (I can see the other end of it in my analytics: real sessions arriving from claude.ai and gemini.google.com.)
So the mental model "AI bot = scraper to block" is wrong for a big chunk of the traffic. There are training crawlers (GPTBot, ClaudeBot, CCBot) and there are live answer-engine fetchers (ChatGPT-User, Perplexity-User, DuckAssistBot). Treating them the same is the mistake.
Surprise #2: robots.txt behaviour is all over the place
I checked who actually requests /robots.txt:
- ClaudeBot: 233 — diligent, checks constantly.
- PerplexityBot: 56 — checks in.
- Bytespider: 77 — does fetch it (whether it honours the contents is its own reputation).
- GPTBot: 0 — didn't pull it once in this window (low volume, granted).
- ChatGPT-User: 0 — never. And that's correct: it's user-initiated, like a browser. Browsers don't read robots.txt, and a live fetch on a human's behalf shouldn't either.
The practical upshot: a blanket Disallow is obeyed by the well-behaved training crawlers and ignored by the user-fetchers — because robots.txt was never meant for them. If your goal is "don't feed training, but keep appearing in answers," the default already does roughly the right thing — but only for the bots that honour it.
The gap nobody mentions: these crawlers don't run JavaScript
This is the one that actually cost me, and the reason I ended up building a little tool around it.
Almost every AI crawler fetches your raw server HTML and does not execute JavaScript. So if your framework injects JSON-LD structured data on the client, or your streaming-SSR setup flushes meta tags into the <body> instead of the initial <head>, those signals are invisible to the crawler — even though Google renders them and every SEO browser extension tells you you're fine.
I only found six pages on my own site doing exactly that, because I built a crawler that deliberately parses the JS-less view and diffs it against the hydrated DOM. Googlebot renders; GPTBot and ClaudeBot mostly don't. If you care about being represented correctly in AI answers, your structured data and metadata have to live in the server HTML, not get painted on after hydration.
What I'd actually do with this
- Don't reflexively block "AI bots." Separate training crawlers (GPTBot, ClaudeBot, CCBot — robots.txt works) from live answer-fetchers (ChatGPT-User, Perplexity-User — blocking them costs real referrals).
- Verify by ASN / reverse DNS, not the user-agent. The UA string is trivially spoofed; a "GPTBot" from a random consumer IP is not GPTBot.
- Put structured data + metadata in the server-rendered HTML. If it only appears after JS, the AI crawlers never see it.
- Watch Bytespider if load is a concern — it's the most aggressive of the genuine crawlers.
For a blog with a few dozen human readers a day, AI crawlers are now the single largest non-search audience hitting the server. They aren't going away — so it's worth knowing which ones are reading you, and making sure they can actually see what you publish.
The JS-less-view crawler I used is open-source (seo-geo-audit) — it flags exactly this gap, plus the usual SEO checks, in plain Node with no paid dependencies.
Top comments (0)