DEV Community

Cover image for The Bot at My Door Picks Its Own Name
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

The Bot at My Door Picks Its Own Name

I opened the access log this morning with a different intention. Back in June I wrote a policy into my robots.txt — "yes to citation, no to training" — and I wanted to count the parties it addresses. What stopped me wasn't the policy. It was the name.

Because every request arriving at the door carries a name on it, and the visitor writes that name itself. When a request says ClaudeBot, two things are in my hand: a string and an IP address. The string is free. In thirty days of logs, eleven addresses share sixteen different official bot names between them: the same IP is ClaudeBot, and OAI-SearchBot, and Googlebot, and bingbot, and meta-externalagent, and DeepSeek. Nine of them carry thirteen names each; one carries fourteen.

This article is about those eleven addresses and what they taught me: tie a rule to names, and the rule holds only as far as the name can be verified.

What I counted, and how

The blog's nginx access log is per vhost and includes the user agent. To stay on whole days, the window is pinned to the thirty days from 19 October to 17 November 2026: 741,578 requests reaching the origin.

Four limits, up front:

  • The site sits behind Cloudflare; requests served from the edge cache never land in the origin log. Every figure is a floor.
  • Of those 741,578, some 231,512 carry a recognizable bot token in the user agent. Most of the remaining 510,066 arrive with browser agent strings, and some of them are mine: alert-monitor at 40,119 requests and assorted curl versions at 67,832. (When I wrote about the watchdog that downloads the feed every minute two weeks ago I couldn't pin down which machine one curl version belongs to; I still haven't.)
  • Token matching is substring-based and case-insensitive. That method itself caused me trouble halfway through this article.
  • Addresses were checked against the vendors' published lists. "Not in the list" does not mean "fake," and I separate the two below.

There is a way to verify a visitor who states its name

All the large vendors publish their crawlers' addresses. Anthropic offers a single file at claude.com/crawling/bots.json, and its documentation says a crawler with a source IP on that list is coming from Anthropic; it doesn't say which bots the list covers. OpenAI publishes four separate files — searchbot, gptbot, chatgpt-user, adsbot. Perplexity publishes two, Google and Apple their own. I downloaded all five vendors' lists and checked every request in the thirty days:

Claimed name In published list Outside Share outside Outside, served 200
ClaudeBot 251 1 0.4% 0
PerplexityBot 19,879 90 0.5% 2
Applebot 8,282 70 0.8% 1
OAI-SearchBot 4,602 76 1.6% 3
Googlebot 29,045 458 1.6% 124
ChatGPT-User 499 122 19.6% 19
Claude-User 12 150 92.6% 61
Perplexity-User 0 75 100% 1
OAI-AdsBot 0 75 100% 0
Google-CloudVertexBot 0 4 100% 3

(Googlebot is not named in my file at all; it's in the table for comparison.)

The top rows are reassuring: automated crawlers verify almost perfectly. The bottom rows all point at one pool.

Eleven addresses, sixteen names

Pulling out the addresses that fall outside the lists, the same handful of IPs kept coming back around. Eleven addresses, 1,092 requests; most on Google Cloud, one on a small hosting provider. Between them those eleven carry sixteen official bot names, and the same address changes its name within a day.

The upshot: for four bots I either banned or allowed, all of the traffic comes from this pool.

Arriving under the name Requests From the pool
meta-externalagent 68 68
Meta-ExternalFetcher 94 94
Perplexity-User 75 75
OAI-AdsBot 75 75
Google-CloudVertexBot 4 4
DeepSeek 88 78

The Google-CloudVertexBot row is the clearest of them. Per Google's documentation, that bot only performs crawls requested by the site owner, for building Vertex AI agents. I requested nothing. Three of the four requests under that name took content, and all four came from the same pool.

The scale belongs here too, otherwise this reads as "most of my bot traffic is fake": 1,092 requests is 0.15% of the thirty-day total. This isn't a flood. The problem isn't volume, it's the ground the rule stands on. Meta's own documentation marks identifying its crawlers by IP rather than user agent as "more secure" — I understand the reasoning better now. Cloudflare's definition of a "verified bot" lands in the same place: a signed agent, a published IP list with a stable user agent, or reverse DNS.

An honesty note: Claude-User, ChatGPT-User and Perplexity-User are user-triggered fetchers — a person asks a model something and the fetch follows. OpenAI's documentation states outright that robots.txt rules may not apply to ChatGPT-User. Of Claude-User's 162 requests, 12 sit in Anthropic's list, 70 come from the pool above, and the remaining 80 arrive from scattered addresses — and all 61 requests that got a 200 are in that scattered set. Anthropic's documentation doesn't say which bots the list covers; for those 80 I can call them neither fake nor genuine with the data in hand, so I won't.

My own counting method fell for a name as well

The part that stings most is that the first thing caught by this trap was my measurement script.

Claude-User and Claude-SearchBot both carry the string claudebot inside their user agents, as a contact address. Substring matching therefore filed 70 Claude-User requests and 62 Claude-SearchBot requests under ClaudeBot; on the first pass ClaudeBot showed 384 requests, and separating them dropped it to 252. So while making the gap between "every request whose name contains X" and "X" the subject of the article, I had walked straight past the same gap in my own table.

Once separated, the ClaudeBot picture is interesting in its own right: 251 of its 252 requests went straight to /robots.txt. It read my file 251 times and came for content exactly once.

Diagram

Some of the names in the file have no addressee

The second problem with a rule built on names isn't verification, it's inventory. My file holds one * group and seventeen names. Six of those seventeen never showed up in thirty days: GPTBot, Amazonbot, anthropic-ai, Claude-Web, Google-Extended, Applebot-Extended.

For four of them "didn't show up" is the wrong phrase — they cannot. Google's documentation is explicit about Google-Extended: the token has no separate HTTP user agent, "crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." Apple's Applebot-Extended is the same species. anthropic-ai and Claude-Web appear nowhere in Anthropic's current documentation, which describes ClaudeBot, Claude-User and Claude-SearchBot.

Meanwhile the arriving crowd isn't named in the file at all. The seventeen tokens made 34,231 requests in thirty days; crawlers with no name in the file made 141,504. The largest eight (the remaining 319 come from four small crawlers):

Not named in my file Requests How it introduces itself
meta-webindexer 50,099 in Meta's docs: improves Meta AI search results
CorsairBot 37,663 only "web indexing" — no URL, no identity page
ExaSearchBot 16,668 crawler.exa.ai in its UA
ShapBot 12,071 a parallel.ai URL on 92 requests, just the name on the rest
LinkupBot 9,474 linkup.so/bot and an email in its UA
ElevenlabsCrawler 7,283 elevenlabs.io/docs/bots in its UA
Amzn-SearchBot 7,128 arrives under Amazon's name, addresses on AWS
DuckAssistBot 799 DuckDuckGo's bot page in its UA

My vocabulary has aged: I told Amazonbot no, and what walked in was Amzn-SearchBot, which got a 200 on 6,809 of its requests. Anthropic's search bot Claude-SearchBot isn't in the file — it came by 62 times in thirty days. Neither is OpenAI's OAI-AdsBot.

The list I wrote in June holds six names that never arrive and misses eleven crawlers I'd never heard of. Five months is enough time for a permission list to go stale.

The one bot that broke the rule wasn't hiding its name

My file has two Disallow lines: /tags/ and /en/tags/. In thirty days those two directories took 78,115 requests, 42,575 of which got a 200.

The breakdown: 19,927 requests from a recognizable bot token, 58,188 with browser agent strings. On the bot side there's one big item, CorsairBot: of its 37,663 requests, 17,529 landed in the forbidden directories, and it never fetched /robots.txt once in thirty days. Twelve addresses, all Google Cloud. Nothing surprising about a bot not obeying a rule it never read.

Part of the 58,188 browser-agent requests draws a pattern of its own: 3,780 requests with one identical Chrome string, from 3,678 distinct addresses. Starlink customer addresses, CGNAT blocks, home connections; one request per address. I can't make a claim about intent, only this: the pattern doesn't look like a person browsing, and robots.txt isn't part of the conversation on that side.

Then there are the bots I named, and they aren't breaking the rule — the rule failing to reach them is my own drafting error. Google's robots.txt documentation is clear: "only one group is valid for a particular crawler," "other groups are ignored," "user agent specific groups and global groups (*) are not combined." RFC 9309 doesn't go that far; it only requires obeying the * group when no group matches, and separately requires that several groups matching the same token be combined. By giving ten bots their own group containing nothing but Allow: /, I put the * group's deny out of their reach: OAI-SearchBot walked the tag directories 936 times, PerplexityBot 925, Applebot 258.

My first instinct was "repeat the deny in every group." In the article where I counted my tags I had already written why that's wrong: for noindex to work the page must not be blocked in robots.txt, because a crawler that can't enter never reads the noindex line. The correct fix isn't to spread the deny but to remove it — noindex is already what drops the tag pages from the index.

The line I deleted

I re-read the policy comment at the top of the file today and saw two things.

First, while saying "no to training," that comment puts Claude and Gemini on the citation side: "AI search/assistant bots may crawl and quote the content (ChatGPT search, Perplexity, Claude, Gemini, Apple)." So ClaudeBot: Allow and Google-Extended: Allow aren't a contradiction I overlooked, they're a deliberate classification. The trouble is that the vendors file those same names in a different box: Anthropic's documentation calls ClaudeBot "our training crawler" outright, and Google defines Google-Extended as the preference governing whether crawled content "may be used for training future generations of Gemini models." This isn't a contradiction, it's a misclassification — and counting anthropic-ai and Claude-Web, there are four training tokens on the allow side.

The second thing is more annoying. In the file's original 20 June state, directly under the * group, sat this line:

Content-Signal: search=yes, ai-input=yes, ai-train=no
Enter fullscreen mode Exit fullscreen mode

That was the only machine-readable statement of the policy. It was deleted on 11 August inside a commit titled "fix: harden blog publishing and validation." So I was going to open this article with "I didn't touch the file for five months"; the truth is that seven weeks later I touched it and removed the one sentence of my policy that machines could read. What's left is a comment block written for humans, and seventeen lines tied to names.

The file that actually holds the door is a different one — and this is the third time

There was a status code in the log I couldn't account for: all 47 of Bytespider's requests returned 444. The cause isn't robots.txt; an nginx file carries a thirty-token user-agent list and closes a matching connection without answering. Per the file's own header it was added on 31 May, three weeks before my policy. AhrefsBot's 3,272 requests and SemrushBot's 5,397 went the same way.

Putting the two lists side by side leaves a single overlap: seven tokens in robots.txt, thirty in nginx, one shared word — bytespider.

And there isn't even an accidentally correct outcome here. All 47 of Bytespider's requests went to /robots.txt, and since nginx closes the connection it never saw the file. RFC 9309 requires a crawler facing an unreachable robots.txt to assume a complete disallow — but the same section continues: if the file stays undefined for a reasonably long period (thirty days, for example), a crawler may treat it as unavailable, in which case it may access any resource on the server. The list has been in place since May. So what I'm shutting out isn't only the scraper; it's my own rule.

This is the third article in which I've looked at the gap between a written rule and a running system. Two weeks ago the same lens went over my commit rule: nine of 3,580 commits followed the documented rule, while a rule I never wrote down held at 96.9% because it lived inside the template. Ten days ago it was my two CSPs' turn: the only enforced policy was the one I hadn't written. The shared lesson there was "put the rule in the path," and I won't repeat it.

The difference here is this: in those two the gap between rule and system was on my side — I had written the rule in the wrong place. Here the rule can be in exactly the right place and still fail, because its subject is something I don't control: the name the visitor declares. There's a class of rule you cannot bake into a template or hang off a hook; verification is all they have.

What I'll do

Three items, all open, all out of this measurement:

  1. Tie verification to the rule. The published lists are already JSON: claude.com/crawling/bots.json, OpenAI's four files, Perplexity's two, Google's and Apple's. Pulling them daily and flagging requests that claim a name while sitting outside its list is about a cron job's worth of work. If the name isn't verified, there is no rule.
  2. Put the Content-Signal line back. Without a machine-readable statement, the policy is a comment only I read.
  3. Drop the two Disallow lines. The noindex that keeps tag pages out of the index only works if the crawler can read the page.

I didn't change the file for this article; the robots.txt that was measured is still the one in production. Showing what I wrote and then quietly fixing it would have made the measurement itself a lie.

The sentence left over is this: the list I hung on the door rests on the assumption that whoever arrives states their name truthfully. Eleven addresses broke that assumption over thirty days — with very little traffic, but completely. A rule written without verifying the name records not the rule, only the intention behind it.

Official Sources

Top comments (0)