Is that really GPTBot? What a week of server logs says about AI crawler claims

crawlerpeek — Sun, 25 Oct 2026 21:15:26 +0000

I maintain a small CLI called crawlerpeek that checks whether AI crawlers can
reach a site. While building it I wanted to know how much of the "bot traffic"
in my own logs is what it says it is. This post has the numbers, what they do
and do not show, and then how to run the same kind of check on your site.

What I measured

Period: 19 to 25 October 2026. Four new, small sites that I operate. For every
request whose user agent contains a bot token, I checked whether the client IP
lies inside the IP list that the operator publishes (Google, OpenAI, Anthropic,
Perplexity, Bing, Apple, Common Crawl, Ahrefs), fetched the same day. I removed
my own server's requests and my own manual tests.

Claimed bot	In published ranges	Share
OAI-SearchBot	2,315 of 2,356	98%
Googlebot	1,923 of 2,003	96%
ClaudeBot	719 of 748	96%
GPTBot	493 of 524	94%
ChatGPT-User	162 of 200	81%
Bingbot	133 of 168	79%
PerplexityBot	12 of 54	22%
CCBot	0 of 36	0%

Applebot and AhrefsBot were 100%.

There were 332 requests that did not match. 287 of them came from a single
hosting network, and that network claimed eight different bots: Bingbot, CCBot,
ChatGPT-User, ClaudeBot, Googlebot, GPTBot, OAI-SearchBot and PerplexityBot.
Its reverse DNS did not confirm forward. A further 45 had no reverse DNS at all.

What this does not show

It is one week on four new, small sites. Bigger or older sites will see a different mix.
Published IP lists can lag behind new crawler addresses. "Unverified" means "not in the list", not "proven fake". The PerplexityBot and ChatGPT-User numbers in particular could be partly list lag.
One host claiming eight unrelated crawlers from non-operator addresses is hard to explain as anything but user-agent spoofing. That is the only claim here I would call solid.

The practical point: a user-agent string is a claim, not an identity. If you
decide anything based on it (allow, block, rate limit, count it in analytics),
the operator's IP list or reverse DNS is the check that matters. Google and
others document reverse-DNS verification, and OpenAI, Anthropic, Perplexity,
Bing, Apple and Common Crawl publish JSON lists of ranges.

The other half: what do real bots get from your site?

Separately from spoofing, there is a more common problem: robots.txt says one
thing and the server does another. A WAF rule returns 403 to a user agent that
robots.txt allows. A redirect sends a bot to a login page. A User-agent: *
group blocks a path, and a bot-specific group further down quietly overrides it
(a bot with its own group ignores the * group).

crawlerpeek checks both layers for each bot: the robots.txt decision under RFC
9309 matching rules, and what the page returns when fetched with that bot's
user agent.

npx crawlerpeek check https://example.com

Example output for a site with a WAF rule and a restrictive robots.txt:

https://example.com
robots.txt: ok (HTTP 200) at https://example.com/robots.txt

Bot                 robots    HTTP
GPTBot              blocked   200  FAIL
ClaudeBot           allowed   403 (block page)  FAIL
Google-Extended     allowed   -

Here GPTBot is closed by robots.txt even though the server would answer, and
ClaudeBot is allowed by robots.txt but gets a block page. Both are things you
would want to know before wondering why a product is never cited.

More forms:

npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked
npx crawlerpeek check https://example.com --bots all --json
npx crawlerpeek check https://example.com --no-http          # robots.txt only
npx crawlerpeek see https://example.com --bot ClaudeBot      # one bot, full detail

One limit matters for the topic of this post: crawlerpeek sends the bot's
user-agent string from your machine. It is not the real bot and does not come
from the operator's IPs. If your site verifies bot IPs, a request that only
claims to be Googlebot may be blocked, so a 403 for Googlebot can be correct.
A 403 for an unusual bot while a browser gets 200 usually points at a WAF or CDN
rule.

Put it in CI

Exit code 0 means ok, 1 means the --fail-on condition was met, 2 means a usage
or request error. Unknown options are an error, so a typo like --failon cannot
pass a build silently.

With the composite action:

- uses: crawlerpeek/crawlerpeek/action@v0
  with: { url: "https://example.com", fail-on: blocked }

Or call the CLI yourself after a deploy:

- run: npx crawlerpeek check https://example.com --fail-on blocked

--fail-on blocked fails when robots.txt closes the path for a bot, or the page
answers that bot with 401, 403, 406, 451 or a challenge page. any-non-200 is
stricter and also fails on 404, 5xx and redirects that end elsewhere. Run it
against the pages you care about, not only the home page.

A robots.txt that cannot be fetched is reported as unknown and does not count
as blocked. Under RFC 9309 a 5xx robots.txt means "disallow all" for many
crawlers, so treat unknown as something to fix.

Use it from Claude Code, Cursor or another MCP client

crawlerpeek mcp starts an MCP server over stdio with three read-only tools:
check_ai_access, see_as_bot and generate_robots_txt. It calls no LLM; the
tools only fetch the URL you give them, and results are cached for 10 minutes.

{ "mcpServers": { "crawlerpeek": { "command": "npx", "args": ["-y", "crawlerpeek", "mcp"] } } }

Then you can ask your assistant "can ClaudeBot read /pricing on my staging
site?" and it runs the check instead of guessing. It is also listed in the
official MCP Registry as io.github.crawlerpeek/crawlerpeek.

To generate a robots.txt that names the AI crawlers explicitly:

npx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xml

Disallowed paths are repeated inside each bot's group on purpose, for the reason
above.

Safety

One request per bot plus one for robots.txt, sequential, with a 300 ms pause; a
10 second timeout; at most 5 redirects, each re-checked; bodies capped at 2 MB;
no cookies. Private, loopback and metadata addresses are refused after DNS
resolution, as are ports other than 80 and 443.

Feedback

Source is MIT on GitHub: https://github.com/crawlerpeek/crawlerpeek. Package:
https://www.npmjs.com/package/crawlerpeek (0.1.2). I would like to hear about
missing bots, wrong user-agent strings and false block-page detections. Issues
are open.

DEV Community: crawlerpeek