Log analysis from a 22-hour window on anclum.com, November 7, 2026.
If you measure how often Google visits your site, stop checking the User-Agent string. It is lying to you.
We run an MCP directory. Like everyone else, we watch how much of it Google actually crawls — that number decides whether anything we publish is ever seen. So we pulled 22 hours of nginx access logs and counted the requests that claim to be Googlebot.
6,121 requests said they were Googlebot.
Then we checked the source IP of every one of them against Google's published crawler ranges.
90 actually were. The other 6,031 — 98.5% of the 6,121 — came from outside Google's published IP ranges. Not Google, not Bing, not any crawler that matters to us. They came from scanners, and they were wearing Google's name.
The 90 and the 258 are two different sets, and mixing them up is how this analysis goes wrong. 258 is the total number of requests Google's own IP ranges sent us in the same window — 168 of them with a User-Agent that does not say "Googlebot" at all. 90 is the overlap: requests that both claimed to be Googlebot and came from Google.
Why this matters more than it sounds
Here is what the fake Googlebot was asking for:
/.env /.env.local /.env.production /.env.bak
/.git/config /.git/head /wp-admin/install.php?step=1
/config.json /api/.env /.well-known/oauth-protected-resource
Every one of those is an attempt to find a leaked secret or an unauthenticated admin panel. It is background radiation on the public internet — every site gets it, every day.
The damage is not the scanning. The damage is what it does to your numbers.
If you count bot traffic by User-Agent, your log says "Google visited 6,121 times today." You feel good. You conclude your crawl budget is healthy. You might even conclude Google likes you.
The truth was 258 requests from Google's ranges — and the quality of what Google actually got looked completely different from what the fake traffic implied. Every share below is out of those 258:
| Status | Requests (of 258) | Share |
|---|---|---|
| 200 OK | 200 | 77.5% |
| 410 Gone | 38 | 14.7% |
| 301 redirect | 13 | 5.0% |
| 404 | 6 | 2.3% |
| 302 redirect | 1 | 0.4% |
We had spent part of the day believing that 45% of those 6,121 requests were hitting dead links. That number came from a User-Agent match. It was wrong by more than an order of magnitude. Of the 258 Google-range requests, the 404 rate was 2.3%.
We had been about to redesign a chunk of the site to fix a problem that did not exist — and to ignore the one that did: Google's ranges sent us only 258 requests in 22 hours.
The mistake we made twice
We first wrote the analysis with a grep 'Googlebot' and a uniq -c. It produced a beautiful, tidy, completely wrong table.
The tell was in the data itself, and we nearly missed it: the "dead links" that accounted for 45% of the 6,121 UA-matched requests were things like /.env.save and /wp-admin/install.php. Googlebot does not hunt for your .env file. Apache and Nginx logs are full of this noise from a long list of botnets, and a large fraction of them now spoof the Googlebot UA — including ancient Android strings, because that is what real Googlebot used to look like.
If your 404 report has /wp-admin in it, you are not looking at a search engine. You are looking at an attacker who read a blog post about SEO.
How to tell who is actually crawling you
Three ways, in increasing order of correctness:
-
Google's published IP ranges. Google serves them as DNS netblocks at
_cloud-netblocks.googleusercontent.com(spf-style TXT records) and asgooglebot.jsonfrom Google's developer docs. Match your log IPs against those. -
Reverse DNS.
dig -x <ip>should return something ending ingooglebot.com(orgoogle.comfor the other Google crawlers). Then forward-resolve that name and confirm it comes back to the same IP — otherwise you are being fooled by a spoofed PTR record. - Search Console's crawl stats. Google reports its own numbers. Compare them to your log-derived count. If your log says Google visited ten times more often than Search Console does, the difference is impostors.
For our first pass we used a hardcoded list of Google's crawler prefixes (66.249., 64.233., 74.125., 209.85., 72.14., 216.239.). That is an approximation, and it can both over- and under-count. The published ranges plus reverse DNS are the correct method — we now use them for the numbers we put on paper.
What Google's crawlers were actually doing
258 requests from Google's ranges over 22 hours, touching 233 distinct URLs. And the distribution was more interesting than the total. Every percentage below is a share of those 258:
-
25.2% (65 requests) went to pagination pages —
/en/t/mcp/p/62/,/zh/t/from-github/az/p/15/and friends. These are pure navigation: the same list of tools, sliced into pages. They are not in our sitemap and never have been, but they are linked from every list page, so Google keeps coming through the door. -
15.5% (40 requests) went to pages we deleted. 40 requests targeted
/ny/contract/...pages that no longer exist, and 38 of the 258 responses overall were410 Gone— Google confirming that content which used to exist does not. That is the correct signal and it decays on its own. Blocking those URLs inrobots.txtat this point would be actively harmful: a blocked page cannot be seen to be gone. - The remaining 59% went to actual content.
65 requests in 22 hours to pagination. That is the single largest waste in an otherwise healthy crawl, and it is fixable: noindex, follow on pagination templates. The follow matters, so the links still pass through.
But here is the part we had to admit to ourselves: moving those 65 requests does not create more crawling. The ceiling is what Google is willing to spend on us, and that ceiling is set by how much other sites point at us — not by how tidy our internal links are. Optimizing the split of a small budget is still a small budget.
The short version
- User-Agent is self-reported. IP is not. If you are counting bots, count IPs.
-
98.5% of the 6,121 requests that claimed to be Googlebot came from scanners looking for
.envfiles. If your dead-link report mentionswp-admin, it is not Google. - Our real crawl was 258 requests in 22 hours, and 200 of those 258 responses were a valid page (77.5%). The problem was never quality — it was volume.
- We nearly rebuilt a working part of the site based on a UA-based number. Same mistake we had made that morning with other measurements: picking the quantity that was easy to read instead of the one that answered the question.
The method above is reproducible from any nginx access log. If your numbers look alarming, check who you are actually counting before you fix anything.
Window: 2026-11-07 00:24:03–22:11:59 (Asia/Shanghai), nginx access log, 9.8 MiB. Real-crawler classification by Google's published crawler IP prefixes, with the caveat noted above. Source: anclum.com access log access.log-20261108.gz.
Top comments (0)