Googlebot ran my analytics script — my unique visitor count was 14x too high

For three weeks, my site's dashboard told a nice story. Traffic was growing. Unique visitors hit 87 on the best day, then 94, then a hundred-odd. New site, indexing starting to happen, numbers going up — exactly what you want to see.

Then I looked at the actual requests and found six humans.

Not six hundred. Six. And two of those six were probably me.

This post is about where the other 81 came from, why the usual bot filters miss them, and the four checks that finally made the number honest. If you run a small site with a hand-rolled analytics endpoint, you have the same problem — you just don't know it yet.

The mechanism: rendered pages run JavaScript

I built the analytics myself. A tiny script posts to /api/pv on page load; a Cloudflare Worker writes a row into D1 with path, referrer, UA, and IP. Fifty lines, no vendor, no cookie banner.

That design assumes a visit is a person. It isn't.

Googlebot fetches a page, renders it in headless Chromium, and executes the JavaScript — my beacon included. So every page Google crawls generates a "visit". On a 380-page site, one crawl pass is up to 380 hits. And because crawlers don't reuse a connection the way a browser session does, each batch can arrive from a different egress IP, so COUNT(DISTINCT ip) treats them as hundreds of separate people.

That's the whole illusion. Nothing was broken. My endpoint correctly recorded what happened. It just recorded the wrong thing as a visitor.

Worth knowing the scale before you go hunting: on my site, Googlebot had crawled 372 of 380 pages, while bingbot had crawled 5. Both are "search engine traffic" in a naive dashboard. One of them is 70x the other, and neither is a person.

Fix attempt #1: filter on user agent (insufficient)

The obvious move — classify at ingest time, store the result, and never reinterpret it later:

const UA_BOT = /bot|crawl|spider|slurp|headless|phantom|puppeteer|playwright|selenium|curl\/|wget|python-|go-http-client|java\/|okhttp|axios|node-fetch|scrapy|semrush|ahrefs|dataforseo|lighthouse/i;

function classifyUA(ua) {
  const s = String(ua || '').slice(0, 200);
  if (!s) return '?';
  if (UA_BOT.test(s)) return 'bot';
  if (/ipad|tablet|silk\//i.test(s)) return 'tb';
  if (/android|iphone|mobile/i.test(s)) return 'mb';
  return 'pc';
}

This caught the honest bots immediately — Googlebot, bingbot, AhrefsBot. It also caught a few things I hadn't thought of:

• GoogleOther, Google-InspectionTool, APIs-Google, FeedFetcher — Google's non-search fetchers. A regex containing only bot misses all of them.
• BuiltWith, DataForSEO, Lighthouse — audit tools that behave exactly like a crawler.
• HeadlessChrome — the giveaway inside otherwise-normal Chrome UAs.

The mediapartners and google-inspectiontool entries were the ones that actually moved my numbers. Add them even if you've never heard of them.

But UA filtering alone fails on two cases, and both were present on my site.

The case that fooled me three times: fake WeChat traffic

I kept seeing a cluster of "WeChat" visitors. Mobile, from Tencent Cloud IP ranges, hundreds of hits. On a Chinese-language engineering site, WeChat referrals are the single most plausible traffic source, so I believed it — three separate times, over three weeks, before I stopped believing it.

What finally exposed it:

Mozilla/5.0 (Linux; Android 5.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36
MicroMessenger/7.0.1

Three independent tells:

1. MicroMessenger/7.0.1 — that's a 2018 client. Real WeChat traffic in 2026 reports 8.x and carries a build identifier like (0x28002d3a). A version with no (0x segment is not the real client.
2. Nexus 5 Build/MRA58N — that is Puppeteer's default device descriptor, straight out of the library's device list. Real Android devices in the wild report their own build string.
3. Tencent Cloud IP ranges — 81.71., 183.60., 42.194., 14.152., 43.248. are datacenter, not mobile carriers.

A single tell is weak. Three together is conclusive. The lesson generalises: a UA string is a claim, not evidence. Check whether the claim is internally consistent before you count it.

The second blind spot: crawlers wearing a normal Chrome UA

The other failure mode is worse, because the UA looks perfect. I had a stream of Macintosh; Intel Mac OS X Safari UAs from 17.166.* — roughly two requests per page, and 159 distinct IPs in a single day. Nothing in the UA string says "bot".

The behaviour does:

• exactly two requests per page (HTML + one asset), never the full asset set
• no referrer, ever, and no session continuity
• 159 IPs, all on the same cloud range, all doing the same thing within hours
• zero interaction — no search query, no second page view, no scroll-triggered event

That's the profile of a distributed fetcher, not of people. Two requests per page means "I parsed your HTML", not "I read your page".

Here's the uncomfortable part: rendering completeness is not a reliable human signal any more. Googlebot loads CSS, executes JS, and would happily pass a "did they load the assets" check. I used to treat "loaded everything" as proof of humanity. It isn't. It only rules out the lazy crawlers.

What actually worked: four layers, in this order

1. Classify at write time, store the verdict. A dev column (bot / tb / mb / pc) written by the Worker. Doing this at query time means every new bot name invalidates three months of history.
2. Named crawlers get IP verification, not just UA matching. Google publishes its ranges; match 66.249., 64.233., 72.14., 74.125., 209.85. alongside the UA. This also removes the impostors: I had a bingbot UA from 209.112.91.75, which is a vulnerability scanner, not Bing. Real Bing crawls from 40.77.167., 52.167.144., 207.46.13., 157.55.39..
3. Consistency checks on the claim. WeChat version vs build number vs IP ownership. Datacenter IP claiming to be a phone is a bot, no matter what the UA says.
4. Request-shape checks for the rest. Assets loaded, referrer present, page-depth, inter-request timing. Any one alone is weak; three together are decisive.

Then every downstream number gets a single filter:

SELECT COUNT(DISTINCT ip) AS humans, COUNT(*) AS hits
FROM hits
WHERE COALESCE(dev, '') <> 'bot'
  AND date(ts, '+8 hours') = date('now', '+8 hours');

That COALESCE(dev, '') <> 'bot' has to be in every query — dashboard, admin panel, the JSON endpoint the frontend calls. I missed it in two places and got a "traffic surged" alert about my own crawler.

One more trap in the same query: my timestamps are stored in UTC. Comparing "today so far" against "yesterday's full day" made the dashboard report -93% on a completely normal morning. Six visits by 11:35 versus thirty-eight across the whole previous day isn't a decline, it's arithmetic. If your period comparison can produce a number that large, the bug is in the comparison, not the site.

What the honest number looks like

After filtering, on a 380-page site with 168 bilingual calculators:

Metric	Before	After
Best-day unique visitors	87	6
Traffic source	"growing"	1 real organic click
Crawl coverage (Googlebot)	unknown	372 / 380 pages
Crawl coverage (bingbot)	unknown	5 / 380 pages

The last row is the one that changed my priorities. With bots stripped out, I could finally see that Google was indexing aggressively and Bing essentially wasn't crawling at all — a fact completely invisible in the inflated dashboard, where both were just "traffic".

You can't diagnose a funnel you can't measure. My dashboard was measuring machine labour and calling it demand.

Takeaway

If your analytics are hand-rolled, assume they're counting crawlers until you've proven otherwise. The cheap version of the proof:

1. Log the UA and IP at write time — you can't retroactively classify what you didn't record.
2. Treat every UA as a claim, then check it against IP ownership and request shape.
3. Add <> 'bot' to every query, not just the pretty one.
4. Compare like periods, in the same timezone, or your percentages will lie to you.

The site in question is jianzhujisuan.com — 168 construction-material calculators, 380 static pages, no framework. If you want to see what the crawling looks like from the other side, pick a page and view source; the result table is already in the HTML, which is why crawlers execute it and why my visitor count was fiction for three weeks.

Comments

[Om Yaduvanshi]

the nexus 5 detail stayed with me. a normal looking ua can still carry a device descriptor that gives the whole visit away, and checking that against the ip range is a much kinder test than trusting the label on its own. classifying at write time feels like the part i would borrow first, since it stops the same visit being reinterpreted every time a new bot name turns up.