DEV Community: Jianzhujisuan

Googlebot ran my analytics script — my unique visitor count was 14x too high

Jianzhujisuan — Fri, 30 Oct 2026 04:37:18 +0000

For three weeks, my site's dashboard told a nice story. Traffic was growing. Unique visitors hit 87 on the best day, then 94, then a hundred-odd. New site, indexing starting to happen, numbers going up — exactly what you want to see.

Then I looked at the actual requests and found six humans.

Not six hundred. Six. And two of those six were probably me.

This post is about where the other 81 came from, why the usual bot filters miss them, and the four checks that finally made the number honest. If you run a small site with a hand-rolled analytics endpoint, you have the same problem — you just don't know it yet.

The mechanism: rendered pages run JavaScript

I built the analytics myself. A tiny script posts to /api/pv on page load; a Cloudflare Worker writes a row into D1 with path, referrer, UA, and IP. Fifty lines, no vendor, no cookie banner.

That design assumes a visit is a person. It isn't.

Googlebot fetches a page, renders it in headless Chromium, and executes the JavaScript — my beacon included. So every page Google crawls generates a "visit". On a 380-page site, one crawl pass is up to 380 hits. And because crawlers don't reuse a connection the way a browser session does, each batch can arrive from a different egress IP, so COUNT(DISTINCT ip) treats them as hundreds of separate people.

That's the whole illusion. Nothing was broken. My endpoint correctly recorded what happened. It just recorded the wrong thing as a visitor.

Worth knowing the scale before you go hunting: on my site, Googlebot had crawled 372 of 380 pages, while bingbot had crawled 5. Both are "search engine traffic" in a naive dashboard. One of them is 70x the other, and neither is a person.

Fix attempt #1: filter on user agent (insufficient)

The obvious move — classify at ingest time, store the result, and never reinterpret it later:

const UA_BOT = /bot|crawl|spider|slurp|headless|phantom|puppeteer|playwright|selenium|curl\/|wget|python-|go-http-client|java\/|okhttp|axios|node-fetch|scrapy|semrush|ahrefs|dataforseo|lighthouse/i;

function classifyUA(ua) {
  const s = String(ua || '').slice(0, 200);
  if (!s) return '?';
  if (UA_BOT.test(s)) return 'bot';
  if (/ipad|tablet|silk\//i.test(s)) return 'tb';
  if (/android|iphone|mobile/i.test(s)) return 'mb';
  return 'pc';
}

This caught the honest bots immediately — Googlebot, bingbot, AhrefsBot. It also caught a few things I hadn't thought of:

GoogleOther, Google-InspectionTool, APIs-Google, FeedFetcher — Google's non-search fetchers. A regex containing only bot misses all of them.
BuiltWith, DataForSEO, Lighthouse — audit tools that behave exactly like a crawler.
HeadlessChrome — the giveaway inside otherwise-normal Chrome UAs.

The mediapartners and google-inspectiontool entries were the ones that actually moved my numbers. Add them even if you've never heard of them.

But UA filtering alone fails on two cases, and both were present on my site.

The case that fooled me three times: fake WeChat traffic

I kept seeing a cluster of "WeChat" visitors. Mobile, from Tencent Cloud IP ranges, hundreds of hits. On a Chinese-language engineering site, WeChat referrals are the single most plausible traffic source, so I believed it — three separate times, over three weeks, before I stopped believing it.

What finally exposed it:

Mozilla/5.0 (Linux; Android 5.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36
MicroMessenger/7.0.1

Three independent tells:

MicroMessenger/7.0.1 — that's a 2018 client. Real WeChat traffic in 2026 reports 8.x and carries a build identifier like (0x28002d3a). A version with no (0x segment is not the real client.
Nexus 5 Build/MRA58N — that is Puppeteer's default device descriptor, straight out of the library's device list. Real Android devices in the wild report their own build string.
Tencent Cloud IP ranges — 81.71., 183.60., 42.194., 14.152., 43.248. are datacenter, not mobile carriers.

A single tell is weak. Three together is conclusive. The lesson generalises: a UA string is a claim, not evidence. Check whether the claim is internally consistent before you count it.

The second blind spot: crawlers wearing a normal Chrome UA

The other failure mode is worse, because the UA looks perfect. I had a stream of Macintosh; Intel Mac OS X Safari UAs from 17.166.* — roughly two requests per page, and 159 distinct IPs in a single day. Nothing in the UA string says "bot".

The behaviour does:

exactly two requests per page (HTML + one asset), never the full asset set
no referrer, ever, and no session continuity
159 IPs, all on the same cloud range, all doing the same thing within hours
zero interaction — no search query, no second page view, no scroll-triggered event

That's the profile of a distributed fetcher, not of people. Two requests per page means "I parsed your HTML", not "I read your page".

Here's the uncomfortable part: rendering completeness is not a reliable human signal any more. Googlebot loads CSS, executes JS, and would happily pass a "did they load the assets" check. I used to treat "loaded everything" as proof of humanity. It isn't. It only rules out the lazy crawlers.

What actually worked: four layers, in this order

Classify at write time, store the verdict. A dev column (bot / tb / mb / pc) written by the Worker. Doing this at query time means every new bot name invalidates three months of history.
Named crawlers get IP verification, not just UA matching. Google publishes its ranges; match 66.249., 64.233., 72.14., 74.125., 209.85. alongside the UA. This also removes the impostors: I had a bingbot UA from 209.112.91.75, which is a vulnerability scanner, not Bing. Real Bing crawls from 40.77.167., 52.167.144., 207.46.13., 157.55.39..
Consistency checks on the claim. WeChat version vs build number vs IP ownership. Datacenter IP claiming to be a phone is a bot, no matter what the UA says.
Request-shape checks for the rest. Assets loaded, referrer present, page-depth, inter-request timing. Any one alone is weak; three together are decisive.

Then every downstream number gets a single filter:

SELECT COUNT(DISTINCT ip) AS humans, COUNT(*) AS hits
FROM hits
WHERE COALESCE(dev, '') <> 'bot'
  AND date(ts, '+8 hours') = date('now', '+8 hours');

That COALESCE(dev, '') <> 'bot' has to be in every query — dashboard, admin panel, the JSON endpoint the frontend calls. I missed it in two places and got a "traffic surged" alert about my own crawler.

One more trap in the same query: my timestamps are stored in UTC. Comparing "today so far" against "yesterday's full day" made the dashboard report -93% on a completely normal morning. Six visits by 11:35 versus thirty-eight across the whole previous day isn't a decline, it's arithmetic. If your period comparison can produce a number that large, the bug is in the comparison, not the site.

What the honest number looks like

After filtering, on a 380-page site with 168 bilingual calculators:

Metric	Before	After
Best-day unique visitors	87	6
Traffic source	"growing"	1 real organic click
Crawl coverage (Googlebot)	unknown	372 / 380 pages
Crawl coverage (bingbot)	unknown	5 / 380 pages

The last row is the one that changed my priorities. With bots stripped out, I could finally see that Google was indexing aggressively and Bing essentially wasn't crawling at all — a fact completely invisible in the inflated dashboard, where both were just "traffic".

You can't diagnose a funnel you can't measure. My dashboard was measuring machine labour and calling it demand.

Takeaway

If your analytics are hand-rolled, assume they're counting crawlers until you've proven otherwise. The cheap version of the proof:

Log the UA and IP at write time — you can't retroactively classify what you didn't record.
Treat every UA as a claim, then check it against IP ownership and request shape.
Add <> 'bot' to every query, not just the pretty one.
Compare like periods, in the same timezone, or your percentages will lie to you.

The site in question is jianzhujisuan.com — 168 construction-material calculators, 380 static pages, no framework. If you want to see what the crawling looks like from the other side, pick a page and view source; the result table is already in the HTML, which is why crawlers execute it and why my visitor count was fiction for three weeks.

Your sitemap's lastmod is lying — one CSS tweak marked all 380 pages as updated

Jianzhujisuan — Thu, 29 Oct 2026 17:00:23 +0000

Last winter I added one line to a stylesheet on my site — letter-spacing: -0.01em on a heading — and redeployed. Nothing else changed. And yet, according to my sitemap, all 380 pages had been updated that day.

bingbot, being reasonable, went off to recrawl 380 pages. Almost none of them had changed.

This is the quiet way a static site wastes its crawl budget: not through broken links or slow responses, but through a lastmod that lies.

Why `lastmod` matters more than people think

Google has said for years that it mostly ignores <lastmod> unless it can trust it. Bing is blunter about it: crawl scheduling uses change history per URL. If your sitemap claims everything changed today, every day, then either:

the crawler recrawls a lot of unchanged pages, or
it learns your lastmod is noise and stops using it — so genuinely new pages wait behind nothing.

Either outcome is bad, and both are invisible in your analytics because nothing errors. You just notice, months later, that a batch of pages never got indexed.

The naive fix, and why it backfires

The obvious improvement over file mtimes is to hash the rendered output:

def lastmod_for(path):
    digest = hashlib.sha1(open(path, "rb").read()).hexdigest()
    return state.setdefault(path, today()) if digest != state.get(path) else state[path]

That's roughly what I shipped first. It's correct in principle and useless in practice, because the bytes of a rendered page are not stable even when the content is.

Two things move on every single deploy:

1. Cache-busting query strings. My asset references look like /assets/style.css?v=3f9a1c. The build rewrites that hash whenever the stylesheet changes. So a one-line CSS edit changes the bytes of every page that links the stylesheet — which is all of them.

2. The analytics beacon. A small script tag with a site token gets injected into <head>. Harmless, identical on every page — but if the injection order or whitespace ever shifts (a template refactor, a new minifier version, a different Python release writing attributes in a different order), the hash changes for the whole site at once.

Neither is a content change. Both are pure output noise. A hash of the raw file can't tell the difference.

Hash the meaning, not the bytes

The fix is to normalise before hashing — strip the parts that are allowed to vary without the page meaning anything different:

import re, hashlib

NOISE = [
    re.compile(r'\?v=[0-9a-zA-Z]+'),                 # cache-busting hashes
    re.compile(r'<script[^>]*data-analytics[^>]*>.*?</script>', re.S),
    re.compile(r'\s+'),                              # whitespace
]

def fingerprint(html):
    for pat in NOISE:
        html = pat.sub("", html)
    return hashlib.sha1(html.encode("utf-8")).hexdigest()

Then keep a state file mapping path → fingerprint, and only touch lastmod when the fingerprint moves:

state = json.load(open("_lastmod.json"))
changed = []
for page, html in built_pages:
    fp = fingerprint(html)
    if state.get(page, {}).get("fp") != fp:
        state[page] = {"fp": fp, "lastmod": today()}
        changed.append(page)
json.dump(state, open("_lastmod.json", "w"))
print(f"{len(changed)} pages changed")

The lastmod values in the sitemap now come from that state file, not from the clock.

The test that keeps it honest

A fingerprint system is easy to write and easy to break silently. The assertion I run in CI is deliberately dumb:

Build twice in a row. The second build must report zero changed pages.

If it reports one page, something in the pipeline is non-deterministic. If it reports 380, the normaliser stopped working — usually because someone added a new injected block and didn't add it to NOISE.

That single assertion has caught three real regressions for me:

a minifier that started emitting a random class-name suffix
an analytics snippet that included a build timestamp
a Python upgrade that changed attribute order in the HTML writer

All three would have shipped an "everything changed" sitemap, and none of them would have raised anything in a normal test suite, because the pages were correct. They were just dishonest about it.

Where the state file lives

One trap worth calling out: if _lastmod.json lives only in CI and the cache is wiped, every page looks new again — the failure mode you were trying to fix.

Two options that work:

commit it to the repo (small, boring, effective — 380 lines of JSON for my site)
store it as a build artefact with a long retention and restore it before building

What does not work: recomputing it from git history. Git tells you when a source file changed, not when a rendered page's output changed — and one shared layout file touches all 380 outputs.

What actually changed

On my site, jianzhujisuan.com — 168 bilingual engineering calculators, 380 static pages — the effect was immediate and measurable in the logs rather than in rankings:

deploys that change one article now mark 1 page as updated instead of 380
the second build in any CI run reports 0 changed pages
crawlers spend their budget on the pages that actually moved

No ranking jump. No dramatic story. Just a sitemap that stopped lying, which is what lastmod was supposed to be all along.

If you want to see the output side of this, open any tool page and view source — say the brick count calculator — the result table is in the HTML, with a fingerprint attribute that's the same on every build until the formula changes.

Your calculator returns nothing to crawlers. Here's how I fixed it with Playwright.

Jianzhujisuan — Thu, 29 Oct 2026 16:19:41 +0000

The problem

If your site is "user types numbers → JavaScript computes a result", your HTML source probably looks like this:

<div id="result"></div>
<script src="calc.js"></script>

Users see nothing wrong. Crawlers download 30KB of HTML containing no answer at all.

Googlebot does execute JavaScript, but only when it decides the page is worth rendering — and the render queue can lag days to weeks. bingbot and most other crawlers do not execute JavaScript at all. They get the blank div, index it, and your page competes for "concrete volume calculator" with zero matching text.

I hit this building jianzhujisuan.com — 168 construction calculators, 380 static pages, every result computed in the browser.

The fix: don't solve it at runtime, solve it at build time

One extra build step: open each page in a real browser with default inputs, wait for the JS to finish, capture #result's innerHTML, write it back into the same container in the HTML file.

const { chromium } = require('playwright');

async function prerender(file) {
  const page = await browser.newPage();
  await page.goto('file://' + file);        // local file, no server needed
  await page.waitForSelector('#result');

  // Note the (expr)() — see trap 1
  const inner = await page.evaluate(`(${config.capture})()`);

  injectIntoContainer(file, inner);         // into the container, see trap 2
}

Output:

<div id="result">
  <!-- prerender:start fp=a3f8c91b2e4d7a60 -->
  <table><tr><td>Concrete volume</td><td>3.24 m³</td></tr></table>
  <!-- prerender:end -->
</div>

For visitors: calc.js loads and overwrites #result's innerHTML — the static copy disappears, nothing changes for them.
For crawlers: no JS execution, but the table is right there in the source.

Single acceptance test — open the page with javaScriptEnabled: false:

const ctx = await browser.newContext({ javaScriptEnabled: false });
const page = await ctx.newPage();
await page.goto(url);
console.log(await page.textContent('#result'));  // must contain real numbers

Trap 1: `page.evaluate("() => x")` evaluates, doesn't call

I used a "formula fingerprint" to decide whether a page needs re-rendering — hash the compute function's source, compare against a stored fingerprint, skip if unchanged.

// WRONG
const fp = await page.evaluate('() => window.CALC.fp');

The fingerprint came back empty every time. When page.evaluate receives a string, it evaluates the expression and returns the resulting function object — it never calls it.

function callExpr(expr) {
  return `(${expr})()`;     // becomes (() => x)()
}
const fp = await page.evaluate(callExpr('() => window.CALC.fp'));

The same trap exists in waitForFunction, and it's worse there: a string that evaluates truthy means you waited for nothing and captured a half-rendered result.

Trap 2: injecting before `</body>` shows users two copies

My first version inserted the block before </body>. Locally, with JS enabled, users saw two result tables — the static one sat outside #result, so the live innerHTML overwrite never removed it.

You have to splice into the container's interior, matching tag depth:

function spliceInner(html, id, inner) {
  const open = new RegExp(`<div[^>]*id=["']${id}["'][^>]*>`);
  const m = open.exec(html);
  if (!m) return null;

  let i = m.index + m[0].length;
  let depth = 1;
  const tagRe = /<div\b|<\/div>/g;
  tagRe.lastIndex = i;

  let t;
  while ((t = tagRe.exec(html))) {
    depth += t[0] === '</div>' ? -1 : 1;
    if (depth === 0) return html.slice(0, i) + inner + html.slice(t.index);
  }
  return null;
}

Now the static block lives inside #result and gets replaced along with everything else.

Trap 3: re-running stacks blocks

A build step must be idempotent. Inserting before </body> gives you three copies after three runs. Wrap the block in markers and replace on re-run:

const BLOCK_RE = /<!--\s*prerender:start[^>]*-->[\s\S]*?<!--\s*prerender:end\s*-->/;
const after = BLOCK_RE.test(before)
  ? before.replace(BLOCK_RE, () => block)
  : inject(before, block);

Second run should report 0 rendered, 1 unchanged. If every run reports "all rendered", your fingerprint is empty — go back to trap 1.

One more thing: lastmod ordering

If your sitemap carries lastmod, the prerender injection must run before lastmod is computed.

Otherwise adding prerender marks all 380 pages as "modified today". Run it again the next day — all 380 pages modified again. From the crawler's perspective your site claims a full-site rewrite every single day, which destroys your sitemap's credibility.

If you must run it daily, strip the injected block from the content fingerprint — the block changed, the page's substance didn't.

Results

380 pages, on a single reused browser instance:

Build time: noticeably longer, but it's a one-off cost with a reused single browser instance — never in the user request path
Every page returns its full result table with JS disabled
Median real body content: 1,655 CJK characters (was ~400 characters of template skeleton)

What prerendering does not fix

It's worth being blunt here. Prerendering fixes "can the crawler see the content". It does not fix "does the search engine want to index you".

I assumed fixing the blank-div problem would fix indexing. It didn't. Two weeks later, bingbot had fetched 2 pages out of 380 and indexed zero. The reason had nothing to do with my HTML — it was a new domain with zero inbound links, and no crawler algorithm allocates crawl budget for a site nobody mentions.

Content visibility and site trust are two separate problems. Prerender solves the first one properly. The second one is a marketing problem, and you solve it by writing things people link to.

Code and verification

The code: calc-prerender — MIT licensed, ~180 lines of Node, a runnable example, and the three traps written up in the README. It is not a demo: this is the script that runs in the production build.

Production case: jianzhujisuan.com — 168 bilingual construction calculators, 380 pages, all built this way. The whole implementation (~180 lines) is what runs in that build pipeline.

Quick verification: open a page like brick count calculator, disable JavaScript, refresh. If the result table is still there, it worked. If you see an empty div, prerendering never ran.