Our affiliate redirect at /go/<slug> classifies every click with two checks: is the Referer header missing, and does the User-Agent match a crawler regex. That is the entire function.
export function isBotClick(args: { referer?: string; userAgent?: string }): boolean {
if (!args.referer) return true;
return CRAWLER_UA.test(args.userAgent ?? '');
}
On 2026-08-18 that filter reported 234 of 506 clicks in the trailing 30 days as human. Applying one further exclusion — dropping the two countries whose pageviews carried a datacentre signature — took the same 234 down to 117. Half of everything the header check called "qualified" was a crawler that had sent a referer. Three days later the gap was wider: 273 qualified, 128 after the country cut, 145 removed.
Nothing in that function is wrong as written. The problem is that what it measures — did this request arrive from a link on our own site — is not what we were reporting, which was whether a person was on the other end. Those two coincided for long enough to look like the same number.
Why a same-origin referer stopped being evidence
The heuristic had a real justification when it went in. /go/ links are same-origin with the article that contains them, and the default referrer policy in current browsers sends a referer on same-origin navigation. A reader clicking a link in an article therefore always carries one. In the first 30-day sample, 305 of 334 clicks had no referer at all — direct hits on a redirect URL nobody has a reason to type. Filtering those out moved the human share from 100% to roughly 9%, which was the right direction and a large correction.
What the check cannot do is distinguish that browser from an HTTP client that fetches the article HTML, parses out the hrefs, and requests each one with Referer: https://pickuma.com/for-dev/<slug>/ attached. The header is set by the client. A crawler that follows internal links produces a byte-identical request. There is no server-side way to separate the two from headers alone, because a browser's referer is not a signed assertion — it is a courtesy the client chooses to extend.
The UA regex has the same shape of problem. It matches self-identifying strings: bot, crawl, spider, slurp, curl, wget, python-requests, headless, scrapy, axios, okhttp. Every entry on that list is a client that told us what it was. A scraper with a copied Chrome UA string passes both halves of the filter. The whole thing depends on the other side volunteering the truth.
If your bot flag is a single boolean written at insert time, you cannot re-run it. We could reclassify 30 days retroactively only because the same row also stores
referer,country, and a saltedua_hash. Write the raw signals; compute the verdict at read time.
The split came from the country column, not the headers
The tell was not in the click table at all. It was in the pageview series: Singapore had accumulated 4,940 pageviews at 100% direct and 0% mobile. No population of real readers is 0% mobile, and none is 100% direct. That is a datacentre fingerprint, and it was unambiguous in a way no header was.
Once we had the two-country signature, applying it to clicks was one predicate. On 2026-08-18, China alone accounted for 52% of what the header check had called qualified. The 234-to-117 collapse was almost entirely those two countries arriving with a referer we had decided to trust.
The second-order damage was worse than the raw miscount. Pageviews already excluded Singapore and China; clicks did not. Any click-through rate computed across those two series divided a filtered numerator by an unfiltered denominator, and every such ratio we had looked at for weeks was off by roughly a factor of two — in the flattering direction. Two filters that disagree are worse than no filter, because the error hides inside a ratio instead of sitting in plain sight in a count.
We did not redefine qualified. The snapshot file now records total, qualified, and qualifiedClean side by side, so rows written before the fix still mean what they meant when they were written. Overwriting a metric's definition in place is how you lose the ability to say when something changed.
What breaks this heuristic next
The country cut is blunt, and it costs us real readers in Singapore and China. We accepted that because the alternative — no comparable series at all — was worse at our volume, which is roughly 500 clicks per 30 days. Below about 50 clicks a month, a country exclusion is noise dressed up as rigour. Do not bother.
Above that, the thing to reach for is a classification made at the edge before your handler runs. Cloudflare exposes bot scoring to Workers through request.cf.botManagement, which is a better input than anything you can reconstruct from request headers. We have not moved to it, and the condition that would flip us is plan availability: the useful part of that field set is a paid Bot Management feature, and we did not verify what our own plan returns. Check that before designing around it rather than after.
Boundaries on what we actually measured. We did not identify the crawlers. ua_hash is salted over UA and IP, so we can count distinct clients but cannot name one or reverse it. We store cf-ipcountry, not the source IP, so reverse-DNS verification was not possible on rows we already had. We did not test whether the same clients inflate pageviews outside the two excluded countries — they probably do, and our "clean" pageview number is therefore an upper bound, not a true one. This is one site's data over 30 days, not a general result about referer heuristics.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)