On 2026-08-18 our Cloudflare RUM panel listed Singapore as one of the site's largest traffic sources: 4,940 pageviews, 100% direct, 0% mobile. No human population produces that shape. Real readers arrive mixed — some from a link, some on a phone, some with a referer their browser happened to keep. A bucket that is exactly 100% direct and exactly 0% mobile is a headless browser fleet or a scraper pool sitting in a datacentre region, and it had been padding our numbers for weeks before anyone looked at the columns side by side.
We have no server logs to check it against. Astro static output on Cloudflare Workers with Static Assets means the edge serves the file and the application never sees a request line, an IP, or a user agent. What we do have is browser-side RUM, which only fires when JavaScript runs, and a clicks table in Supabase written by the /go/[slug] affiliate redirect. That is a thin instrument. It still works, because the three fields it does capture — country, referer, device class — disagree in a specific and repeatable way when the visitor is not a person.
The signal is the disagreement, not any single field
Each field on its own is defensible. A 100% direct bucket could be dark social: newsletter clients, Slack, an in-app browser that strips the referer. A 0% mobile bucket could just be a developer audience on desktops. One country dominating could be a genuine regional hit — a local aggregator picked you up.
What has no innocent reading is all three at once, at their extremes, in the same bucket. Dark social still produces mobile traffic. A desktop-heavy developer audience still produces some referers. A regional hit still shows a device split. The combination is the tell, and it is visible in any analytics product that will break pageviews down by country, and cross it with referer type and device class.
The thresholds we settled on, and these are judgement calls rather than anything derived from a labelled dataset:
- The country accounts for more than roughly 5% of total pageviews
- Direct share within that country is above 95%
- Mobile share within that country is below 5%
Singapore cleared all three by a wide margin. It is also, not coincidentally, one of the densest cloud regions in Asia-Pacific — AWS, GCP, Azure, DigitalOcean and Vultr all have capacity there. Country in RUM is derived from IP geolocation, so a scraper running on a rented box in ap-southeast-1 reports as Singapore in exactly the way a reader in Singapore does. The country field is not measuring readership, it is measuring where the compute is.
Once you exclude a country from pageviews, you have created two populations that are no longer comparable. If your pageview number excludes Singapore and China but your click number does not, any click-through rate you compute across them is wrong in your favour. Keep both counts named and separate — we track
qualified(all non-bot clicks) andqualifiedClean(the same, minus the excluded regions) and never quote one where the other belongs.
Why our crawler flag missed about half of it
The clicks table already had a bot column. Its rule: mark the click as a bot if the request has no referer, or if the user agent matches a known crawler string. That is the standard cheap heuristic and it does catch a lot.
It also has a blind spot that turned out to be large. On the 2026-08-18 window, 222 clicks passed the bot filter as qualified. Excluding Singapore and China took that to 115. Roughly half of what the flag called human traffic came from two buckets whose country-referer-device signature said otherwise, with China the largest single bucket.
The reason is structural. A crawler that reaches an affiliate link by following an internal link from an article sends a referer — your own domain. A "has referer" test passes it cleanly. So the flag is weakest against exactly the automation that crawls your site properly, page by page, which is also the automation most likely to hit an outbound link. The naive scrapers that hit a URL cold get caught; the well-behaved ones that walk your navigation do not.
That is why the country signature is worth computing even when you already have a UA-based flag. They fail on different populations, and the overlap between them is smaller than you would guess.
What this cannot tell you
This is a heuristic over three coarse fields, and it is wrong in known directions.
It cannot see non-JS traffic at all. RUM fires from a script tag. A curl loop, a Python requests scraper, or anything that pulls HTML without a browser engine never appears in the dataset. Whatever bot share this method reports, your real share is higher — this measures only the subset of automation that bothers to execute JavaScript.
It produces false positives you have to accept. A reader on a datacentre-hosted VPN, behind a corporate proxy egress, or on a privacy browser that strips referers will look like a bot on two of the three axes. We accept that cost because the residual is small against 4,940 pageviews from a single country. At smaller volumes the ratio flips and the method stops being safe.
We did not test the things that would actually settle it. No user-agent entropy analysis, no TLS fingerprinting (JA3/JA4), no ASN lookup on the source IP. Those give you a per-request answer instead of a per-bucket guess, and they are what you should reach for if you can.
The alternative we would pick given the option: put a Worker in front of the origin and read Cloudflare's bot score and the client ASN per request, or ship request logs to somewhere queryable. ASN plus bot score beats country heuristics on every axis — it is per-request, it does not confuse a Singaporean reader with a Singaporean EC2 instance, and it survives someone routing their scrapers through residential proxies far better. The condition that flips it back to the heuristic: you are on a fully static host with no request-level logging, no paid analytics tier, and no budget for either. Then three fields is what you have.
The other hard constraint is time. Cloudflare RUM keeps a rolling window and Search Console shows no history at all. If you do not append a dated row somewhere durable each week, the series has a hole in it that can never be filled. We write ours to a committed JSON file in the repo; the point is that it is dated, append-only and outside the tool that expires it.
Before you exclude a country permanently, check whether the signature holds across two or three separate weeks. A one-off spike with a clean device mix is more likely a real referral you have not identified yet. The 100%/0% pattern is stable week over week when it is infrastructure.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)