On October 30, Umami reported 212 visits on apprendre.forelse.fr. A good day, on the face of it. Looking closer, 203 of those visits came from bots. That left 9 people.
To measure the site's audience, I use Umami, self-hosted on my own server: no cookies, no IP addresses kept, nothing sent to a third party. But numbers that are 95% wrong are worse than no numbers at all. I was making decisions based on them: which article worked, which exercise gets used, whether a LinkedIn post brings anyone in.
Here is how I got rid of them, in three steps, and what it costs.
What Umami already does
Umami ignores visits whose User-Agent says it is a bot (Googlebot, bingbot, LinkedInBot…), using the isbot library. That takes care of the polite bots.
The problem is the others. The ones that run a real Chrome, execute the page's JavaScript, and therefore send a page view like any visitor would. To Umami, they look exactly like a human.
Step 1: robots.txt
The site's tracker is served under /mesure/, and robots.txt contains:
Disallow: /mesure/
It's the bare minimum, and it only helps with bots that read robots.txt. The ones causing me trouble don't: one of them followed the "Log in" link on every page, even though /connexion is disallowed there.
Step 2: automated browsers (navigator.webdriver)
The first bot I spotted was the one that follows LinkedIn. With every post, within a second of LinkedInBot passing by, a "Chrome on Windows" opened the page, then the links on the page. It counted as visits and, worse, as failed starts of the PHP playground: it left before PHP had finished loading. I thought I had a bug.
The server logs settled it. Some User-Agents were simply impossible: Chrome 140 on Windows 7, when Chrome hasn't shipped for Windows 7 since version 109.
A browser driven by Selenium, Playwright or Puppeteer usually exposes navigator.webdriver === true. So the site doesn't load the tracker in that case:
if (navigator.webdriver) return; // no tracker for an automated browser
The next day, no more failed starts after a LinkedIn post.
Step 3: wait for a gesture from the visitor
The biggest one was still there: a bot that came every night, always with the same fingerprint. Chrome, Linux, a 1920x1080 screen, en-US, addresses in several countries. 163 visits in five days. Its User-Agent claimed "Chrome/155", a version that doesn't exist yet. And it doesn't set navigator.webdriver.
My first idea was to filter out that fingerprint. But fingerprints change. And 1920x1080 on Linux in English is also the screen of a lot of developers.
Looking at its visits, one thing stood out: one page, no events, ever. It renders the page and leaves. It doesn't move the mouse, touch the screen, turn the wheel or press a key.
So the site waits for that first gesture before loading the tracker:
const GESTURES = ['pointerdown', 'pointermove', 'touchstart', 'wheel', 'keydown'];
const load = () => {
GESTURES.forEach((gesture) => window.removeEventListener(gesture, load, true));
document.head.append(script); // Umami's tracker, and the page view with it
};
GESTURES.forEach((gesture) =>
window.addEventListener(gesture, load, { capture: true, passive: true }),
);
A few details matter:
-
No
scrollin the list. A page can scroll on its own, with nobody there: the code editor, or the browser restoring the position. A bot would then have loaded the tracker. - The gesture isn't sent. It's only a trigger: Umami receives exactly the same data as before, nothing more.
- Events wait for the tracker. An event fired before the first gesture ("exercise opened", "account created" right after sign-up) is queued, then sent as soon as the tracker has loaded. Without that, those events were lost.
The result
The change has been in production since November 3. Since November 4, I haven't found a single bot in Umami. No more SQL queries excluding fingerprints by hand: the raw numbers are the real ones.
They're a lot more modest: a few people a day when I don't publish anything, more on article days. Less flattering, but it's exactly what I wanted to know.
What it costs
Nothing comes for free:
- A visitor who arrives, reads without touching anything and leaves isn't counted. On a computer, that's rare: the mouse almost always moves. On mobile, the first swipe fires
touchstart. - A bot that fakes mouse movements will get through. This isn't a protection, it's a filter against lazy bots, which is almost all of the ones that came here.
- The page view is sent a little later. For Umami, that changes nothing.
For a site that wants to know how many real people read its articles, it's a good trade in my view. For a shop that needs to count every visit, I'd think twice.
The full code lives in the site's engine, which is open source: a 75-line file, playground/src/mesure.ts.
What about you: do you filter bots out of your analytics, or do you live with them?
Top comments (0)