Four layers of bot detection, and why disabling the first one changes nothing

Hong Ye — Sun, 15 Nov 2026 22:01:37 +0000

Body

Most people fighting anti-bot systems do the same thing: they find the check that's failing, they defeat it, and they're surprised when it still doesn't work.

That's because detection isn't one check. It's four layers, and they fail independently.

Layer 1: Signature

The crude stuff. navigator.webdriver, cdc_ variables injected by ChromeDriver, the HeadlessChrome string in the user agent.

This is what undetected-chromedriver fixes, and it's why people think they've solved the problem.

Defeating it takes ten minutes. Patching navigator.webdriver can be done with a single Page.addScriptToEvaluateOnNewDocument call.

If a site still catches you after this, the site is not checking signatures. Stop looking here.

Layer 2: Fingerprint

The browser tells the truth about itself, and the truth is inconsistent.

A real Chrome has plugins. An automated one usually has an empty plugin array.

A real screen is 1920x1080 with a window slightly smaller. An automated one often reports a viewport that doesn't match the window.

WebGL reports a renderer string. On headless it says SwiftShader, or ANGLE with a suspicious driver.

Fonts installed. Timezone versus IP geolocation. Intl locale. Canvas hash. Audio context hash.

No single item is damning. The problem is that in aggregate they form a signature, and signatures are compared against a population.

If your fingerprint matches 0.01% of real users, you get flagged — even though every individual value looks plausible.

This is why stealth plugins work less well than they used to. They make every value look normal, but they don't make the combination look common.

The reliable fix here is unglamorous: run the browser with a real user profile. --user-data-dir pointed at a directory where someone actually browsed. You inherit real plugins, real fonts, real history.

It beats any stealth library, and it costs nothing.

Layer 3: Behaviour

This is the layer most people never think about, and increasingly the one that matters.

A real person logging in:

Moves the mouse in a curve with deceleration and overshoot
Takes 200-800ms to move between fields
Types with inter-key intervals that vary (40-300ms), with occasional pauses
Makes a typo occasionally and corrects it
Scrolls in chunks, sometimes back up

An automated login:

element.click() teleports the cursor and fires a click in one frame
send_keys types at a constant interval
No mouse movement at all if you used JS to click
Scroll is window.scrollTo(0, 5000) — one jump

Some sites now weight behaviour above fingerprint. Because fingerprints are hard to keep consistent across a fleet, but behaviour is trivially cheap to check.

Fixing this is not about adding random delays. Random delays are still detectable by distribution — real typing has structure (bursts, pauses before difficult words, correction patterns) that uniform randomness does not.

The cheap version that works: use mouse.move(x, y, steps=N) to generate interpolation, and type with a log-normal delay distribution. It's not perfect but it clears the first-order checks.

Layer 4: Session and reputation

Not about the browser at all.

IP reputation (datacenter ranges, VPN exit nodes, prior abuse)
Cookie age (a brand-new cookie jar logging in is suspicious)
Login velocity from that ASN
Whether the account has history

This is why your perfectly-stealthed browser still gets challenged on a fresh IP with a fresh cookie jar.

And this is the layer you cannot code around. You can't fabricate account age.

The ordering matters

When something fails, the correct diagnostic order is:

1. Does a real browser from the same IP and same cookie jar work?
   → No: it's Layer 4. Your IP or session is burned.
   → Yes: continue.

2. Does your automation work if you point it at the real user profile?
   → Yes: it was Layer 2. Keep the profile approach.
   → No: continue.

3. Does it work if you slow everything down and add real mouse movement?
   → Yes: it was Layer 3.
   → No: continue.

4. It's Layer 1, or it's not anti-bot at all (you have a logic bug).

Most debugging time is wasted because people start at 4 and work backwards.

The honest conclusion

If a site has invested seriously in Layers 2-4, defeating it is not a ten-minute fix. It's a maintenance project — profiles to keep warm, proxies to rotate, behaviour to tune, and a re-testing loop because they change things.

That's the thing to tell a client before taking the job. Not "I can bypass it," but "here's what bypassing it costs per month to keep working."

The number of sites where the answer is "don't scrape this, use their API or partner with them" is larger than most people want to admit.

Your scraper is lying to you (and it's your fault)

Hong Ye — Sun, 15 Nov 2026 20:05:22 +0000

The pipeline described here runs on Hacker News' public API and is open source. It's deliberately small — 200 lines, no dependencies — because the point is the error handling, not the parsing.

DEV Community: Hong Ye

Four layers of bot detection, and why disabling the first one changes nothing

Body

Layer 1: Signature

Layer 2: Fingerprint

Layer 3: Behaviour

Layer 4: Session and reputation

The ordering matters

The honest conclusion

Your scraper is lying to you (and it's your fault)