DEV Community

Vhub Systems
Vhub Systems

Posted on

How Sites Detect Headless Browsers (And How to Evade Each Signal) — 2026 Guide

#webscraping #python #javascript #security

Websites can detect headless browsers through dozens of signals. Most scraper tutorials show you how to launch Playwright — none explain why it still gets blocked. Here's what sites are actually checking and how to evade each signal.

How headless detection works

Detection is probabilistic, not binary. Sites like Cloudflare, Akamai, and DataDome maintain a "bot score" based on many signals. Each anomaly adds to your score. Enough anomalies = block.

The most common signals:

  1. navigator.webdriver = true
  2. Missing browser plugins
  3. Specific headless Chrome flags
  4. Canvas/WebGL fingerprint anomalies
  5. Chrome DevTools Protocol (CDP) exposure
  6. User-agent / platform inconsistencies
  7. Missing media codecs
  8. Behavioral signals (mouse movement, timing)

Signal 1: navigator.webdriver

The most obvious signal. Chrome headless sets navigator.webdriver = true by default.

How to fix:

// Playwright
const context = await browser.newContext();
await context.addInitScript(() => {
    Object.defineProperty(navigator, 'webdriver', {
        get: () => undefined,
        configurable: true,
    });
});
Enter fullscreen mode Exit fullscreen mode 
# Playwright Python
async with async_playwright() as p:
    browser = await p.chromium.launch(headless=True)
    context = await browser.new_context()
    await context.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        })
    """)
Enter fullscreen mode Exit fullscreen mode

Why this isn't enough: Sites also check for chrome.runtime, window.chrome, and CDP artifacts.

Signal 2: Chrome DevTools Protocol (CDP) detection

Playwright communicates with Chrome via CDP. Some sites detect open CDP connections:

// Detection technique used by some sites
const ws = window.WebSocket;
window.WebSocket = function(...args) {
    if (args[0].includes('devtools')) {
        throw new Error('DevTools detected!');
    }
    return new ws(...args);
};
Enter fullscreen mode Exit fullscreen mode

Fix: Use --remote-debugging-port=0 to disable CDP discovery, or better — use camoufox which patches CDP exposure:

from camoufox.async_api import AsyncCamoufox

async with AsyncCamoufox(headless=True) as browser:
    page = await browser.new_page()
    await page.goto("https://bot-test.com")
    # CDP not detectable from page JavaScript
Enter fullscreen mode Exit fullscreen mode

Signal 3: Missing Chrome plugins and extensions

Real Chrome has plugins (navigator.plugins). Headless Chrome has zero:

// Detection: checks for plugins
if (navigator.plugins.length === 0) {
    // Likely headless
}
Enter fullscreen mode Exit fullscreen mode

Fix: Spoof plugins array:

// Add fake plugins
Object.defineProperty(navigator, 'plugins', {
    get: () => [
        { name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer' },
        { name: 'Chrome PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai' },
        { name: 'Native Client', filename: 'internal-nacl-plugin' },
    ]
});
Enter fullscreen mode Exit fullscreen mode

Signal 4: Canvas fingerprinting

Sites draw to canvas and compare the output hash. Headless Chrome produces a different hash than normal Chrome due to rendering differences.

Detection:

const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');
ctx.fillText('fingerprint', 10, 50);
const hash = canvas.toDataURL();
// Compare hash to known-bot hashes
Enter fullscreen mode Exit fullscreen mode

Fix: Use noise injection to randomize canvas output:

# Playwright: add canvas noise
await context.add_init_script("""
    const originalFillText = CanvasRenderingContext2D.prototype.fillText;
    CanvasRenderingContext2D.prototype.fillText = function(...args) {
        const noise = Math.random() * 0.001;
        this.globalAlpha = 1 - noise;
        return originalFillText.apply(this, args);
    };
""")
Enter fullscreen mode Exit fullscreen mode

Or use playwright-stealth which handles this automatically:

from playwright_stealth import stealth_async

browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await stealth_async(page)  # Patches all known detection vectors
await page.goto("https://bot.sannysoft.com")
Enter fullscreen mode Exit fullscreen mode

Signal 5: Headless-specific user agent

Chrome headless used to include "HeadlessChrome" in the UA string. Now it doesn't, but other UA anomalies remain:

# BAD: Default headless UA has inconsistencies
browser = await p.chromium.launch(headless=True)

# GOOD: Use a real Chrome UA that matches the browser version
context = await browser.new_context(
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
)
Enter fullscreen mode Exit fullscreen mode

Also match the platform:

context = await browser.new_context(
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
    # navigator.platform must match the UA
    extra_http_headers={"sec-ch-ua-platform": '"Windows"'}
)
Enter fullscreen mode Exit fullscreen mode

Signal 6: Timing and behavioral signals

Bots move the mouse in straight lines, click instantly, and scroll at constant speeds. Sites use mouse heuristics to distinguish humans from bots.

Fix: Add realistic mouse movement:

import asyncio
import random

async def human_mouse_move(page, target_x, target_y, steps=10):
    """Move mouse in a curved path"""
    box = await page.query_selector("body")
    start = await page.evaluate("() => ({x: window.innerWidth/2, y: window.innerHeight/2})")

    for i in range(steps):
        # Bezier curve interpolation
        t = i / steps
        x = start['x'] + (target_x - start['x']) * t + random.gauss(0, 5)
        y = start['y'] + (target_y - start['y']) * t + random.gauss(0, 5)

        await page.mouse.move(x, y)
        await asyncio.sleep(random.uniform(0.01, 0.05))

async def human_click(page, selector):
    element = await page.query_selector(selector)
    box = await element.bounding_box()

    # Click slightly off-center (humans rarely click exact center)
    x = box['x'] + box['width'] * random.uniform(0.3, 0.7)
    y = box['y'] + box['height'] * random.uniform(0.3, 0.7)

    await human_mouse_move(page, x, y)
    await asyncio.sleep(random.uniform(0.1, 0.3))  # Pause before click
    await page.mouse.click(x, y)
Enter fullscreen mode Exit fullscreen mode

Signal 7: Chrome launch flags

Some --disable-* flags that make scraping easier also make you more detectable:

# Flags that increase detection risk:
args = [
    "--disable-blink-features=AutomationControlled",  # This is fine
    "--disable-dev-shm-usage",  # Low risk
    "--no-sandbox",  # INCREASES detection - only use in Docker
    "--disable-setuid-sandbox",  # Same
    "--disable-web-security",  # High detection risk - don't use
]

# Recommended minimal flags:
safe_args = [
    "--disable-blink-features=AutomationControlled",
    "--disable-infobars",
    "--window-size=1280,800",
]
Enter fullscreen mode Exit fullscreen mode

The complete stealth setup for Playwright Python 

from playwright.async_api import async_playwright
import asyncio
import random

STEALTH_JS = """
// 1. Remove webdriver
Object.defineProperty(navigator, 'webdriver', { get: () => undefined });

// 2. Add Chrome object
window.chrome = { runtime: {} };

// 3. Add fake plugins
Object.defineProperty(navigator, 'plugins', {
    get: () => [
        { name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer' },
        { name: 'Chromium PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai' },
    ]
});

// 4. Add fake languages
Object.defineProperty(navigator, 'languages', {
    get: () => ['en-US', 'en']
});

// 5. Fix permissions API
const originalQuery = window.navigator.permissions.query;
window.navigator.permissions.query = (parameters) => (
    parameters.name === 'notifications'
        ? Promise.resolve({ state: Notification.permission })
        : originalQuery(parameters)
);
"""

async def create_stealth_browser():
    p = await async_playwright().start()
    browser = await p.chromium.launch(
        headless=True,
        args=[
            "--disable-blink-features=AutomationControlled",
            "--disable-infobars",
            "--window-size=1366,768",
        ]
    )
    context = await browser.new_context(
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
        viewport={"width": 1366, "height": 768},
        locale="en-US",
        timezone_id="America/New_York",
        extra_http_headers={
            "Accept-Language": "en-US,en;q=0.9",
            "sec-ch-ua": '"Chromium";v="122", "Not(A:Brand";v="24"',
            "sec-ch-ua-mobile": "?0",
            "sec-ch-ua-platform": '"Windows"',
        }
    )
    await context.add_init_script(STEALTH_JS)
    return context

# Test against bot detection
async def test_stealth():
    context = await create_stealth_browser()
    page = await context.new_page()
    await page.goto("https://bot.sannysoft.com")
    await page.screenshot(path="stealth-test.png")
    print("Check stealth-test.png for results")

asyncio.run(test_stealth())
Enter fullscreen mode Exit fullscreen mode

camoufox: the nuclear option

camoufox is a Firefox fork specifically built for stealth scraping. It modifies Firefox at the binary level to eliminate detection signals:

from camoufox.sync_api import Camoufox

with Camoufox(headless=True, os="windows") as browser:
    page = browser.new_page()
    page.goto("https://bot.sannysoft.com")
    # Passes most detection tests
Enter fullscreen mode Exit fullscreen mode

Install: pip install camoufox && python -m camoufox fetch

camoufox passes Cloudflare's bot checks in headless mode — something no Playwright configuration can do reliably.

What still doesn't work

Even with perfect browser fingerprinting, some sites detect bots via:

  • IP reputation: residential proxies required
  • Account behavior: new accounts with no history get restricted
  • JavaScript execution patterns: some sites analyze CPU timing
  • Network timing: consistent request intervals are a bot signal

For the hardest sites (LinkedIn, Google, major e-commerce), managed actors handle all of this:

Apify Scrapers Bundle ($29) — pre-built actors with stealth, proxy rotation, and behavioral randomization built in.

Quick reference: detection evasion checklist

  • [ ] navigator.webdriver = undefined
  • [ ] Add window.chrome.runtime
  • [ ] Spoof navigator.plugins (≥2 plugins)
  • [ ] Set realistic navigator.languages
  • [ ] Match UA string to actual Chrome version
  • [ ] Set timezone_id to match locale
  • [ ] Use --disable-blink-features=AutomationControlled
  • [ ] Avoid --no-sandbox in non-Docker environments
  • [ ] Add canvas noise injection
  • [ ] Use human-like mouse movement for click actions
  • [ ] Use residential proxies for IP reputation
  • [ ] Or use playwright-stealth / camoufox for automated patching

n8n AI Automation Pack ($39) — 5 production-ready workflows

Related Tools

Pre-built actors for this use case:

Top comments (0)