DEV Community

JINFENG LI
JINFENG LI

Posted on

Running Playwright behind residential proxies: 7 things that actually break

Most tutorials stop at proxy={"server": "..."}. In production, that gets you a 60% success rate and a support ticket.

Here are the seven things that actually decide whether your scraper survives — each one learned the hard way.

1. Authenticated proxies must be configured at launch, not per request

browser = p.chromium.launch(
    proxy={
        "server": "http://gateway.example.com:8000",
        "username": "user",
        "password": "pass",
    },
)
Enter fullscreen mode Exit fullscreen mode

If you instead set HTTP_PROXY env vars or route requests manually, you'll hit the classic split: plain HTTP works, HTTPS handshakes fail, because proxy auth headers aren't attached on CONNECT.

2. Timezone, locale and IP geolocation must agree

A browser exiting in Los Angeles with Asia/Shanghai timezone and zh-CN locale is one of the cheapest bot signals there is.

ctx = browser.new_context(
    locale="en-US",
    timezone_id="America/Los_Angeles",
    viewport={"width": 1440, "height": 900},
)
Enter fullscreen mode Exit fullscreen mode

When you're doing geo price comparison, these three values aren't environment setup — they're the variables you're testing.

3. WebRTC leaks the real IP even through a proxy

args=[
    "--disable-webrtc",
    "--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
]
Enter fullscreen mode Exit fullscreen mode

Verify by visiting any WebRTC leak test page: the reported IP must equal your proxy exit IP. If it doesn't, you're leaking.

4. navigator.webdriver is only the first layer

Modern Chromium reports false in headless=new mode, but there are dozens of other signals: plugin count, languages, hardwareConcurrency, canvas fingerprint, presence of chrome.runtime.

Don't hand-patch these — the maintenance cost is brutal. Use a maintained stealth layer, or accept that some targets will block you and invest in realistic environments (residential IPs, real UA strings, human pacing) instead of browser patching.

And to be clear: passing a bot check is not permission. Respect robots.txt and the target's terms of service.

5. One context per task, never reuse

browser.new_context() is cheap — far cheaper than launching a new browser. Reusing contexts leaks cookies between tasks: task A logs in, task B rides A's session and gets flagged instantly.

def fetch(browser, url):
    ctx = browser.new_context(locale="en-US", timezone_id="America/Los_Angeles")
    try:
        page = ctx.new_page()
        page.goto(url, wait_until="domcontentloaded", timeout=30000)
        page.wait_for_selector("main", timeout=15000)
        return page.content()
    finally:
        ctx.close()
Enter fullscreen mode Exit fullscreen mode

6. Stop waiting for networkidle

Sites with polling, WebSockets or analytics beacons never go idle. You wait 30 seconds and get a timeout.

Wait for DOM, then for the one element that proves success:

page.goto(url, wait_until="domcontentloaded", timeout=30000)
page.wait_for_selector("main", timeout=15000)
Enter fullscreen mode Exit fullscreen mode

Knowing exactly what "success" looks like beats waiting for "nothing is happening".

7. Define "blocked" explicitly, or your retries are just burning money

Residential traffic is billed per GB. Every blind retry costs real money.

BLOCK_SIGNALS = ("captcha", "verify you are human", "access denied",
                 "unusual traffic", "are you a robot")

def is_blocked(page):
    html = page.content().lower()
    return any(s in html for s in BLOCK_SIGNALS)
Enter fullscreen mode Exit fullscreen mode

When blocked, escalate in cost order: new session IP → lower concurrency → new UA/fingerprint. Most people do it backwards and burn the expensive knob first.

Bonus: cut traffic by 40–70% for free

If you're billed per GB, blocking images and fonts is the single highest-leverage change you'll make:

ctx.route("**/*", lambda route: route.abort()
          if route.request.resource_type in ("image", "font", "media")
          else route.continue_())
Enter fullscreen mode Exit fullscreen mode

Two more cost items worth watching:

  • Concurrency: 8–12 Chromium instances on an 8-core box, 3–5 contexts each. More isn't faster — residential exits are real home connections with limited bandwidth, and timeouts will queue up.
  • --disable-dev-shm-usage: without it, Chromium crashes randomly in containers where /dev/shm is 64MB.

Does the proxy vendor matter?

Less than you'd think, once the seven items above are handled. What matters is: can you control session stickiness, and can you see the exit IP per session when debugging. If you can't answer "was that 403 the proxy or my code?", you'll burn days.

I run these patterns on IPLight's residential pool (dynamic from $1.20/GB, 100 GB minimum; static ISP from $3/IP/mo). The reason isn't price — it's that session IDs are explicit in the username and the console keeps a per-session exit IP record, which turns debugging from guesswork into a diff.

Start with a small quota, run your real workload for a week, and measure success rate per dollar. Any spec sheet is a worse predictor than that number.

Top comments (0)