DEV Community

RoamProxy
RoamProxy

Posted on

Playwright + Proxies: Sticky Sessions, Rotation, and the SOCKS5 Trap

#python #playwright #webscraping #tutorial

You wired a proxy into requests in one line. Then you tried the same thing with Playwright and got a soup of ERR_TUNNEL_CONNECTION_FAILED, mystery CAPTCHAs, and a bandwidth bill that made no sense. Been there.

Headless browsers talk to proxies differently than HTTP clients do, and most of the "playwright proxy" snippets floating around skip the three things that actually bite you in production. Here's the setup I wish someone had handed me.

The baseline: one browser, one proxy 

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://gw.example.com:41080",
        "username": "USER-session-alpha",
        "password": "PASS",
    })
    page = browser.new_page()
    page.goto("https://httpbin.org/ip")
    print(page.content())
    browser.close()
Enter fullscreen mode Exit fullscreen mode

Two details people miss:

  1. Auth goes in the launch config, not the URL. http://user:pass@host:port as the server string works in some HTTP clients, but Chromium wants credentials separately. If your password has special characters (#, @), the URL form will silently mangle them.
  2. The proxy applies to everything the browser loads — every image, every font, every tracking pixel. Remember that; it matters in a minute.

Trap #1: SOCKS5 auth doesn't work in Chromium

This one costs people hours. Chromium — and therefore Playwright's default browser — does not support username/password authentication over SOCKS5. Your provider's SOCKS5 endpoint works fine in curl, so you assume your Playwright config is wrong. It isn't. The browser just never sends the credentials.

Fixes, in order of sanity:

  • Use the HTTP proxy port with auth instead. Any decent provider serves HTTP CONNECT and SOCKS5 (ideally on the same port), and CONNECT tunnels HTTPS just fine. There is no meaningful difference for scraping.
  • IP-allowlist auth on the SOCKS5 side, if your provider supports it and your egress IP is stable.
  • A local forwarder (e.g. a tiny pproxy sidecar) that holds the credentials — works, but now you're running infrastructure to avoid a one-line fix.

Trap #2: rotation granularity is per-context, not per-request

With an HTTP client, rotating proxies per request is trivial. A browser is a different animal: a page load is 50–200 requests, and they must all exit from the same IP or you'll trip every anti-bot system on earth (real users don't load HTML from Berlin and CSS from Osaka).

So the unit of rotation is the browser context, not the request. With a session-based proxy gateway, you pin one session ID per context:

def new_identity(p, session_id: str):
    browser = p.chromium.launch(proxy={
        "server": "http://gw.example.com:41080",
        "username": f"USER-session-{session_id}",
        "password": "PASS",
    })
    return browser

# identity per worker: same IP for the whole browsing session
for task_id, urls in enumerate(url_batches):
    browser = new_identity(p, f"worker{task_id}")
    page = browser.new_page()
    for url in urls:
        page.goto(url)
        ...
    browser.close()
Enter fullscreen mode Exit fullscreen mode

Same session ID = same exit IP for the lifetime of the session; new ID = fresh IP. One identity per context, rotate by spawning a new context with a new session ID. That's the whole model. (Playwright also accepts proxy= on new_context(), so you can run several identities inside one browser process — cheaper than one process per identity.)

Trap #3: you're paying residential rates for webfonts

Residential bandwidth is billed by the GB. A single product page can pull 3–8 MB, of which the HTML you actually want is maybe 80 KB. Multiply by 100k pages and you've spent real money loading hero images for nobody.

Route-block everything you don't parse:

BLOCK = {"image", "media", "font", "stylesheet"}

def register_blocking(page):
    page.route("**/*", lambda route:
        route.abort() if route.request.resource_type in BLOCK
        else route.continue_())
Enter fullscreen mode Exit fullscreen mode

In my runs this cuts bandwidth 60–85% depending on the site, with zero effect on the DOM you scrape. Two caveats: keep stylesheets if you rely on :visible selectors (layout needs CSS), and don't block XHR/fetch — that's usually where the data lives.

Putting it together

The full pattern — sticky identity per context, auth done right, resource blocking on every page — is about 40 lines. I keep a copy-pasteable version (plus requests/httpx/Scrapy equivalents) in a public examples repo: github.com/roamproxy/proxy-examples.

If you want to test against a real residential pool without a subscription, the network I work on (Roam) gives 300MB free on signup — enough to verify your Playwright setup against your actual target before spending anything.

Questions about a setup that's misbehaving? Drop it in the comments — proxy-through-browser bugs are weirdly fun to debug.

Top comments (0)