Web Scraping With Playwright and Residential Proxies: The Complete Configuration Guide

HTTP-client scraping (requests, httpx, aiohttp) is the right default — fast, cheap, parallel. But some targets simply cannot be scraped without a real browser: heavy client-side rendering, anti-bot JavaScript challenges, canvas fingerprinting, WebSocket-driven content. That's when you reach for Playwright.

And that's when most people hit a wall, because browser-level proxying works fundamentally differently from HTTP-client proxying, and the IP is only one of a dozen fingerprint dimensions a browser leaks. This post is the guide I wish had existed when I first wired these together: proxy configuration, session management, and the fingerprint hygiene that decides whether your headless browser gets served content or a challenge page.

Part 1: Proxy Configuration Done Right

The Basic Setup

Playwright's launch() accepts proxy config at the browser level:

from playwright.sync_api import sync_playwright

PROXY = {
    "server": "http://p.thordata.com:9000",
    "username": "thor_user",
    "password": "thor_pass",
}

with sync_playwright() as p:
    browser = p.chromium.launch(proxy=PROXY, headless=True)
    page = browser.new_page()
    page.goto("https://httpbin.org/ip")
    print(page.inner_text("body"))   # verify the exit IP
    browser.close()

Always — always — verify the exit IP on httpbin (or similar) before pointing your scraper at a real target. Half of all "Playwright proxy doesn't work" issues are actually credential or whitelist misconfigurations that a 5-second IP check would have caught.

The Trap: Browser-Level Proxies Don't Rotate Per-Page

Here's the first big difference from HTTP clients. When you set the proxy at launch(), every context and page in that browser instance shares one proxy exit. If your provider rotates per-connection, you'll get bizarre behavior: the browser's main connection uses one IP, but a WebSocket or a retry uses another, and cookies get orphaned mid-session.

The correct unit of proxy control in Playwright is the browser context, not the browser:

import itertools

class ProxyContextFactory:
    """One browser context = one proxy exit = one 'user'."""

    def __init__(self, playwright, proxy_template):
        self.pw = playwright
        self.template = proxy_template
        self.counter = itertools.count()

    def new_context(self, geo="us"):
        n = next(self.counter)
        proxy = {
            "server": self.template["server"],
            # session token in username => sticky exit for this context's lifetime
            "username": f'{self.template["username"]}'
                        f'-session-ctx{n}-cc-{geo}',
            "password": self.template["password"],
        }
        browser = self.pw.chromium.launch(proxy=proxy, headless=True)
        context = browser.new_context(
            locale="en-US",
            timezone_id="America/New_York",
            geolocation={"latitude": 40.71, "longitude": -74.01},
            permissions=["geolocation"],
        )
        return browser, context

The pattern to internalize: launch a browser per context when you need per-context proxies. It feels heavy, but Chromium shares most binaries between launches, and the isolation you get — one exit IP, one cookie jar, one fingerprint per "user" — is exactly what you want. Tear the whole thing down when the task completes and the next launch gets a fresh IP automatically.

Region Pinning

Most residential proxy providers accept country/city codes in the username (Thordata, for example, uses a -cc-us style suffix). Pin the region and then make the context agree with it:

GEO_PRESETS = {
    "us": ("en-US", "America/New_York", (40.71, -74.01)),
    "de": ("de-DE", "Europe/Berlin", (52.52, 13.40)),
    "uk": ("en-GB", "Europe/London", (51.51, -0.13)),
}

def new_context_for(browser, geo):
    locale, tz, (lat, lon) = GEO_PRESETS[geo]
    return browser.new_context(
        locale=locale, timezone_id=tz,
        geolocation={"latitude": lat, "longitude": lon},
        viewport={"width": 1366, "height": 768},
    )

The single most common geo-mistake I see: US exit IP + timezone_id: "Asia/Shanghai" + Chinese Accept-Language. Your IP says New Jersey, your clock says Shanghai. No bot detector needs more than that.

Part 2: Fingerprint Management — Where Headless Browsers Get Caught

Here's the uncomfortable truth: a naive chromium.launch(headless=True) is more detectable than plain requests, because it presents a browser-shaped fingerprint full of headless tells. The IP gets you in the door; the fingerprint decides whether you're served.

Step 1: Kill the Headless Tells

Modern Playwright handles a lot of this via its bundled Chromium, but you should be explicit:

browser = p.chromium.launch(
    headless=True,
    args=[
        "--disable-blink-features=AutomationControlled",  # navigator.webdriver
        "--no-sandbox",
    ],
)

context = browser.new_context(
    user_agent=(
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    viewport={"width": 1366, "height": 768},
    device_scale_factor=1,
    locale="en-US",
)

Note the user_agent: never ship the default, which contains "HeadlessChrome" — an instant tell.

Step 2: Patch the JavaScript-Visible Surface

Anti-bot scripts probe properties the CDP flags don't cover. The standard fix is an init script that runs before any page JS:

context.add_init_script("""
    Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
    Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']});
    Object.defineProperty(navigator, 'plugins', {get: () => [1, 2, 3, 4, 5]});
    window.chrome = { runtime: {} };
    const origQuery = navigator.permissions.query;
    navigator.permissions.query = (params) =>
        params.name === 'notifications'
            ? Promise.resolve({state: Notification.permission})
            : origQuery(params);
""")

Run your patched browser against a detection test page (bot.sannysoft.com is the classic) and fix every red row before touching production. Seriously — treat that as a build gate, not a debugging step.

Step 3: Make It Behave Like a Human

The best fingerprint in the world fails if the behavior is robotic. Three habits worth building into a helper:

import random

def human_goto(page, url):
    page.goto(url, wait_until="domcontentloaded")
    page.wait_for_timeout(random.randint(800, 2200))

    # scroll like a person: variable steps, occasional pause
    total = page.evaluate("document.body.scrollHeight")
    pos = 0
    while pos < total:
        step = random.randint(300, 700)
        pos += step
        page.mouse.wheel(0, step)
        page.wait_for_timeout(random.randint(150, 600))
        if random.random() < 0.15:
            page.wait_for_timeout(random.randint(1000, 3000))  # "reading"
    return page

def human_click(page, selector):
    box = page.locator(selector).bounding_box()
    if box:
        x = box["x"] + box["width"] * random.uniform(0.3, 0.7)
        y = box["y"] + box["height"] * random.uniform(0.3, 0.7)
        page.mouse.move(x - random.randint(0, 40), y + random.randint(0, 40))
        page.mouse.move(x, y, steps=random.randint(5, 15))
        page.mouse.click(x, y)
        page.wait_for_timeout(random.randint(400, 1200))

Momentum-based scrolling, curved mouse paths, and randomized think-time between actions. Behavioral detection models score these signals heavily — a session with pixel-perfect instant scrolls and zero dwell time is a bot regardless of what its canvas fingerprint says.

Step 4: Keep Context, Credentials, and IP in Lockstep

Just like with HTTP clients (I covered the session-alignment principle in my post on IP rotation strategies), everything must rotate as a unit. One context = one exit IP = one fingerprint = one cookie jar. Never reuse a logged-in storage state across contexts with different exit IPs — that's the signature of a stolen session, and fraud systems flag it harder than plain botting.

Part 3: A Complete Scraping Skeleton

Putting it all together — this is the shape of a production Playwright worker:

from playwright.sync_api import sync_playwright

TEMPLATE = {
    "server": "http://p.thordata.com:9000",
    "username": "thor_user",
    "password": "thor_pass",
}

def scrape_product(url: str, geo: str = "us") -> dict:
    proxy = dict(TEMPLATE)
    proxy["username"] += f"-session-{id(url):x}-cc-{geo}"

    with sync_playwright() as p:
        browser = p.chromium.launch(proxy=proxy, headless=True)
        context = new_context_for(browser, geo)   # from Part 1
        apply_stealth(context)                     # Part 2 steps 1-2
        page = context.new_page()

        human_goto(page, url)
        title = page.locator("h1").first.inner_text()
        price = page.locator("[data-price]").first.get_attribute("data-price")

        browser.close()     # context dies => session ends => IP released
        return {"url": url, "title": title, "price": price}

One task, one context, one IP, one teardown. Scale horizontally — many processes, not many pages in one browser — and your per-"user" isolation stays clean at any volume.

When Not to Do Any of This

A closing reality check: Playwright is 20–50x more expensive per request than httpx. Use it only when the target forces you to. My decision rule: try the HTTP client first; if you get blocked or the content isn't in the initial HTML, escalate to Playwright for that domain — and only that domain. Most scraping fleets I run are 90% HTTP clients and 10% browsers, with the browser share reserved for the genuinely hard targets.

Disclosure: I use Thordata's residential proxies for all my Playwright-based scraping — the username-suffix session control shown in this post is how I bind one exit IP to one browser context. You can find them at thordata.com, and the code **thor020* gets you 10% off.*

Comments

[Luis Cruz]

Your explanation of how browser-level proxying differs from HTTP-client proxying is incredibly insightful, especially the emphasis on managing proxies at the browser context level. This approach not only prevents issues with connection mismatches but also aligns well with ensuring a cleaner fingerprint management strategy. One thing I’ve found helpful is to implement logging for proxy connection attempts and failures, which can significantly aid in debugging during development. If you're looking for extra hands to explore further optimizations in this area, I'd be happy to discuss a paid collaboration! What specific challenges have you faced with managing multiple browser contexts?