DEV Community: RoamProxy

HTTP 200 Is Not Success: Write a Contract Per Host

RoamProxy — Sat, 12 Sep 2026 22:39:08 +0000

The last two posts were about spending an IP well: pace the requests, and hold a sticky session exactly as long as one unit of work. Both assume something I never stated out loud — that you can tell when a request worked.

Most pipelines can't. They check the status code, see 200, and move on. And 200 is exactly what a target returns when it has decided to stop giving you real answers.

The failure mode nobody alerts on

Hard failures are easy. A 403, a connection reset, a captcha page that throws on parse — your retry logic catches those, your dashboard turns red, you go look.

Soft degradation is the expensive one:

The page returns 200 with a challenge shell and none of the content.
The search endpoint returns 200 with a well-formed, empty result list.
The API returns 200 with a cached response from six hours ago.
The listing returns 200 with 3 rows where it used to return 50.

Every one of those sails through a status-code check. Your scraper reports a clean run. Your dataset quietly rots, and you find out days later when someone downstream asks why a whole region went empty.

Worse, you keep spending the IP. The target has already decided you're not a person; you just haven't been told. Every request after that point is paid for and worthless — and it's building exactly the kind of profile that turns a soft block into a hard one.

Write the contract before you write the parser

For each host you scrape, write down what a good response looks like. Three assertions is usually enough:

Shape — a structural element that only exists on a real page. Not <title>, which survives on challenge pages. Pick the container that holds the thing you came for: the results table, the JSON key with the records in it.
Volume — a plausible range, not a minimum of one. If a category page has returned between 20 and 60 rows every day for a month, then 3 rows is a failure even though it parsed fine.
Identity — proof the response is about what you asked for. The canonical URL, the product ID, the echoed query string. This is what catches stale caches and silent redirects to a generic page.

A response that fails any of the three is a failure, no matter what the status line says.

from dataclasses import dataclass
from typing import Callable

@dataclass
class Contract:
    host: str
    shape: Callable[[dict], bool]      # the container exists
    volume: range                      # plausible record count
    identity: Callable[[dict, str], bool]  # response matches request

    def check(self, parsed: dict, requested_id: str) -> str | None:
        if not self.shape(parsed):
            return "shape"
        if len(parsed.get("records", [])) not in self.volume:
            return "volume"
        if not self.identity(parsed, requested_id):
            return "identity"
        return None  # passed

Writing this down takes ten minutes per host and changes what "success rate" means in your metrics. Most people discover their real success rate was never the number on the dashboard.

The canary tells you whose fault it is

A contract tells you a response was bad. It doesn't tell you why, and the difference decides what you do next:

If that page is broken, skip it and move on. Rotating your IP is a waste.
If you are blocked, moving on is the worst possible move. Every subsequent request digs the hole deeper.

You separate the two with a canary: one URL per host whose correct answer you already know and that essentially never changes. A stable category page, a documented API example, an "about" endpoint with fixed fields. Run it through the same session, same headers, same exit IP as your real traffic, and check it against the same contract.

Now the diagnosis is mechanical:

Canary	Target page	Diagnosis	Action
pass	pass	healthy	continue
pass	fail	that page is genuinely bad or changed	log it, skip, don't rotate
fail	fail	you're degraded	stop the unit of work, rotate at the seam
fail	pass	flaky canary — fix your canary	investigate offline

That third row is the one that saves money. A canary failure is the earliest honest signal that your session has stopped being trusted — usually well before anything returns a 403.

Run the canary like a visitor, not like a monitor

Two rules, both inherited from the earlier posts.

Don't run it per request. One canary per unit of work — at the start, so you don't waste a journey, and once more before you commit expensive writes. That's a 1–2% overhead, not 50%.

Don't run it on a timer. A request to the same URL every 60.0 seconds, forever, from an IP that otherwise browses like a human is the single most machine-shaped pattern you can emit. The canary exists to detect that you look like a bot; it must not be the reason you do. Same jitter, same pacing, same session as everything else.

What this buys you

The contract turns "did it 200?" into "did I get what I came for?". The canary turns a bad response into an attributable one. Together they change the failure from silent, days-long data rot into an event you can act on in the same minute it happens.

And it changes what proxy quality even means to you. Once every response is checked against a contract, you stop arguing about pool sizes and start measuring the thing that matters: how many units of work complete, per IP, before the canary goes quiet. That number is comparable across providers, across weeks, and across your own config changes — which is the whole point.

I work on Roam, residential and static residential proxies billed per GB. The measurement habits above are how we evaluate our own pools; they apply no matter whose IPs you're renting.

How Long Should You Hold a Sticky Session?

RoamProxy — Sat, 05 Sep 2026 17:11:53 +0000

The last post argued that a residential IP has to behave like a household — cap the concurrency, pace the rate, jitter the spacing. A natural follow-up question landed in my inbox: "OK, so I'm holding one sticky session and pacing it politely. How long am I supposed to keep it?"

It's the right question, because both of the obvious answers are wrong.

Rotate every request, and you throw away everything a session accumulates: cookies issued against one IP, a TLS session the server has seen before, the slow-built trust of a consistent story. You also walk straight back into the geo-mismatch problem — a login minted in one country and replayed from another is exactly the "mixed-currency" signal I keep writing about.

Hold one session forever, and two clocks are ticking against you. The first is the behavior budget from the last post: every request you send accumulates on that IP, and the longer you hold it, the more implausible its "household" story gets. The second clock isn't yours at all.

The session can die without you

A residential exit IP is a real device on a real home connection. That device goes to sleep. Its DHCP lease renews. Someone reboots the router. When that happens, your "sticky" session doesn't politely tell you — depending on the provider, the session either dies or is quietly re-pinned to a different device.

In practice, most residential sticky sessions are only reliably stable on the order of minutes, not hours. Some survive much longer; you just can't build on it. If your pipeline assumes one IP for a two-hour crawl, you haven't designed a session strategy — you've bought a lottery ticket.

The corollary: verify, don't assume. A cheap periodic origin check (an httpbin-style echo through the same session) tells you whether the IP under your session silently changed. If it did, everything from the last post applies to a new IP now — and the cookies you're carrying were minted on the old one. That mismatch is worth knowing about before the target notices it for you.

Rotate between units of work, not inside them

The useful framing isn't a number of minutes. It's this:

One sticky session should live exactly as long as one unit of work — and rotate at the seam.

A unit of work is one coherent journey a single visitor could plausibly have: log in, walk a pagination trail, extract a batch, leave. Everything inside that journey shares state — cookies, referer chain, server-side session — so it must share an IP. Splitting it across exits is self-sabotage.

Between journeys, there's nothing connecting you. That's the free rotation point. You pay no continuity cost, and you reset the per-IP behavior budget to zero.

This immediately gives you sizing rules that are about your workload, not magic numbers:

If your unit of work is 30 seconds of requests, don't hold the session for 10 minutes "to be efficient." You're accumulating budget for nothing.
If your unit of work takes 40 minutes, that's longer than a residential session reliably lives. Don't reach for a longer TTL — shrink the unit of work. Checkpoint state so a journey can resume, or split the trail into chunks that each fit comfortably inside a session lifetime.
If your units of work are independent, run them on different concurrent sessions rather than queuing them through one. Five journeys through five IPs looks like five visitors. Five journeys back-to-back through one IP looks like a shift worker.

When to rotate early

Holding a session to the end of its unit of work is the default, not a vow. Three signals justify cutting it short:

1. An IP-scoped block. I wrote a whole post on diagnosing block scope; the short version is that rotation only fixes blocks that are actually pinned to the IP. If your probes say the IP itself is burned — fresh session, clean headers, still walled — finish the journey later on a new exit. Retrying through the burned one just documents your persistence.

2. The IP changed under you. Your origin check came back different. The sticky abstraction already broke; carrying on pretends it didn't. Restart the unit of work cleanly on the new session rather than dragging old-IP cookies onto a new-IP story.

3. Degradation, not denial. Responses through this session are getting slower, challenge pages more frequent, while a control request through a fresh session is fine. You're being softly bucketed. The polite exit is to finish the current page, not the current journey, and re-enter from a new IP with the pacing lessons applied.

Note what's not on the list: a timer. "Rotate every N minutes" is the sticky-session equivalent of metronomic request spacing — a rule that ignores what's actually happening. Every rotation decision above is driven by an observable, and all three observables are cheap to collect.

The shape of the code

None of this needs a framework. It's a small wrapper that owns three things — a session name, a birth-time origin check, and a rotate-at-seam method:

import time, uuid, httpx

GATE = "http://user-country-us-session-{sid}:pass@gateway:7777"

class StickyUnit:
    def __init__(self):
        self.sid = uuid.uuid4().hex[:8]
        self.client = httpx.Client(proxy=GATE.format(sid=self.sid))
        self.origin = self._origin()

    def _origin(self):
        return self.client.get("https://httpbin.org/ip", timeout=10).json()["origin"]

    def drifted(self):
        return self._origin() != self.origin

    def close(self):
        self.client.close()

def run_journey(urls):
    unit = StickyUnit()
    try:
        for i, url in enumerate(urls):
            if i and i % 10 == 0 and unit.drifted():
                raise RuntimeError("exit IP changed mid-journey; restart unit")
            yield unit.client.get(url)
    finally:
        unit.close()  # next journey constructs a fresh unit = fresh seam

The session ID is random per unit of work, the drift check is periodic and cheap, and rotation isn't an event you schedule — it's just what happens between constructors.

Full runnable versions of this and the earlier posts' probes live in our examples repo: https://github.com/roamproxy/proxy-examples

I work on Roam, a pay-as-you-go proxy network. Sticky sessions there are named exactly like the snippet above — the session suffix pins the exit until you change it, which makes "rotate at the seam" a one-line decision.

Your Residential IP Is Clean. Your Request Rate Gives You Away.

RoamProxy — Tue, 01 Sep 2026 04:49:28 +0000

The last post I wrote here was about geo signals — how a German exit IP means nothing if your Accept-Language, timezone and locale all still say "US developer laptop." A reader pushed back with a fair point: they'd fixed all of that, every signal agreed, and they were still getting throttled through clean residential IPs. So what was left?

The exit IP was clean. Every static signal agreed. What gave them away was how the IP behaved — the rate, the concurrency, and the timing of the requests coming out of it. A residential IP doesn't just claim to be a household. It has to act like one.

An IP is a story about a person

When a site sees a residential IP, the implicit claim is "there is a human on a home connection here." Humans, and the households behind them, have a shape:

They open a few tabs, not forty.
They read a page for some seconds before clicking the next one.
They go quiet for hours and then come back.
Their requests arrive one after another with human gaps, not in perfectly-spaced 200ms intervals and not in a thundering burst.

None of this is exotic bot detection. It's arithmetic the server was already doing for capacity planning. A single residential IP that sustains 40 concurrent connections and 30 requests a second is not a suspicious fingerprint — it's a physically implausible household. The IP is clean. The story it's telling is not.

The three numbers that matter per IP

Forget global rate limits for a second. The unit that matters is per exit IP, because that's the unit the target attributes behavior to.

1. Concurrency. How many requests are in flight through one IP at once. This is the one people get most wrong, because async frameworks make it invisible. asyncio.gather over 500 URLs with one sticky session doesn't feel like 500 concurrent requests when you write it — but that's exactly what leaves the IP. Cap it. A real browser rarely has more than ~6 connections open to one host.

2. Rate. Requests per second per IP, sustained. The safe number depends entirely on the target, but the useful mental model is "how fast would one interested person click?" — not "how fast can my machine send." For most sites, sustained single-digit requests per second per IP is already faster than any human.

3. Spacing. The gap between requests. Two scrapers can have the same average rate and look completely different: one sends a request exactly every 500ms, the other sends them with jittered human-ish gaps. Metronomic timing is itself a signal — real traffic is bursty and irregular. A little randomization in the delay does more than shaving the average rate.

The trap is optimizing the first two and ignoring the third. A perfectly rate-limited scraper that fires on a clockwork interval still reads as automation, because nothing that has a human in the loop is that regular.

Why this fails softly (again)

If you've read the earlier posts you know where this goes. Cross the rate a target actually enforces and you get a 429 — loud, obvious, easy to back off from. But most sites don't hard-limit at the first sign; they degrade you. Slower responses. More challenges. Shorter sticky-session lifetimes. You get quietly moved into a bucket for traffic that's probably automated, and you stay there. Your success rate is 10% lower than it should be and you blame the proxies.

And rotating the IP doesn't buy back what you spent. If you burn an IP by hammering it, the next IP gets the same treatment the moment it behaves the same way. You're not running out of IPs; you're running the same implausible household story from a new address.

The fix is a budget, not a sleep

The instinct is to sprinkle time.sleep() around and call it rate limiting. That controls the average and nothing else. What you actually want is a small budget enforced per IP: a cap on in-flight requests, a token-bucket rate with jitter, and a rule that when one IP starts getting slow or challenged, you pace it down before you rotate it away.

Here's the shape of it — a per-IP limiter you acquire before every request through that exit:

import asyncio, random, time

class IPBudget:
    """Per-exit-IP pacing: concurrency cap + jittered token-bucket rate."""
    def __init__(self, max_concurrency=4, rps=2.0, jitter=0.4):
        self._sem = asyncio.Semaphore(max_concurrency)
        self._min_gap = 1.0 / rps
        self._jitter = jitter
        self._next_at = 0.0
        self._lock = asyncio.Lock()

    async def __aenter__(self):
        await self._sem.acquire()
        async with self._lock:
            now = time.monotonic()
            wait = max(0.0, self._next_at - now)
            gap = self._min_gap * (1 + random.uniform(0, self._jitter))
            self._next_at = max(now, self._next_at) + gap
        if wait:
            await asyncio.sleep(wait)
        return self

    async def __aexit__(self, *exc):
        self._sem.release()

One IPBudget per exit IP, not one global one. If you hold a sticky residential session, that session gets its own budget for its whole life; when you rotate, the new IP starts fresh. The concurrency cap keeps gather honest, the token bucket keeps the sustained rate human, and the jitter keeps the spacing from looking like a machine.

A runnable version — with the per-host bookkeeping and a "slow down before you rotate" hook — is in our examples repo: github.com/roamproxy/proxy-examples.

Putting it with the rest

Across these posts the theme keeps coming back: a clean IP is table stakes, and it's the cheapest part of not getting blocked. The IP has to look like a household, the client's geo signals have to agree with it, and the behavior coming out of it has to be physically plausible for one person on one connection. Get all three right and the exit IP stops being the thing anyone notices — which is the whole point.

We run Roam, a pay-as-you-go residential/datacenter/mobile proxy network — real home-broadband exits, per-GB pricing, balance never expires. The examples above run against any proxy; the repo is provider-agnostic.

Your Proxy Is in Germany. The Rest of Your Client Isn't.

RoamProxy — Tue, 25 Aug 2026 22:18:26 +0000

A reader on an earlier post described a scraper that was pulling Amazon listings through a rotating residential proxy and getting prices back in three different currencies inside one run. The thread treated it as a data-cleaning problem — normalise the currency, move on. I think it was the earliest visible symptom of something that gets scrapers blocked a lot more quietly than any IP list does: the exit IP said one country, and everything else about the client said another.

Geo-targeting a proxy is one line. country-de in the username, done. But the target isn't looking at one signal. It's looking at half a dozen, and if the other five still say "US developer laptop," the German IP doesn't make you look German. It makes you look like a US developer laptop using a German proxy, which is a much more specific — and much more suspicious — thing to look like.

The signals that have to agree

Here's what a target can read, without any JavaScript trickery, from a single request that arrives via a German residential exit.

Accept-Language. Python's requests sends nothing. httpx sends nothing. Playwright sends en-US,en;q=0.9 by default. A real German Chrome sends de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7. This header alone splits "German visitor" from "someone routing through Germany" on the first request, before any fingerprinting happens.

Timezone. If there's JS in the picture, Intl.DateTimeFormat().resolvedOptions().timeZone is one call. A headless browser inherits the host machine's zone. Your CI runner is in UTC; your laptop is in America/Los_Angeles; your exit is in Frankfurt. Two of those three will disagree with the IP, and the disagreement is a stronger signal than any single one of them.

Locale-derived formatting. navigator.language, navigator.languages, the date and number formats the page's own scripts produce when they run in your context. These all come from the browser's locale, not from the IP.

The currency and store you get served. This one's the tell that actually shows up in your data. Big e-commerce sites resolve your storefront from a combination of IP geolocation, Accept-Language, and any prior locale cookie. When those disagree, different code paths win on different page types — the search page keys on IP, the product page keys on the cookie, the cart keys on the account. That's how one run ends up with EUR, USD and GBP in the same output file. The mixed currencies aren't noise; they're a log of which signal each endpoint trusted.

DNS resolution location. Easy to forget. With an HTTP proxy, the proxy resolves the hostname and you're fine. With a SOCKS5 proxy configured as socks5:// rather than socks5h://, your machine resolves the hostname, so a CDN sees a DNS query from California and a TCP connection from Frankfurt. That's not a fingerprint most sites act on, but it does mean you get routed to the wrong edge, which is its own source of inconsistency (different A/B buckets, different cache, sometimes different content).

TLS session resumption. If you reuse a client across exits, the TLS session ticket from the previous connection can be presented on the next one. A session ticket issued to an IP in Ohio, resumed from Frankfurt, is a small oddity — but it's an oddity the server sees at the handshake, before your first byte of HTTP.

Why it's worse than a bad IP

An IP on a blocklist fails loudly. You get a 403, you notice, you deal with it.

Inconsistent geo fails softly. You get served a slightly different page. You land in a "we're not sure where you are" bucket that gets more challenges and fewer cached responses. Sticky sessions expire faster. A/B tests assign you to the control group. Nothing tells you this is happening; your success rate is just 8% lower than it should be, permanently, and the data has small irregularities you'll attribute to the target.

The other reason it's worse: rotating the IP doesn't fix it. It can't. The IP was the one signal that was right. If you read the previous post on block scopes, this is the fingerprint-scoped case — it fails identically from every exit because the exit isn't the problem.

Make the client agree with the exit

The fix is boring: derive every locale-ish setting from the exit's country, in one place, and never let anything else set them.

# geo_profile.py — one source of truth per exit country
import httpx

PROFILES = {
    "de": {"lang": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
           "tz": "Europe/Berlin", "locale": "de-DE"},
    "gb": {"lang": "en-GB,en;q=0.9",
           "tz": "Europe/London", "locale": "en-GB"},
    "us": {"lang": "en-US,en;q=0.9",
           "tz": "America/New_York", "locale": "en-US"},
    "jp": {"lang": "ja-JP,ja;q=0.9,en-US;q=0.8,en;q=0.7",
           "tz": "Asia/Tokyo", "locale": "ja-JP"},
}

def client_for(country: str, proxy_url: str) -> httpx.Client:
    p = PROFILES[country]
    return httpx.Client(
        proxy=proxy_url,
        headers={"Accept-Language": p["lang"]},
        http2=True, timeout=20,
    )

def playwright_context_kwargs(country: str) -> dict:
    p = PROFILES[country]
    return {"locale": p["locale"], "timezone_id": p["tz"],
            "extra_http_headers": {"Accept-Language": p["lang"]}}

For Playwright that's browser.new_context(proxy=..., **playwright_context_kwargs("de")) — locale and timezone_id are first-class context options and they fix navigator.language, Intl, and the header in one go. For plain HTTP clients it's just the header, plus using socks5h:// if you're on SOCKS.

Verify it before you trust it

Configuration you haven't checked is a guess. This is the audit I run when a new exit or a new client build goes into rotation — it asks a geo-echo endpoint what the server sees and compares that to what the client intends:

# geo_audit.py
import json, sys
import httpx

ECHO = "https://httpbin.org/anything"   # any endpoint that reflects headers + origin

def audit(client: httpx.Client, expect_country: str, expect_lang_prefix: str):
    r = client.get(ECHO)
    seen = r.json()
    origin = seen["origin"].split(",")[0]
    lang = seen["headers"].get("Accept-Language", "")
    geo = httpx.get(f"https://ipinfo.io/{origin}/json", timeout=10).json()
    problems = []
    if geo.get("country", "").lower() != expect_country:
        problems.append(f"exit country {geo.get('country')} != {expect_country}")
    if not lang.lower().startswith(expect_lang_prefix):
        problems.append(f"Accept-Language {lang!r} does not start with {expect_lang_prefix!r}")
    return origin, geo.get("country"), lang, problems

if __name__ == "__main__":
    from geo_profile import client_for
    country, proxy = sys.argv[1], sys.argv[2]
    with client_for(country, proxy) as c:
        origin, cc, lang, problems = audit(c, country, country if country != "gb" else "en-gb")
    print(json.dumps({"exit": origin, "country": cc, "lang": lang,
                      "ok": not problems, "problems": problems}, indent=2))

Run it once per country you target and once per client build. If ok is false, the request never reaches the real target — a client that disagrees with its own exit is not allowed into the pool. The audit takes about two seconds and it has caught, in order: a CI image that reset TZ=UTC after the context was created, a socks5:// that should have been socks5h://, and a "German" profile that was still sending en-US because the header was set on the session and then overwritten per-request by a helper someone added later.

The short version

An IP is one vote. Accept-Language, timezone, locale, DNS origin and TLS state are five more, and a target that geolocates at all is counting them. Pick the exit country, derive everything else from it, and assert the agreement before the first real request. Mixed currencies in your output aren't a cleaning job — they're the target telling you which of your signals it didn't believe.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.

Before You Rotate the IP, Find Out What Actually Got Blocked

RoamProxy — Sun, 23 Aug 2026 20:03:27 +0000

Something in your pipeline just started returning 403s, or a challenge page, or an empty 200 where there used to be JSON. The reflex — in every tutorial, every Stack Overflow answer, every proxy vendor's docs — is to rotate the IP.

Most of the time that's the wrong first move, and it's expensive in a way that isn't obvious: rotating throws away everything the old identity had accumulated. Session cookies, a solved challenge, whatever internal trust score the target had assigned you after 40 well-behaved requests. If the IP wasn't the thing that got flagged, you paid all of that and fixed nothing — and you now have a second identity showing the same pattern, which is a worse signal than the first one alone.

The useful question isn't "how do I get unblocked." It's what scope did the block apply to. There are five, they're cheap to tell apart, and the right response is different for each.

The five scopes

Request-scoped. One specific request was rejected — a missing header, a bad referer, an expired token in a query param, a URL that requires a POST. The next request works fine. This is the most common "block" and it isn't a block at all.

Session-scoped. Your cookie jar / TLS session got flagged. A challenge interstitial that keeps re-serving, a cf_clearance that went stale, a rate limiter keyed on session ID. Everything else about you is fine.

Fingerprint-scoped. Your client is being rejected for what it is, not where it's from. Automation-flavoured TLS handshake, a headless-shaped JS environment, a header order no real browser emits. The tell: it fails instantly and identically no matter how long you wait or what IP you use.

IP-scoped. The address is on a list. Every request from it fails, from every session, with every client, immediately.

Account-scoped. You're logged in and the account is limited. Rotating the IP here is actively harmful — a logged-in account that suddenly appears from a new city is a much stronger abuse signal than one that stays put.

Four probes that separate them

You don't need to guess. Four requests, under a minute, and the answer falls out:

Same IP, fresh session (new cookie jar, same client, same exit).
Same IP, different client (curl if you were using a browser, or vice versa — this varies the TLS/JA3 and header order).
Different IP, same cookies.
Different IP, everything fresh.

import httpx

def probe(url, *, proxy=None, cookies=None, headers=None, http2=True):
    """One probe. Returns (status, len(body), challenged)."""
    with httpx.Client(proxy=proxy, cookies=cookies or {},
                      headers=headers or {}, http2=http2,
                      timeout=20, follow_redirects=True) as c:
        r = c.get(url)
    body = r.text
    challenged = any(s in body.lower() for s in
                     ("just a moment", "un momento", "checking your browser",
                      "attention required", "verify you are human"))
    return r.status_code, len(body), challenged


def diagnose(url, old_proxy, new_proxy, old_cookies, browser_headers):
    p = {}
    p["fresh_session_same_ip"] = probe(url, proxy=old_proxy, headers=browser_headers)
    p["diff_client_same_ip"]   = probe(url, proxy=old_proxy, http2=False, headers={})
    p["old_cookies_new_ip"]    = probe(url, proxy=new_proxy, cookies=old_cookies,
                                       headers=browser_headers)
    p["all_fresh_new_ip"]      = probe(url, proxy=new_proxy, headers=browser_headers)
    return p

Read it like this:

What happens	Scope	What to actually change
Fresh session on the same IP works	Session	Drop the cookie jar. Keep the IP.
Same IP fails, new IP with old cookies also fails	Session or account	Cookies. Not the IP.
Everything fails instantly from both IPs, but a plain `curl` works	Fingerprint	Your client, not your route.
Every request from the old IP fails, everything from the new one works	IP	Rotate — this is the one case where it's right.
Only some URLs fail, others are fine	Request	Fix the headers/method/token.
Fails only while logged in	Account	Slow down. Do not move the account to a new IP.

The row people get wrong is the second one. A stale session on a target that keys its rate limiter to a session ID looks exactly like an IP ban from the inside — every request fails — and rotating the IP "fixes" it, because the new exit also comes with a new cookie jar. So you conclude the IP was burned when the cookie was, and you burn an address per incident forever after.

The waiting case

There's a fifth outcome the probes will show you and most people won't believe: temporary, and the correct action is nothing.

I hit this on Medium's Cloudflare edge a couple of weeks ago. Mid-session, after an import that touched several endpoints in quick succession, every page turned into an interstitial. Same tab, same everything, for something like half an hour — reloading did nothing, four attempts did nothing. The exit IP was verifiably unchanged the whole time (I checked, precisely because rotating was the tempting move and that session had a login I didn't want to re-establish). Then it cleared on its own and the next request went through normally.

If I'd rotated at minute two I would have concluded the IP was blocked, thrown away a good address and a live session, and learned the wrong lesson permanently. The signal that it was temporary was there: the challenge was serving (200 with a challenge body), not refusing (403 with nothing). A challenge is an invitation to prove yourself. A 403 with an empty body is a door. They deserve opposite responses.

Change the cheapest thing first

Order your remediations by what they cost you if you're wrong:

Headers / request shape — free.
Cookie jar — cheap, unless you're logged in.
Wait — costs only time, and time is what half of these need.
Client fingerprint — moderate; you're rebuilding a client, not an identity.
Exit IP — expensive. You lose the session, the accumulated trust, and (if you were logged in) you hand the target a "this account moved cities" event.
Account — most expensive, obviously.

Rotation belongs at position five for a reason. It's the loudest change you can make and it's the one people reach for first, because it's the one the tooling makes easiest.

A practical version of this: keep the diagnostic in your codebase, not in your head. When a run starts failing, diagnose() runs before any remediation logic does, logs which scope it found, and only the IP-scoped branch is allowed to call rotate(). It takes an afternoon to write and it stops the reflex from making decisions for you.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.

Your Scraper's Retry Logic Is Probably Making the Block Worse

RoamProxy — Fri, 21 Aug 2026 04:43:36 +0000

Most scrapers have one retry policy: catch the exception, sleep, try again, give up after N attempts. It's the default in every tutorial, it's what tenacity gives you in three lines, and on a healthy target it's fine.

It stops being fine the moment the failures stop being random. A rate limit, a hard block, a TLS reset and a slow origin all arrive as "the request failed," and treating them identically means you spend your entire run budget hammering something that was never going to open — while the one failure that would have succeeded on retry gets three attempts and then gets dropped.

Here's the taxonomy I actually use, and the part that took the longest to learn: the counter matters more than the backoff.

The four failures, and what each one means

429 with a Retry-After header. The friendliest failure you will ever get. The server is telling you exactly what it wants. Honour it — sleep the stated duration, then continue on the same connection and the same identity. Do not rotate anything. Rotating here is how a temporary throttle turns into a fingerprinted pattern.

429 with no header. The server wants you slower but won't say by how much. Exponential backoff with jitter is genuinely the right tool. Start higher than you think — 5 seconds, not 0.5.

403 / 401 / a challenge page. This is not a retryable condition. Something about the request was rejected, and repeating it byte-for-byte will be rejected the same way. Every retry here is pure waste, and worse, it's the pattern that gets a durable block instead of a temporary one. Short-circuit straight to whatever your fallback is — different identity, different route, or shelve the URL and move on.

Connection reset / timeout / TLS handshake failure. This one is retryable, and it's the one people wrongly lump in with 403. Transport failures are frequently just noise. Retry immediately, twice, then treat it as a hard failure.

The distinction that matters: HTTP-level rejections tell you something about your request. Transport-level failures usually tell you nothing at all.

import random, time
import httpx

RETRYABLE_STATUS = {429, 500, 502, 503, 504}
HARD_BLOCK = {401, 403, 407, 451}

def classify(exc_or_resp):
    if isinstance(exc_or_resp, httpx.HTTPError):
        return "transport"          # retry fast, twice
    code = exc_or_resp.status_code
    if code in HARD_BLOCK:
        return "blocked"            # do not retry, change something
    if code == 429:
        return "throttled"
    if code in RETRYABLE_STATUS:
        return "server"
    return "ok" if code < 400 else "fatal"

def sleep_for(kind, attempt, resp=None):
    if kind == "throttled" and resp is not None:
        ra = resp.headers.get("retry-after")
        if ra and ra.isdigit():
            return int(ra)
        return min(60, 5 * (2 ** attempt)) + random.uniform(0, 2)
    if kind == "server":
        return min(30, 2 ** attempt) + random.uniform(0, 1)
    if kind == "transport":
        return 0.5
    return 0

The counter is the part everyone skips

Backoff handles the failure you already got. It does nothing about the one you're about to get, and on a lot of targets the limit isn't time-based at all — it's a volume budget. You get N requests, and the block arrives on N+1 regardless of how politely you spaced them.

I learned this on a regional grocery chain's stock checker. It would 429 you politely for about an hour, and then flip to a permanent 403 once you crossed some invisible total. Every backoff strategy in the world is useless against that, because the thing being counted isn't your rate — it's your volume.

The fix is a rolling per-host counter that you enforce on yourself, set well below whatever number killed the last run:

import time
from collections import defaultdict, deque

class HostBudget:
    """Rolling request counter per host. Refuses before the server does."""

    def __init__(self, max_requests=200, window_seconds=3600):
        self.max = max_requests
        self.window = window_seconds
        self.hits = defaultdict(deque)

    def allow(self, host):
        now = time.time()
        q = self.hits[host]
        while q and now - q[0] > self.window:
            q.popleft()
        if len(q) >= self.max:
            return False
        q.append(now)
        return True

    def spent(self, host):
        return len(self.hits[host])

Once that exists, a 403 stops being a signal you react to and becomes a signal you record: whatever the counter said when the block landed is your new ceiling for that host, minus a healthy margin. Write it to disk. The next run starts already knowing.

The reason the 403 is a bad signal on its own is timing. By the time you see it, you're usually already benched — the block is applied, the cooldown clock started without telling you, and nothing you do in the next hour matters. The counter is the only thing that fires before the damage.

Putting it together

def fetch(client, url, budget, max_attempts=4):
    host = httpx.URL(url).host
    if not budget.allow(host):
        raise RuntimeError(f"self-imposed budget hit for {host} ({budget.spent(host)})")

    for attempt in range(max_attempts):
        try:
            resp = client.get(url, timeout=20)
        except httpx.HTTPError as exc:
            kind = classify(exc)
            if attempt >= 2:
                raise
            time.sleep(sleep_for(kind, attempt))
            continue

        kind = classify(resp)
        if kind == "ok":
            return resp
        if kind == "blocked":
            # record the ceiling, then hand off — retrying byte-for-byte is waste
            raise BlockedError(host, budget.spent(host))
        if kind == "fatal":
            resp.raise_for_status()
        time.sleep(sleep_for(kind, attempt, resp))

    raise RuntimeError(f"exhausted attempts for {url}")

Three rules, in order of how much run time they save:

Never retry a hard block unchanged. It cannot succeed and it makes the block stickier.
Count your own requests per host and stop before the server stops you.
Retry transport errors fast and cheaply. They're usually nothing.

The retry loop is the least glamorous part of a scraper and the one that quietly decides whether a run finishes. Worth twenty minutes.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.

Stop Reaching for a Headless Browser to Scrape Documentation Sites

RoamProxy — Tue, 11 Aug 2026 23:45:28 +0000

Every few days someone in a scraping forum asks a version of the same question: "I'm collecting documentation text for an AI tool, but the pages render with JavaScript. What's the lightest way to get the content?"

The answers are always the same — open DevTools, check the Network tab, find the XHR. That advice is correct, and for documentation sites specifically it's usually unnecessary work.

Documentation sites are not arbitrary web apps. They're overwhelmingly built by a handful of static site generators, and those generators leave the content sitting in predictable places. Three routes cover most of what you'll hit, and none of them need a browser.

Route 1: the JSON is already in the HTML

Next.js-based docs (which includes a large share of company developer portals) embed the full page payload in a __NEXT_DATA__ script tag. It's in the initial HTML response — no JavaScript execution needed.

import json, re, httpx

html = httpx.get(url, follow_redirects=True).text
m = re.search(r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', html, re.S)
if m:
    data = json.loads(m.group(1))
    # content location varies by site; dump the tree once and look around
    print(json.dumps(data["props"]["pageProps"], indent=2)[:2000])

Docusaurus (the other big one) doesn't use __NEXT_DATA__, but it pre-renders the full article text into the static HTML. A plain httpx.get plus a main or article selector gets you everything. The "JavaScript rendering" you see in the browser is hydration for navigation and search — the prose was already there in the initial response.

Thirty-second check: curl -s <url> | grep -c "some sentence you can see on the page". If that returns 1 or more, the content is in the HTML and you're done. This one check resolves the majority of "it's JavaScript-rendered" cases, because people judge from DevTools' Elements panel — which shows the hydrated DOM, not what the server actually sent.

Route 2: the markdown is in a public repo

Most open-source project documentation is markdown in the same repository as the code, usually under docs/. Scraping the rendered HTML means fetching pages one at a time, parsing them, and stripping navigation chrome you didn't want. Cloning gets you the clean source in one shot:

git clone --depth 1 --filter=blob:none --sparse https://github.com/org/project
cd project && git sparse-checkout set docs

You get the original markdown — headings intact, code fences intact, no nav sidebars, no cookie banners, no per-page rate limiting. For a RAG pipeline this is strictly better input than parsed HTML, and it's one request instead of several hundred.

Worth saying plainly: check the license before you ingest. Documentation is frequently licensed separately from the code, and "the repo is public" is not the same as "you may redistribute this."

Route 3: the site publishes a text endpoint

A growing number of documentation hosts expose plain-text views:

/llms.txt — an emerging convention where a site publishes a curated, plain-text map of itself specifically for LLM consumption. Fast-moving developer-tool companies have adopted it quickly. Always worth one request.
/sitemap.xml — not text content, but it gives you the complete URL list without crawling, which means you never have to discover pages by following links.
ReadTheDocs projects usually offer downloadable HTML and often PDF/ePub builds of the entire docs set from the version menu. One artifact, complete content.

Picking a route in under a minute

Signal	Route
`curl` output contains visible page text	Parse the static HTML — done
`__NEXT_DATA__` in the HTML	Extract and walk the JSON
Public repo with a `docs/` directory	Sparse-clone the markdown
`/llms.txt` returns 200	Start there
ReadTheDocs / GitBook host	Look for the download build
None of the above	Now open DevTools

When you actually do need a browser

Some cases are genuinely dynamic, and it's worth knowing them so you don't over-apply the above:

Docs behind authentication where the session is established by client-side JavaScript.
Content assembled from several API calls at runtime with no single payload — more common in interactive API explorers than in prose documentation.
Sites that gate on a JavaScript challenge before serving anything, where the challenge is the point.

For a few hundred pages of prose, though, these are the exception. The default assumption should be that the text is already reachable, and the browser is the fallback.

The part that actually costs you

The reason this matters isn't purity — it's that headless browsers change the shape of your project. You go from a script anyone can run to a pipeline with a browser binary, a memory ceiling, per-page startup cost, and a new class of flaky failures that only reproduce sometimes. On a few hundred documentation pages, Route 1 or Route 2 typically finishes before a browser-based run has finished launching.

Check whether the content is already sitting there in plain text. Most of the time, it is.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.

We Tried to Beat Cloudflare With Proxies and TLS Tricks. Here's What Actually Failed.

RoamProxy — Tue, 28 Jul 2026 04:41:54 +0000

Most advice about scraping Cloudflare-fronted sites is asserted, not measured. "Use residential IPs." "Match Chrome's JA3." People repeat these like folklore. We run a proxy network, so we had the tooling to actually test them — and the results contradicted the two most common pieces of advice.

Here's what we ran, and what failed.

The test

Two variables, four combinations:

Exit IP: a Psychz datacenter address vs a Frontier Communications residential line (both US).
TLS stack: plain Python requests vs curl_cffi impersonating Chrome.

Seven Cloudflare-fronted sites, two passes each — 28 requests total, US exits, 27 July 2026. A "pass" required HTTP 200 and a body without challenge markers. A 200 that returns "Just a moment…" is a block wearing a success code, and it does not count.

The numbers

Combination	Passed
Datacenter IP + default Python TLS	0 / 7
Datacenter IP + Chrome-impersonated TLS	0 / 7
Residential IP + default Python TLS	0 / 7
Residential IP + Chrome-impersonated TLS	0 / 7

Every combination failed identically. Two things in that table contradict advice you'll read everywhere.

The residential IP bought us nothing

Not a single site treated the Frontier residential exit differently from the Psychz datacenter one. That does not mean IP reputation is a myth — it means these sites never got as far as judging our IP. A request that fails the bot-management check is refused on other grounds first.

IP quality decides outcomes in the middle of the difficulty curve. At the hard end it isn't the binding constraint, and buying better IPs won't move it. If you're stuck at 403 on a genuinely aggressive Cloudflare config, a residential proxy is not the missing piece.

Chrome TLS impersonation didn't flip a single site either

curl_cffi with impersonate="chrome" is great tooling, but on these targets it changed nothing. The sites require executing the JavaScript challenge, which no HTTP client does, no matter how convincing its TLS handshake. If you're stuck, adding a TLS library isn't the fix — you need something that runs JS.

For calibration: the same script sailed through Cloudflare-fronted sites that aren't running aggressive bot mode, on every combination including plain Python from a bare datacenter host. "Protected by Cloudflare" spans an enormous range of strictness. Establish which end your target sits at before spending money on tooling.

Bonus: stop chasing a specific JA3 hash

While measuring, we checked what fingerprint our own requests presented. Three consecutive curl_cffi requests with the same impersonate="chrome" setting, from the same host, produced three different JA3 hashes (a0052cf3…, 0899dce7…, d2de58db…).

That's not a bug. Real Chrome shuffles TLS extensions and injects GREASE values, so its JA3 changes per connection — and a good impersonation library reproduces that. Any guide telling you to match one specific JA3 string is describing a browser that stopped existing years ago.

What did stay identical across all our requests — direct, via residential proxy, and via datacenter proxy — was the HTTP/2 fingerprint (52d84b11…), derived from the SETTINGS frame, header table size, window size and pseudo-header order. It's far more stable than JA3, which is exactly why it's worth more to the people fingerprinting you. Check both when you debug a silent block; we used tls.browserleaks.com/json.

So what actually works?

Match the tool to where your target sits on the curve:

Soft Cloudflare (rate limiting, light bot mode): an HTTP client with a real browser TLS impersonation and a decent IP is plenty. This is most of the web.
Hard Cloudflare (interactive challenge, Turnstile): you need a real browser that executes the JS — Playwright or Puppeteer. The proxy's job there is IP diversity, not passing the check.

The mistake I see most often is spending money on better proxies to solve a problem that lives in the JavaScript layer. Test the cheap hypothesis first: does plain Python already work? Does a headless browser with no proxy work? Only then start paying for exit quality.

Reproduce it

The method is boring on purpose: 28 requests, 7 sites, 4 tool combinations, pass = 200 with no challenge markers in the body. If you want to run the same matrix against your own targets, the request-side code (requests, httpx, curl_cffi, Playwright) is in a public examples repo: github.com/roamproxy/proxy-examples.

If you want a residential pool to test the "does a better IP help my target" hypothesis without a subscription, the network I work on (Roam) gives 300MB free on signup — enough to run the matrix above against your actual target before spending anything.

Got a Cloudflare block that behaves differently from this? I'd genuinely like to hear it — drop the details in the comments.

Your proxy connected. That doesn't mean it hid your IP.

RoamProxy — Sat, 25 Jul 2026 16:25:52 +0000

Every proxy checker I've come across answers one question: did the connection open?

That's the least interesting thing about a proxy. A proxy can connect perfectly and still hand your real IP address to the destination server in a header. If you're scraping, testing geo-restricted behaviour, or just trying not to be identified, "it connected" tells you almost nothing.

Here's how to actually check, with code you can run.

Three ways a proxy gives you away

1. It forwards your real IP in a header

The most direct leak. The proxy adds your address to the request:

GET /whatever HTTP/1.1
Host: example.com
X-Forwarded-For: 203.0.113.44     <- your actual IP

X-Forwarded-For is the common one, but X-Real-IP and the RFC 7239 Forwarded header do the same job. This is normal, correct behaviour for a reverse proxy sitting in front of your server — and completely wrong for a forward proxy you're using to not be identified.

A proxy that does this is called transparent. The destination sees both the proxy's IP and yours.

2. It announces that it's a proxy

Sometimes your IP is safely hidden, but the request still carries:

Via: 1.1 squid-cache
Proxy-Connection: keep-alive

Your identity is intact, but the server knows the traffic is proxied. Plenty of anti-bot systems treat that alone as reason enough to block, rate-limit, or serve you different content. So it matters even though nothing personal leaked.

3. The exit is not where it claims to be

A proxy sold as Frankfurt that egresses in Singapore isn't just a labelling problem — it breaks anything geo-dependent, and the latency will be nothing like what you planned for.

Checking it properly

The key insight: you need a baseline. Without knowing your own IP first, you can't tell "the proxy leaked my address" apart from "there happens to be an IP in this header."

import httpx

# Baseline: who am I without a proxy?
direct_ip = httpx.get("https://api.ipify.org", timeout=10).text.strip()

PROXY = "http://user:pass@proxy.example.com:8080"

with httpx.Client(proxy=PROXY, timeout=10) as client:
    exit_ip = client.get("https://api.ipify.org").text.strip()
    seen_headers = client.get("https://httpbin.org/headers").json()["headers"]

print(f"me:   {direct_ip}")
print(f"exit: {exit_ip}")
print(f"headers the server saw: {seen_headers}")

Now grade it:

PROXY_HEADERS = (
    "via", "x-forwarded-for", "x-real-ip",
    "forwarded", "proxy-connection",
)

def grade(headers: dict, direct_ip: str | None) -> str:
    lowered = {k.lower(): str(v) for k, v in headers.items()}

    # Our real IP showed up somewhere in what the server received.
    if direct_ip and any(direct_ip in v for v in lowered.values()):
        return "transparent"

    # Identity is safe, but the request is visibly proxied.
    if any(h in lowered for h in PROXY_HEADERS):
        return "anonymous"

    return "elite"

Three outcomes:

Grade	What it means
`transparent`	Your real IP is visible through the proxy. Useless for anonymity.
`anonymous`	Real IP hidden, but headers announce a proxy is in use.
`elite`	Looks like an ordinary direct request.

The part that's easy to get wrong

Note the direct_ip and ... guard. If fetching your baseline fails — no network, endpoint down, rate limited — direct_ip is None, and a naive implementation that does if direct_ip in value will either crash or silently match nothing.

Either way you must not report transparent. Claiming a proxy is safe when you simply failed to check is the one error mode that actually hurts someone. Degrade to anonymous, which is what the header evidence alone supports.

Same reasoning applies to the header check being case-insensitive. Servers and proxies are inconsistent about casing, and "Via" vs "via" deciding whether you flag a leak is not a distinction you want in security-adjacent code.

Doing this to a whole list

One proxy at a time is fine for debugging. For a list of a few hundred you want concurrency, timeouts, and output you can pipe somewhere.

I put the above into a single-file CLI called proxyprobe — MIT, one dependency:

pip install "httpx[socks]"
curl -O https://raw.githubusercontent.com/roamproxy/proxyprobe/main/proxyprobe.py
python proxyprobe.py proxies.txt --geo

PROXY                              OK   MS    EXIT IP        ANONYMITY    LOCATION
---------------------------------  ---  ----  -------------  -----------  -----------------
http://user:***@gw.example.com:80  yes  312   203.0.113.44   elite        Tokyo Japan
socks5://198.51.100.7:1080         yes  1180  198.51.100.7   transparent  Frankfurt Germany
http://10.0.0.9:3128               no   -     -              -

2/3 working  |  312ms median  |  1 elite, 1 transparent

It exits 0 if anything worked and 1 if nothing did, so it drops into CI:

proxyprobe proxies.txt --working-only --json > alive.json || echo "all proxies down"

Two implementation details worth stealing even if you write your own:

Mask passwords greedily. Proxy credentials routinely contain @ and :. A lazy regex like ://([^:/@]+):([^@]+)@ stops at the first @ and leaves the tail of the password in your output:

http://u:p@ss:word@host:80   ->   http://u:***@ss:word@host:80   # leaked

Match greedily to the last @ instead. A unit test caught this one for me, which is a good argument for unit-testing the boring string function that touches secrets.

Make the endpoints configurable. Anything hardcoded means someone routes their entire proxy list through a third party they never chose. --echo-url and --headers-url take about four lines and remove that objection entirely.

What this doesn't cover

This is an HTTP-layer check. If you're driving a real browser, there's a separate set of leaks that live above it — WebRTC handing out local candidates, DNS resolving outside the tunnel, timezone and locale disagreeing with the exit IP. None of that shows up in request headers, and none of it is detectable with the code above.

Worth knowing where the boundary is: passing this check means the proxy isn't betraying you at the HTTP layer. It doesn't mean a browser sitting on top of it is quiet.

Disclosure: I work on Roam, a proxy provider. proxyprobe works with proxies from anyone and has no vendor lock-in — we open-sourced it because we were writing this script over and over internally. If you use it against a competitor's proxies and find a bug, I'd genuinely like the issue.

Playwright + Proxies: Sticky Sessions, Rotation, and the SOCKS5 Trap

RoamProxy — Sat, 18 Jul 2026 08:51:03 +0000

You wired a proxy into requests in one line. Then you tried the same thing with Playwright and got a soup of ERR_TUNNEL_CONNECTION_FAILED, mystery CAPTCHAs, and a bandwidth bill that made no sense. Been there.

Headless browsers talk to proxies differently than HTTP clients do, and most of the "playwright proxy" snippets floating around skip the three things that actually bite you in production. Here's the setup I wish someone had handed me.

The baseline: one browser, one proxy

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://gw.example.com:41080",
        "username": "USER-session-alpha",
        "password": "PASS",
    })
    page = browser.new_page()
    page.goto("https://httpbin.org/ip")
    print(page.content())
    browser.close()

Two details people miss:

Auth goes in the launch config, not the URL. http://user:pass@host:port as the server string works in some HTTP clients, but Chromium wants credentials separately. If your password has special characters (#, @), the URL form will silently mangle them.
The proxy applies to everything the browser loads — every image, every font, every tracking pixel. Remember that; it matters in a minute.

Trap #1: SOCKS5 auth doesn't work in Chromium

This one costs people hours. Chromium — and therefore Playwright's default browser — does not support username/password authentication over SOCKS5. Your provider's SOCKS5 endpoint works fine in curl, so you assume your Playwright config is wrong. It isn't. The browser just never sends the credentials.

Fixes, in order of sanity:

Use the HTTP proxy port with auth instead. Any decent provider serves HTTP CONNECT and SOCKS5 (ideally on the same port), and CONNECT tunnels HTTPS just fine. There is no meaningful difference for scraping.
IP-allowlist auth on the SOCKS5 side, if your provider supports it and your egress IP is stable.
A local forwarder (e.g. a tiny pproxy sidecar) that holds the credentials — works, but now you're running infrastructure to avoid a one-line fix.

Trap #2: rotation granularity is per-context, not per-request

With an HTTP client, rotating proxies per request is trivial. A browser is a different animal: a page load is 50–200 requests, and they must all exit from the same IP or you'll trip every anti-bot system on earth (real users don't load HTML from Berlin and CSS from Osaka).

So the unit of rotation is the browser context, not the request. With a session-based proxy gateway, you pin one session ID per context:

def new_identity(p, session_id: str):
    browser = p.chromium.launch(proxy={
        "server": "http://gw.example.com:41080",
        "username": f"USER-session-{session_id}",
        "password": "PASS",
    })
    return browser

# identity per worker: same IP for the whole browsing session
for task_id, urls in enumerate(url_batches):
    browser = new_identity(p, f"worker{task_id}")
    page = browser.new_page()
    for url in urls:
        page.goto(url)
        ...
    browser.close()

Same session ID = same exit IP for the lifetime of the session; new ID = fresh IP. One identity per context, rotate by spawning a new context with a new session ID. That's the whole model. (Playwright also accepts proxy= on new_context(), so you can run several identities inside one browser process — cheaper than one process per identity.)

Trap #3: you're paying residential rates for webfonts

Residential bandwidth is billed by the GB. A single product page can pull 3–8 MB, of which the HTML you actually want is maybe 80 KB. Multiply by 100k pages and you've spent real money loading hero images for nobody.

Route-block everything you don't parse:

BLOCK = {"image", "media", "font", "stylesheet"}

def register_blocking(page):
    page.route("**/*", lambda route:
        route.abort() if route.request.resource_type in BLOCK
        else route.continue_())

In my runs this cuts bandwidth 60–85% depending on the site, with zero effect on the DOM you scrape. Two caveats: keep stylesheets if you rely on :visible selectors (layout needs CSS), and don't block XHR/fetch — that's usually where the data lives.

Putting it together

The full pattern — sticky identity per context, auth done right, resource blocking on every page — is about 40 lines. I keep a copy-pasteable version (plus requests/httpx/Scrapy equivalents) in a public examples repo: github.com/roamproxy/proxy-examples.

If you want to test against a real residential pool without a subscription, the network I work on (Roam) gives 300MB free on signup — enough to verify your Playwright setup against your actual target before spending anything.

Questions about a setup that's misbehaving? Drop it in the comments — proxy-through-browser bugs are weirdly fun to debug.

We Sent 10,000 Requests Through a Residential Proxy Network. Here's the Raw Data.

RoamProxy — Thu, 16 Jul 2026 09:58:07 +0000

Most proxy benchmarks you'll find are vendor marketing: a success rate with no methodology, no failure counts, no raw data. We run a residential proxy network, and we wanted numbers we could actually defend — so we benchmarked ourselves the way we'd want a third party to do it, and published every row.

Here's what 10,000 real requests look like, and what surprised us.

The setup

10,000 HTTPS requests to https://www.cloudflare.com/cdn-cgi/trace — small, anycast, and it echoes the exit IP and country back at you.
5 target countries (US, UK, Germany, Japan, Brazil) × 2,000 requests each.
Every request forced a fresh session (new exit IP requested from the pool each time).
Plain curl, concurrency 20, 30s timeout, zero retries. Every failure stays in the dataset.
Vantage point: a server in Tokyo (AS3258).

A request only counts as a success if it returned HTTP 200 within the timeout — DNS, proxy CONNECT, TLS handshake and body included.

The numbers

Country	Success	Median	p95	Unique exit IPs (of 2,000)
US	99.80%	2.00s	7.27s	1,827
UK	99.80%	2.37s	8.23s	1,533
Germany	99.75%	2.39s	12.22s	1,651
Japan	99.70%	2.03s	14.01s	1,026
Brazil	99.30%	2.73s	17.18s	1,399

Overall: 99.67% success, 7,436 unique exit IPs across 10,000 requests. Geo-targeting accuracy was 98.9–100% per country.

Three things that surprised us

1. The tail is 5–6× the median. Medians cluster tightly at 2.0–2.7s everywhere, but p95 stretches to 12–17s in Germany, Japan and Brazil. If you set client timeouts based on the median, you'll drop a meaningful slice of otherwise-successful requests. Budget for the tail: 20–30s timeouts for residential traffic.

2. 74% of requests got a globally unique IP. With a forced new session per request, 7,436 of 10,000 requests exited through an address no other request in the test saw. The US pool was the deepest: 1,827 unique IPs in 2,000 requests (91%).

3. IPv6 is a big share of real residential exits. Between 42% and 76% of exits per country were IPv6. If your target or your tooling silently breaks on IPv6, that's a chunk of your error budget gone before you've done anything wrong.

Reproduce it (please do)

The whole thing is one loop:

for i in $(seq 1 2000); do
  curl -s -o /dev/null -m 30 \
    -w "%{http_code},%{time_total},%{time_starttransfer}\n" \
    -x "http://USER-country-us-session-$i:PASS@gateway:port" \
    https://www.cloudflare.com/cdn-cgi/trace
done

Point it at any provider, diff the results. Running this against a competitor takes an afternoon, which is exactly why we think publishing raw data should be table stakes in this industry.

Honest limitations

One vantage point (Tokyo), one small target. A heavy page on a slow origin behaves differently — this measures the network, not your target site.
Success against Cloudflare's trace endpoint ≠ success against aggressive bot defenses. That depends on your client fingerprint as much as the IP.
It's our own network, measured by us. That's why the raw CSV (exit IPs anonymized to /24 & /48) and full methodology are public — verify, don't trust.

Full write-up + downloadable CSV (CC BY 4.0): roamproxy.com/residential-proxy-benchmark · code samples: github.com/roamproxy/proxy-examples

Residential vs. Datacenter Proxies: Which One Do You Actually Need?

RoamProxy — Tue, 14 Jul 2026 15:44:48 +0000

If you're scraping the web or managing multiple accounts, the residential-vs-datacenter question decides both your success rate and your bill. Here's the practical breakdown I wish someone had given me earlier — including when the "worse" option is actually the right call.

The core difference

Datacenter proxies come from cloud providers and hosting companies. The IP blocks are publicly registered to companies like OVH or Hetzner, so any target site can (and does) look up the ASN and see "this visitor is a server, not a person."

Residential proxies are IP addresses assigned by consumer ISPs (Comcast, Vodafone, China Telecom...) to real households. To a target site, a residential IP is indistinguishable from a normal visitor on their couch.

That single difference drives everything else:

	Datacenter	Residential
Speed	Fast, stable (~10–50 ms)	Slower, variable (~100–800 ms)
Cost	Cheap (~$0.5–1/GB)	2–5× more (~$2–8/GB)
Block resistance	Weak — ASN is a giveaway	Strong — looks like real users
IP pool size	Thousands	Millions, rotating
Best for	APIs, unprotected sites, speed-critical jobs	Protected sites, sneakers, social, ads verification

A decision rule that actually works

Ask two questions:

1. Does the target actively fight bots?
Run a quick probe through a datacenter IP. If you get 200s all day, congratulations — use datacenter and pocket the savings. If you see 403s, CAPTCHAs, or empty shells of pages, the site checks IP reputation and you need residential.

2. Is per-request latency critical?
Price monitoring across 100k SKUs cares about throughput; a checkout bot cares about a single fast round-trip. Datacenter wins raw speed. Residential wins effective speed once you count retries — a 300 ms residential request beats a 30 ms datacenter request that gets blocked and retried five times.

The hybrid pattern most teams end up with

In practice, mature scraping stacks route per-target:

import requests

DC   = "http://user_dc_1:pass@gw.example.com:41080"    # cheap, fast
RESI = "http://user-country-us:pass@gw.example.com:41080"  # stealthy

ROUTES = {
    "api.competitor.com":   DC,    # plain JSON API, no protection
    "www.sneaker-shop.com": RESI,  # aggressive bot detection
}

def get(url, host):
    proxy = ROUTES.get(host, DC)   # default to cheap
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

Start every new target on datacenter. Promote it to residential the first time you hit sustained 403s. Your blend usually lands around 70/30 datacenter/residential by request count — but 30/70 by importance.

Three mistakes to avoid

Buying residential for everything. You'll pay 3× for traffic that a $0.8/GB datacenter pool would have handled fine.
Buying datacenter for protected sites and "solving" blocks with more IPs. IP reputation databases classify entire datacenter ranges; rotating within a burned ASN just burns money.
Ignoring session stickiness. For logins and carts you need the same IP across a flow. Both proxy types support sticky sessions — use them instead of fighting random rotation.

TL;DR

Unprotected targets + speed → datacenter
Bot detection, geo-restrictions, accounts → residential
Real projects → both, routed per target

I work on RoamProxy, a pay-as-you-go proxy service (residential $2/GB, datacenter $0.8/GB, same gateway endpoint for both). Code examples for Python/Node/Go are on our GitHub.