DEV Community: RoamProxy

Before You Rotate the IP, Find Out What Actually Got Blocked

RoamProxy — Sun, 23 Aug 2026 20:03:27 +0000

Something in your pipeline just started returning 403s, or a challenge page, or an empty 200 where there used to be JSON. The reflex — in every tutorial, every Stack Overflow answer, every proxy vendor's docs — is to rotate the IP.

Most of the time that's the wrong first move, and it's expensive in a way that isn't obvious: rotating throws away everything the old identity had accumulated. Session cookies, a solved challenge, whatever internal trust score the target had assigned you after 40 well-behaved requests. If the IP wasn't the thing that got flagged, you paid all of that and fixed nothing — and you now have a second identity showing the same pattern, which is a worse signal than the first one alone.

The useful question isn't "how do I get unblocked." It's what scope did the block apply to. There are five, they're cheap to tell apart, and the right response is different for each.

The five scopes

Request-scoped. One specific request was rejected — a missing header, a bad referer, an expired token in a query param, a URL that requires a POST. The next request works fine. This is the most common "block" and it isn't a block at all.

Session-scoped. Your cookie jar / TLS session got flagged. A challenge interstitial that keeps re-serving, a cf_clearance that went stale, a rate limiter keyed on session ID. Everything else about you is fine.

Fingerprint-scoped. Your client is being rejected for what it is, not where it's from. Automation-flavoured TLS handshake, a headless-shaped JS environment, a header order no real browser emits. The tell: it fails instantly and identically no matter how long you wait or what IP you use.

IP-scoped. The address is on a list. Every request from it fails, from every session, with every client, immediately.

Account-scoped. You're logged in and the account is limited. Rotating the IP here is actively harmful — a logged-in account that suddenly appears from a new city is a much stronger abuse signal than one that stays put.

Four probes that separate them

You don't need to guess. Four requests, under a minute, and the answer falls out:

Same IP, fresh session (new cookie jar, same client, same exit).
Same IP, different client (curl if you were using a browser, or vice versa — this varies the TLS/JA3 and header order).
Different IP, same cookies.
Different IP, everything fresh.

import httpx

def probe(url, *, proxy=None, cookies=None, headers=None, http2=True):
    """One probe. Returns (status, len(body), challenged)."""
    with httpx.Client(proxy=proxy, cookies=cookies or {},
                      headers=headers or {}, http2=http2,
                      timeout=20, follow_redirects=True) as c:
        r = c.get(url)
    body = r.text
    challenged = any(s in body.lower() for s in
                     ("just a moment", "un momento", "checking your browser",
                      "attention required", "verify you are human"))
    return r.status_code, len(body), challenged


def diagnose(url, old_proxy, new_proxy, old_cookies, browser_headers):
    p = {}
    p["fresh_session_same_ip"] = probe(url, proxy=old_proxy, headers=browser_headers)
    p["diff_client_same_ip"]   = probe(url, proxy=old_proxy, http2=False, headers={})
    p["old_cookies_new_ip"]    = probe(url, proxy=new_proxy, cookies=old_cookies,
                                       headers=browser_headers)
    p["all_fresh_new_ip"]      = probe(url, proxy=new_proxy, headers=browser_headers)
    return p

Read it like this:

What happens	Scope	What to actually change
Fresh session on the same IP works	Session	Drop the cookie jar. Keep the IP.
Same IP fails, new IP with old cookies also fails	Session or account	Cookies. Not the IP.
Everything fails instantly from both IPs, but a plain `curl` works	Fingerprint	Your client, not your route.
Every request from the old IP fails, everything from the new one works	IP	Rotate — this is the one case where it's right.
Only some URLs fail, others are fine	Request	Fix the headers/method/token.
Fails only while logged in	Account	Slow down. Do not move the account to a new IP.

The row people get wrong is the second one. A stale session on a target that keys its rate limiter to a session ID looks exactly like an IP ban from the inside — every request fails — and rotating the IP "fixes" it, because the new exit also comes with a new cookie jar. So you conclude the IP was burned when the cookie was, and you burn an address per incident forever after.

The waiting case

There's a fifth outcome the probes will show you and most people won't believe: temporary, and the correct action is nothing.

I hit this on Medium's Cloudflare edge a couple of weeks ago. Mid-session, after an import that touched several endpoints in quick succession, every page turned into an interstitial. Same tab, same everything, for something like half an hour — reloading did nothing, four attempts did nothing. The exit IP was verifiably unchanged the whole time (I checked, precisely because rotating was the tempting move and that session had a login I didn't want to re-establish). Then it cleared on its own and the next request went through normally.

If I'd rotated at minute two I would have concluded the IP was blocked, thrown away a good address and a live session, and learned the wrong lesson permanently. The signal that it was temporary was there: the challenge was serving (200 with a challenge body), not refusing (403 with nothing). A challenge is an invitation to prove yourself. A 403 with an empty body is a door. They deserve opposite responses.

Change the cheapest thing first

Order your remediations by what they cost you if you're wrong:

Headers / request shape — free.
Cookie jar — cheap, unless you're logged in.
Wait — costs only time, and time is what half of these need.
Client fingerprint — moderate; you're rebuilding a client, not an identity.
Exit IP — expensive. You lose the session, the accumulated trust, and (if you were logged in) you hand the target a "this account moved cities" event.
Account — most expensive, obviously.

Rotation belongs at position five for a reason. It's the loudest change you can make and it's the one people reach for first, because it's the one the tooling makes easiest.

A practical version of this: keep the diagnostic in your codebase, not in your head. When a run starts failing, diagnose() runs before any remediation logic does, logs which scope it found, and only the IP-scoped branch is allowed to call rotate(). It takes an afternoon to write and it stops the reflex from making decisions for you.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.

Your Scraper's Retry Logic Is Probably Making the Block Worse

RoamProxy — Fri, 21 Aug 2026 04:43:36 +0000

Most scrapers have one retry policy: catch the exception, sleep, try again, give up after N attempts. It's the default in every tutorial, it's what tenacity gives you in three lines, and on a healthy target it's fine.

It stops being fine the moment the failures stop being random. A rate limit, a hard block, a TLS reset and a slow origin all arrive as "the request failed," and treating them identically means you spend your entire run budget hammering something that was never going to open — while the one failure that would have succeeded on retry gets three attempts and then gets dropped.

Here's the taxonomy I actually use, and the part that took the longest to learn: the counter matters more than the backoff.

The four failures, and what each one means

429 with a Retry-After header. The friendliest failure you will ever get. The server is telling you exactly what it wants. Honour it — sleep the stated duration, then continue on the same connection and the same identity. Do not rotate anything. Rotating here is how a temporary throttle turns into a fingerprinted pattern.

429 with no header. The server wants you slower but won't say by how much. Exponential backoff with jitter is genuinely the right tool. Start higher than you think — 5 seconds, not 0.5.

403 / 401 / a challenge page. This is not a retryable condition. Something about the request was rejected, and repeating it byte-for-byte will be rejected the same way. Every retry here is pure waste, and worse, it's the pattern that gets a durable block instead of a temporary one. Short-circuit straight to whatever your fallback is — different identity, different route, or shelve the URL and move on.

Connection reset / timeout / TLS handshake failure. This one is retryable, and it's the one people wrongly lump in with 403. Transport failures are frequently just noise. Retry immediately, twice, then treat it as a hard failure.

The distinction that matters: HTTP-level rejections tell you something about your request. Transport-level failures usually tell you nothing at all.

import random, time
import httpx

RETRYABLE_STATUS = {429, 500, 502, 503, 504}
HARD_BLOCK = {401, 403, 407, 451}

def classify(exc_or_resp):
    if isinstance(exc_or_resp, httpx.HTTPError):
        return "transport"          # retry fast, twice
    code = exc_or_resp.status_code
    if code in HARD_BLOCK:
        return "blocked"            # do not retry, change something
    if code == 429:
        return "throttled"
    if code in RETRYABLE_STATUS:
        return "server"
    return "ok" if code < 400 else "fatal"

def sleep_for(kind, attempt, resp=None):
    if kind == "throttled" and resp is not None:
        ra = resp.headers.get("retry-after")
        if ra and ra.isdigit():
            return int(ra)
        return min(60, 5 * (2 ** attempt)) + random.uniform(0, 2)
    if kind == "server":
        return min(30, 2 ** attempt) + random.uniform(0, 1)
    if kind == "transport":
        return 0.5
    return 0

The counter is the part everyone skips

Backoff handles the failure you already got. It does nothing about the one you're about to get, and on a lot of targets the limit isn't time-based at all — it's a volume budget. You get N requests, and the block arrives on N+1 regardless of how politely you spaced them.

I learned this on a regional grocery chain's stock checker. It would 429 you politely for about an hour, and then flip to a permanent 403 once you crossed some invisible total. Every backoff strategy in the world is useless against that, because the thing being counted isn't your rate — it's your volume.

The fix is a rolling per-host counter that you enforce on yourself, set well below whatever number killed the last run:

import time
from collections import defaultdict, deque

class HostBudget:
    """Rolling request counter per host. Refuses before the server does."""

    def __init__(self, max_requests=200, window_seconds=3600):
        self.max = max_requests
        self.window = window_seconds
        self.hits = defaultdict(deque)

    def allow(self, host):
        now = time.time()
        q = self.hits[host]
        while q and now - q[0] > self.window:
            q.popleft()
        if len(q) >= self.max:
            return False
        q.append(now)
        return True

    def spent(self, host):
        return len(self.hits[host])

Once that exists, a 403 stops being a signal you react to and becomes a signal you record: whatever the counter said when the block landed is your new ceiling for that host, minus a healthy margin. Write it to disk. The next run starts already knowing.

The reason the 403 is a bad signal on its own is timing. By the time you see it, you're usually already benched — the block is applied, the cooldown clock started without telling you, and nothing you do in the next hour matters. The counter is the only thing that fires before the damage.

Putting it together

def fetch(client, url, budget, max_attempts=4):
    host = httpx.URL(url).host
    if not budget.allow(host):
        raise RuntimeError(f"self-imposed budget hit for {host} ({budget.spent(host)})")

    for attempt in range(max_attempts):
        try:
            resp = client.get(url, timeout=20)
        except httpx.HTTPError as exc:
            kind = classify(exc)
            if attempt >= 2:
                raise
            time.sleep(sleep_for(kind, attempt))
            continue

        kind = classify(resp)
        if kind == "ok":
            return resp
        if kind == "blocked":
            # record the ceiling, then hand off — retrying byte-for-byte is waste
            raise BlockedError(host, budget.spent(host))
        if kind == "fatal":
            resp.raise_for_status()
        time.sleep(sleep_for(kind, attempt, resp))

    raise RuntimeError(f"exhausted attempts for {url}")

Three rules, in order of how much run time they save:

Never retry a hard block unchanged. It cannot succeed and it makes the block stickier.
Count your own requests per host and stop before the server stops you.
Retry transport errors fast and cheaply. They're usually nothing.

The retry loop is the least glamorous part of a scraper and the one that quietly decides whether a run finishes. Worth twenty minutes.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.

Stop Reaching for a Headless Browser to Scrape Documentation Sites

RoamProxy — Tue, 11 Aug 2026 23:45:28 +0000

Every few days someone in a scraping forum asks a version of the same question: "I'm collecting documentation text for an AI tool, but the pages render with JavaScript. What's the lightest way to get the content?"

The answers are always the same — open DevTools, check the Network tab, find the XHR. That advice is correct, and for documentation sites specifically it's usually unnecessary work.

Documentation sites are not arbitrary web apps. They're overwhelmingly built by a handful of static site generators, and those generators leave the content sitting in predictable places. Three routes cover most of what you'll hit, and none of them need a browser.

Route 1: the JSON is already in the HTML

Next.js-based docs (which includes a large share of company developer portals) embed the full page payload in a __NEXT_DATA__ script tag. It's in the initial HTML response — no JavaScript execution needed.

import json, re, httpx

html = httpx.get(url, follow_redirects=True).text
m = re.search(r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', html, re.S)
if m:
    data = json.loads(m.group(1))
    # content location varies by site; dump the tree once and look around
    print(json.dumps(data["props"]["pageProps"], indent=2)[:2000])

Docusaurus (the other big one) doesn't use __NEXT_DATA__, but it pre-renders the full article text into the static HTML. A plain httpx.get plus a main or article selector gets you everything. The "JavaScript rendering" you see in the browser is hydration for navigation and search — the prose was already there in the initial response.

Thirty-second check: curl -s <url> | grep -c "some sentence you can see on the page". If that returns 1 or more, the content is in the HTML and you're done. This one check resolves the majority of "it's JavaScript-rendered" cases, because people judge from DevTools' Elements panel — which shows the hydrated DOM, not what the server actually sent.

Route 2: the markdown is in a public repo

Most open-source project documentation is markdown in the same repository as the code, usually under docs/. Scraping the rendered HTML means fetching pages one at a time, parsing them, and stripping navigation chrome you didn't want. Cloning gets you the clean source in one shot:

git clone --depth 1 --filter=blob:none --sparse https://github.com/org/project
cd project && git sparse-checkout set docs

You get the original markdown — headings intact, code fences intact, no nav sidebars, no cookie banners, no per-page rate limiting. For a RAG pipeline this is strictly better input than parsed HTML, and it's one request instead of several hundred.

Worth saying plainly: check the license before you ingest. Documentation is frequently licensed separately from the code, and "the repo is public" is not the same as "you may redistribute this."

Route 3: the site publishes a text endpoint

A growing number of documentation hosts expose plain-text views:

/llms.txt — an emerging convention where a site publishes a curated, plain-text map of itself specifically for LLM consumption. Fast-moving developer-tool companies have adopted it quickly. Always worth one request.
/sitemap.xml — not text content, but it gives you the complete URL list without crawling, which means you never have to discover pages by following links.
ReadTheDocs projects usually offer downloadable HTML and often PDF/ePub builds of the entire docs set from the version menu. One artifact, complete content.

Picking a route in under a minute

Signal	Route
`curl` output contains visible page text	Parse the static HTML — done
`__NEXT_DATA__` in the HTML	Extract and walk the JSON
Public repo with a `docs/` directory	Sparse-clone the markdown
`/llms.txt` returns 200	Start there
ReadTheDocs / GitBook host	Look for the download build
None of the above	Now open DevTools

When you actually do need a browser

Some cases are genuinely dynamic, and it's worth knowing them so you don't over-apply the above:

Docs behind authentication where the session is established by client-side JavaScript.
Content assembled from several API calls at runtime with no single payload — more common in interactive API explorers than in prose documentation.
Sites that gate on a JavaScript challenge before serving anything, where the challenge is the point.

For a few hundred pages of prose, though, these are the exception. The default assumption should be that the text is already reachable, and the browser is the fallback.

The part that actually costs you

The reason this matters isn't purity — it's that headless browsers change the shape of your project. You go from a script anyone can run to a pipeline with a browser binary, a memory ceiling, per-page startup cost, and a new class of flaky failures that only reproduce sometimes. On a few hundred documentation pages, Route 1 or Route 2 typically finishes before a browser-based run has finished launching.

Check whether the content is already sitting there in plain text. Most of the time, it is.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.

We Tried to Beat Cloudflare With Proxies and TLS Tricks. Here's What Actually Failed.

RoamProxy — Tue, 28 Jul 2026 04:41:54 +0000

Most advice about scraping Cloudflare-fronted sites is asserted, not measured. "Use residential IPs." "Match Chrome's JA3." People repeat these like folklore. We run a proxy network, so we had the tooling to actually test them — and the results contradicted the two most common pieces of advice.

Here's what we ran, and what failed.

The test

Two variables, four combinations:

Exit IP: a Psychz datacenter address vs a Frontier Communications residential line (both US).
TLS stack: plain Python requests vs curl_cffi impersonating Chrome.

Seven Cloudflare-fronted sites, two passes each — 28 requests total, US exits, 27 July 2026. A "pass" required HTTP 200 and a body without challenge markers. A 200 that returns "Just a moment…" is a block wearing a success code, and it does not count.

The numbers

Combination	Passed
Datacenter IP + default Python TLS	0 / 7
Datacenter IP + Chrome-impersonated TLS	0 / 7
Residential IP + default Python TLS	0 / 7
Residential IP + Chrome-impersonated TLS	0 / 7

Every combination failed identically. Two things in that table contradict advice you'll read everywhere.

The residential IP bought us nothing

Not a single site treated the Frontier residential exit differently from the Psychz datacenter one. That does not mean IP reputation is a myth — it means these sites never got as far as judging our IP. A request that fails the bot-management check is refused on other grounds first.

IP quality decides outcomes in the middle of the difficulty curve. At the hard end it isn't the binding constraint, and buying better IPs won't move it. If you're stuck at 403 on a genuinely aggressive Cloudflare config, a residential proxy is not the missing piece.

Chrome TLS impersonation didn't flip a single site either

curl_cffi with impersonate="chrome" is great tooling, but on these targets it changed nothing. The sites require executing the JavaScript challenge, which no HTTP client does, no matter how convincing its TLS handshake. If you're stuck, adding a TLS library isn't the fix — you need something that runs JS.

For calibration: the same script sailed through Cloudflare-fronted sites that aren't running aggressive bot mode, on every combination including plain Python from a bare datacenter host. "Protected by Cloudflare" spans an enormous range of strictness. Establish which end your target sits at before spending money on tooling.

Bonus: stop chasing a specific JA3 hash

While measuring, we checked what fingerprint our own requests presented. Three consecutive curl_cffi requests with the same impersonate="chrome" setting, from the same host, produced three different JA3 hashes (a0052cf3…, 0899dce7…, d2de58db…).

That's not a bug. Real Chrome shuffles TLS extensions and injects GREASE values, so its JA3 changes per connection — and a good impersonation library reproduces that. Any guide telling you to match one specific JA3 string is describing a browser that stopped existing years ago.

What did stay identical across all our requests — direct, via residential proxy, and via datacenter proxy — was the HTTP/2 fingerprint (52d84b11…), derived from the SETTINGS frame, header table size, window size and pseudo-header order. It's far more stable than JA3, which is exactly why it's worth more to the people fingerprinting you. Check both when you debug a silent block; we used tls.browserleaks.com/json.

So what actually works?

Match the tool to where your target sits on the curve:

Soft Cloudflare (rate limiting, light bot mode): an HTTP client with a real browser TLS impersonation and a decent IP is plenty. This is most of the web.
Hard Cloudflare (interactive challenge, Turnstile): you need a real browser that executes the JS — Playwright or Puppeteer. The proxy's job there is IP diversity, not passing the check.

The mistake I see most often is spending money on better proxies to solve a problem that lives in the JavaScript layer. Test the cheap hypothesis first: does plain Python already work? Does a headless browser with no proxy work? Only then start paying for exit quality.

Reproduce it

The method is boring on purpose: 28 requests, 7 sites, 4 tool combinations, pass = 200 with no challenge markers in the body. If you want to run the same matrix against your own targets, the request-side code (requests, httpx, curl_cffi, Playwright) is in a public examples repo: github.com/roamproxy/proxy-examples.

If you want a residential pool to test the "does a better IP help my target" hypothesis without a subscription, the network I work on (Roam) gives 300MB free on signup — enough to run the matrix above against your actual target before spending anything.

Got a Cloudflare block that behaves differently from this? I'd genuinely like to hear it — drop the details in the comments.

Your proxy connected. That doesn't mean it hid your IP.

RoamProxy — Sat, 25 Jul 2026 16:25:52 +0000

Every proxy checker I've come across answers one question: did the connection open?

That's the least interesting thing about a proxy. A proxy can connect perfectly and still hand your real IP address to the destination server in a header. If you're scraping, testing geo-restricted behaviour, or just trying not to be identified, "it connected" tells you almost nothing.

Here's how to actually check, with code you can run.

Three ways a proxy gives you away

1. It forwards your real IP in a header

The most direct leak. The proxy adds your address to the request:

GET /whatever HTTP/1.1
Host: example.com
X-Forwarded-For: 203.0.113.44     <- your actual IP

X-Forwarded-For is the common one, but X-Real-IP and the RFC 7239 Forwarded header do the same job. This is normal, correct behaviour for a reverse proxy sitting in front of your server — and completely wrong for a forward proxy you're using to not be identified.

A proxy that does this is called transparent. The destination sees both the proxy's IP and yours.

2. It announces that it's a proxy

Sometimes your IP is safely hidden, but the request still carries:

Via: 1.1 squid-cache
Proxy-Connection: keep-alive

Your identity is intact, but the server knows the traffic is proxied. Plenty of anti-bot systems treat that alone as reason enough to block, rate-limit, or serve you different content. So it matters even though nothing personal leaked.

3. The exit is not where it claims to be

A proxy sold as Frankfurt that egresses in Singapore isn't just a labelling problem — it breaks anything geo-dependent, and the latency will be nothing like what you planned for.

Checking it properly

The key insight: you need a baseline. Without knowing your own IP first, you can't tell "the proxy leaked my address" apart from "there happens to be an IP in this header."

import httpx

# Baseline: who am I without a proxy?
direct_ip = httpx.get("https://api.ipify.org", timeout=10).text.strip()

PROXY = "http://user:pass@proxy.example.com:8080"

with httpx.Client(proxy=PROXY, timeout=10) as client:
    exit_ip = client.get("https://api.ipify.org").text.strip()
    seen_headers = client.get("https://httpbin.org/headers").json()["headers"]

print(f"me:   {direct_ip}")
print(f"exit: {exit_ip}")
print(f"headers the server saw: {seen_headers}")

Now grade it:

PROXY_HEADERS = (
    "via", "x-forwarded-for", "x-real-ip",
    "forwarded", "proxy-connection",
)

def grade(headers: dict, direct_ip: str | None) -> str:
    lowered = {k.lower(): str(v) for k, v in headers.items()}

    # Our real IP showed up somewhere in what the server received.
    if direct_ip and any(direct_ip in v for v in lowered.values()):
        return "transparent"

    # Identity is safe, but the request is visibly proxied.
    if any(h in lowered for h in PROXY_HEADERS):
        return "anonymous"

    return "elite"

Three outcomes:

Grade	What it means
`transparent`	Your real IP is visible through the proxy. Useless for anonymity.
`anonymous`	Real IP hidden, but headers announce a proxy is in use.
`elite`	Looks like an ordinary direct request.

The part that's easy to get wrong

Note the direct_ip and ... guard. If fetching your baseline fails — no network, endpoint down, rate limited — direct_ip is None, and a naive implementation that does if direct_ip in value will either crash or silently match nothing.

Either way you must not report transparent. Claiming a proxy is safe when you simply failed to check is the one error mode that actually hurts someone. Degrade to anonymous, which is what the header evidence alone supports.

Same reasoning applies to the header check being case-insensitive. Servers and proxies are inconsistent about casing, and "Via" vs "via" deciding whether you flag a leak is not a distinction you want in security-adjacent code.

Doing this to a whole list

One proxy at a time is fine for debugging. For a list of a few hundred you want concurrency, timeouts, and output you can pipe somewhere.

I put the above into a single-file CLI called proxyprobe — MIT, one dependency:

pip install "httpx[socks]"
curl -O https://raw.githubusercontent.com/roamproxy/proxyprobe/main/proxyprobe.py
python proxyprobe.py proxies.txt --geo

PROXY                              OK   MS    EXIT IP        ANONYMITY    LOCATION
---------------------------------  ---  ----  -------------  -----------  -----------------
http://user:***@gw.example.com:80  yes  312   203.0.113.44   elite        Tokyo Japan
socks5://198.51.100.7:1080         yes  1180  198.51.100.7   transparent  Frankfurt Germany
http://10.0.0.9:3128               no   -     -              -

2/3 working  |  312ms median  |  1 elite, 1 transparent

It exits 0 if anything worked and 1 if nothing did, so it drops into CI:

proxyprobe proxies.txt --working-only --json > alive.json || echo "all proxies down"

Two implementation details worth stealing even if you write your own:

Mask passwords greedily. Proxy credentials routinely contain @ and :. A lazy regex like ://([^:/@]+):([^@]+)@ stops at the first @ and leaves the tail of the password in your output:

http://u:p@ss:word@host:80   ->   http://u:***@ss:word@host:80   # leaked

Match greedily to the last @ instead. A unit test caught this one for me, which is a good argument for unit-testing the boring string function that touches secrets.

Make the endpoints configurable. Anything hardcoded means someone routes their entire proxy list through a third party they never chose. --echo-url and --headers-url take about four lines and remove that objection entirely.

What this doesn't cover

This is an HTTP-layer check. If you're driving a real browser, there's a separate set of leaks that live above it — WebRTC handing out local candidates, DNS resolving outside the tunnel, timezone and locale disagreeing with the exit IP. None of that shows up in request headers, and none of it is detectable with the code above.

Worth knowing where the boundary is: passing this check means the proxy isn't betraying you at the HTTP layer. It doesn't mean a browser sitting on top of it is quiet.

Disclosure: I work on Roam, a proxy provider. proxyprobe works with proxies from anyone and has no vendor lock-in — we open-sourced it because we were writing this script over and over internally. If you use it against a competitor's proxies and find a bug, I'd genuinely like the issue.

Playwright + Proxies: Sticky Sessions, Rotation, and the SOCKS5 Trap

RoamProxy — Sat, 18 Jul 2026 08:51:03 +0000

You wired a proxy into requests in one line. Then you tried the same thing with Playwright and got a soup of ERR_TUNNEL_CONNECTION_FAILED, mystery CAPTCHAs, and a bandwidth bill that made no sense. Been there.

Headless browsers talk to proxies differently than HTTP clients do, and most of the "playwright proxy" snippets floating around skip the three things that actually bite you in production. Here's the setup I wish someone had handed me.

The baseline: one browser, one proxy

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://gw.example.com:41080",
        "username": "USER-session-alpha",
        "password": "PASS",
    })
    page = browser.new_page()
    page.goto("https://httpbin.org/ip")
    print(page.content())
    browser.close()

Two details people miss:

Auth goes in the launch config, not the URL. http://user:pass@host:port as the server string works in some HTTP clients, but Chromium wants credentials separately. If your password has special characters (#, @), the URL form will silently mangle them.
The proxy applies to everything the browser loads — every image, every font, every tracking pixel. Remember that; it matters in a minute.

Trap #1: SOCKS5 auth doesn't work in Chromium

This one costs people hours. Chromium — and therefore Playwright's default browser — does not support username/password authentication over SOCKS5. Your provider's SOCKS5 endpoint works fine in curl, so you assume your Playwright config is wrong. It isn't. The browser just never sends the credentials.

Fixes, in order of sanity:

Use the HTTP proxy port with auth instead. Any decent provider serves HTTP CONNECT and SOCKS5 (ideally on the same port), and CONNECT tunnels HTTPS just fine. There is no meaningful difference for scraping.
IP-allowlist auth on the SOCKS5 side, if your provider supports it and your egress IP is stable.
A local forwarder (e.g. a tiny pproxy sidecar) that holds the credentials — works, but now you're running infrastructure to avoid a one-line fix.

Trap #2: rotation granularity is per-context, not per-request

With an HTTP client, rotating proxies per request is trivial. A browser is a different animal: a page load is 50–200 requests, and they must all exit from the same IP or you'll trip every anti-bot system on earth (real users don't load HTML from Berlin and CSS from Osaka).

So the unit of rotation is the browser context, not the request. With a session-based proxy gateway, you pin one session ID per context:

def new_identity(p, session_id: str):
    browser = p.chromium.launch(proxy={
        "server": "http://gw.example.com:41080",
        "username": f"USER-session-{session_id}",
        "password": "PASS",
    })
    return browser

# identity per worker: same IP for the whole browsing session
for task_id, urls in enumerate(url_batches):
    browser = new_identity(p, f"worker{task_id}")
    page = browser.new_page()
    for url in urls:
        page.goto(url)
        ...
    browser.close()

Same session ID = same exit IP for the lifetime of the session; new ID = fresh IP. One identity per context, rotate by spawning a new context with a new session ID. That's the whole model. (Playwright also accepts proxy= on new_context(), so you can run several identities inside one browser process — cheaper than one process per identity.)

Trap #3: you're paying residential rates for webfonts

Residential bandwidth is billed by the GB. A single product page can pull 3–8 MB, of which the HTML you actually want is maybe 80 KB. Multiply by 100k pages and you've spent real money loading hero images for nobody.

Route-block everything you don't parse:

BLOCK = {"image", "media", "font", "stylesheet"}

def register_blocking(page):
    page.route("**/*", lambda route:
        route.abort() if route.request.resource_type in BLOCK
        else route.continue_())

In my runs this cuts bandwidth 60–85% depending on the site, with zero effect on the DOM you scrape. Two caveats: keep stylesheets if you rely on :visible selectors (layout needs CSS), and don't block XHR/fetch — that's usually where the data lives.

Putting it together

The full pattern — sticky identity per context, auth done right, resource blocking on every page — is about 40 lines. I keep a copy-pasteable version (plus requests/httpx/Scrapy equivalents) in a public examples repo: github.com/roamproxy/proxy-examples.

If you want to test against a real residential pool without a subscription, the network I work on (Roam) gives 300MB free on signup — enough to verify your Playwright setup against your actual target before spending anything.

Questions about a setup that's misbehaving? Drop it in the comments — proxy-through-browser bugs are weirdly fun to debug.

We Sent 10,000 Requests Through a Residential Proxy Network. Here's the Raw Data.

RoamProxy — Thu, 16 Jul 2026 09:58:07 +0000

Most proxy benchmarks you'll find are vendor marketing: a success rate with no methodology, no failure counts, no raw data. We run a residential proxy network, and we wanted numbers we could actually defend — so we benchmarked ourselves the way we'd want a third party to do it, and published every row.

Here's what 10,000 real requests look like, and what surprised us.

The setup

10,000 HTTPS requests to https://www.cloudflare.com/cdn-cgi/trace — small, anycast, and it echoes the exit IP and country back at you.
5 target countries (US, UK, Germany, Japan, Brazil) × 2,000 requests each.
Every request forced a fresh session (new exit IP requested from the pool each time).
Plain curl, concurrency 20, 30s timeout, zero retries. Every failure stays in the dataset.
Vantage point: a server in Tokyo (AS3258).

A request only counts as a success if it returned HTTP 200 within the timeout — DNS, proxy CONNECT, TLS handshake and body included.

The numbers

Country	Success	Median	p95	Unique exit IPs (of 2,000)
US	99.80%	2.00s	7.27s	1,827
UK	99.80%	2.37s	8.23s	1,533
Germany	99.75%	2.39s	12.22s	1,651
Japan	99.70%	2.03s	14.01s	1,026
Brazil	99.30%	2.73s	17.18s	1,399

Overall: 99.67% success, 7,436 unique exit IPs across 10,000 requests. Geo-targeting accuracy was 98.9–100% per country.

Three things that surprised us

1. The tail is 5–6× the median. Medians cluster tightly at 2.0–2.7s everywhere, but p95 stretches to 12–17s in Germany, Japan and Brazil. If you set client timeouts based on the median, you'll drop a meaningful slice of otherwise-successful requests. Budget for the tail: 20–30s timeouts for residential traffic.

2. 74% of requests got a globally unique IP. With a forced new session per request, 7,436 of 10,000 requests exited through an address no other request in the test saw. The US pool was the deepest: 1,827 unique IPs in 2,000 requests (91%).

3. IPv6 is a big share of real residential exits. Between 42% and 76% of exits per country were IPv6. If your target or your tooling silently breaks on IPv6, that's a chunk of your error budget gone before you've done anything wrong.

Reproduce it (please do)

The whole thing is one loop:

for i in $(seq 1 2000); do
  curl -s -o /dev/null -m 30 \
    -w "%{http_code},%{time_total},%{time_starttransfer}\n" \
    -x "http://USER-country-us-session-$i:PASS@gateway:port" \
    https://www.cloudflare.com/cdn-cgi/trace
done

Point it at any provider, diff the results. Running this against a competitor takes an afternoon, which is exactly why we think publishing raw data should be table stakes in this industry.

Honest limitations

One vantage point (Tokyo), one small target. A heavy page on a slow origin behaves differently — this measures the network, not your target site.
Success against Cloudflare's trace endpoint ≠ success against aggressive bot defenses. That depends on your client fingerprint as much as the IP.
It's our own network, measured by us. That's why the raw CSV (exit IPs anonymized to /24 & /48) and full methodology are public — verify, don't trust.

Full write-up + downloadable CSV (CC BY 4.0): roamproxy.com/residential-proxy-benchmark · code samples: github.com/roamproxy/proxy-examples

Residential vs. Datacenter Proxies: Which One Do You Actually Need?

RoamProxy — Tue, 14 Jul 2026 15:44:48 +0000

If you're scraping the web or managing multiple accounts, the residential-vs-datacenter question decides both your success rate and your bill. Here's the practical breakdown I wish someone had given me earlier — including when the "worse" option is actually the right call.

The core difference

Datacenter proxies come from cloud providers and hosting companies. The IP blocks are publicly registered to companies like OVH or Hetzner, so any target site can (and does) look up the ASN and see "this visitor is a server, not a person."

Residential proxies are IP addresses assigned by consumer ISPs (Comcast, Vodafone, China Telecom...) to real households. To a target site, a residential IP is indistinguishable from a normal visitor on their couch.

That single difference drives everything else:

	Datacenter	Residential
Speed	Fast, stable (~10–50 ms)	Slower, variable (~100–800 ms)
Cost	Cheap (~$0.5–1/GB)	2–5× more (~$2–8/GB)
Block resistance	Weak — ASN is a giveaway	Strong — looks like real users
IP pool size	Thousands	Millions, rotating
Best for	APIs, unprotected sites, speed-critical jobs	Protected sites, sneakers, social, ads verification

A decision rule that actually works

Ask two questions:

1. Does the target actively fight bots?
Run a quick probe through a datacenter IP. If you get 200s all day, congratulations — use datacenter and pocket the savings. If you see 403s, CAPTCHAs, or empty shells of pages, the site checks IP reputation and you need residential.

2. Is per-request latency critical?
Price monitoring across 100k SKUs cares about throughput; a checkout bot cares about a single fast round-trip. Datacenter wins raw speed. Residential wins effective speed once you count retries — a 300 ms residential request beats a 30 ms datacenter request that gets blocked and retried five times.

The hybrid pattern most teams end up with

In practice, mature scraping stacks route per-target:

import requests

DC   = "http://user_dc_1:pass@gw.example.com:41080"    # cheap, fast
RESI = "http://user-country-us:pass@gw.example.com:41080"  # stealthy

ROUTES = {
    "api.competitor.com":   DC,    # plain JSON API, no protection
    "www.sneaker-shop.com": RESI,  # aggressive bot detection
}

def get(url, host):
    proxy = ROUTES.get(host, DC)   # default to cheap
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

Start every new target on datacenter. Promote it to residential the first time you hit sustained 403s. Your blend usually lands around 70/30 datacenter/residential by request count — but 30/70 by importance.

Three mistakes to avoid

Buying residential for everything. You'll pay 3× for traffic that a $0.8/GB datacenter pool would have handled fine.
Buying datacenter for protected sites and "solving" blocks with more IPs. IP reputation databases classify entire datacenter ranges; rotating within a burned ASN just burns money.
Ignoring session stickiness. For logins and carts you need the same IP across a flow. Both proxy types support sticky sessions — use them instead of fighting random rotation.

TL;DR

Unprotected targets + speed → datacenter
Bot detection, geo-restrictions, accounts → residential
Real projects → both, routed per target

I work on RoamProxy, a pay-as-you-go proxy service (residential $2/GB, datacenter $0.8/GB, same gateway endpoint for both). Code examples for Python/Node/Go are on our GitHub.

Building a Rotating Proxy Pool in Python That Survives Failures

RoamProxy — Mon, 13 Jul 2026 10:35:37 +0000

If you scrape the web at any real scale, a single proxy will not cut it. IPs get rate-limited, temporarily banned, or simply time out. The durable fix is a pool of proxies plus logic that rotates through them and retries failed requests on a fresh IP.

Here is how to build one in Python that does not fall over the first time a proxy misbehaves.

Why one proxy is never enough

A single IP hitting a site hundreds of times per minute looks nothing like a human. Target sites respond with 429 Too Many Requests, throw CAPTCHAs, or quietly start returning empty pages. Even a "good" residential IP has a bad day: the upstream node drops, the TCP connection hangs, DNS flakes.

So two things are non-negotiable in production:

Rotation — spread requests across many IPs.
Retry on a different IP — when a request fails, don't just retry the same dead proxy; pick a new one.

The naive version (and why it breaks)

import requests

proxy = "http://user:pass@gw.roamproxy.com:41080"
r = requests.get("https://example.com", proxies={"http": proxy, "https": proxy})

One bad IP and this raises, with no rotation and no retry. Wrap it in a loop and you're already reinventing a pool — badly. Let's do it properly.

A resilient pool

The core idea: on every attempt, pick a proxy, and treat both exceptions and retryable status codes (429, 5xx) as "try again on a fresh IP," with exponential backoff.

import random
import time
import requests


class ProxyPool:
    RETRYABLE = {429, 500, 502, 503, 504}

    def __init__(self, proxies, max_retries=3, timeout=15):
        self.proxies = list(proxies)
        self.max_retries = max_retries
        self.timeout = timeout

    def _pick(self):
        return random.choice(self.proxies)

    def get(self, url, **kwargs):
        last_exc = None
        for attempt in range(self.max_retries):
            proxy = self._pick()
            try:
                r = requests.get(
                    url,
                    proxies={"http": proxy, "https": proxy},
                    timeout=self.timeout,
                    **kwargs,
                )
                if r.status_code in self.RETRYABLE:
                    raise requests.HTTPError(f"retryable status {r.status_code}")
                return r
            except requests.RequestException as e:
                last_exc = e
                time.sleep(2 ** attempt)  # 1s, 2s, 4s...
        raise last_exc


pool = ProxyPool([
    "http://user:pass@gw.roamproxy.com:41080",
    # ...add as many gateway endpoints or credentials as you like
])

resp = pool.get("https://httpbin.org/ip")
print(resp.json())

A few things worth calling out:

Rotate on every attempt, not just on success. The whole point is that attempt 2 uses a different IP than attempt 1.
Exponential backoff (2 ** attempt) keeps you from hammering a struggling target.
Treat 429/5xx as retryable, but let real client errors like 404 return normally — retrying them just wastes IPs.

With most gateway providers you don't even maintain a list of raw IPs. You point at one endpoint and get a new exit IP per request automatically. Rotation for free:

# rotating: a fresh IP on every request
proxy = "http://user-country-us:pass@gw.roamproxy.com:41080"

When you actually want the same IP

Rotation is the default, but some flows break if the IP changes mid-way: logins, add-to-cart, anything multi-step behind a session cookie. That's a sticky session — pin a session id in the username so the gateway keeps handing you the same exit IP until you rotate it:

# sticky: same IP while the session id stays constant
sticky = "http://user-country-us-session-abc123:pass@gw.roamproxy.com:41080"

s = requests.Session()
s.proxies = {"http": sticky, "https": sticky}
s.post("https://example.com/login", data=creds)   # same IP...
s.get("https://example.com/dashboard")             # ...as this one

Change the session id and you get a new IP. That single lever — rotate vs. pin — covers the vast majority of scraping needs.

Don't reinvent the whole thing

The pattern above (rotate, retry-on-failure with backoff, plus sticky sessions and a pool that recycles slots) is common enough that we packaged it into a tiny zero-dependency SDK: roamproxy-python. It gives you a Client, a StickySession context manager, and a SessionPool that retries on 5xx/429/connection errors out of the box:

from roamproxy import SessionPool

pool = SessionPool(retry_on_failure=True)
resp = pool.get("https://httpbin.org/ip")

It's MIT-licensed — copy the ideas, or pip install it, whatever's faster for you.

Wrapping up

A production scraper isn't "requests + a proxy." It's a pool that rotates, retries on a fresh IP, backs off, and can pin a sticky session when a flow needs one. Build those four behaviors in and most "the site keeps blocking me" problems quietly disappear.

If you'd rather not run your own IP infrastructure, that's the problem RoamProxy solves — pay-as-you-go residential and datacenter proxies from $2/GB, with the rotating/sticky gateway shown above. Either way, the engineering above is the part that matters.