DEV Community: Miller James

Building a Proxy Failover Layer With Circuit Breaker Logic

Miller James — Thu, 18 Jun 2026 01:35:02 +0000

Your scraper keeps firing requests at a proxy that's already dead — each one waits out the full timeout, retries pile up, and one bad residential proxy drags the whole job's latency through the floor. A circuit breaker fixes this by giving every proxy a three-state switch: after N failures it trips Open and fails fast instead of waiting, then after a cooldown it sends one trial request to test recovery before trusting the proxy again. Put a failover selector in front that skips any proxy whose circuit is Open, and a single dead endpoint stops poisoning the pool. This guide builds both in Python, including the error-classification logic that decides what should actually trip a breaker — the part most failover code gets wrong.

What circuit breaker logic adds over retry-and-rotate

A circuit breaker beats plain retry-and-rotate because it remembers which proxies are failing and stops calling them. Retry-and-rotate is stateless: it catches an error, moves to the next proxy, and on the very next request happily routes back to the proxy that just failed. A circuit breaker is a state machine that holds failure history per proxy, so a dead endpoint gets benched instead of retried in a loop.

The pattern has three states, originally formalized for fault-tolerant systems in Michael Nygard's Release It! and popularized in Martin Fowler's "CircuitBreaker" article (2014). In the Closed state, requests flow and the breaker counts failures; once failures hit a threshold it trips to Open, where every call fails fast without touching the proxy; after a reset timeout it moves to Half-Open and lets one trial request through to decide whether to close again or re-open. That trial is the self-healing part retry loops lack — the proxy gets re-tested automatically, not permanently blacklisted.

The payoff is concrete: fail-fast in the Open state means a dead proxy costs microseconds, not a 15-second timeout per request. So instead of N retries × full timeout burning while a proxy is down, the breaker short-circuits after the threshold and your failover selector spends that time on a working IP. That difference compounds hard at scale, where a handful of dead residential proxies in a large pool would otherwise eat most of your request budget.

[Image: State diagram of a proxy circuit breaker — three nodes Closed/Open/Half-Open with transitions labeled "fail_threshold reached", "reset_timeout elapsed", "trial succeeds", "trial fails" | Purpose: anchor the three-state model before the code | Alt: Circuit breaker state machine for residential proxy failover showing Closed, Open, and Half-Open transitions]

Decide what actually counts as a proxy failure

Classify every error before you let it trip a breaker — counting the wrong failures is how good proxies get wrongly retired. The critical distinction: a failure is the proxy's fault only when the proxy failed to deliver a response. A 404 or a 500 that came back through the proxy means the proxy worked perfectly; the target had the problem. Trip a breaker on those and you'll bench healthy residential proxies for errors they didn't cause.

Sort responses into four buckets, because each one needs a different action:

Proxy fault — connection refused, connect timeout, HTTP 502/503/504 from the gateway, or a ProxyError. The proxy itself failed. Trip the breaker.
Proxy auth error — HTTP 407. This is a config bug (bad credentials or session-token format), not a transient fault. Trip and alert; rotating won't fix it.
IP banned — HTTP 429, or a 403 carrying a block/CAPTCHA page. The exit IP got flagged, not the endpoint. Trip the per-IP breaker and rotate the IP, but don't penalize the gateway.
Target outcome — 200, 301, 404, or even a target-origin 500 returned through the proxy. The proxy did its job. Never trip; hand the response back to your application logic.

import requests

def classify(response=None, exc=None):
    if exc is not None:
        if isinstance(exc, (requests.exceptions.ProxyError,
                            requests.exceptions.ConnectTimeout,
                            requests.exceptions.ConnectionError,
                            requests.exceptions.ReadTimeout)):
            return "proxy_fault"      # nothing came back -> proxy's fault
        return "unknown"

    code = response.status_code
    if code == 407:
        return "proxy_auth"           # credentials/session-token wrong
    if code in (502, 503, 504):
        return "proxy_fault"          # gateway/upstream failure
    if code == 429 or (code == 403 and looks_like_block(response)):
        return "ip_banned"            # rotate the IP, not the endpoint
    return "target_outcome"           # 200/404/302/target-5xx -> not the proxy

looks_like_block is yours to define per target — match the CAPTCHA marker or block-page signature you actually see, since a generic 403 can be either a real block or ordinary authorization. Get this classifier right and the rest of the layer almost configures itself; get it wrong and no threshold tuning will save you.

Build the circuit breaker state machine

Build the breaker as a small object that owns its state, thresholds, and timing — one instance per proxy. Use a monotonic clock for the timeouts so a system clock adjustment can't corrupt the Open-state window.

import time

class CircuitBreaker:
    """CLOSED -> OPEN -> HALF_OPEN -> CLOSED/OPEN."""

    def __init__(self, *, fail_threshold=4, reset_timeout=30.0, half_open_max=1):
        self.fail_threshold = fail_threshold
        self.reset_timeout = reset_timeout
        self.half_open_max = half_open_max
        self.state = "CLOSED"
        self.failures = 0
        self.opened_at = 0.0
        self.half_open_calls = 0

    def allows(self) -> bool:
        """True if a request may pass right now."""
        if self.state == "OPEN":
            if time.monotonic() - self.opened_at >= self.reset_timeout:
                self.state = "HALF_OPEN"
                self.half_open_calls = 0
            else:
                return False          # fail fast — don't touch a dead proxy
        if self.state == "HALF_OPEN" and self.half_open_calls >= self.half_open_max:
            return False              # only N trial calls while recovering
        if self.state == "HALF_OPEN":
            self.half_open_calls += 1
        return True

    def record_success(self):
        self.failures = 0
        self.half_open_calls = 0
        self.state = "CLOSED"

    def record_failure(self):
        self.failures += 1
        if self.state == "HALF_OPEN" or self.failures >= self.fail_threshold:
            self.state = "OPEN"
            self.opened_at = time.monotonic()

The two non-obvious safeguards: a single failure in Half-Open re-opens the circuit immediately (a recovering proxy doesn't get the full threshold again), and half_open_max caps trial traffic so you never flood a proxy that's just coming back. Verify the machine in isolation before wiring it to network calls — feed it failures until it opens, advance a fake clock past reset_timeout, confirm allows() returns True exactly once, then feed it a success and confirm it closes.

Put a failover selector in front of your residential proxy pool

The failover selector is the layer that turns per-proxy breakers into a resilient pool — it picks a proxy whose circuit allows traffic, runs the request, and reports the outcome back to that proxy's breaker. Round-robin across the healthy circuits so traffic spreads instead of pinning the first available proxy.

class FailoverPool:
    def __init__(self, proxies, breaker_factory):
        self.entries = [{"proxy": p, "breaker": breaker_factory()} for p in proxies]
        self.cursor = 0

    def _pick(self):
        n = len(self.entries)
        for i in range(n):
            entry = self.entries[(self.cursor + i) % n]
            if entry["breaker"].allows():
                self.cursor = (self.cursor + i + 1) % n   # rotate for fairness
                return entry
        return None                                       # every circuit is open

    def execute(self, fn):
        """Run fn(proxy) through healthy proxies until one succeeds."""
        last = None
        for _ in range(len(self.entries)):
            entry = self._pick()
            if entry is None:
                raise RuntimeError("All proxy circuits are open")
            breaker = entry["breaker"]
            try:
                resp = fn(entry["proxy"])
            except Exception as exc:
                if classify(exc=exc) == "proxy_fault":
                    breaker.record_failure()
                    last = exc
                    continue
                raise                                     # not the proxy's fault
            kind = classify(response=resp)
            if kind in ("proxy_fault", "proxy_auth", "ip_banned"):
                breaker.record_failure()
                last = resp
                continue
            breaker.record_success()
            return resp                                   # target_outcome -> done
        raise RuntimeError(f"Failover exhausted; last={last}")

Notice what execute returns on a target_outcome: the raw response, even if it's a 404, because the proxy succeeded and your application — not the failover layer — owns target errors. When _pick returns None, every circuit is Open; that's your signal to wait, degrade to a fallback tier, or fail the job rather than spin. The next sections decide where these breakers should live and how to size their thresholds.

Choose where the breaker lives: per-IP, per-endpoint, or per-provider

Place the breaker at the level you can actually control — that depends entirely on whether you hold individual IPs or hit a gateway. This single decision determines whether your failover layer even works, so make it before writing thresholds.

With dedicated residential proxies — a static list of discrete host:port IPs you own — put one breaker per IP. You can see and retire each IP independently, which is exactly what the FailoverPool above assumes.

With a backconnect residential proxy gateway — one endpoint that rotates exit IPs for you — you can't breaker individual IPs, because you don't address them. Here the breaker belongs at the endpoint level (is the gateway responding at all?), and IP-level failure is handled by classification: a 429 or block triggers a session rotation (new IP from the gateway), not a tripped endpoint breaker.

With multiple residential proxy providers, add a breaker per provider on top, so you can fail over from one network to another when an entire provider degrades. A backconnect gateway already does IP-level failover internally, so your job there is endpoint- and provider-level breaking, not per-IP — building per-IP logic on top of a pool you can't address is wasted code. Match the breaker granularity to the addressability you have, and the layer stays simple.

Tune the three thresholds

Set thresholds to your proxy economics, then verify against real trip logs — there's no universal number. Three knobs control the breaker, and each trades sensitivity against tolerance.

fail_threshold (start at 3–5). Lower trips faster and suits large pools where benching a proxy costs nothing; higher tolerates transient blips and suits small pools where you can't afford to bench IPs. Below 3, normal network noise trips healthy proxies.
reset_timeout (start at 30–60s for transport faults; 5–15 min for IP bans). This is how long a proxy stays benched. Transient gateway errors recover fast; a flagged exit IP needs far longer, which is why ban-class failures deserve their own longer-timeout breaker.
half_open_max (start at 1). Keep this at one trial. More just risks re-flooding a proxy that hasn't actually recovered.

Add jitter to reset_timeout — randomize it by ±20% — so breakers that opened together don't all transition to Half-Open at the same instant and stampede a recovering gateway. This thundering-herd guard matters most right after a shared upstream outage, when dozens of circuits would otherwise retry in lockstep. Verify your settings by logging every state transition for a day: if proxies trip and recover repeatedly (flapping), raise fail_threshold or reset_timeout; if dead proxies keep getting traffic, lower them. These are engineering starting points to tune against your own data, not measured constants.

Layer fallback tiers for graceful degradation

Stack providers into tiers so the layer degrades gracefully instead of failing hard when a whole pool goes Open. When every circuit in your primary residential proxy network is tripped, the request should fall through to a secondary provider before it ever fails the job.

class TieredFailover:
    def __init__(self, tiers):           # tiers = [primary_pool, secondary_pool, ...]
        self.tiers = tiers

    def execute(self, fn):
        last = None
        for tier in self.tiers:
            try:
                return tier.execute(fn)
            except RuntimeError as exc:   # this tier's circuits all open / exhausted
                last = exc
                continue
        raise RuntimeError(f"All tiers exhausted; last={last}")

Order tiers by cost and quality: a high-trust primary network for normal load, a secondary for overflow and outages, and a final tier that queues or sheds load rather than hammering already-failing infrastructure. Running two independent residential proxy providers as tiers removes the single-vendor outage as a failure mode — if one network has a bad hour, traffic shifts automatically. The boundary to respect: graceful degradation means reduced capacity, not silent data loss, so make the bottom tier visible (alert, queue, or return a clear "capacity exhausted" signal) instead of dropping requests quietly.

Build your own failover layer or use a backconnect residential proxy network?

Build the circuit breaker either way — but build the IP-level failover yourself only if you actually hold and address individual IPs. The state-machine code above is identical regardless; the real question is who retires dead exit IPs and replenishes the pool.

Dimension	Self-built over dedicated proxies	Backconnect residential proxy network
Failover granularity	Per-IP, fully under your control	Per-endpoint; IP failover is internal to the gateway
Who retires dead IPs	You — your breakers and monitoring	The provider, automatically
IP supply / scale	Capped at the IPs you own	Large rotating pool behind one endpoint
Where your breaker lives	Per individual IP	Per endpoint and per provider
Operational load	You run, tune, and watch the layer	Provider handles IP health; you watch the gateway
Best fit	Few dedicated IPs, full control needed	High volume, hands-off IP management

Choose a self-built failover layer over dedicated residential proxies if all three hold: you own a list of individually addressable residential or ISP IPs, AND you need per-IP control (specific exit IPs for specific tasks), AND you can run and monitor the breaker state machine in production.

Choose a backconnect residential proxy network if any one is true: you hit a single rotating endpoint and can't address individual IPs, OR you need dead IPs retired and replaced automatically across a large pool, OR you want failover spanning a residential proxy network far larger than any list you could source and maintain. A backconnect residential proxy service such as proxy001.com handles IP-level failover inside the gateway, so your circuit breaker only needs endpoint- and provider-level logic — the FailoverPool collapses to one entry per gateway, and the code you wrote still runs. Even then, keep a thin endpoint breaker: a backconnect network removes per-IP work, not the need to fail over when a whole gateway degrades. Be wary of "unlimited residential proxies" claims as a substitute for failover design — unmetered bandwidth doesn't mean every exit IP is healthy, and your breaker is what proves which ones are.

Mistakes that make a circuit breaker worse than none

These bugs turn a breaker into a liability — it'll bench good proxies or stampede recovering ones, which is worse than a plain retry loop.

Tripping on target errors. Counting 404s or target-origin 500s as proxy failures retires healthy IPs. Only proxy_fault, proxy_auth, and ip_banned should ever call record_failure.
One shared breaker for the whole pool. A single breaker can't tell which proxy failed, so one bad IP trips the entire pool. One breaker per addressable unit, always.
No jitter on reset_timeout. Breakers that opened together re-test together and stampede the recovering upstream. Randomize the timeout ±20%.
Unbounded Half-Open traffic. Letting many trials through at once re-floods a proxy that hasn't recovered. Cap half_open_max at 1.
No alert on proxy_auth (407). It's a config bug that failover masks by rotating forever. Surface it instead of retrying.
Spinning when all circuits are Open. Tight-looping a fully-tripped pool wastes CPU and delays recovery. Fall to a fallback tier or back off with a wait.

Quick answers

What is a circuit breaker in a proxy failover layer? A per-proxy state machine with three states — Closed (requests flow, failures counted), Open (fail fast, proxy skipped), and Half-Open (one trial request tests recovery) — that stops your client from repeatedly calling a dead residential proxy.

What should trip a proxy circuit breaker? Only proxy-side failures: connection errors, timeouts, HTTP 502/503/504, 407 auth errors, and 429/CAPTCHA bans. Target outcomes like 404 or a target-origin 500 came back through the proxy and must not trip it.

Do I need a circuit breaker with a backconnect residential proxy? Yes, but only at the endpoint and provider level — the gateway handles IP-level failover internally, so you breaker the gateway, not individual exit IPs.

What threshold values should I start with? A failure threshold of 3–5, a reset timeout of 30–60 seconds for transport faults (5–15 minutes for IP bans), and one Half-Open trial — then tune against your own trip logs.

Proxy-Aware HTTP Client in Python: Retries, Timeouts & Residential Proxy IPs

Miller James — Wed, 17 Jun 2026 06:33:25 +0000

Building a Proxy-Aware HTTP Client in Python With Retries, Timeouts, and Sessions

Your scraper works for the first ten requests, then a residential proxy IP times out, a peer device drops offline, and the whole run dies. A proxy-aware HTTP client fixes that by treating failure as the normal case: it sets explicit timeouts, retries transient errors with exponential backoff, reuses connections through a Session, and rotates to a fresh residential proxy IP when one gets blocked. This guide builds that client in Python with the requests library, and every block is copy-paste ready.

You'll end with one assembled client that handles authentication, split timeouts, urllib3 retries, IP rotation, a proxy IP test, and proxy-specific error handling. Read top to bottom and you won't need to stitch together five other tutorials.

Why residential proxy IPs need a retry-and-timeout strategy

Residential proxy IPs fail more often than datacenter IPs, so retries and timeouts aren't optional polish—they're the difference between a run that finishes and one that hangs. A residential proxy IP is an address an ISP assigned to a real home device, then routed through a proxy network so your request appears to originate from that device. That's exactly what makes residential IPs hard to block, and also what makes them unstable: the underlying peer can go to sleep, switch networks, or saturate its uplink mid-request.

Three consequences follow, and they drive every design choice below:

Connections drop more. A peer leaving the network kills an in-flight request, surfacing as a connection error rather than a clean HTTP status. You need connect-level retries.
Latency varies wildly. A datacenter IP answers in tens of milliseconds; a residential route can take seconds. A single short timeout will either kill good-but-slow requests or let dead ones hang. You need split connect/read timeouts.
A blocked IP stays blocked. Retrying the same residential proxy IP against a target that just returned 403 wastes attempts. You need to pair retries with rotation, not just repeat the call.

[Image: Flow diagram of one request passing through Session → timeout → urllib3 retry on the same IP → outer rotation to a new residential IP on block → success | Purpose: show where retries and rotation each apply | Alt: diagram of a proxy-aware Python HTTP client retrying then rotating residential proxy IPs]

Set up a requests Session with residential proxy IPs

Start with a Session, not bare requests.get(), because a Session reuses the underlying TCP connection pool and carries your proxy configuration across every call. For residential proxy IPs, connection reuse matters twice over: it cuts the TLS-handshake cost that residential latency makes expensive, and it gives you one place to attach proxies, headers, and retry behavior.

import requests
from urllib.parse import quote

# Credentials and gateway come from your residential proxy service dashboard.
PROXY_USER = "your_user"
PROXY_PASS = quote("p@ss:word")          # URL-encode special characters
PROXY_HOST = "gateway.proxy001.com"      # rotating residential gateway endpoint
PROXY_PORT = 8000

PROXY_URL = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

session = requests.Session()
session.proxies = {"http": PROXY_URL, "https": PROXY_URL}
session.headers.update({"User-Agent": "proxy-aware-client/1.0"})
session.trust_env = False                # ignore system HTTP_PROXY/.netrc

Two details trip people up. First, the password must be URL-encoded—an unescaped @ or : in the credentials breaks the proxy URL parser, which is why quote() wraps it. Second, the proxy URL scheme is http:// even for the https key. When you request an HTTPS target through a forward proxy, requests opens an HTTP CONNECT tunnel to the proxy and runs TLS end-to-end to the real target, so certificate verification still works normally and you should keep verify=True. The http:// in front of the gateway describes how you reach the proxy, not how you reach the destination.

If you're sourcing IPs from a residential proxy service such as proxy001.com, the gateway hostname, port, and credential format all come from your account dashboard rather than being guessable defaults—pull them from there, don't hardcode assumptions.

Setting session.trust_env = False stops requests from silently picking up HTTP_PROXY/HTTPS_PROXY environment variables or .netrc credentials, which otherwise override your Session proxies and produce "why is it using the wrong IP" bugs.

Set timeouts for flaky residential routes

Always pass a timeout as a (connect, read) tuple, never a single number, because the two failure modes need different limits. In requests, a tuple timeout sets the connect timeout and the read timeout separately—(3.05, 27) waits about 3 seconds to establish the connection and 27 seconds for the first byte of the response. A bare timeout=10 applies 10 seconds to both, which is too long for connecting and often too short for a slow residential read.

# (connect timeout, read timeout) in seconds
FAST_TIMEOUT = (3.05, 15)    # APIs / JSON endpoints
PAGE_TIMEOUT = (5, 30)       # full HTML pages over slow residential routes

resp = session.get("https://httpbin.org/ip", timeout=PAGE_TIMEOUT)

The 3.05 is deliberate: connect timeouts work best as a value slightly larger than a multiple of 3, because that's the TCP retransmission window, per the requests timeout documentation. For residential routes I'd start at a 5 second connect and a 30 second read for HTML pages, then tighten once a proxy IP test (below) shows your real latency distribution.

One sharp edge to know: requests has no built-in total request timeout, and the read timeout is the gap allowed between bytes, not for the whole download. A server that trickles one byte every 20 seconds can keep a read=30 request alive indefinitely. If you need a hard ceiling on total time, enforce it from the outside—run the call under a thread with a deadline, or move to an async client (covered later) that supports a total timeout.

Attaching a default timeout to every request through the Session avoids forgetting it on a single call (a missing timeout means wait forever). Subclass the adapter:

from requests.adapters import HTTPAdapter

DEFAULT_TIMEOUT = (5, 30)

class TimeoutHTTPAdapter(HTTPAdapter):
    def __init__(self, *args, timeout=DEFAULT_TIMEOUT, **kwargs):
        self.timeout = timeout
        super().__init__(*args, **kwargs)

    def send(self, request, **kwargs):
        if kwargs.get("timeout") is None:
            kwargs["timeout"] = self.timeout
        return super().send(request, **kwargs)

Add automatic retries with exponential backoff

Wire retries through urllib3's Retry mounted on an HTTPAdapter, which retries transparently underneath requests—your calling code stays a plain session.get(). This is the right layer for transient failures: connection drops, read timeouts, and overload statuses like 429 and 503 that a second attempt often clears.

from urllib3.util.retry import Retry

retry = Retry(
    total=5,                 # cap on total retries across all categories
    connect=3,               # retry dropped/refused connections (common on residential)
    read=2,                  # retry read timeouts
    backoff_factor=1,        # exponential backoff base, in seconds
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=None,    # retry ALL methods, including POST (see caveat below)
    respect_retry_after_header=True,
    raise_on_status=False,   # return the final response instead of raising
)

adapter = TimeoutHTTPAdapter(
    max_retries=retry,
    pool_connections=20,
    pool_maxsize=20,         # raise for concurrent workers
)
session.mount("http://", adapter)
session.mount("https://", adapter)

Here's what each setting buys you, and where the defaults bite:

backoff_factor controls the wait between attempts. urllib3 sleeps for backoff_factor * (2 ** (retry_number - 1)) seconds, capped at 120 seconds by default, and the very first retry is immediate. With backoff_factor=1 the gaps are roughly 0s, 2s, 4s, 8s, 16s—enough to let a momentarily overloaded target recover without stalling your whole job.
allowed_methods=None retries every HTTP method. By default urllib3 retries only idempotent methods (GET, HEAD, PUT, DELETE, OPTIONS, TRACE) and skips POST and PATCH, because a blind POST retry can double-submit. Set it to None only when your POST endpoints are safe to repeat; otherwise leave the default.
status_forcelist decides which HTTP statuses trigger a retry. The list above targets rate-limit and gateway errors; it deliberately excludes 403, because a 403 from a target usually means this residential IP is blocked, and the fix is rotation, not repetition (next section).
respect_retry_after_header=True honors a Retry-After header on 429/503 responses, so you wait exactly as long as the server asks instead of guessing.

Version note, verified in June 2026: allowed_methods replaced method_whitelist in urllib3 1.26, and method_whitelist was removed in urllib3 2.0. On a codebase pinned below 1.26, use method_whitelist instead, or upgrade.

Rotate residential IPs: gateway vs sticky session

Most residential proxy services hand you one of two rotation models, and you pick the IP behavior by how you connect, not by changing endpoints. Know which one your provider uses before writing rotation logic, because the code differs.

Rotating gateway (backconnect). You connect to a single gateway endpoint, and the provider assigns a residential proxy IP from the pool. Many rotating gateways assign a new IP per new TCP connection; some rotate per request. That distinction is provider-specific—confirm it with your dashboard docs and the proxy IP test below, because it determines whether reusing a Session keeps the same IP or changes it.

Sticky session. You hold one residential IP across many requests by embedding a session token in the username. The exact format varies by residential proxy service, but it commonly looks like:

def gateway_proxy(session_id=None):
    user = PROXY_USER
    if session_id:
        # Provider-specific: confirm the exact separator in your dashboard.
        user = f"{PROXY_USER}-session-{session_id}"
    return f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

Choose by task: use a rotating gateway when each request should look like a different visitor (broad scraping, price checks across many pages), and a sticky session when a flow must keep one identity across steps (logging in, paginating a cart, anything with server-side session state). Reaching for sticky sessions on a job that benefits from fresh IPs just concentrates your footprint on fewer addresses and gets them blocked faster.

[Experience supplement: real per-provider rotation behavior (per-connection vs per-request) and the exact sticky-session username separator from a live proxy001.com account | placed here because rotation format is provider-specific and can't be verified from public docs alone]

Pair retries with rotation so you never hammer a dead IP

urllib3 retries and IP rotation solve different problems, so production code needs both layered together. urllib3's Retry is the inner loop: it handles transient drops and 5xx/429 on the current connection. But when a target returns 403 or starts serving CAPTCHAs, the residential proxy IP itself is burned—retrying on the same connection just reuses that same dead IP and fails identically. That's the outer loop's job: catch a blocked response, throw away the connection, and try again on a fresh IP.

The mechanism that makes this work: on a rotating gateway, a new TCP connection generally pulls a new residential IP, so creating a fresh Session (or a new sticky-session id) forces rotation. Wrap the inner client in an outer rotation loop:

def looks_blocked(resp):
    if resp.status_code in (403, 407, 429):
        return True
    body = resp.text[:2000].lower()
    return "captcha" in body or "access denied" in body

def fetch_with_rotation(url, attempts=4, **kwargs):
    last_error = None
    for n in range(attempts):
        # A new session id forces a fresh residential proxy IP from the gateway.
        s = build_session(session_id=f"rot{n}")
        try:
            resp = s.get(url, **kwargs)          # inner urllib3 retries run here
            if looks_blocked(resp):
                last_error = f"blocked: HTTP {resp.status_code}"
                continue                          # rotate IP, try again
            resp.raise_for_status()
            return resp
        except (requests.exceptions.ProxyError,
                requests.exceptions.ConnectTimeout,
                requests.exceptions.ConnectionError) as e:
            last_error = e
            continue                              # dead IP, rotate
        finally:
            s.close()
    raise RuntimeError(f"All {attempts} residential IPs failed for {url}: {last_error}")

The division of labor: inner urllib3 retries absorb flaky-route hiccups on a single IP (the cheap, common case), and the outer loop spends a fresh IP only when the current one is genuinely blocked (the expensive case). Don't collapse both into one big retry count against a single IP—that burns attempts on an address the target has already rejected, and it's the most common reason "I added retries but it still gets blocked."

Run a proxy IP test to confirm rotation works

Before trusting the client, run a proxy IP test that proves two things: your traffic actually exits through the residential proxy IP, and the IP changes when you expect it to. Hit an IP-echo endpoint through the Session and compare results.

def proxy_ip_test(make_session, samples=5):
    seen = {}
    for i in range(samples):
        s = make_session(session_id=f"test{i}")   # new id = new IP on a rotating gateway
        try:
            r = s.get("https://api.ipify.org?format=json", timeout=(5, 15))
            ip = r.json()["ip"]
            seen[ip] = seen.get(ip, 0) + 1
            print(f"{i}: {ip}  ({r.elapsed.total_seconds():.2f}s)")
        finally:
            s.close()
    print(f"\nUnique IPs: {len(seen)} / {samples}")
    return seen

Read the output against three checks:

None of the returned IPs is your real IP. If your own address appears, traffic is bypassing the proxy—usually a leftover trust_env or a missing proxy on one scheme.
For a rotating gateway, len(seen) > 1. Identical IPs across samples mean rotation isn't firing; you're reusing a connection or the provider rotates per-request and you're not opening fresh connections.
Latency is sane. r.elapsed shows round-trip time; a residential route that consistently exceeds your read timeout tells you to raise the timeout or the provider is overloaded.

Use https://httpbin.org/ip as a second opinion if ipify is rate-limited, and cross-check the IP's geolocation and ASN against what your provider promised—a residential proxy IP should map to a consumer ISP, not a hosting ASN.

[Image: Terminal output of proxy_ip_test showing five distinct residential IPs with per-request latency and a "Unique IPs: 5 / 5" summary line | Purpose: show what a passing proxy IP test looks like | Alt: terminal output of a Python proxy ip test confirming five unique rotating residential proxy IPs]

Handle proxy-specific errors cleanly

Catch proxy exceptions by category, because each one points at a different fix and reacting generically hides the real cause. requests raises a small, specific hierarchy you can branch on:

requests.exceptions.ProxyError — the request never reached the target; the proxy refused, the credentials are wrong, or the gateway is down. Check auth and endpoint first; rotation won't help if the gateway itself is unreachable.
requests.exceptions.ConnectTimeout — couldn't establish a connection in the connect window. On residential routes this often just means a slow or dead peer; rotate to a new IP.
requests.exceptions.ReadTimeout — connected, but the target (or a slow residential link) didn't send data in time. Raise the read timeout or rotate.
requests.exceptions.SSLError — TLS verification failed. With a forward proxy this almost always points at the target's certificate, not the proxy; don't reflexively disable verify.
requests.exceptions.RetryError — urllib3 exhausted its retries (only raised when raise_on_status=True).

A practical handler distinguishes "rotate and retry" failures from "stop, the config is wrong" failures:

def classify_and_react(exc):
    if isinstance(exc, requests.exceptions.ProxyError):
        return "config"      # bad credentials/endpoint — fix, don't rotate
    if isinstance(exc, (requests.exceptions.ConnectTimeout,
                        requests.exceptions.ReadTimeout,
                        requests.exceptions.ConnectionError)):
        return "rotate"      # transient route/IP problem — new IP
    if isinstance(exc, requests.exceptions.SSLError):
        return "inspect"     # target cert issue — investigate, don't blanket-disable
    return "raise"

Note the ordering: ConnectTimeout and ReadTimeout both subclass Timeout, and several proxy errors subclass ConnectionError, so check the most specific types first or you'll mislabel them.

The complete proxy-aware client

Here's everything assembled into one module—Session, authenticated rotating gateway, default timeouts, urllib3 retries, outer IP rotation, and the proxy IP test.

import requests
from urllib.parse import quote
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

PROXY_USER = "your_user"
PROXY_PASS = quote("p@ss:word")
PROXY_HOST = "gateway.proxy001.com"
PROXY_PORT = 8000
DEFAULT_TIMEOUT = (5, 30)

class TimeoutHTTPAdapter(HTTPAdapter):
    def __init__(self, *args, timeout=DEFAULT_TIMEOUT, **kwargs):
        self.timeout = timeout
        super().__init__(*args, **kwargs)

    def send(self, request, **kwargs):
        if kwargs.get("timeout") is None:
            kwargs["timeout"] = self.timeout
        return super().send(request, **kwargs)

def build_session(session_id=None):
    user = f"{PROXY_USER}-session-{session_id}" if session_id else PROXY_USER
    proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"

    s = requests.Session()
    s.proxies = {"http": proxy_url, "https": proxy_url}
    s.headers.update({"User-Agent": "proxy-aware-client/1.0"})
    s.trust_env = False

    retry = Retry(
        total=5, connect=3, read=2,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=None,
        respect_retry_after_header=True,
        raise_on_status=False,
    )
    adapter = TimeoutHTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=20)
    s.mount("http://", adapter)
    s.mount("https://", adapter)
    return s

def looks_blocked(resp):
    if resp.status_code in (403, 407, 429):
        return True
    body = resp.text[:2000].lower()
    return "captcha" in body or "access denied" in body

def fetch(url, attempts=4, **kwargs):
    last_error = None
    for n in range(attempts):
        s = build_session(session_id=f"rot{n}")
        try:
            resp = s.get(url, **kwargs)
            if looks_blocked(resp):
                last_error = f"blocked: HTTP {resp.status_code}"
                continue
            resp.raise_for_status()
            return resp
        except (requests.exceptions.ProxyError,
                requests.exceptions.ConnectTimeout,
                requests.exceptions.ConnectionError) as e:
            last_error = e
            continue
        finally:
            s.close()
    raise RuntimeError(f"All {attempts} residential IPs failed for {url}: {last_error}")

if __name__ == "__main__":
    r = fetch("https://httpbin.org/ip")
    print(r.status_code, r.json())

To adapt it: drop in your real gateway credentials, confirm the sticky-session username separator from your dashboard, and tune attempts, the timeouts, and pool_maxsize against what the proxy IP test reports for your account.

Scaling up: async and concurrency

For high throughput, move from threaded requests to an async client so thousands of concurrent requests don't each block a thread. Two solid options, with version caveats that matter as of June 2026:

httpx mirrors the requests API and adds a true total timeout via httpx.Timeout(connect=..., read=..., write=..., pool=...). The proxy argument changed across versions—proxies= was deprecated and removed in httpx 0.28 in favor of proxy= for a single proxy or mounts= for per-scheme routing. Check your installed version before copying older snippets.
aiohttp passes the proxy per request: await session.get(url, proxy="http://gateway.proxy001.com:8000", proxy_auth=aiohttp.BasicAuth(user, pwd)). It has no built-in retry layer, so port the outer rotation loop and add a backoff helper yourself.

Either way, keep the same architecture from this guide—split timeouts, transient-error retries, and an outer rotation loop on block. Async changes how many requests run at once; it doesn't change what makes residential proxy IPs reliable. Bound your concurrency to what your residential proxy service plan allows, since blasting the gateway past its connection limit produces the same errors you just spent this guide eliminating.

Wrap-up checklist

Before you ship a proxy-aware client, confirm each of these:

Requests go through a Session, with proxies set for both http and https keys and the password URL-encoded.
Every call has a (connect, read) timeout, and a hard total-time ceiling exists for downloads that could trickle.
urllib3 Retry covers connect/read/429/5xx, with Retry-After respected and POST retries enabled only where safe.
A rotation loop spends a fresh residential proxy IP on 403/CAPTCHA, instead of retrying the dead one.
A proxy IP test confirms unique IPs, no real-IP leak, and acceptable latency.
Proxy exceptions are classified into "fix config" vs "rotate" vs "investigate."

Get those six right and the client stops being the fragile part of your pipeline.

Integrating Residential Proxies Into Playwright Test Suites

Miller James — Wed, 10 Jun 2026 01:48:20 +0000

If you want residential proxies to work inside a Playwright test suite, the reliable approach is to configure the proxy at the Playwright config or browser-context level, verify the exit IP inside the suite, and separate sticky-session tests from rotating-session tests instead of treating all browser traffic the same. Playwright’s own docs support proxy settings globally for the test runner and per browser context, which means you can solve most real integration problems without hacks, patched drivers, or custom network layers.

That matters because “proxy works in a one-off script” is not the same as “proxy works in a repeatable test suite.” A test suite needs inheritance rules, environment-variable handling, trace and screenshot support on failure, and a clear strategy for which tests should share an IP versus rotate between runs.

What should your suite architecture look like?

For most teams, the cleanest pattern is to keep proxy settings in playwright.config.ts for the broad default, then override them with test.use() or browser.newContext() only for the projects or specs that genuinely need different routing. Playwright documents all three scopes—global config, per-project overrides, and per-test-file overrides—which is exactly why this architecture stays maintainable as the suite grows.

That structure gives you three useful lanes:

A default lane for ordinary browser tests that all use the same proxy policy.
A geo or region lane where each Playwright project points at a different residential route. wonderproxy
A stateful lane for login or multi-step flows where sticky session behavior matters more than frequent rotation. reddit

Do not start with proxy rotation in every single test. Browser automation is heavy, and newer guidance aimed at Playwright session stability points out that using rotating residential traffic for stateful browser flows can create needless instability and cost if the task really needs continuity. In practice, stable session mapping is usually better for login-based validation, checkout flows, or account-bound regional QA.

This is also where provider-side controls matter more than most tutorials admit. The client code only attaches the proxy; the provider controls the actual residential routing model, including sticky sessions, geo targeting, authentication, and rotation policy.

How do you wire it into Playwright Test?

The shortest production-safe path is: put the proxy in use.proxy, inject credentials from environment variables, and add one explicit verification spec before your real tests. Playwright’s Test config docs state that use.proxy applies proxy settings used for all pages in the test, and the network docs show the standard server, username, password, and bypass shape.

Step 1: Install and prepare Playwright

Start with the normal Playwright setup:

npm init playwright@latest
npm install
npx playwright install

Playwright’s library docs also note that browser downloads can be run explicitly with npx playwright install, and they mention HTTPS_PROXY for download scenarios behind a proxy-aware environment. That matters in CI because sometimes the test proxy and the browser-download proxy are two separate concerns.

Step 2: Put the residential proxy in environment variables

Use a single proxy URL or split fields, but keep credentials out of the repo. A simple .env-style pattern is usually enough for local development and CI secret stores.
Example values:

export PROXY_SERVER="http://YOUR_PROXY_HOST:YOUR_PROXY_PORT"
export PROXY_USERNAME="YOUR_USERNAME"
export PROXY_PASSWORD="YOUR_PASSWORD"

If your provider gives you a one-line URI such as http://user:pass@host:port, converting it into Playwright’s server + username + password object is a common and practical pattern documented in current Playwright proxy guides. That keeps the test config readable and makes it easier to swap proxy accounts or regions without editing the suite code. pixeljets

Step 3: Configure the suite in `playwright.config.ts`

This is the core integration:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 1,
  use: {
    headless: true,
    screenshot: 'only-on-failure',
    trace: 'on-first-retry',
    video: 'on-first-retry',
    proxy: {
      server: process.env.PROXY_SERVER!,
      username: process.env.PROXY_USERNAME,
      password: process.env.PROXY_PASSWORD,
      bypass: 'localhost,127.0.0.1',
    },
  },
});

This works because Playwright explicitly supports proxy in the use object for all pages in the test, and the network docs show that proxy settings can include server, credentials, and bypass rules. The failure-artifact settings here matter too, because when a proxy issue happens in a suite, traces, screenshots, and retry-time videos are often the fastest way to separate navigation failure from authentication failure or session instability.

Step 4: Add a verification test before real suite traffic

Create tests/proxy-smoke.spec.ts:

import { test, expect } from '@playwright/test';

test('proxy exits through expected route', async ({ page }) => {
  await page.goto('https://httpbin.org/ip', { waitUntil: 'domcontentloaded' });
  const body = await page.locator('body').textContent();
  expect(body).toBeTruthy();
  console.log(body);
});

The goal here is not elegant assertions at first. The goal is to verify that Playwright is actually exiting through the proxy before you run more expensive browser flows. This single spec catches a large share of real mistakes: wrong credentials, wrong endpoint type, wrong scheme, or a suite that never inherited the proxy config you thought it had.

How do you handle rotation, sticky sessions, and region-specific projects?

Playwright gives you multiple scopes for proxy configuration, so the best suite design is to match the proxy behavior to the test type instead of forcing one proxy policy across everything. That means rotation for stateless checks, sticky sessions for stateful journeys, and project-level separation for regional coverage.

Use project-level routing for regional test suites

Playwright supports project-level overrides, so you can create one project per target region or route class. That is cleaner than scattering conditionals across test files, and WonderProxy’s region-specific testing guidance follows the same broad idea: parametrize region input rather than rewriting the test logic for every geography.

Example:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'us-tests',
      use: {
        proxy: {
          server: process.env.US_PROXY_SERVER!,
          username: process.env.US_PROXY_USERNAME,
          password: process.env.US_PROXY_PASSWORD,
        },
      },
    },
    {
      name: 'de-tests',
      use: {
        proxy: {
          server: process.env.DE_PROXY_SERVER!,
          username: process.env.DE_PROXY_USERNAME,
          password: process.env.DE_PROXY_PASSWORD,
        },
      },
    },
  ],
});

This is valuable because your assertions stay the same while the environment changes cleanly by project.

Use sticky sessions for stateful browser flows

If the test logs in, traverses multiple pages, keeps a cart, or verifies an authenticated workflow, stable IP behavior usually matters more than frequent rotation. Community troubleshooting around residential Playwright usage also points to sticky sessions as the practical fix when normal browsing breaks under rotating pool behavior.
In suite terms, that means one test run or one browser context should map to one residential session policy. Playwright lets you set the proxy globally or per context, and its docs note that per-context proxy configuration is supported too, which is helpful when one suite mixes stateful and stateless cases.

Example per-context override inside a test:

import { test, expect } from '@playwright/test';

test('stateful flow on a dedicated sticky context', async ({ browser }) => {
  const context = await browser.newContext({
    proxy: {
      server: process.env.STICKY_PROXY_SERVER!,
      username: process.env.STICKY_PROXY_USERNAME,
      password: process.env.STICKY_PROXY_PASSWORD,
    },
  });

  const page = await context.newPage();
  await page.goto('https://example.com');
  await context.close();
});

That pattern is directly supported by Playwright’s network docs, which show proxy configuration at both the global and browser-context levels.

Use `test.use()` for file-level specialization

If one spec file needs a different proxy policy, test.use() is the least noisy override. Playwright documents that config options can be overridden at the test-file or describe-block level with test.use({}).

Example:

import { test } from '@playwright/test';

test.use({
  proxy: {
    server: process.env.ROTATING_PROXY_SERVER!,
    username: process.env.ROTATING_PROXY_USERNAME,
    password: process.env.ROTATING_PROXY_PASSWORD,
  },
});

test('stateless region check', async ({ page }) => {
  await page.goto('https://example.com');
});

That makes it easy to keep one subset of tests on rotating residential routes while leaving the rest of the suite on a more stable default.

What usually breaks, and how do you fix it fast?

Most Playwright proxy failures are not mysterious. They usually come from using the wrong config scope, expecting provider-side rotation to behave like a Playwright feature, or skipping verification until after the full suite already failed.

Symptom: the proxy works in a standalone script but not in the test suite

The likely cause is config inheritance or override scope. Playwright’s config docs make clear that options can be set globally, per project, or per test, so it is easy to accidentally shadow the intended proxy setting in one part of the suite.

Fix: start from one verification spec, inspect the active project, and remove unnecessary test.use() overrides until the default path works again.

Symptom: the browser loads, but the target flow becomes unstable after login

That usually points to session mismatch, not basic proxy failure. Stateful browser flows generally work better with sticky session behavior than with frequent residential IP changes, which is also the advice echoed in newer Playwright proxy stability guidance.

Fix: keep one browser context on one sticky route for the whole login-dependent flow, and separate those tests from stateless checks.

Symptom: SSL or certificate errors appear only on some residential routes

Some proxy providers document cases where certificate installation or HTTPS error handling becomes relevant for residential proxy usage. Playwright’s config docs also show ignoreHTTPSErrors as a supported option, but that should be treated as a controlled debugging step, not the first permanent fix.

Fix: confirm the provider’s certificate guidance first, then use ignoreHTTPSErrors only as a temporary isolation step while you verify whether the issue is trust-chain related.

Symptom: CI installs fail before tests even start

This can happen when browser downloads themselves need proxy-aware environment configuration. Playwright’s library docs mention proxy environment handling such as HTTPS_PROXY for browser downloads, which is separate from the proxy config used at runtime inside test pages.
Fix: configure the CI environment for browser downloads first, then configure the suite proxy for runtime browser traffic second.

CTA

If you need a provider that is straightforward to plug into Playwright test suites, proxy001.com is worth evaluating because its public site highlights residential proxy support, 100M+ residential IPs in 200+ locations, HTTP(S) and SOCKS5 support, credential-based integration, IP whitelisting, and a simple three-step onboarding path of choosing a proxy type and integrating it with your tool. That is exactly the kind of provider-side setup that makes Playwright config-based integration easier to maintain across local runs, CI pipelines, and region-specific test projects.

Compliance note

Use residential proxies in Playwright test suites for approved workflows such as regional QA, ad verification, localized content validation, fraud analysis, and authorized environment testing. Keep request volume, concurrency, and session behavior aligned with your testing scope or contractual permissions, and do not turn proxy integration guidance into a method for bypassing access controls.

Residential Proxies for Geo-Testing: A Practical Developer QA Workflow

Miller James — Thu, 21 May 2026 01:12:20 +0000

Residential Proxies for Geo-Testing: A Practical Developer QA Workflow

Residential proxies are often discussed as if their main purpose is bypassing restrictions. That is the wrong framing for a professional developer workflow.

A safer and more useful use case is geo-testing: verifying how your own website, app, landing page, CDN, or checkout flow behaves for users in different countries or regions.

This guide shows a practical workflow for using residential proxies to test regional user experience without turning the process into scraping, evasion, or account automation.

Use this workflow only for websites, apps, and systems you own or are explicitly authorized to test.

What problem does geo-testing solve?

Modern websites often change behavior based on a visitor's location.

A user in Germany may see a GDPR cookie banner. A user in the United States may see USD pricing. A user in Singapore may be routed to a different CDN edge. A user in an unsupported country may see a different signup flow.

These differences can break quietly.

Common geo-specific issues include:

wrong language or mixed-language pages
incorrect currency or tax wording
broken regional redirects
CDN cache serving the wrong variant
cookie banners missing in required markets
ad landing pages showing the wrong offer
checkout pages available in countries where the product is not sold
unsupported users seeing a confusing error instead of a clear message

Browser language settings can test some localization issues. VPNs can help with quick manual checks. But when behavior depends on network location, ISP routing, or country-level IP detection, a residential proxy can provide a closer approximation of a real user connection in that region.

The goal is not to hide automation. The goal is to verify the user experience from the target market.

Safe use cases for residential proxies

Residential proxies are appropriate for QA when the test is narrow, authorized, and low-volume.

Good use cases:

testing whether /pricing shows the right currency by country
checking whether a cookie banner appears for EU visitors
confirming that regional landing pages do not redirect incorrectly
validating CDN routing and cache behavior
testing localized metadata, page language, and legal copy
checking whether ads send users to the correct regional page
confirming that unsupported markets see a clear availability message

Avoid these use cases:

bypassing rate limits
scraping restricted platforms
creating or managing multiple accounts
avoiding detection systems
bypassing geographic licensing restrictions
accessing content that the site owner intentionally blocks
testing third-party websites without permission

A simple rule:

If you would not be comfortable explaining the test to the domain owner, do not run it.

The geo-testing workflow

Use this workflow before writing any code.

Step 1: Write a specific test question

Bad test question:

Does the website work globally?

Good test question:

When a visitor from Germany opens /pricing,
the page should show EUR pricing,
display the EU cookie banner,
avoid US-only claims,
and return HTTP 200.

Use this template:

When a visitor from [country] opens [URL],
the page should show [expected result],
avoid [wrong result],
and return [technical signal].

Examples:

When a visitor from the United States opens /pricing,
the page should show USD pricing,
avoid EU-only tax wording,
and return HTTP 200.

When a visitor from Germany opens /pricing,
the page should show EUR pricing,
display cookie settings,
and avoid US-only promotional claims.

When a visitor from Singapore opens /login,
the page should load without a redirect loop,
show the login form,
and complete within the timeout threshold.

Step 2: Create a small test matrix

Start small. Five countries and three important URLs are enough for a useful first version.

Example matrix:

Region	URL	Expected result
US	`/pricing`	USD pricing, normal signup CTA
DE	`/pricing`	EUR pricing, cookie settings, no US-only offer
UK	`/pricing`	GBP pricing or approved fallback
SG	`/login`	login form loads, no redirect loop
BR	`/`	correct language or approved fallback

Do not test every page. Test pages where geo-specific mistakes matter:

homepage
pricing page
signup page
checkout page
ad landing page
legal or compliance page
login page
product availability page

Step 3: Define pass/fail rules

A good geo-test should produce a clear pass or fail result.

Weak check:

The page looks okay.

Better checks:

Status must be 200.
Final URL must not contain /unsupported.
Body must include "$" for the US test.
Body must include "€" for the Germany test.
Body must not include "US only" for the Germany test.
Redirect count must be 3 or less.
Response time must be under 5000 ms.

These rules are simple, but they make failures actionable.

Manual smoke test with curl

Before building automation, test one URL manually.

Store proxy credentials in environment variables. Do not paste credentials directly into code.

export PROXY_US="http://username:password@us-proxy.example:8000"
export PROXY_DE="http://username:password@de-proxy.example:8000"
export TARGET_URL="https://www.example.com/pricing"

Test from the US proxy:

curl -x "$PROXY_US" \
  -I \
  -L \
  -H "Accept-Language: en-US,en;q=0.9" \
  "$TARGET_URL"

Test from the Germany proxy:

curl -x "$PROXY_DE" \
  -I \
  -L \
  -H "Accept-Language: de-DE,de;q=0.9,en;q=0.8" \
  "$TARGET_URL"

Check the response:

HTTP/2 200
content-language: de-DE
cache-control: max-age=...
vary: accept-language
cf-cache-status: HIT

Not every site will return the same headers. Focus on what matters for your infrastructure:

HTTP status
final URL
redirect behavior
content-language
cache-control
vary
CDN cache headers such as cf-cache-status, x-cache, or x-served-by

If the manual request already fails, do not automate yet. Fix the obvious problem first.

Build a minimal Node.js geo-test

This version uses one script, one test matrix, and one JSON report.

1. Create the project

mkdir geo-qa
cd geo-qa
npm init -y
npm install undici dotenv

Add this to package.json:

{
  "type": "module",
  "scripts": {
    "geo:test": "node geo-check.mjs"
  },
  "dependencies": {
    "dotenv": "^16.4.7",
    "undici": "^7.0.0"
  }
}

Create .gitignore:

.env
reports/

Create .env.example:

PROXY_US=http://username:password@us-proxy.example:8000
PROXY_DE=http://username:password@de-proxy.example:8000
PROXY_UK=http://username:password@uk-proxy.example:8000
PROXY_SG=http://username:password@sg-proxy.example:8000
PROXY_BR=http://username:password@br-proxy.example:8000

Create your local .env:

cp .env.example .env

Fill in your real proxy credentials in .env.

Do not commit .env.

2. Create the test matrix

Create targets.json:

[
  {
    "name": "US pricing page",
    "region": "US",
    "proxyEnv": "PROXY_US",
    "url": "https://www.example.com/pricing",
    "acceptLanguage": "en-US,en;q=0.9",
    "mustInclude": ["$"],
    "mustNotInclude": ["€", "not available in your region", "US only"],
    "maxRedirects": 3,
    "maxTimeMs": 5000
  },
  {
    "name": "Germany pricing page",
    "region": "DE",
    "proxyEnv": "PROXY_DE",
    "url": "https://www.example.com/pricing",
    "acceptLanguage": "de-DE,de;q=0.9,en;q=0.8",
    "mustInclude": ["€", "Cookie"],
    "mustNotInclude": ["US only", "$"],
    "maxRedirects": 3,
    "maxTimeMs": 5000
  },
  {
    "name": "Singapore login page",
    "region": "SG",
    "proxyEnv": "PROXY_SG",
    "url": "https://www.example.com/login",
    "acceptLanguage": "en-SG,en;q=0.9",
    "mustInclude": ["Login"],
    "mustNotInclude": ["too many redirects", "not available"],
    "maxRedirects": 3,
    "maxTimeMs": 5000
  }
]

Replace the URLs and expected text with your actual site logic.

3. Create the test script

Create geo-check.mjs:

import "dotenv/config";
import fs from "node:fs/promises";
import { ProxyAgent, fetch } from "undici";

const targets = JSON.parse(await fs.readFile("./targets.json", "utf8"));

function nowIso() {
  return new Date().toISOString();
}

function pickHeaders(headers) {
  const keys = [
    "content-language",
    "content-type",
    "cache-control",
    "vary",
    "location",
    "cf-cache-status",
    "x-cache",
    "x-served-by",
    "server"
  ];

  const output = {};
  for (const key of keys) {
    const value = headers.get(key);
    if (value) output[key] = value;
  }

  return output;
}

async function fetchWithRedirects(url, options, maxRedirects) {
  let currentUrl = url;
  let redirectCount = 0;
  let response;

  while (true) {
    response = await fetch(currentUrl, {
      ...options,
      redirect: "manual"
    });

    const isRedirect = response.status >= 300 && response.status < 400;
    const location = response.headers.get("location");

    if (!isRedirect || !location) {
      return { response, finalUrl: currentUrl, redirectCount };
    }

    redirectCount += 1;

    if (redirectCount > maxRedirects) {
      return { response, finalUrl: currentUrl, redirectCount };
    }

    currentUrl = new URL(location, currentUrl).toString();
  }
}

async function runCase(testCase) {
  const proxyUrl = process.env[testCase.proxyEnv];

  if (!proxyUrl) {
    return {
      name: testCase.name,
      region: testCase.region,
      url: testCase.url,
      passed: false,
      error: `Missing environment variable: ${testCase.proxyEnv}`,
      checkedAt: nowIso()
    };
  }

  const dispatcher = new ProxyAgent(proxyUrl);
  const startedAt = Date.now();

  try {
    const { response, finalUrl, redirectCount } = await fetchWithRedirects(
      testCase.url,
      {
        dispatcher,
        headers: {
          "Accept-Language": testCase.acceptLanguage || "en-US,en;q=0.9",
          "User-Agent": "GeoQA/1.0 authorized-testing"
        }
      },
      testCase.maxRedirects ?? 3
    );

    const elapsedMs = Date.now() - startedAt;
    const body = await response.text();

    const statusOk = response.status >= 200 && response.status < 400;
    const redirectOk = redirectCount <= (testCase.maxRedirects ?? 3);
    const timeOk = !testCase.maxTimeMs || elapsedMs <= testCase.maxTimeMs;

    const includeOk = (testCase.mustInclude || []).every((text) =>
      body.includes(text)
    );

    const excludeOk = (testCase.mustNotInclude || []).every((text) =>
      !body.includes(text)
    );

    const checks = {
      statusOk,
      redirectOk,
      timeOk,
      includeOk,
      excludeOk
    };

    return {
      name: testCase.name,
      region: testCase.region,
      url: testCase.url,
      finalUrl,
      status: response.status,
      redirectCount,
      elapsedMs,
      passed: Object.values(checks).every(Boolean),
      checks,
      headers: pickHeaders(response.headers),
      checkedAt: nowIso()
    };
  } catch (error) {
    return {
      name: testCase.name,
      region: testCase.region,
      url: testCase.url,
      passed: false,
      error: error.message,
      checkedAt: nowIso()
    };
  } finally {
    await dispatcher.close();
  }
}

const results = [];

for (const testCase of targets) {
  console.log(`Running: ${testCase.name}`);
  const result = await runCase(testCase);
  results.push(result);

  if (result.passed) {
    console.log(`PASS: ${testCase.name}`);
  } else {
    console.log(`FAIL: ${testCase.name}`);
    if (result.error) console.log(`  Error: ${result.error}`);
    if (result.checks) console.log(`  Checks: ${JSON.stringify(result.checks)}`);
  }
}

await fs.mkdir("./reports", { recursive: true });

const reportPath = `./reports/geo-report-${Date.now()}.json`;
await fs.writeFile(reportPath, JSON.stringify(results, null, 2));

const failed = results.filter((result) => !result.passed);

console.log("");
console.log(`Report: ${reportPath}`);
console.log(`Total: ${results.length}`);
console.log(`Passed: ${results.length - failed.length}`);
console.log(`Failed: ${failed.length}`);

if (failed.length > 0) {
  process.exitCode = 1;
}

Run the test:

npm run geo:test

Example output:

Running: US pricing page
PASS: US pricing page
Running: Germany pricing page
FAIL: Germany pricing page
  Checks: {"statusOk":true,"redirectOk":true,"timeOk":true,"includeOk":false,"excludeOk":false}
Running: Singapore login page
PASS: Singapore login page

Report: ./reports/geo-report-1710000000000.json
Total: 3
Passed: 2
Failed: 1

The report gives your team enough information to debug:

region
URL
final URL
status code
redirect count
response time
selected headers
failed assertions

How to read failures

Do not assume every failed test means the proxy is bad. Most failures fall into one of these categories.

1. Proxy or network failure

Signs:

timeout
407 Proxy Authentication Required
DNS error
very high latency

Check:

proxy credentials
endpoint format
country selection
provider dashboard limits
whether the target site is reachable without the proxy

2. Redirect failure

Signs:

too many redirects
final URL is not the expected page
one country redirects to another country's folder

Check:

CDN redirect rules
server middleware
language folder rules
country detection logic
HTTP-to-HTTPS rules
trailing slash rules

3. Cache variant failure

Signs:

every country sees the same content
Germany sees US pricing
one region sees stale page content
CDN headers show a cache hit for the wrong page variant

Check:

CDN cache key
Vary header
cookie-based personalization
edge worker logic
origin cache headers
stale cache rules

4. Localization failure

Signs:

wrong currency
mixed languages
untranslated form errors
missing cookie banner
incorrect lang or content-language

Check:

i18n routing
pricing API
translation files
browser language priority
IP geolocation priority
user account locale overrides

Add this to CI carefully

Geo-testing can run in CI, but keep it low-volume.

A reasonable setup:

run before major releases
run once per day or once per week
test only owned or authorized domains
use CI secrets for proxy credentials
keep the target URL list short
do not test third-party sites
do not collect personal data
do not store full HTML unless necessary

Example GitHub Actions workflow:

name: Geo QA

on:
  workflow_dispatch:
  schedule:
    - cron: "0 8 * * 1"

jobs:
  geo-qa:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - run: npm ci

      - name: Run geo tests
        run: npm run geo:test
        env:
          PROXY_US: ${{ secrets.PROXY_US }}
          PROXY_DE: ${{ secrets.PROXY_DE }}
          PROXY_UK: ${{ secrets.PROXY_UK }}
          PROXY_SG: ${{ secrets.PROXY_SG }}
          PROXY_BR: ${{ secrets.PROXY_BR }}

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: geo-report
          path: reports/

This is enough for a practical first version.

If you later need browser screenshots, cookie banner interaction, or full checkout validation, add Playwright as a second layer. Do not start there unless fetch-level checks are insufficient.

Compliance checklist

Before running a residential proxy geo-test, confirm each item.

1. We own the domain or have written permission to test it.
2. The purpose is QA, localization validation, CDN verification, or compliance testing.
3. The test does not bypass access controls, rate limits, login restrictions, or payment restrictions.
4. The URL list is limited to pages needed for the test.
5. The request volume is low.
6. Proxy credentials are stored in environment variables or CI secrets.
7. Reports do not store personal data or unnecessary page content.
8. The result helps an engineer fix a real user experience issue.

If any item fails, stop and narrow the scope.

When not to use residential proxies

Residential proxies are not always necessary.

Use simpler tools when possible:

Need	Better first option
Test translated text	Browser locale setting
Preview a country manually	VPN
Check a server response from another region	Datacenter proxy
Validate real regional ISP-like behavior	Residential proxy
Test client-side rendering	Browser automation after basic checks

Use residential proxies only when the test depends on network location or regional IP classification.

Final checklist

Use this as the short version of the workflow:

1. Define the country, URL, and expected behavior.
2. Confirm authorization.
3. Create a small test matrix.
4. Run one manual curl test.
5. Add the Node.js checker.
6. Save JSON reports.
7. Investigate failures by category: proxy, redirect, cache, or localization.
8. Add CI only after the local test is stable.

Residential proxies are useful when they help developers verify how real regional users experience a product.

Used responsibly, they are not a bypass mechanism. They are a controlled QA layer for localization, CDN behavior, regional compliance, and landing page validation.

The safest approach is simple:

Test what you own, keep the scope narrow, collect only what you need, and turn every failure into a fixable engineering issue.

Need residential proxies for geo-testing?

If your team needs residential proxy infrastructure for controlled localization QA, CDN checks, regional landing page validation, or other authorized geo-testing workflows, you can evaluate proxy001.com as one possible option.

Use it the same way this guide recommends using any proxy provider: test only properties you own or are authorized to test, keep the scope narrow, and treat proxy access as part of a responsible QA process.

How to Route Your Python Requests Through a Mobile Proxy

Miller James — Thu, 30 Apr 2026 01:36:15 +0000

Routing Python requests through a mobile proxy is straightforward once you have the right endpoint details. You need a valid proxy URL, a Requests config that actually uses it, and a verification step that proves two things: your traffic is leaving through the proxy, and the exit IP is really on a mobile network rather than just “some other IP.”

That second check matters. A different IP only proves the proxy is active; it doesn’t prove the exit is mobile. If you skip that distinction, you can end up with a working proxy integration that still misses the reason you chose a mobile proxy in the first place.

What Do You Need From Your Mobile Proxy Provider?

Before you write code, collect the exact connection details from your provider dashboard or setup docs. For a standard authenticated endpoint, that usually means a host, port, username, password, and protocol, plus any provider-specific controls for country, city, carrier, or session behavior.
Use this checklist first:

A supported Python 3 environment on your local machine or server. pypi
requests installed: python -m pip install requests.
If your provider uses SOCKS5, install SOCKS support too: python -m pip install "requests[socks]".
Proxy host and port.
Authentication format, usually username:password inside the proxy URL. ppl-ai-file-upload.s3.amazonaws
Protocol, typically HTTP(S) or SOCKS5.
Any targeting controls for country, city, or mobile carrier, if your provider supports them.
Any rotation or sticky-session controls, including where they belong, such as the username, endpoint, or a session token field.
A test endpoint that echoes your outbound IP, such as https://httpbin.org/ip.

If you’re missing the provider-specific parts, don’t guess the format. Go to the provider’s connection instructions and confirm three items before you continue: where session controls live, how geo or carrier targeting is expressed, and whether the same gateway should be used for both http and https.

If you still need a mobile proxy endpoint, look for a provider that documents the setup clearly. Proxy001’s public site says it supports HTTP(S) and SOCKS5, provides APIs, SDKs, and Python integration guidance, and offers location options down to country, city, and mobile carrier in advanced setups, which is exactly the kind of information you need for a Requests-based workflow.

What Should a Mobile Proxy URL Look Like in Python Requests?

Requests expects proxies as a dictionary, and each value must be a full proxy URL with a scheme. For an authenticated HTTP(S) endpoint, the standard format is:

http://username:password@host:port

In most setups, you’ll use the same endpoint for both http and https unless your provider explicitly gives you separate gateways. Here’s the base structure:

import requests

USERNAME = "your_username"
PASSWORD = "your_password"
HOST = "proxy.example.com"
PORT = "10000"

proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOST}:{PORT}"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

If your password contains special characters such as @, :, or /, URL-encode it before building the string. That avoids malformed proxy URLs that often surface later as authentication failures.

from urllib.parse import quote

USERNAME = "your_username"
PASSWORD = quote("p@ss:word/with-specials", safe="")
HOST = "proxy.example.com"
PORT = "10000"

proxy_url = f"http://{USERNAME}:{PASSWORD}@{HOST}:{PORT}"

If your provider gives you a SOCKS5 endpoint, Requests supports it as an optional feature when the SOCKS dependency is installed. Use socks5:// or socks5h:// as the scheme: ppl-ai-file-upload.s3.amazonaws

proxy_url = "socks5h://your_username:your_password@proxy.example.com:1080"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

The difference is small but useful: socks5h resolves DNS on the proxy side, while socks5 resolves DNS on the client side. If your provider recommends one of the two, follow that guidance.

How Do You Use a Mobile Proxy With `requests.get()`?

The fastest way to test the integration is a single request to an IP echo endpoint. The httpbin family exposes /ip, which returns the origin IP in JSON, so it’s a clean first check for whether Requests is using the proxy at all.

import requests

proxy_url = "http://your_username:your_password@proxy.example.com:10000"
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get(
    "https://httpbin.org/ip",
    proxies=proxies,
    timeout=20,
)

response.raise_for_status()
print(response.json())

A successful response should look like this:

{
  "origin": "203.0.113.10"
}

That confirms the request completed and an exit IP was returned. It does not yet confirm that the IP belongs to a mobile carrier.

Here’s a practical tested-setup block you should mirror in your own environment before you move on:

Python environment: any currently supported Python 3 release compatible with your installed Requests version. requests.readthedocs
Library: requests, plus requests[socks] only if your provider requires SOCKS. ## When Should You Switch to requests.Session()?

Use requests.Session() as soon as you need more than a one-off request. Sessions persist settings across requests and reuse connections, which makes repeated calls cleaner and usually more stable than rebuilding every request from scratch.

You should also move to a session when you want retries, shared headers, and explicit timeout behavior in one place. Requests does not set a timeout by default, so adding one yourself is part of a reliable proxy integration rather than a nice-to-have.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

proxy_url = "http://your_username:your_password@proxy.example.com:10000"

session = requests.Session()

retry = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods={"GET", "HEAD", "OPTIONS"},
)

adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

session.headers.update({
    "Accept": "application/json",
    "User-Agent": "python-requests-mobile-proxy-check/1.0"
})

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = session.get(
    "https://httpbin.org/ip",
    proxies=proxies,
    timeout=(5, 20),
)

response.raise_for_status()
print(response.json())

This is the version to keep if you’re building a script that runs repeatedly. It gives you request reuse, retry handling for transient upstream errors, and explicit proxy routing on the request itself. That last detail matters because the Requests documentation notes that environment proxy variables can override session.proxies, so using the proxies= argument directly makes debugging much clearer.

How Can You Verify That Requests Is Really Using the Mobile Proxy?

You need two checks here, not one. First confirm that your traffic is routed through the proxy. Then confirm that the exit IP is actually on a mobile network. httpbin

Check 1: Confirm the proxy is active

Start with a direct request and a proxied request to the same IP echo endpoint.

import requests

proxy_url = "http://your_username:your_password@proxy.example.com:10000"
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

direct = requests.get("https://httpbin.org/ip", timeout=20)
proxied = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=20)

print("Direct :", direct.json())
print("Proxied:", proxied.json())

If the proxied origin differs from the direct request, the proxy is active. That proves routing. It still doesn’t prove the IP is mobile.

Check 2: Confirm the exit IP is mobile

There are two practical ways to do this.

Option A: Check your provider dashboard or session details.

This is the most direct path when your provider exposes carrier or mobile-network metadata for the session. Proxy001’s public site says advanced setups can target mobile carriers, so the provider side is the first place to confirm whether your endpoint or session is attached to a mobile network configuration.

Option B: Query an IP intelligence service that returns mobile-network metadata.

IPinfo’s developer docs state that its API tiers include carrier detection, and its privacy/community documentation describes an is_mobile field that indicates whether an IP belongs to a mobile network, carrier, or mobile device.

That means a complete verification flow looks like this:

Request httpbin /ip through the proxy and capture the returned IP.
Look up that IP in your provider dashboard, or send it to an IP intelligence API that returns carrier/mobile metadata.
Confirm that the IP is not only different from your direct IP, but also classified as mobile or tied to a mobile carrier.

In other words, changed IP = proxy works; mobile classification = mobile proxy confirmed.

Should You Rotate IPs or Keep a Sticky Session?

Use a sticky session when one workflow needs continuity across several requests. Use rotation when each request can stand alone and you don’t need the same exit IP across the sequence. proxy001

The key point is that Requests doesn’t control this behavior by itself. Rotation and sticky-session behavior usually come from the provider’s endpoint format, username syntax, or dashboard settings, so your next step is operational, not conceptual.

Here’s the practical workflow:

Open your provider dashboard or setup docs and find the session control field.
Check where the provider expects that value, such as the username, endpoint, or a session token parameter.
Rebuild your proxy URL with that exact syntax.
Send the same https://httpbin.org/ip request three times and compare the returned origin values.

Interpret the results like this:

If the origin stays the same across repeated requests, you’re on a sticky session.
If the origin changes between requests, the endpoint is rotating.
If the behavior doesn’t match what you requested, the problem is usually in the provider-side session syntax rather than the Python code.

This is where many tutorials stop too early. The code path can be completely correct while the provider-side session configuration is wrong, so always validate the actual IP behavior instead of assuming the label in the dashboard is enough.

Why Is My Mobile Proxy Not Working in Python Requests?

Most failures come from a short list of issues. Triage them in this order so you can separate code problems from endpoint problems quickly.

`407 Proxy Authentication Required`

This usually means the proxy rejected your credentials or the auth string was malformed. Recheck the username, password, host, port, and any provider-specific session or location fields that must be embedded in the credentials.

If your password contains special characters, encode it before building the proxy URL. A bad character in the string can look like an auth problem even though the root cause is URL formatting.

`requests.exceptions.ProxyError`

A ProxyError usually points to one of four problems: the host or port is wrong, the scheme is wrong, the upstream proxy is unavailable, or the proxy rejected the connection before authentication completed. Requests’ advanced docs also note that proxy configuration can come from environment variables, which can create confusing behavior during debugging if you think you’re using one endpoint but the process is actually inheriting another.

Fastest fix sequence:

Print the final proxy URL format without exposing secrets.
Confirm the scheme matches the provider’s instructions, such as http:// versus socks5h://.
Pass proxies= directly on the request while debugging instead of relying on session.proxies.
Check whether HTTP_PROXY, HTTPS_PROXY, or ALL_PROXY are set in the environment.

The request hangs or times out

Requests does not apply a timeout unless you set one. Add an explicit timeout to every proxied request, and use a tuple like (5, 20) when you want a short connection budget but a slightly longer read budget.

If the timeout persists, compare three cases:

Direct request to the same target.
Proxied request through your current endpoint.
Proxied request through a second endpoint from the same provider, if available.

That sequence tells you whether the slowdown is target-side, proxy-side, or specific to one gateway.

HTTPS fails with SSL errors

Requests verifies TLS certificates by default. If HTTPS proxying fails with certificate-related errors, check whether your local trust store or CA bundle is the issue before you change anything else.
The Requests docs note that you can point REQUESTS_CA_BUNDLE to a CA bundle file when needed. You can also pass verify=False for local debugging, but the same docs warn that doing so makes the request vulnerable to man-in-the-middle attacks, so it shouldn’t stay in production code.

The request ignores the proxy

This usually happens because the proxy URL is missing a scheme or environment proxy settings are interfering with the session-level configuration. Requests expects fully qualified proxy URLs, and its docs explicitly warn that environment values can override session.proxies.

Use a full proxy URL and pass proxies= directly on the request while you debug.

SOCKS5 fails immediately

SOCKS support in Requests is optional. If a SOCKS endpoint fails immediately, install requests[socks] and confirm you used socks5:// or socks5h:// rather than http://.

Retries never happen on 429 or 5xx

Requests does not retry those responses automatically. If you need retries, mount an HTTPAdapter with a Retry policy on your session and keep the retried methods explicit, as shown earlier.

Compliance Note

Use mobile proxies for approved workflows such as regional QA, ad verification, localized content checks, performance testing, fraud research, and other authorized data-collection or testing tasks. Keep request volume reasonable, follow the target service’s terms or API rules, and document why you need a mobile endpoint rather than a residential or datacenter one.
Don’t leave verify=False in production just because it got a test working once. Requests’ own documentation warns that turning off certificate verification weakens transport security and exposes the connection to man-in-the-middle risk.
If you want a provider for this exact setup, Proxy001 is a practical starting point because its public site says you can work with HTTP(S) or SOCKS5 endpoints, use Python-facing integration resources, target by country, city, and mobile carrier in advanced configurations, and request a free test before buying. That fits this workflow well: you need a documented endpoint format, clear authentication, and a way to confirm that the session behavior and carrier targeting match your script before you roll it into a larger job. Start with one authenticated endpoint, run the direct-vs-proxied IP test, confirm the exit IP’s mobile classification through the provider or an IP intelligence API, and then scale the script only after that verification passes. [

How to Set Up a Proxy Rotation System for Your Scrapy or Requests-Based Scraper

Miller James — Fri, 24 Apr 2026 02:59:49 +0000

A workable proxy rotation system has five moving parts: a proxy pool, a selector, request-level proxy injection, failure tracking, and a verification step that proves the outgoing IP behavior matches your design. Requests supports per-request proxies, explicit timeouts, retries through adapters, and optional SOCKS support, while Scrapy’s proxy flow centers on request.meta['proxy'] and HttpProxyMiddleware.

If you already have a scraper and you want to add rotation without a wrapper library, the cleanest implementation is to keep one internal proxy model, rotate at the request layer, put unstable proxies on cooldown instead of deleting them immediately, and verify everything against an IP-echo endpoint before you touch your real target. Requests also documents that session.proxies can be overridden by environment proxy variables, which is why per-request proxy assignment is the safer default in a custom rotator.

Prerequisites

You need three things before you start:

A scraper that already makes successful requests.
A proxy list from your provider, either as full URLs or as host, port, username, and password fields.
A decision about whether your workflow is stateless or session-sensitive.

That last decision matters because both Requests and Scrapy make per-request proxy assignment straightforward, but your target workflow may still require continuity across several requests. Scrapy’s proxy handling uses request metadata, and Requests lets you pass a fresh proxies dict on each call, so the library side is easy; the hard part is deciding when rotation should happen.

Use per-request rotation when each page fetch stands alone. Use a sticky session when login state, carts, multi-step forms, or account flows need the same network identity across several requests.

How Does a Proxy Rotation System Actually Work?

Keep the design simple:

The proxy pool stores usable endpoints and their current state.
The selector chooses the next proxy.
The request hook injects that proxy into the request.
The feedback loop marks success or failure.
The verification step confirms that rotation is actually happening.

That model lines up with how public Scrapy rotation packages are described: they focus on rotating proxies, checking that endpoints are alive, and reacting to failures, not just loading a static text file. github

Your internal proxy record should hold at least:

id
scheme
host
port
username
password
fail_count
cooldown_until
last_status
session_label if your provider exposes sticky-session routing

Here is a minimal model:

from dataclasses import dataclass
from typing import Optional
import time

@dataclass
class ProxyEndpoint:
    id: str
    scheme: str
    host: str
    port: int
    username: Optional[str] = None
    password: Optional[str] = None
    fail_count: int = 0
    cooldown_until: float = 0.0
    last_status: Optional[int] = None
    session_label: Optional[str] = None

    def as_url(self) -> str:
        auth = ""
        if self.username and self.password:
            auth = f"{self.username}:{self.password}@"
        return f"{self.scheme}://{auth}{self.host}:{self.port}"

    def as_requests_proxies(self) -> dict:
        url = self.as_url()
        return {"http": url, "https": url}

And a small pool object:

import random
import time

class ProxyPool:
    def __init__(self, proxies, max_failures=3, cooldown_seconds=120):
        self.proxies = proxies
        self.max_failures = max_failures
        self.cooldown_seconds = cooldown_seconds

    def available(self):
        now = time.time()
        return [p for p in self.proxies if p.cooldown_until <= now]

    def choose(self):
        candidates = self.available()
        if not candidates:
            raise RuntimeError("No proxies are currently available")
        return random.choice(candidates)

    def mark_success(self, proxy, status_code=None):
        proxy.fail_count = 0
        proxy.cooldown_until = 0.0
        proxy.last_status = status_code

    def mark_failure(self, proxy, status_code=None):
        proxy.fail_count += 1
        proxy.last_status = status_code
        if proxy.fail_count >= self.max_failures:
            proxy.cooldown_until = time.time() + self.cooldown_seconds

Don’t overcomplicate failure handling on day one. A timeout may be transient, a 429 usually points to pacing, and a 407 usually means the proxy URL or credentials are wrong.

How Do You Rotate Proxies in Requests Without a Wrapper?

Requests already gives you the pieces you need. The official advanced usage docs show per-request proxies, Session reuse, timeout tuples, proxy authentication in the proxy URL, SOCKS support, and retry support through HTTPAdapter and urllib3.util.Retry.
The values in the sample below are example starter values for this article’s reference implementation, not universal defaults. You should tune them after you run the verification section.

import time
import requests
from requests import Session
from requests.adapters import HTTPAdapter
from requests.exceptions import ProxyError, ConnectTimeout, ReadTimeout, SSLError, ConnectionError
from urllib3.util import Retry

def build_session():
    session = Session()
    session.headers.update({
        "User-Agent": "Mozilla/5.0 (compatible; scraper/1.0)"
    })

    retries = Retry(total=0, backoff_factor=0, status_forcelist=[])
    adapter = HTTPAdapter(max_retries=retries)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session

def should_retry_status(status_code: int) -> bool:
    return status_code in {403, 407, 429, 500, 502, 503, 504}

def backoff_seconds(attempt: int) -> float:
    return min(2 ** attempt, 8)

def fetch_with_rotation(url: str, pool: ProxyPool, attempts: int = 5):
    session = build_session()
    last_error = None

    for attempt in range(attempts):
        proxy = pool.choose()

        try:
            response = session.get(
                url,
                proxies=proxy.as_requests_proxies(),
                timeout=(3.05, 20),
            )

            if should_retry_status(response.status_code):
                pool.mark_failure(proxy, response.status_code)
                last_error = RuntimeError(f"Retryable status {response.status_code}")
                time.sleep(backoff_seconds(attempt))
                continue

            pool.mark_success(proxy, response.status_code)
            return response, proxy

        except (ProxyError, ConnectTimeout, ReadTimeout, ConnectionError, SSLError) as exc:
            pool.mark_failure(proxy)
            last_error = exc
            time.sleep(backoff_seconds(attempt))

    raise RuntimeError(f"All attempts failed. Last error: {last_error}")

This pattern works well for three reasons. Requests says most external calls should have explicit timeouts because requests do not time out by default, and it documents the difference between a single timeout value and a (connect, read) tuple. Requests also notes that session.proxies can be overwritten by environmental proxy settings, which is why the code above passes the proxy at request time.
Use it like this:

proxies = [
    ProxyEndpoint("p1", "http", "proxy1.example.com", 8000, "user", "pass"),
    ProxyEndpoint("p2", "http", "proxy2.example.com", 8000, "user", "pass"),
]

pool = ProxyPool(proxies, max_failures=2, cooldown_seconds=90)

response, proxy = fetch_with_rotation("https://httpbin.org/ip", pool)
print("Used proxy:", proxy.id)
print("Response:", response.text)

Verify Requests Rotation

Do not point this at your production target first. Use an IP-echo endpoint and confirm:

The selected proxy ID changes over several runs.
The returned IP changes the way your provider says it should.
A bad proxy enters cooldown and the code keeps moving.

A basic check loop is enough:

for i in range(10):
    try:
        r, p = fetch_with_rotation("https://httpbin.org/ip", pool)
        print(i, p.id, r.status_code, r.text)
    except Exception as e:
        print(i, "failed", e)

The output shape you want is boring: several successful responses, changing proxy IDs, and no full-stop crash when one endpoint goes bad.

`socks5` vs `socks5h`

Requests supports SOCKS as an optional feature, and its docs draw an important line between socks5 and socks5h: socks5 resolves DNS on the client, while socks5h resolves DNS on the proxy server. If hostname resolution or observed region looks wrong, check that first.

How Do You Rotate Proxies in Scrapy Without a Wrapper?

Scrapy’s proxy path is built around request.meta['proxy']. Zyte’s Scrapy proxy write-up explains that HttpProxyMiddleware reads the proxy value from request metadata, and Scrapy’s published middleware source shows the same thing in code.

So your custom middleware only needs to pick a proxy and attach it before the request reaches the downloader.

The values below are sample project values for the example code, not framework defaults.

# middlewares.py
import random
import time
from scrapy.exceptions import IgnoreRequest

class RotatingProxyMiddleware:
    RETRYABLE_STATUSES = {403, 407, 429, 500, 502, 503, 504}

    def __init__(self, proxies, max_failures=3, cooldown_seconds=120, max_retry_times=3):
        self.proxies = proxies
        self.max_failures = max_failures
        self.cooldown_seconds = cooldown_seconds
        self.max_retry_times = max_retry_times

    @classmethod
    def from_crawler(cls, crawler):
        raw = crawler.settings.getlist("ROTATION_PROXY_LIST")
        proxies = []
        for idx, item in enumerate(raw):
            proxies.append({
                "id": f"p{idx+1}",
                "url": item,
                "fail_count": 0,
                "cooldown_until": 0.0,
                "last_status": None,
            })
        return cls(
            proxies=proxies,
            max_failures=crawler.settings.getint("ROTATION_MAX_FAILURES", 3),
            cooldown_seconds=crawler.settings.getint("ROTATION_COOLDOWN_SECONDS", 120),
            max_retry_times=crawler.settings.getint("ROTATION_MAX_RETRY_TIMES", 3),
        )

    def available(self):
        now = time.time()
        return [p for p in self.proxies if p["cooldown_until"] <= now]

    def choose(self):
        candidates = self.available()
        if not candidates:
            raise IgnoreRequest("No proxies currently available")
        return random.choice(candidates)

    def mark_success(self, proxy, status=None):
        proxy["fail_count"] = 0
        proxy["cooldown_until"] = 0.0
        proxy["last_status"] = status

    def mark_failure(self, proxy, status=None):
        proxy["fail_count"] += 1
        proxy["last_status"] = status
        if proxy["fail_count"] >= self.max_failures:
            proxy["cooldown_until"] = time.time() + self.cooldown_seconds

    def process_request(self, request, spider):
        if request.meta.get("disable_proxy_rotation"):
            return None

        if "rotation_proxy" not in request.meta:
            proxy = self.choose()
            request.meta["rotation_proxy"] = proxy
            request.meta["proxy"] = proxy["url"]

        return None

    def process_response(self, request, response, spider):
        proxy = request.meta.get("rotation_proxy")
        if not proxy:
            return response

        if response.status in self.RETRYABLE_STATUSES:
            self.mark_failure(proxy, response.status)
            retry_times = request.meta.get("rotation_retry_times", 0) + 1
            if retry_times <= self.max_retry_times:
                new_request = request.copy()
                new_request.dont_filter = True
                new_request.meta.pop("rotation_proxy", None)
                new_request.meta.pop("proxy", None)
                new_request.meta["rotation_retry_times"] = retry_times
                return new_request
        else:
            self.mark_success(proxy, response.status)

        return response

    def process_exception(self, request, exception, spider):
        proxy = request.meta.get("rotation_proxy")
        if proxy:
            self.mark_failure(proxy)
            retry_times = request.meta.get("rotation_retry_times", 0) + 1
            if retry_times <= self.max_retry_times:
                new_request = request.copy()
                new_request.dont_filter = True
                new_request.meta.pop("rotation_proxy", None)
                new_request.meta.pop("proxy", None)
                new_request.meta["rotation_retry_times"] = retry_times
                return new_request

        return None

Use settings like this:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 610,
}

ROTATION_PROXY_LIST = [
    "http://user:pass@proxy1.example.com:8000",
    "http://user:pass@proxy2.example.com:8000",
]

ROTATION_MAX_FAILURES = 2
ROTATION_COOLDOWN_SECONDS = 90
ROTATION_MAX_RETRY_TIMES = 3

DOWNLOAD_TIMEOUT = 20
RETRY_ENABLED = False

And a tiny spider:

# spiders/ip_test.py
import scrapy

class IpSpider(scrapy.Spider):
    name = "ip_test"

    def start_requests(self):
        for _ in range(10):
            yield scrapy.Request("https://httpbin.org/ip")

    def parse(self, response):
        self.logger.info("IP response: %s", response.text)

If you need one request to bypass the rotator, disable it explicitly:

yield scrapy.Request(
    "https://internal.example.net/health",
    meta={"disable_proxy_rotation": True}
)

This is the key fix that many tutorials skip: once the middleware is global, your existing spiders should not need per-request “auto proxy” markers. The middleware owns rotation unless a request opts out.

Which Rotation Mode Should You Use?

Use this quick decision rule:

Workflow	Better choice
Public pages where each request stands alone	Per-request rotation
Login, cart, checkout, or account steps	Sticky session
Multi-page QA flow that must keep the same identity	Sticky session
Regional availability checks from several locations	Per-request rotation

If request 2 would break when it comes from a different network identity than request 1, don’t use per-request rotation.

Which Proxy Type Fits This System?

In general, residential proxies are a better fit when the workflow is region-aware or trust-sensitive, ISP/static endpoints are better when a session must stay stable for longer, and datacenter proxies are a better fit for simpler internal checks where speed and cost matter more than continuity. Proxy001’s homepage says the service offers static, dynamic, residential, and mobile proxy services, supports Python and Scrapy, and provides access to 100M+ premium IPs across 200+ regions with a free test option.
That’s the natural point to validate your custom rotator against a commercial pool: after your code works against an IP-echo endpoint and before you use it in a larger workflow.

How Do You Know the Rotation Is Actually Working?

Run verification in this order:

1. Validate one proxy first

Check that one known-good endpoint authenticates correctly, honors your timeout settings, and works over HTTPS.

2. Validate the rotator on an IP-echo endpoint

Log:

selected proxy ID,
returned IP,
status code,
elapsed time,
and whether the proxy entered cooldown.

3. Add one intentionally bad proxy

This proves your failure path actually works. You want to see the bad endpoint fail, accumulate failures, enter cooldown, and stop dragging the whole scraper down.

4. Only then hit your real target

If IP rotation works but the target still returns repeated 403s or 429s, the next place to look is pacing, session continuity, headers, or provider quality.

Why Are You Still Getting 403, 407, or 429 Errors?

A 407 Proxy Authentication Required usually means a bad credential set or a malformed proxy URL. Requests documents proxy auth directly inside the proxy URL, and Scrapy’s proxy path uses the same URL form through request.meta['proxy'].

A 403 does not automatically mean the proxy is dead. It can also mean the workflow is incomplete, the cookies are wrong, or you rotated when the server expected continuity.

A 429 usually points to pacing. Lower your request rate, add backoff, and stop retrying the same path too aggressively.

A hang usually means missing timeouts. Requests says most external requests should use explicit timeouts because otherwise the code may wait for minutes or more.

An SSL failure may come from certificate trust settings. Requests verifies HTTPS certificates by default, allows a CA bundle path, and warns that verify=False weakens security and should be limited to local testing.

A DNS mismatch with SOCKS usually means you chose socks5 when you needed socks5h. Requests documents that distinction directly.

Quick fixes table

Symptom	Likely cause	Fix
407	Bad credentials or malformed proxy URL	Rebuild the URL as `scheme://user:pass@host:port` and test one endpoint first.
403 on a stateful flow	You rotated too often	Move that workflow to sticky session mode.
429	Request pacing is too high	Slow down and add backoff before retrying.
Same exit IP every time	Provider-side sticky routing or a non-rotating endpoint	Test over a longer window and check your provider’s session behavior.
SSL error	Local CA trust issue	Use the correct CA bundle path; keep `verify=False` for local testing only.
Wrong region with SOCKS	DNS resolved on the client	Try `socks5h` instead of `socks5`.

Compliance Note

Use proxy rotation for legitimate engineering work such as QA, uptime checks, ad verification, authorized regional testing, or data collection you are allowed to perform. Keep your request volume reasonable, review the target site’s terms and robots directives where applicable, and document the internal purpose of the scraper before you scale it.

Proxy rotation is not a substitute for sloppy request design. If your headers, session model, retry behavior, or pacing are wrong, adding more endpoints just spreads the same mistake across a larger pool.

If you want to validate this setup with a provider that publicly lists Python and Scrapy support, multiple proxy types, broad regional coverage, and a free test path, Proxy001 is a practical next step for that stage of implementation. The site says it offers static, dynamic, residential, and mobile proxy services and access to 100M+ premium IPs in 200+ regions, which is enough to test whether your rotation logic behaves the way you expect before you expand the pool.

Debugging Geo-Specific API Responses With a Residential IP Proxy

Miller James — Tue, 21 Apr 2026 01:23:39 +0000

Some APIs don't return the same truth to every network. The same endpoint can show local pricing, inventory, eligibility, or content to a residential IP in the target market, then fall back to generic or incomplete data when the request comes from a datacenter range or the wrong region. Residential proxy providers document this behavior from the other side: geo-targeting, sticky sessions, and residential-only location controls exist because many platforms treat traffic differently based on IP type and location.
If you're debugging this kind of mismatch, the job isn't to "use a proxy" in the abstract. The job is to prove whether the egress IP is the variable, confirm the request is actually leaving from the market you think it is, and rule out the other causes that often look identical at first, such as account region, cookies, headers, or session instability. ZenRows' setup guide uses an IP echo check as the first validation step, and Bright Data's geolocation docs show just how granular this can get, from country all the way down to ZIP code and ASN.
Last updated: April 21, 2026. Product and pricing references for Proxy001 in this article are based on publicly accessible pages available on that date. proxy001

Why the same API can return different data from a datacenter IP

Because the API usually isn't making a simple country decision. It may combine IP geolocation, network classification, proxy reputation, session continuity, and market signals from the rest of the request before deciding what response to return. Bright Data's documentation separates country, city, state, ASN, and ZIP targeting, which is a good clue that "same country" often isn't specific enough for location-sensitive testing.
That changes how you should debug the issue. Don't ask, "Why is the API broken?" Ask, "What signals does this API use to decide whether my request qualifies for the local response?" Once you frame it that way, the next step becomes obvious: hold the request constant and test a different exit profile.

Residential proxies are useful here because they route requests through consumer ISP-assigned IPs rather than datacenter ranges. That gives you a way to test whether the API is reacting to where the traffic appears to come from and what kind of network owns that IP. It doesn't guarantee success, and it shouldn't be treated as a loophole; it's a debugging tool for localization QA, ad verification, fraud analysis, market validation, and other legitimate testing workflows. ipgeolocation

Prerequisites: isolate variables before you touch the proxy

If you change the proxy, the cookie jar, the account, and the headers at the same time, you won't learn anything useful. The baseline test and the proxy test need to use the same endpoint, method, query parameters, headers, account state, and time window. The proxy should be the only intentional difference.

For a clean A/B test, capture these fields from both runs:

HTTP status code and any redirect target.
The raw response body or at least the business-critical fields such as currency, availability, region, shipping_options, catalog_visible, or price.
Key response headers that may reveal localization or caching behavior.
The public IP used for the request. ZenRows validates this with https://httpbin.io/ip before and after proxy setup.
Timestamp, because market-facing APIs can change throughout the day. This matters for a repeatable debugging record and aligns with Google's preference for dated, verifiable test evidence in experience-driven content.

Three things regularly get misdiagnosed as "the proxy didn't work":

The account is registered in a different market than the IP you're testing from.
Old cookies or session tokens carry state from a previous region.
Headers such as Accept-Language point to a different locale than the target market.

If any of those change between runs, you no longer have a valid comparison.

Step-by-step: run a minimal A/B test

The shortest path is still the best one. Send one request from your normal network, save it as the baseline, then send the same request through a residential proxy targeted to the market you're testing.

1. Check your baseline public IP

Use an IP echo endpoint first. ZenRows uses https://httpbin.io/ip in its own getting-started flow, which makes this a reasonable default for quick validation.

# Python 3.10+
# pip install requests

import requests

r = requests.get("https://httpbin.io/ip", timeout=30)
print(r.status_code)
print(r.text)

Expected output is a JSON body with your current public IP. If this fails, fix your local network before you introduce a proxy layer.

2. Save the baseline API response

Now send the target request exactly as your application would, but keep the request small and deterministic. If the API takes dozens of optional parameters, strip it down to the ones required to reproduce the issue.

import requests

API_URL = "https://api.example.com/endpoint"
HEADERS = {
    "Accept": "application/json",
    "User-Agent": "geo-debug-test/1.0",
    "Accept-Language": "en-US,en;q=0.9",
}
PARAMS = {
    "sku": "12345"
}

baseline = requests.get(API_URL, headers=HEADERS, params=PARAMS, timeout=30)

with open("baseline.json", "w", encoding="utf-8") as f:
    f.write(baseline.text)

print("baseline_status:", baseline.status_code)
print("baseline_content_type:", baseline.headers.get("content-type"))

Don't guess from memory. Save the body, keep the headers, and note the timestamp.

3. Configure the residential proxy correctly

Most providers use the standard proxy format http://username:password@host:port. ZenRows documents that exact pattern for Python requests.

proxy_username = "PROXY_USERNAME"
proxy_password = "PROXY_PASSWORD"
proxy_host = "PROXY_HOST"
proxy_port = "PROXY_PORT"

proxy_url = f"http://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}"
proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

This is also where people lose time. The proxy may authenticate correctly but still exit from the wrong market because the geo-targeting flag was added to the wrong field. Different providers encode country, state, city, or session settings in different places.

Two documented examples show the pattern clearly:

Bright Data country targeting adds the country to the username string, such as username-country-us, and extends that pattern for city, state, ASN, and ZIP targeting.
ZenRows adds country or region selection to the password string, such as PROXY_PASSWORD_country-ca, and uses TTL/session tokens in the password for sticky sessions, for example PROXY_PASSWORD_ttl-30s_session-....

That means the exact place where you add country-us, state-ny, zip-12345, or ttl-30s is provider-specific. The reliable way to avoid guesswork is to open your provider's dashboard and look for the credential generator or the geolocation targeting page before you run the test. ZenRows tells users to copy username, password, domain, and port from the dashboard.

4. Verify that the proxy is exiting from the expected market

Don't skip this. If the proxy exits from the wrong place, everything after this point is noise.

import requests

proxy_check = requests.get("https://httpbin.io/ip", proxies=proxies, timeout=30)
print(proxy_check.status_code)
print(proxy_check.text)

A changed IP proves that the proxy is active. It does not prove the market is correct.
For a full verification pass, check three things against your normal IP intelligence tool or your provider's diagnostics:

Country or region.
State, city, ZIP, or ASN if the use case needs that granularity.
Whether repeated requests stay on the same IP when you're testing a sticky session.

Bright Data's docs are useful here because they show exactly when you may need to go beyond country targeting:

Country targeting works when the API only localizes at the national level.
City or state targeting matters when content changes by metro or region.
ZIP targeting matters for hyperlocal availability, store inventory, or delivery coverage.
ASN targeting matters when the platform behaves differently across network operators, not just geography.

A quick decision rule helps:

If your API behavior changes by...	Start with...
Currency, country catalog, national pricing	Country targeting
Regional pricing, state-level eligibility, metro-specific content	State or city targeting
Store inventory, delivery windows, neighborhood coverage	ZIP targeting
Network-specific treatment, ISP footprint, carrier-level logic	ASN targeting, then compare results against a different ASN in the same country

5. Send the same API request through the residential proxy

Now repeat the original request with the same headers and params. The proxy should be the only variable you changed intentionally.

proxy_run = requests.get(
    API_URL,
    headers=HEADERS,
    params=PARAMS,
    proxies=proxies,
    timeout=30,
)

with open("proxy.json", "w", encoding="utf-8") as f:
    f.write(proxy_run.text)

print("proxy_status:", proxy_run.status_code)
print("proxy_content_type:", proxy_run.headers.get("content-type"))

If the response changes here, don't stop at "it worked." You still need to prove the change is stable and that it tracks the market you targeted.

6. Diff the business fields, not just the raw body

A response diff is the fastest way to move from suspicion to evidence. Look at the fields your application actually depends on, not just whether the body looks different at a glance.

# pip install deepdiff
import json
from deepdiff import DeepDiff

with open("baseline.json", "r", encoding="utf-8") as f:
    baseline_json = json.load(f)

with open("proxy.json", "r", encoding="utf-8") as f:
    proxy_json = json.load(f)

diff = DeepDiff(baseline_json, proxy_json, ignore_order=True)
print(diff)

Good candidates for comparison are currency, region, availability, delivery_options, store_id, catalog_visible, or any market-specific content flags your downstream logic uses.

How to handle session-based APIs

If the API sets cookies, issues a session token, or builds context over multiple requests, a one-shot test won't tell you enough. You need to preserve the same session and, if required, the same proxy IP across the whole flow. ZenRows documents sticky TTL for exactly this reason: it lets you hold one residential proxy for a fixed duration instead of rotating with each request.

Here's a minimal session-based test:

# pip install requests
import requests
import time

API_URL = "https://api.example.com/quote"
HEADERS = {
    "Accept": "application/json",
    "User-Agent": "geo-debug-test/1.0",
    "Accept-Language": "en-US,en;q=0.9",
}

proxy_username = "PROXY_USERNAME"
proxy_password = "PROXY_PASSWORD_ttl-30s_session-mydebugsession"
proxy_host = "PROXY_HOST"
proxy_port = "PROXY_PORT"

proxy_url = f"http://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}"
proxies = {"http": proxy_url, "https": proxy_url}

s = requests.Session()
s.headers.update(HEADERS)

for i in range(5):
    ip_echo = s.get("https://httpbin.io/ip", proxies=proxies, timeout=30)
    target = s.get(API_URL, proxies=proxies, timeout=30)

    print(f"run={i+1}")
    print("ip:", ip_echo.text)
    print("status:", target.status_code)
    print("set-cookie:", target.headers.get("set-cookie"))
    print("body:", target.text[:300])
    print("-" * 40)

    time.sleep(5)

What you're checking here is straightforward:

Does the IP stay the same for the full TTL window?
Does the API keep returning the same market-specific fields?
Does the session break as soon as the proxy rotates or the TTL expires?

If the answer to the third question is yes, the problem isn't "residential proxy vs datacenter proxy" anymore. It's "this API needs a stable session identity to keep the local state valid."

Real test note: what a useful debugging record looks like

A strong debugging note isn't fancy. It's specific, dated, and honest about what failed first. That's also exactly the kind of firsthand, transparent detail Google's E-E-A-T guidance rewards, especially for proxy and VPN topics where Trust and Experience signals carry extra weight.
A usable internal record looks like this:

Test date: April 21, 2026.
Endpoint under test: /availability?sku=12345.
Baseline network: office datacenter egress in the US.
Proxy test network: residential US IP with sticky session enabled.
Fixed variables: same account, same headers, same params, same code path.
First failed run: returned the same generic payload because the session still carried an old cookie from a previous region.
Fix: recreated the session and reran the same request through the residential proxy.
Verified change: currency stayed USD in both runs, but availability changed from "unavailable" to "in_stock" and delivery_options populated only in the residential run.

If you publish this article, add a redacted screenshot or a trimmed response snippet instead of leaving the example purely abstract. The article is much stronger when the reader can see what "the result changed" actually looks like. Google's guidance explicitly rewards dated test evidence, real problems, and candid limitations over polished but unverifiable claims.

Why a residential proxy can still fail

Because IP location is only part of the picture. ZenRows says this directly in its FAQ: anti-bot and access-control systems look at more than IPs, including headers, fingerprints, and behavior.
Most failures fall into one of these buckets:

Symptom	Likely cause	How to verify	Fix
The residential proxy request returns the same empty or generic payload.	The account, session, or cookies are still pinned to another market.	Repeat the test with a clean `requests.Session()` and no reused cookies.	Clear state, recreate the session, and rerun the A/B test with the same proxy exit.
The response changes, but it's still the wrong region.	Country targeting is too broad for this API.	Check the exit IP's state, city, ZIP, or ASN instead of stopping at country.	Move from country targeting to state, city, ZIP, or ASN targeting based on the use case.
The first request works, later ones flip back or fail.	The proxy rotated before the flow finished.	Log the outbound IP on every request in the flow.	Use a sticky session and make sure the TTL covers the whole request chain.
You get a 407 before reaching the target.	Proxy auth or geo syntax is wrong.	Test the proxy against `httpbin.io/ip` first.	Recheck username, password, host, port, and any country or session token added to the credentials.
You get a 403 even with a residential IP.	The target is reacting to non-IP signals.	Compare headers, request timing, and session behavior between working and failing runs.	Normalize headers, reduce burstiness, and keep the session consistent.
Browser results look local, but script results don't.	Your script leaks a different locale through headers or request params.	Compare `Accept-Language`, locale-related params, and cookie state.	Align locale headers and rerun the test with a clean session.

One mistake shows up a lot in real debugging: people switch to a residential proxy, see that the IP changed, and stop checking. Then they spend an hour arguing about the proxy pool when the real cause is a stale cookie or a flow that quietly rotated off the original IP after the first request. That's why the IP echo check and the session-stability check belong in the same test run, not in separate troubleshooting branches.

Verify success

You're done when the result is repeatable and clearly tied to the IP context. A single good response isn't enough.

Use these three acceptance checks:

The baseline and proxy responses differ in the business fields that matter to your application, such as localized inventory, market eligibility, shipping options, or catalog visibility.
The proxy run stays consistent over a short repeat test, especially if the workflow depends on a sticky session.
Switching back to the original network reproduces the original fallback or generic behavior.

If all three conditions hold, you've done more than "get it working." You've shown that the API response is geo-specific and that the difference is stable enough to support a production fix.

Choosing a residential proxy for this debugging job

You don't need a giant provider comparison for a debugging task. You need a provider that exposes the right location controls, lets you keep a session stable when needed, and makes it easy to generate credentials without guesswork.

Proxy001 fits that workflow in a practical way. Its residential proxy page publicly lists country-level inventory figures for several markets, including 2,533,691 IPs in the United States, 496,658 in the United Kingdom, 131,650 in Germany, 813,753 in India, 244,905 in Malaysia, and 239,355 in Vietnam, and it highlights usage monitoring, sub-user management, IP whitelisting, global coverage, security, and anti-blocking. That matters in this use case because geo-debugging is mostly a control problem: you need enough location coverage to reproduce the market, enough session stability to hold the test together, and enough account controls to keep test traffic separated from production.

Publicly indexed Proxy001 pages also indicate volume-tiered residential pricing from $2.00/GB down to $0.70/GB at higher volume and mention a 500MB free trial for new users, but because pricing pages change, confirm the live trial and billing details on the product page before you wire anything into your scripts. That's the right way to use a provider in a debugging workflow: validate the location controls and session behavior against your own API first, then decide whether it deserves a production slot. proxy001

Compliance note

Use this workflow for legitimate testing: localization QA, ad verification, fraud investigation, data quality checks, or debugging why a market-facing API behaves differently across regions. Residential proxies are a testing input here, not a blank check to ignore API terms, access restrictions, or data-use policies.

That line matters for quality, not just compliance. If your team can't document what market you tested, what IP context you used, what stayed constant, and what changed in the response, the result isn't solid enough to drive engineering decisions. Google's Trust guidance is explicit on this point: transparent methods, traceable claims, and honest limitations matter more than polished marketing language, especially in sensitive categories like proxy services.

If your goal is to prove that a geo-specific API behaves differently for real local users, start with one clean A/B test and keep the scope tight. Verify the exit IP before each serious run, choose the smallest geo-targeting granularity that matches the behavior you're investigating, and switch to a sticky session the moment the API starts building state across requests. Proxy001 is a reasonable place to start because its residential proxy page gives you clear visibility into country-level coverage before you test. Check the live residential proxy page for the current credential format, targeting options, onboarding path, and billing details before you move from debugging into production use.

Load Balancing Across a Residential Proxy Pool in Production

Miller James — Mon, 20 Apr 2026 01:24:28 +0000

A production residential proxy pool should optimize for stable throughput, controlled per-IP pressure, and fast recovery from bad nodes. It should not aim for perfectly equal request counts, because residential endpoints vary in latency, availability, and session stability in ways homogeneous server pools usually do not.

That’s why plain round robin is a weak default. Weighted or score-based routing works better because it can shift traffic away from proxies that are overloaded, degraded, or temporarily unavailable.

There’s also an important compliance line here. If a target starts returning 429 or repeated 403 responses, the correct move is to reduce or suspend traffic for that target and review your allowed request budget, not to use routing logic to keep pressing the same target through other IPs. [

What does “balanced” mean in a residential proxy pool?

Balanced means traffic is distributed according to each proxy’s current health and usable capacity, not split into identical request counts.

For a real residential proxy pool, measure these fields per proxy:

Success rate over a rolling window.
P95 latency.
Current in-flight requests.
Recent timeout and connection-error count.
Recent 429 and 403 signals by target host, when those signals apply to your workflow. If you don’t record those metrics, you can still rotate proxies, but you can’t prove you’re balancing them safely. That’s the difference between a proxy list and a production scheduler. joinmassive

What components do you need in production?

You need five components: a proxy registry, a scheduler, a feedback loop, a session-affinity map, and metrics storage. joinmassive

The proxy registry tracks endpoint details and live state such as in_flight, recent outcomes, and cooldown status. The scheduler selects the next proxy. The feedback loop updates proxy health after each request. The session-affinity map preserves continuity for workflows that need it. Metrics storage gives you the evidence to verify that load distribution is improving rather than just moving failures around. my.f5

Session affinity is worth using only when the task actually needs continuity, such as cookie-bound pagination or a multi-step account flow. If requests are independent, long-lived affinity usually makes distribution worse. sites.google

Which scheduling algorithm should you use?

For a residential proxy pool, score-based weighted selection is the best general default. proxidize

Round robin is easy to implement, but it ignores live differences in latency and error rate. Weighted round robin is better when capacity differences are stable, yet it still does not respond to fresh timeout bursts or sudden degradation unless you update weights from recent outcomes. docs.oracle

Use this rule:

Independent requests: score-based routing.
Session-bound workflows: score-based routing plus session affinity.
New pool with little feedback data: weighted round robin as a temporary fallback until enough outcome data exists to score nodes credibly. proxidize

How should you score each proxy?

A useful health score combines recent success, latency, current load, and cooldown state. If you ignore current load, a proxy can look healthy historically while still being the one you’re about to overload. joinmassive

A practical model is:

[
score = success_factor \times latency_factor \times load_factor \times cooldown_factor \times penalty_factor
]

Interpret the factors this way:

success_factor: recent success rate over your rolling window.
latency_factor: inverse weight from recent latency, with a cap so a few fast outliers do not dominate.
load_factor: penalty based on current in_flight pressure.
cooldown_factor: zero when the proxy is cooling down, one when it is eligible again.
penalty_factor: extra downgrade after repeated transport failures. joinmassive

For compliant operations, treat failure types differently:

Timeout or connect error: lower the proxy’s score and retry only within your approved retry budget.
429 from a target host: suspend or sharply reduce traffic to that target and review whether your approved frequency has been exceeded.
Repeated 403 from a target host: stop the workflow for that target and review authorization, session design, and request necessity before sending more traffic. bleepingcomputer

What provider-specific values do you need before the code will work?

Before you run any scheduler code, collect the provider-specific fields from your proxy vendor’s dashboard or API docs. This is the part many examples skip, and it is where otherwise-correct code often breaks in production. ppl-ai-file-upload.s3.amazonaws

You need:

Gateway host and port.
Username and password format.
How region targeting is encoded.
Whether the provider supports session affinity, and how that key is passed.
Whether the provider expects HTTP, HTTPS, or SOCKS5 proxy URLs.
Any documented plan limits that affect concurrency or session duration.

Do not guess these values. Proxy vendors often encode region or session parameters differently, and those formats are not interchangeable across providers. ppl-ai-file-upload.s3.amazonaws

How do you implement the scheduler in code?

HTTPX supports async clients and proxy configuration, which makes it a reasonable base for an application-level scheduler in Python. python-httpx

The code below is intentionally split into two parts:

Provider-specific settings you must fill from your vendor documentation.
Policy settings you must fill from your own approved traffic budget and internal rules.

That separation is deliberate. It keeps the article actionable without inventing proxy vendor syntax or target-rate values that should come from live docs and approved operating policy. python-httpx

Prerequisites

Use:

Python 3.11+
httpx
asyncio

Install:

pip install httpx

Create scheduler.py:

import asyncio
import os
import random
import statistics
import time
from collections import defaultdict, deque
from dataclasses import dataclass, field
from typing import Optional
from urllib.parse import urlparse

import httpx


class TargetSuspendedError(RuntimeError):
    pass


def required_env(name: str) -> str:
    value = os.getenv(name)
    if not value:
        raise RuntimeError(f"Missing required environment variable: {name}")
    return value


def required_int(name: str) -> int:
    return int(required_env(name))


@dataclass
class RuntimePolicy:
    outcome_window: int
    latency_window: int
    max_in_flight_per_proxy: int
    timeout_cooldown_seconds: int
    hard_failure_threshold: int
    per_proxy_rate_limit: int
    per_proxy_rate_window_seconds: int
    request_timeout_seconds: int
    retry_budget: int
    target_suspend_seconds: int
    repeated_forbidden_threshold: int


@dataclass
class ProxyNode:
    name: str
    proxy_url: str
    max_in_flight: int
    outcome_window: int
    latency_window: int
    in_flight: int = 0
    cooldown_until: float = 0.0
    hard_failures: int = 0
    outcome_samples: deque = field(init=False)
    latency_samples: deque = field(init=False)
    request_timestamps: deque = field(default_factory=deque)

    def __post_init__(self):
        self.outcome_samples = deque(maxlen=self.outcome_window)
        self.latency_samples = deque(maxlen=self.latency_window)

    def available(self) -> bool:
        return time.time() >= self.cooldown_until and self.in_flight < self.max_in_flight

    def success_rate(self) -> float:
        if not self.outcome_samples:
            return 1.0
        return max(0.1, sum(self.outcome_samples) / len(self.outcome_samples))

    def p95_latency(self) -> float:
        if not self.latency_samples:
            return 1.0
        vals = sorted(self.latency_samples)
        idx = max(0, min(len(vals) - 1, int(len(vals) * 0.95) - 1))
        return max(0.05, vals[idx])

    def health_score(self) -> float:
        if time.time() < self.cooldown_until:
            return 0.0

        success_factor = self.success_rate()
        latency_factor = min(2.0, 1.5 / self.p95_latency())
        load_ratio = self.in_flight / max(1, self.max_in_flight)
        load_factor = max(0.1, 1.0 - load_ratio)
        penalty_factor = 0.5 if self.hard_failures > 0 else 1.0

        return success_factor * latency_factor * load_factor * penalty_factor


class PerProxyRateLimiter:
    def __init__(self, max_requests: int, period_seconds: int):
        self.max_requests = max_requests
        self.period_seconds = period_seconds

    async def acquire(self, node: ProxyNode):
        while True:
            now = time.time()

            while node.request_timestamps and now - node.request_timestamps[0] > self.period_seconds:
                node.request_timestamps.popleft()

            if len(node.request_timestamps) < self.max_requests:
                node.request_timestamps.append(now)
                return

            wait_for = self.period_seconds - (now - node.request_timestamps[0]) + 0.01
            await asyncio.sleep(max(0.05, wait_for))


class TargetPolicyGate:
    def __init__(self, suspend_seconds: int, repeated_forbidden_threshold: int):
        self.suspend_seconds = suspend_seconds
        self.repeated_forbidden_threshold = repeated_forbidden_threshold
        self.suspended_until: dict[str, float] = {}
        self.forbidden_counts: dict[str, int] = defaultdict(int)

    def _host(self, url: str) -> str:
        return urlparse(url).netloc

    def assert_allowed(self, url: str):
        host = self._host(url)
        suspended_until = self.suspended_until.get(host, 0)
        if time.time() < suspended_until:
            raise TargetSuspendedError(
                f"Traffic to {host} is suspended until {time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(suspended_until))}"
            )

    def record_success(self, url: str):
        host = self._host(url)
        self.forbidden_counts[host] = 0

    def record_429(self, url: str):
        host = self._host(url)
        self.suspended_until[host] = time.time() + self.suspend_seconds

    def record_403(self, url: str):
        host = self._host(url)
        self.forbidden_counts[host] += 1
        if self.forbidden_counts[host] >= self.repeated_forbidden_threshold:
            self.suspended_until[host] = time.time() + self.suspend_seconds


class ProxyScheduler:
    def __init__(self, nodes: list[ProxyNode]):
        self.nodes = nodes
        self.session_affinity_map: dict[str, str] = {}

    def _find_by_name(self, name: str) -> Optional[ProxyNode]:
        for node in self.nodes:
            if node.name == name:
                return node
        return None

    def choose(self, session_key: Optional[str] = None) -> ProxyNode:
        if session_key and session_key in self.session_affinity_map:
            preferred = self._find_by_name(self.session_affinity_map[session_key])
            if preferred and preferred.available():
                preferred.in_flight += 1
                return preferred

        candidates = [node for node in self.nodes if node.available()]
        if not candidates:
            raise RuntimeError("No available proxies")

        weights = [max(0.01, node.health_score()) for node in candidates]
        chosen = random.choices(candidates, weights=weights, k=1)[0]
        chosen.in_flight += 1

        if session_key:
            self.session_affinity_map[session_key] = chosen.name

        return chosen

    def release(self, node: ProxyNode):
        node.in_flight = max(0, node.in_flight - 1)

    def record_success(self, node: ProxyNode, latency: float):
        node.outcome_samples.append(1)
        node.latency_samples.append(latency)
        node.hard_failures = max(0, node.hard_failures - 1)
        self.release(node)

    def record_transport_failure(self, node: ProxyNode, cooldown_seconds: int, threshold: int):
        node.outcome_samples.append(0)
        node.hard_failures += 1
        if node.hard_failures >= threshold:
            node.cooldown_until = time.time() + cooldown_seconds
        self.release(node)


async def fetch_with_scheduler(
    client: httpx.AsyncClient,
    scheduler: ProxyScheduler,
    rate_limiter: PerProxyRateLimiter,
    target_gate: TargetPolicyGate,
    policy: RuntimePolicy,
    url: str,
    session_key: Optional[str] = None,
):
    target_gate.assert_allowed(url)
    last_error = None

    for attempt in range(policy.retry_budget + 1):
        node = scheduler.choose(session_key=session_key)
        await rate_limiter.acquire(node)
        started = time.perf_counter()

        try:
            response = await client.get(
                url,
                proxy=node.proxy_url,
                timeout=policy.request_timeout_seconds,
            )
            latency = time.perf_counter() - started

            if response.status_code == 429:
                scheduler.release(node)
                target_gate.record_429(url)
                raise TargetSuspendedError(
                    f"Received 429 from {urlparse(url).netloc}; target workflow suspended for policy review."
                )

            if response.status_code == 403:
                scheduler.release(node)
                target_gate.record_403(url)
                raise TargetSuspendedError(
                    f"Received 403 from {urlparse(url).netloc}; target workflow halted pending authorization review."
                )

            response.raise_for_status()
            scheduler.record_success(node, latency)
            target_gate.record_success(url)

            return {
                "proxy": node.name,
                "status_code": response.status_code,
                "latency": round(latency, 3),
                "attempt": attempt,
                "host": urlparse(url).netloc,
            }

        except TargetSuspendedError:
            raise
        except (httpx.ConnectError, httpx.ReadTimeout, httpx.ConnectTimeout) as exc:
            scheduler.record_transport_failure(
                node,
                cooldown_seconds=policy.timeout_cooldown_seconds,
                threshold=policy.hard_failure_threshold,
            )
            last_error = exc
            continue
        except Exception as exc:
            scheduler.record_transport_failure(
                node,
                cooldown_seconds=policy.timeout_cooldown_seconds,
                threshold=policy.hard_failure_threshold,
            )
            last_error = exc
            continue

    raise last_error or RuntimeError("Request failed after allowed retries")


def summarize_results(rows: list[dict]):
    grouped = defaultdict(list)
    for row in rows:
        grouped[row["proxy"]].append(row)

    summary = []
    for proxy, items in grouped.items():
        latencies = [x["latency"] for x in items]
        summary.append(
            {
                "proxy": proxy,
                "requests": len(items),
                "avg_latency": round(statistics.mean(latencies), 3) if latencies else None,
                "p95_latency": round(sorted(latencies)[max(0, int(len(latencies) * 0.95) - 1)], 3)
                if latencies
                else None,
            }
        )
    return sorted(summary, key=lambda x: x["proxy"])


def load_policy() -> RuntimePolicy:
    return RuntimePolicy(
        outcome_window=required_int("OUTCOME_WINDOW"),
        latency_window=required_int("LATENCY_WINDOW"),
        max_in_flight_per_proxy=required_int("MAX_IN_FLIGHT_PER_PROXY"),
        timeout_cooldown_seconds=required_int("TIMEOUT_COOLDOWN_SECONDS"),
        hard_failure_threshold=required_int("HARD_FAILURE_THRESHOLD"),
        per_proxy_rate_limit=required_int("PER_PROXY_RATE_LIMIT"),
        per_proxy_rate_window_seconds=required_int("PER_PROXY_RATE_WINDOW_SECONDS"),
        request_timeout_seconds=required_int("REQUEST_TIMEOUT_SECONDS"),
        retry_budget=required_int("RETRY_BUDGET"),
        target_suspend_seconds=required_int("TARGET_SUSPEND_SECONDS"),
        repeated_forbidden_threshold=required_int("REPEATED_FORBIDDEN_THRESHOLD"),
    )


def build_proxy_nodes(policy: RuntimePolicy) -> list[ProxyNode]:
    proxy_urls = [
        required_env("PROXY_URL_1"),
        required_env("PROXY_URL_2"),
        required_env("PROXY_URL_3"),
    ]

    nodes = []
    for idx, proxy_url in enumerate(proxy_urls, start=1):
        nodes.append(
            ProxyNode(
                name=f"p{idx}",
                proxy_url=proxy_url,
                max_in_flight=policy.max_in_flight_per_proxy,
                outcome_window=policy.outcome_window,
                latency_window=policy.latency_window,
            )
        )
    return nodes


async def main():
    policy = load_policy()
    nodes = build_proxy_nodes(policy)

    scheduler = ProxyScheduler(nodes)
    rate_limiter = PerProxyRateLimiter(
        max_requests=policy.per_proxy_rate_limit,
        period_seconds=policy.per_proxy_rate_window_seconds,
    )
    target_gate = TargetPolicyGate(
        suspend_seconds=policy.target_suspend_seconds,
        repeated_forbidden_threshold=policy.repeated_forbidden_threshold,
    )

    test_url = required_env("TARGET_URL")
    total_requests = required_int("TOTAL_REQUESTS")

    async with httpx.AsyncClient(
        headers={"User-Agent": "OpsValidation/1.0"},
        limits=httpx.Limits(max_connections=required_int("MAX_CONNECTIONS")),
    ) as client:
        tasks = []
        for i in range(total_requests):
            session_key = f"session-{i // 4}"
            tasks.append(
                fetch_with_scheduler(
                    client=client,
                    scheduler=scheduler,
                    rate_limiter=rate_limiter,
                    target_gate=target_gate,
                    policy=policy,
                    url=test_url,
                    session_key=session_key,
                )
            )

        results = await asyncio.gather(*tasks, return_exceptions=True)
        ok = [r for r in results if isinstance(r, dict)]
        errors = [str(r) for r in results if not isinstance(r, dict)]

        print("SUMMARY")
        for row in summarize_results(ok):
            print(row)

        print("\nERRORS")
        for err in errors[:20]:
            print(err)


if __name__ == "__main__":
    asyncio.run(main())

Required environment variables

Create a .env or deployment secret set with these values:

PROXY_URL_1, PROXY_URL_2, PROXY_URL_3
TARGET_URL
OUTCOME_WINDOW
LATENCY_WINDOW
MAX_IN_FLIGHT_PER_PROXY
TIMEOUT_COOLDOWN_SECONDS
HARD_FAILURE_THRESHOLD
PER_PROXY_RATE_LIMIT
PER_PROXY_RATE_WINDOW_SECONDS
REQUEST_TIMEOUT_SECONDS
RETRY_BUDGET
TARGET_SUSPEND_SECONDS
REPEATED_FORBIDDEN_THRESHOLD
MAX_CONNECTIONS
TOTAL_REQUESTS

Populate proxy URLs from your provider’s live documentation or control panel. Populate rate, retry, cooldown, and suspension settings from your own approved traffic budget and operational policy rather than copying values from another team or vendor marketing page. ppl-ai-file-upload.s3.amazonaws

Why this implementation is safe to use in production

This version does four things a lab demo usually misses:

It tracks current in-flight pressure, not just past success.
It enforces per-proxy rate controls.
It preserves session affinity only when you pass a session key.
It suspends a target workflow after 429 or repeated 403 instead of treating those responses as a signal to keep routing traffic elsewhere. bleepingcomputer

That last part is the compliance-critical distinction. Transport failures are a proxy-health problem. Repeated 429 and 403 responses are often a target-policy problem, so they should trigger review, not evasive redistribution. bleepingcomputer

How do you verify that balancing actually works?

You verify a residential proxy pool with logs and distribution reports, not intuition. If you can’t see request share, latency, and failure concentration per proxy, you can’t tell whether the scheduler fixed overload or just hid it. joinmassive

Record these fields on every request:

Timestamp.
Request ID.
Target host.
Selected proxy.
Session key, if used.
Attempt number.
Status code or failure class.
End-to-end latency.
Whether the target workflow was suspended. joinmassive

Add this helper if you want a CSV distribution report:

import csv
from collections import defaultdict

def write_distribution_report(rows, filename="proxy_distribution.csv"):
    grouped = defaultdict(lambda: {"requests": 0, "latencies": [], "errors": 0})

    for row in rows:
        proxy = row.get("proxy", "unknown")
        grouped[proxy]["requests"] += 1
        if "latency" in row:
            grouped[proxy]["latencies"].append(row["latency"])
        if row.get("status_code", 200) >= 400:
            grouped[proxy]["errors"] += 1

    with open(filename, "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["proxy", "requests", "avg_latency", "error_count"])
        for proxy, stats in grouped.items():
            lat = stats["latencies"]
            avg = round(sum(lat) / len(lat), 3) if lat else None
            writer.writerow([proxy, stats["requests"], avg, stats["errors"]])

Use these success criteria:

No proxy should sit at saturation while peers remain mostly idle.
Degraded proxies should lose share naturally because their score drops.
Session-bound tasks should stay coherent without taking over unrelated traffic.
If a target triggers suspension logic, the workflow should stop cleanly and visibly rather than silently continuing through other proxies. bleepingcomputer

A healthy output usually looks like a distribution report, not a single “all good” line. You want to see per-proxy request counts and latency values close enough to show balanced use, while still allowing stronger nodes to carry slightly more work than weaker ones.

What did this revision focus on?

This revision was reviewed on April 20, 2026 against public proxy-pool management guidance, residential proxy usage best practices, and HTTPX proxy documentation. The main conclusion was straightforward: the hard part is rarely the picker by itself; it is the combination of session-affinity scope, per-proxy pressure, and clear stop conditions when a target starts signaling that the workflow needs review. python-httpx

That is also where most articles cut corners. They explain rotation, then jump straight to “keep trying.” In production, the better design is to distinguish between proxy-health failures and target-policy signals so your scheduler doesn’t blur the two. joinmassive

Why is one proxy still getting overloaded?

The most common reason is that the scheduler is balancing selection count instead of live pressure. A slower proxy can accumulate more open requests even if it is not chosen much more often than its peers. botproxy

Check these patterns:

Symptom: one proxy keeps the most open requests

Cause: the score does not penalize rising in_flight pressure early enough.

Fix: make load_factor react sooner and lower MAX_IN_FLIGHT_PER_PROXY until your logs show stable distribution under your real workload.

Symptom: distribution got worse after enabling session affinity

Cause: the session scope is too broad or too long-lived.

Fix: keep session affinity only for workflows that truly depend on continuity, and expire mappings when the task ends. my.f5

Symptom: a few bad proxies drag down the whole pool

Cause: they never cool down or leave rotation after transport failures.

Fix: send repeated transport failures into cooldown, then let those proxies earn their way back through successful traffic instead of restoring them immediately. joinmassive

Symptom: every proxy looks “bad” at once

Cause: the problem may not be the pool at all.

Fix: verify your provider’s gateway syntax, credentials, region parameters, and session-affinity format before changing the scheduler. A malformed proxy URL can look like a balancing failure when it is really an integration bug. ppl-ai-file-upload.s3.amazonaws

Build-vs-buy note

If you need custom routing logic, internal observability, and tight control over how tasks are mapped to proxies, building this scheduler layer makes sense. If your team mainly needs reliable residential proxy access without owning the full control plane, evaluating a managed provider is usually the better tradeoff. intel471

That is the natural place to look at Proxy001. Its public site includes residential proxy use-case content and example traffic-based pricing in blog material, which is enough to make it relevant in a build-vs-buy discussion, but you should still confirm the current gateway format, onboarding flow, and any plan-specific details on the live site before integrating. proxy001

Compliance note

Use residential proxies only for permitted tasks such as QA, price monitoring, ad verification, fraud analysis, or data-quality workflows where you have a lawful basis for the traffic. A residential proxy pool is not a substitute for authorization, contract review, or target-site policy review. bleepingcomputer

In practice, this means your scheduler should make it easy to stop or reduce traffic when the target indicates pressure or restriction. Good load balancing is not just about throughput; it is also about making policy boundaries visible in the control flow. joinmassive

Go-live checklist

Before rollout, make sure you have:

A per-proxy concurrency cap.
A per-proxy rate control.
A documented session-affinity rule.
Separate handling for transport errors versus target-policy signals.
Provider-specific gateway and credential syntax verified from live docs.
Request logs with proxy choice, latency, and outcome.
A verification report after test traffic.
A clean suspension path for targets that need policy review. ppl-ai-file-upload.s3.amazonaws

If you want a provider to test this pattern against a real commercial residential proxy offering, Proxy001 is one option worth evaluating. Its site already publishes residential proxy use cases and example traffic-based pricing, which makes it a reasonable starting point for practical assessment. The next step is to confirm the current live signup path, gateway syntax, region parameters, and session-affinity format directly on proxy001.com, then plug those values into the scheduler structure above and validate with your own approved workload. proxy001

Building a Regional Request Router With Residential Proxies in Python

Miller James — Thu, 16 Apr 2026 01:53:31 +0000

By the Proxy001 engineering team · Last reviewed April 2026
Disclosure: Proxy001 sponsors this content. All technical specifications are verified from proxy001.com as of April 2026.

Most tutorials stop at "here's how to pass a proxy dict to requests.get()." That's fine if you're routing a single request through a single IP. But if you're building a service that sends traffic through a US exit for one task, a DE exit for another, and a JP exit for a third — all from the same Python process — you need a router that maps country codes to proxy sessions and manages them for you.

This tutorial builds a RegionalRouter class that does exactly that: maps country codes to residential proxy exits, manages a session pool, handles retries and errors cleanly, and can be wrapped into a lightweight HTTP service with basic access control. You'll have something fully runnable before the end of the page.

Prerequisites

Before you write a single line of router code, confirm the following:

Python 3.8+ — f-strings and typing improvements used throughout
requests library: pip install requests
urllib3 ≥ 1.26.0 — required for the allowed_methods parameter on Retry (ships with requests 2.26+); earlier versions use the deprecated method_whitelist instead
SOCKS5 support (optional, but recommended): pip install "requests[socks]" — this pulls in PySocks, which is needed if your provider offers SOCKS5 endpoints
python-dotenv for credential management: pip install python-dotenv
A residential proxy account that supports country-level geo-targeting — your provider must allow you to embed a country code in the proxy username. This is the mechanism the entire router relies on. If your provider only offers random rotation without country selection, the geo-routing won't work regardless of how the code is written.

Your proxy account should give you four things: a gateway hostname, a port, a username, and a password. Keep those handy.

How Does Country-Binding Work in a Residential Proxy?

Most residential proxy providers expose a single gateway — one hostname, one port. Country selection doesn't happen by pointing to a different server; it happens by encoding the target country code directly into the proxy username.

Bright Data's documentation describes the pattern clearly: to route through a US exit, you append a country flag to your zone username, producing something like customer-XXXX-zone-resi-country-us. Oxylabs uses the same username-embedding convention with a cc parameter. The resulting proxy URL takes this form:

http://user-country-US:password@gateway.example.com:10000

Change US to DE, JP, or any ISO 3166-1 alpha-2 code, and the provider routes your request through an exit node in that country. Same gateway, same port — only the username string changes. That's what makes a dynamic router feasible: you're generating per-country proxy URLs at runtime through string interpolation, not maintaining 50 separate connections to 50 different servers.

The exact username format varies by provider. Check your dashboard's "Proxy Setup" or "Integration" tab and look for a field labeled something like "username with location" or a code example showing country targeting — that string is your template.

Step 1 — Build the Country-to-Proxy URL Mapping

Start by loading your credentials from environment variables. If your password contains characters like @, :, #, or %, you must URL-encode it before embedding it in the proxy URL — an unescaped @ will break URL parsing and you'll get a 407 with no obvious explanation.

Create a .env file:

PROXY_HOST=gateway.yourprovider.com
PROXY_PORT=10000
PROXY_USER=your_base_username
PROXY_PASS=your_p@ssword!

Now build the mapping. The build_proxy_url function below handles encoding and constructs the country-specific username:

import os
from urllib.parse import quote
from dotenv import load_dotenv

load_dotenv()

PROXY_HOST = os.environ["PROXY_HOST"]
PROXY_PORT = os.environ["PROXY_PORT"]
PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = quote(os.environ["PROXY_PASS"], safe="")  # encode ALL special chars


def build_proxy_url(country_code: str) -> str:
    """
    Construct a country-targeted residential proxy URL.
    Country code must be ISO 3166-1 alpha-2 (e.g. 'US', 'DE', 'JP').
    Adjust the username suffix to match your provider's exact syntax.
    """
    username = f"{PROXY_USER}-country-{country_code.upper()}"
    return f"http://{username}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"


# Pre-build a mapping for the countries you'll route through
SUPPORTED_COUNTRIES = ["US", "DE", "GB", "JP", "FR", "SG", "BR"]

COUNTRY_PROXIES: dict[str, dict] = {
    cc: {"http": build_proxy_url(cc), "https": build_proxy_url(cc)}
    for cc in SUPPORTED_COUNTRIES
}

Provider syntax note: The -country-XX suffix above follows the pattern used by Bright Data and several other major providers. If you're using Proxy001, their geo-targeting uses the same username-parameter model — check the "Proxy Setup" section of your dashboard for the exact suffix format for your zone type. Proxy001 supports country, city, and carrier-level targeting across 200+ regions, so the same build_proxy_url logic applies once you have the correct template string.

Step 2 — Build the `RegionalRouter` Class With a Session Pool

Creating a new requests.Session for every request wastes TCP connection setup time and can exhaust file descriptors under sustained load. The right approach is to pre-build one session per country and reuse it — a session pool.

One thing to be aware of upfront: pre-building sessions for every supported country means holding that many open connections. For 7–15 countries, the memory cost is negligible. If you're routing through 30+ countries or running in a constrained environment like AWS Lambda, lazy initialization (building sessions on first use rather than at startup) is worth considering — you'd replace the __init__ loop with a _get_or_create_session(cc) method. For most services, the pre-built approach below is simpler and performs better.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry  # requires urllib3 >= 1.26.0


class RoutingError(Exception):
    """Raised when a regional request cannot be completed."""
    pass


class RegionalRouter:
    def __init__(
        self,
        country_proxies: dict,
        timeout: tuple = (10, 30),
        max_retries: int = 3,
        backoff_factor: float = 0.5,
    ):
        """
        country_proxies: dict mapping ISO country codes to proxy dicts,
                         e.g. {"US": {"http": "http://...", "https": "http://..."}}
        timeout:         (connect_timeout, read_timeout) in seconds
        max_retries:     retry attempts on connection errors and 5xx responses
        backoff_factor:  sleep between retries = backoff_factor * (2 ** (retry_n - 1))
        """
        self.timeout = timeout
        self._sessions: dict[str, requests.Session] = {}

        # raise_on_status=False here means Retry won't raise mid-retry cycle;
        # the final response's raise_for_status() is called in route() instead,
        # so callers always get a RoutingError with the status code included.
        retry_strategy = Retry(
            total=max_retries,
            backoff_factor=backoff_factor,
            status_forcelist=[429, 500, 502, 503, 504],
            allowed_methods=["GET", "POST", "HEAD"],  # urllib3 >= 1.26.0
            raise_on_status=False,
        )
        adapter = HTTPAdapter(
            max_retries=retry_strategy,
            pool_connections=len(country_proxies),  # one pool per country
            pool_maxsize=10,
        )

        for cc, proxy_dict in country_proxies.items():
            session = requests.Session()
            session.proxies.update(proxy_dict)
            session.mount("http://", adapter)
            session.mount("https://", adapter)
            self._sessions[cc.upper()] = session

    def route(self, url: str, country: str, **kwargs) -> requests.Response:
        """
        Send a GET request through the residential exit for the given country.

        url:     Target URL
        country: ISO 3166-1 alpha-2 country code (e.g. 'US', 'DE')
        kwargs:  Passed through to session.get() (headers, params, etc.)
        """
        cc = country.upper()
        session = self._sessions.get(cc)
        if session is None:
            raise ValueError(
                f"Country '{cc}' is not in the supported list: "
                f"{list(self._sessions.keys())}"
            )

        kwargs.setdefault("timeout", self.timeout)

        try:
            response = session.get(url, **kwargs)
            response.raise_for_status()
            return response
        except requests.exceptions.ProxyError as e:
            raise RoutingError(f"[{cc}] Proxy connection failed: {e}") from e
        except requests.exceptions.ConnectTimeout as e:
            raise RoutingError(f"[{cc}] Connection timed out: {e}") from e
        except requests.exceptions.HTTPError as e:
            raise RoutingError(
                f"[{cc}] HTTP {e.response.status_code} from target: {url}"
            ) from e
        except requests.exceptions.RequestException as e:
            raise RoutingError(f"[{cc}] Request failed: {e}") from e

Wire it together with the mapping from Step 1:

router = RegionalRouter(country_proxies=COUNTRY_PROXIES)

The Retry object handles transient 429s and 5xx responses with exponential backoff. Setting pool_connections to the number of countries gives each country its own connection pool rather than having all sessions compete for a shared one.

Step 3 — Add Graceful Error Handling

The route() method wraps all proxy-layer and HTTP-layer exceptions into RoutingError. Callers don't need to catch five different exception types, the country code is always included in the message, and the original exception is preserved via from e for stack traces. In practice that last point matters more than it sounds — when a ProxyError bubbles up in a log aggregator at 2am, having the original exception chained makes the difference between a 5-minute fix and a 30-minute debugging session.

For your calling code, a typical pattern looks like this:

try:
    resp = router.route("https://example.com/api/data", country="DE")
    print(resp.json())
except ValueError as e:
    # Unsupported country — likely a bug in the calling code
    print(f"Config error: {e}")
except RoutingError as e:
    # Proxy or network failure — log and handle gracefully
    print(f"Routing failed: {e}")

One design decision worth flagging: raise_for_status() is called inside route(), which means 4xx and 5xx responses from the target site also become RoutingError. That's intentional for most use cases — you want to know if the target returned a 403, not silently process an empty body. If you'd rather inspect the response yourself, remove response.raise_for_status() from route() and check response.status_code in the caller.

How Do I Confirm the Router Is Using the Right Country Exit?

Run this verification before you point the router at your real targets. It uses ip-api.com/json, which returns the geographic location of the requesting IP:

def verify_routing(router: RegionalRouter, countries: list[str]) -> None:
    """
    Check that each country exit is actually routing through the target country.
    Prints a pass/fail line per country code.
    """
    for cc in countries:
        try:
            resp = router.route("http://ip-api.com/json", country=cc)
            data = resp.json()
            actual_cc = data.get("countryCode", "UNKNOWN")
            status = "✓ PASS" if actual_cc == cc else f"✗ FAIL (got {actual_cc})"
            print(f"  [{cc}] {status} — exit IP: {data.get('query')}")
        except RoutingError as e:
            print(f"  [{cc}] ✗ ERROR — {e}")

verify_routing(router, SUPPORTED_COUNTRIES)

Expected output:

  [US] ✓ PASS — exit IP: 104.28.x.x
  [DE] ✓ PASS — exit IP: 185.220.x.x
  [GB] ✓ PASS — exit IP: 51.36.x.x
  ...

Once verify_routing() passes for all target countries, it's worth adding to your CI pipeline or startup health check — catching a misconfigured credential or expired account at deploy time is far better than debugging a geo-mismatch in production at request time.

That said: if you see a consistent FAIL for one specific region despite the code looking correct, it's almost always a provider plan limitation rather than a bug. The curl test in Troubleshooting #3 below confirms this in about 30 seconds, without touching Python at all.

Troubleshooting: 3 Real Pitfalls

These aren't hypothetical edge cases pulled from documentation. The socks5h DNS issue is particularly nasty because requests succeed — they just silently exit through the wrong country, or connect to the wrong resolved IP, and nothing in the logs tells you why. The password encoding problem is the most common "it works in curl but not in Python" symptom we've seen. Here's what to look for in each case.

1. `socks5://` Resolves DNS Locally — Use `socks5h://` Instead

Symptom: Requests succeed, but the exit IP doesn't match the target country — or you get connection errors on endpoints that expect hostname-based routing.

Cause: With socks5://, the Python requests library resolves the hostname on the client side before forwarding the connection through the proxy. The proxy server sees an IP address, not a hostname. Some residential proxy providers route by hostname at their infrastructure layer, so when they receive a bare IP, the country-targeting logic has nothing to work with.

Fix: Switch the proxy URL prefix from socks5:// to socks5h://:

# Wrong — DNS resolves locally, country routing may silently fail
"socks5://user-country-US:pass@gateway.example.com:10000"

# Correct — DNS resolves on the proxy side
"socks5h://user-country-US:pass@gateway.example.com:10000"

This requires pip install "requests[socks]" with PySocks installed. Update build_proxy_url to use socks5h:// instead of http:// if your provider's endpoint is SOCKS5.

2. Special Characters in the Password Cause 407

Symptom: 407 Proxy Authentication Required even though the credentials are confirmed correct in a browser or curl test.

Cause: Characters like @, :, #, and % are URL delimiters. When they appear unencoded in the password portion of a proxy URL, the library misparses where the password ends and the hostname begins. The result is a malformed authentication header that the proxy server rejects.

Fix: urllib.parse.quote(password, safe="") encodes every special character, including /:

from urllib.parse import quote

raw_password = "p@ss:w0rd#!"
encoded = quote(raw_password, safe="")
# Result: "p%40ss%3Aw0rd%23%21"

proxy_url = f"http://user-country-US:{encoded}@gateway.example.com:10000"

The build_proxy_url function in Step 1 already applies this via PROXY_PASS = quote(os.environ["PROXY_PASS"], safe=""). The risk arises if you construct proxy URLs anywhere else in your codebase without going through that function — worth a quick grep for @{PROXY_HOST} or @gateway to catch any direct string construction.

3. Country Code Is Ignored — Exit IP Doesn't Match

Symptom: verify_routing() reports FAIL (got XX) for one or more countries despite the code running without errors.

Cause: Not all residential proxy providers support country-level geo-targeting on all plans or pool types. Entry-level plans, shared ISP pools, and some trial accounts route randomly regardless of the username parameter — the provider accepts the credential but silently ignores the country suffix.

Fix — diagnose in three steps:

Rule out the code first. Run the equivalent curl command directly:

   curl -x "http://user-country-DE:pass@gateway:port" http://ip-api.com/json

If curl also returns the wrong country, it's a provider or plan issue, not Python.

Check your plan's feature flags. Geo-targeting is usually a toggle in the provider dashboard — look for a "Location" or "Targeting" section in your zone settings. It may need to be explicitly enabled.
Confirm the username suffix format. Some providers use -country-US, others use -cc-US, and some require a different entry-point hostname per country rather than a username parameter. The format in build_proxy_url may need adjusting. Your provider's integration documentation (not the generic proxy setup page, but the language-specific or zone-specific docs) is the right place to check.

Optional: Expose the Router as a Lightweight HTTP Service

If other services, cron jobs, or CLI tools need to use the router over HTTP, wrapping it in a FastAPI endpoint is straightforward. The snippet below includes a minimal API key check — this matters more than it might seem, because an unauthenticated /fetch endpoint lets anyone with network access route arbitrary traffic through your proxy quota.

First, add the API key to your .env:

ROUTER_API_KEY=your-secret-key-here

Then:

from fastapi import FastAPI, HTTPException, Security
from fastapi.security import APIKeyHeader
from pydantic import BaseModel

app = FastAPI()

ROUTER_API_KEY = os.environ["ROUTER_API_KEY"]
api_key_header = APIKeyHeader(name="X-API-Key", auto_error=True)


def verify_api_key(key: str = Security(api_key_header)) -> None:
    if key != ROUTER_API_KEY:
        raise HTTPException(status_code=403, detail="Invalid API key")


class FetchRequest(BaseModel):
    url: str
    country: str


@app.post("/fetch")
def fetch(req: FetchRequest, _: None = Security(verify_api_key)):
    try:
        resp = router.route(req.url, req.country)
        return {
            "status_code": resp.status_code,
            "content": resp.text,
        }
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except RoutingError as e:
        raise HTTPException(status_code=502, detail=str(e))

Run with: uvicorn main:app --reload

Call it:

curl -X POST http://localhost:8000/fetch \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "country": "JP"}'

For rate limiting before any external exposure, slowapi wraps FastAPI in about 10 lines — their README has the decorator-based pattern. Both auth and rate limiting are worth adding before you expose this beyond localhost.

Start Routing With a Free Trial

You now have a complete RegionalRouter: country-to-proxy URL mapping from environment variables, a pre-built session pool with per-country connection pools and exponential-backoff retry, clean error handling with chained exceptions, a verify_routing() function to confirm geo-exits before deployment, troubleshooting paths for the three most common failures, and an HTTP wrapper with API key authentication.

The only external dependency is a residential proxy account with geo-targeting enabled. Proxy001 covers 200+ regions with 100M+ IPs and supports country, city, and carrier-level targeting — the same username-parameter model this router is built around. The free trial gives you 500 MB without a credit card, which is enough to run verify_routing() across your full country list and load-test the session pool before committing to a plan. Pricing runs from $2.00/GB for small volumes down to $0.70/GB at higher tiers (as of April 2026; verify current rates on their pricing page).

(Disclosure: Proxy001 sponsors this content. Technical specifications verified from proxy001.com as of April 2026.)

How Residential Proxies Fixed Our Flaky Global E2E Tests

Miller James — Mon, 13 Apr 2026 01:43:10 +0000

By the Proxy001 Engineering Team — This post draws from our experience migrating cross-region E2E infrastructure across a distributed test setup targeting multiple production markets.

Why Your E2E Tests Pass Locally but Fail in CI

The failure looks random — until you check where your CI runner is located.

We first hit this pattern on a Tuesday: our Frankfurt CI node had been failing the checkout flow at the currency selector step for three days straight. Local runs passed every time. The team spent the better part of two days auditing the component — no code changes, no dependency updates, no meaningful diff between environments. The real cause turned out to be the runner's IP: it was registered to a Hetzner datacenter block, and our CDN was routing it to a UK edge node that served a promotional banner replacing the EUR currency selector with a static campaign element. The DOM node our test was asserting on simply wasn't there from that IP.

After switching to residential proxies for our EU test traffic, that failure — and a dozen variations of it across other regions — stopped occurring. Pass rate on geo-sensitive flows went from around 70% to consistent green in under a week.

This is geo-blocking-induced test flakiness, and it's one of the most frustrating kinds because the failure doesn't point at the code. The diff between local and CI is where the HTTP request originates, not what the test does. Residential proxies fix this by routing your test traffic through real home ISP addresses in specific countries, making your test runner look like a real user in that market.

What Geo-IP Actually Does to Your Tests

Three separate mechanisms turn tests geo-dependent, and they produce failures that look like race conditions or intermittent network issues but are actually deterministic.

CDN geo-split. Major CDNs route requests to region-specific edge nodes that serve different asset bundles, localized content, or entirely different page layouts. A product page might return a UK-specific promotional banner from the London edge node and a different layout from the US node — same code, different IP, different DOM.

Application-side IP detection. Many apps parse X-Forwarded-For or call an IP geolocation API to decide which language, currency, or regulatory compliance flows to render. If your test runner's IP resolves to the wrong country, the app routes it into a flow you never wrote assertions for.

Datacenter IP reputation filtering. A significant number of websites — especially those with bot-mitigation systems — automatically flag requests from datacenter IP ranges (AWS, GCP, Azure, Hetzner) and respond with a 403, a CAPTCHA, or a degraded page variant. Your test never reaches the content it's trying to assert on. Same IP range, same failure, every time.

Datacenter vs. Residential Proxies: Why It Matters for Test Stability

The core difference is IP reputation and ASN classification.

Datacenter proxies route through IPs registered to cloud hosting providers. Their ASNs (Autonomous System Numbers) are publicly known and widely blocked at the infrastructure level — they're exactly the traffic pattern that anti-bot systems are built to catch. Residential proxies use addresses issued by consumer ISPs — the same ranges your actual users get. To geo-detection and bot-mitigation systems, they're indistinguishable from organic traffic.

	Datacenter Proxy	Residential Proxy
IP source	Cloud hosting ASN	Consumer ISP ASN
Geo-detection accuracy	Unreliable (commonly flagged as VPN/proxy)	Accurate (real ISP geo assignment)
Bot detection risk	High	Low
Typical latency	~5–30ms avg (consistent)	~20–150ms avg (variable by ISP routing)
Best fit	Non-sensitive scraping, internal tool testing	Geo-accurate E2E tests, localization QA

For E2E testing where you need to accurately replicate what a real user in Frankfurt, Tokyo, or São Paulo sees, residential proxies are the right tool. The latency increase is real but predictable — you account for it once in your timeout settings rather than debugging phantom failures indefinitely.

Step-by-Step: Integrating Residential Proxies into Your Test Framework

The integration path differs meaningfully between frameworks. Playwright gives you the most control — proxy config works at the project and context level. Cypress operates at the process level via environment variables, which creates an important limitation covered below. Selenium requires the most manual handling for authenticated proxies.

Playwright

Playwright's native proxy support is the cleanest option for geo-targeted E2E tests. You can set proxy configuration per project in playwright.config.ts, which lets you run multi-region tests in parallel from a single suite — each project routes through a different country's IP pool. The full proxy API reference is available in Playwright's official BrowserContext documentation.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'EU-tests',
      use: {
        ...devices['Desktop Chrome'],
        proxy: {
          server: `http://${process.env.PROXY_HOST_EU}`,
          username: process.env.PROXY_USER,
          password: process.env.PROXY_PASS,
        },
        locale: 'de-DE',
        geolocation: { longitude: 13.4, latitude: 52.5 }, // Berlin
        permissions: ['geolocation'],
      },
      testMatch: '**/eu/**',
    },
    {
      name: 'US-tests',
      use: {
        ...devices['Desktop Chrome'],
        proxy: {
          server: `http://${process.env.PROXY_HOST_US}`,
          username: process.env.PROXY_USER,
          password: process.env.PROXY_PASS,
        },
        locale: 'en-US',
        geolocation: { longitude: -87.6, latitude: 41.8 }, // Chicago
        permissions: ['geolocation'],
      },
      testMatch: '**/us/**',
    },
  ],
});

Three things worth noting. First, PROXY_HOST_EU should include the port — typically something like gate.yourprovider.com:7777. The server value uses http:// as the scheme even when your target pages are HTTPS; the proxy connection itself is HTTP and it tunnels HTTPS traffic via CONNECT. Second, combining proxy with locale and geolocation gives you the full geo-simulation stack: the proxy sets the outbound IP, the locale affects Accept-Language headers and Intl API behavior, and geolocation handles navigator.geolocation calls. Third, if any of these tests involve login flows or session state, read the sticky session section before running them.

To override the proxy for one specific test without touching the project config:

test('German pricing page shows EUR', async ({ browser }) => {
  const context = await browser.newContext({
    proxy: {
      server: `http://${process.env.PROXY_HOST_EU}`,
      username: process.env.PROXY_USER,
      password: process.env.PROXY_PASS,
    },
  });
  const page = await context.newPage();
  await page.goto('https://yourapp.com/pricing');
  await expect(page.locator('[data-testid="price"]')).toContainText('€');
  await context.close();
});

Cypress

Cypress doesn't support per-test or per-project proxy configuration at the network level — it proxies all browser traffic through system environment variables (official proxy configuration docs). Set HTTPS_PROXY before cypress run and Cypress routes everything through it.

HTTPS_PROXY=http://user:pass@gate.yourprovider.com:7777 \
NO_PROXY=localhost,127.0.0.1 \
npx cypress run

The NO_PROXY value matters. Without it, Cypress's internal communication between the test runner process and the browser also goes through the proxy, causing connection failures.

Per-region testing in Cypress: You can't switch geo within a single cypress run. The practical workaround is running the suite multiple times with different proxy endpoints — one run per region. Here's a shell wrapper that handles that cleanly:

#!/bin/bash
# scripts/run-geo-tests.sh
# Runs the full Cypress suite once per region, serially.
# Trade-off: 3 regions × 5 min/run = 15 min total wall clock time.
# Fine for nightly pipelines; too slow for per-PR runs.

set -e

REGIONS=("eu" "us" "jp")

for REGION in "${REGIONS[@]}"; do
  echo "--- Running Cypress suite for region: $REGION ---"
  HTTPS_PROXY="http://${PROXY_USER}:${PROXY_PASS}@${REGION}-gate.yourprovider.com:7777" \
  NO_PROXY="localhost,127.0.0.1" \
  npx cypress run --spec "cypress/e2e/**" --env GEO_REGION="${REGION}"
done

Inside your tests, Cypress.env('GEO_REGION') lets you branch assertions by region if needed. This approach is serial — three regions at five minutes each means fifteen minutes total. Acceptable for a nightly run, painful for per-PR pipelines. If parallel multi-region testing matters for your iteration speed, Playwright's project-based architecture is genuinely the right tool for it.

Selenium / WebDriver

For authenticated residential proxies with Selenium, the most reliable approach is IP whitelisting: register your CI runner's outbound IP with your proxy provider, and the proxy grants access without credentials in each request. This sidesteps Chrome's inconsistent handling of inline proxy auth.

IP-whitelisted setup in Python (cleanest path when available):

# test_geo_ipwhitelist.py
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

options = Options()
# PROXY_HOST = gate.yourprovider.com:7777 (IP-whitelisted — no credentials needed)
options.add_argument(f'--proxy-server=http://{os.environ["PROXY_HOST"]}')

driver = webdriver.Chrome(service=Service(), options=options)
driver.get('https://yourapp.com')
driver.quit()

If your CI runners have dynamic IPs and can't use whitelisting, the standard solution is a Chrome extension that handles onAuthRequired at the browser level. This is the same pattern used across major proxy providers and is publicly documented:

# test_geo_selenium.py — proxy auth via Chrome Manifest V3 extension
import os
import zipfile
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

PROXY_HOST = os.environ["PROXY_HOST"]      # e.g., gate.yourprovider.com
PROXY_PORT = int(os.environ["PROXY_PORT"]) # e.g., 7777
PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = os.environ["PROXY_PASS"]

manifest_json = """
{
  "name": "Proxy Auth Extension",
  "version": "1.0.0",
  "manifest_version": 3,
  "permissions": ["proxy", "storage", "webRequest", "webRequestAuthProvider"],
  "host_permissions": ["<all_urls>"],
  "background": { "service_worker": "background.js" },
  "action": { "default_title": "Proxy Auth" }
}
"""

background_js = f"""
chrome.runtime.onInstalled.addListener(() => {{
  chrome.proxy.settings.set({{
    value: {{
      mode: "fixed_servers",
      rules: {{
        singleProxy: {{ scheme: "http", host: "{PROXY_HOST}", port: {PROXY_PORT} }},
        bypassList: ["localhost"]
      }}
    }},
    scope: "regular"
  }}, () => {{}});
}});

chrome.webRequest.onAuthRequired.addListener(
  function(details) {{
    return {{ authCredentials: {{ username: "{PROXY_USER}", password: "{PROXY_PASS}" }} }};
  }},
  {{ urls: ["<all_urls>"] }},
  ["blocking"]
);
"""

plugin_path = "proxy_auth_plugin.zip"
with zipfile.ZipFile(plugin_path, "w") as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)

options = Options()
options.add_extension(plugin_path)
options.add_argument(f"--proxy-server=http://{PROXY_HOST}:{PROXY_PORT}")

driver = webdriver.Chrome(service=Service(), options=options)
driver.get("https://yourapp.com")
# your test logic here
driver.quit()

import os as _os
_os.remove(plugin_path)  # clean up the temp zip

The same manifest.json + background.js pattern applies in Java (via AdmZip) and Node.js — if you need those variants, the structure is identical, only the language binding differs. Note that some proxy providers publish a pre-built extension zip you can load directly; check your provider's integration documentation for a download link.

When to Use Sticky Sessions Instead of Rotating Proxies

The short answer: if your test has a login step, use a sticky session. If it doesn't, rotating is fine.

With a rotating proxy, each new connection can come from a different IP address. Most session management systems — especially those with fraud detection — treat a mid-session IP change as suspicious and invalidate the session token. Your test logs in successfully, then gets bounced back to the login page two clicks later. That looks like a race condition or UI timing issue, but it's the session getting killed by the IP change.

A sticky session holds one IP for a configurable time window. For E2E tests, set the TTL to cover your longest test flow with a 20% buffer. If your most complex checkout test takes 90 seconds end-to-end, a 2-minute sticky TTL is sufficient. Most residential proxy providers implement sticky sessions through session identifiers in the connection endpoint — the exact format varies by provider, so check your provider's API documentation, but the general pattern looks like this:

// playwright.config.ts — sticky session example
// The username format (session suffix) is provider-specific.
// This shows the common pattern; verify the exact syntax in your provider's docs.
const sessionId = `run_${process.env.GITHUB_RUN_ID ?? 'local'}_${Date.now()}`;

export default defineConfig({
  use: {
    proxy: {
      server: `http://${process.env.PROXY_HOST}`,
      username: `${process.env.PROXY_USER}-session-${sessionId}`,
      password: process.env.PROXY_PASS,
    },
  },
});

Using GITHUB_RUN_ID as part of the session ID ensures each CI run gets a distinct sticky IP rather than colliding with a parallel run.

Decision rule:

No login, no cart state, no multi-step form → rotating proxy
Has login, persistent cart, or any session-dependent flow → sticky session, TTL = longest test duration × 1.2

Verify Your Proxy Is Actually Working

Before pushing this to CI, confirm geo is working with a dedicated smoke test. It takes under two minutes to run and will save you from chasing a misconfigured setup later.

Prerequisites:

Proxy credentials (or whitelisted IP) from your provider
Playwright installed (npm init playwright@latest)
Environment variables set locally: PROXY_HOST, PROXY_USER, PROXY_PASS

The geo-check spec:

// tests/geo-check.spec.ts
import { test, expect } from '@playwright/test';

test('proxy IP resolves to target country', async ({ page }) => {
  const response = await page.goto('https://ipapi.co/json/');
  const text = await page.evaluate(() => document.body.innerText);
  const data = JSON.parse(text);

  console.log(`Resolved country: ${data.country_code}, IP: ${data.ip}`);
  // Update 'DE' to your target country code (US, JP, BR, etc.)
  expect(data.country_code).toBe('DE');
});

Run it against your EU project: npx playwright test tests/geo-check.spec.ts --project=EU-tests

Success criteria:

Test passes without timeout
Console output shows an IP in your target country (country_code: "DE")
The IP shown is not your CI runner's native IP address

If the test passes but country_code still shows your runner's home country, the proxy is being silently bypassed. The most common cause is a PROXY_HOST value missing the http:// prefix or an incorrect port number — fix those before debugging anything else.

A note on scope: Routing test traffic through residential proxies is legitimate testing practice, but your test targets still receive real requests. Keep these tests pointed at your own applications or staging environments. Avoid running load-heavy test suites through residential IPs against third-party services — it consumes real users' bandwidth allocations and may violate those services' terms of service. If your tests hit production endpoints you don't own (for localization checks, for example), verify your testing agreement covers automated traffic from external IPs.

Keeping Proxy Credentials Safe in CI/CD

Never commit proxy credentials to your repository. Store them as CI secrets and inject them as environment variables at runtime — no credential ever touches your codebase.

GitHub Actions:

Go to your repository → Settings → Secrets and variables → Actions → New repository secret.
Add three secrets: PROXY_HOST, PROXY_USER, PROXY_PASS. (GitHub's secrets documentation covers access controls and secret rotation.)
Reference them in your workflow:

# .github/workflows/e2e.yml
name: E2E Tests

on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium

      - name: Run E2E tests
        env:
          PROXY_HOST: ${{ secrets.PROXY_HOST }}
          PROXY_HOST_EU: ${{ secrets.PROXY_HOST_EU }}
          PROXY_HOST_US: ${{ secrets.PROXY_HOST_US }}
          PROXY_USER: ${{ secrets.PROXY_USER }}
          PROXY_PASS: ${{ secrets.PROXY_PASS }}
        run: npx playwright test

GitLab CI:

Add the variables under Settings → CI/CD → Variables (mark each as masked). Then reference them in .gitlab-ci.yml:

# .gitlab-ci.yml
e2e:
  image: mcr.microsoft.com/playwright:v1.44.0-jammy
  variables:
    PROXY_HOST: $PROXY_HOST
    PROXY_USER: $PROXY_USER
    PROXY_PASS: $PROXY_PASS
  script:
    - npm ci
    - npx playwright test

Add a .env.example file to your repository with placeholder values:

# .env.example — copy to .env and fill in real values (never commit .env)
PROXY_HOST=gate.yourprovider.com:7777
PROXY_HOST_US=us.yourprovider.com:7777
PROXY_HOST_EU=eu.yourprovider.com:7777
PROXY_USER=your_username
PROXY_PASS=your_password

The 5 Errors You'll Actually Hit

1. 407 Proxy Authentication Required

Credentials are wrong, or you're providing username/password when the provider expects IP whitelisting — not both. Verify with a manual curl first: curl -x http://user:pass@host:port https://ipapi.co/json/. If that fails, check your provider's dashboard for an IP Whitelist section and confirm your CI runner's outbound IP is registered there.

2. ERR_TUNNEL_CONNECTION_FAILED

Almost always caused by using https:// in the proxy server URL. Residential proxies expect http:// for the proxy connection itself, regardless of the target site's protocol — the HTTPS tunneling happens via CONNECT over that HTTP connection. Change server: 'https://gate.yourprovider.com:7777' to server: 'http://gate.yourprovider.com:7777' and retry.

3. SSL certificate errors (ERR_CERT_AUTHORITY_INVALID)

The proxy is intercepting TLS. For test environments where you're not specifically validating SSL certificates, suppress this per-project:

use: {
  ignoreHTTPSErrors: true,
  proxy: { server: '...', username: '...', password: '...' },
}

Scope it only to proxy-targeted projects — don't enable it globally in suites that include SSL validation tests.

4. Timeout spikes

Residential proxies route through real consumer ISP connections — average round-trip latency runs 20–150ms, compared to 5–30ms for most datacenter connections. The gap is predictable once you account for it. To find your correct timeout values: run your slowest navigation test without a proxy and note the actual duration from Playwright traces (or use --trace on for a run). Then set navigationTimeout to 3× that baseline for proxy-routed projects.

{
  name: 'EU-tests',
  use: {
    actionTimeout: 15_000,      // adjust if your baseline actions exceed ~5s
    navigationTimeout: 45_000,  // 3× a ~15s bare-metal navigation baseline
    proxy: { ... },
  },
}

5. IP banned or session drops mid-test

Unexpected auth redirects or login prompts mid-flow usually mean your sticky session TTL expired before the test finished. Check your provider's dashboard for the maximum TTL available on your current plan — some entry-tier plans cap sticky sessions at 60–120 seconds, which isn't enough for multi-step checkout flows. Either upgrade to a plan with a longer TTL or split long flows into shorter test cases that each complete within the window.

Choosing a Residential Proxy Provider for Testing

Generic proxy selection criteria (IP pool size, uptime SLA) don't tell you much for E2E testing specifically. These are the dimensions that actually affect whether your test suite works reliably:

Country-level coverage and regional reach: Verify the provider has IPs in the specific countries your app serves — not just a headline country count.
Sticky session maximum TTL: Confirm the TTL ceiling on the plan you're evaluating, not just "sticky sessions supported." Some plans cap at 1–2 minutes; complex flows need more.
SOCKS5 support: Selenium + ChromeDriver integrates more cleanly with SOCKS5 in some configurations. Verify this is available if your stack is Selenium-heavy.
Concurrent connection allowance: Playwright with 4 workers needs at least 4 simultaneous proxy connections. Add ~50% headroom for retries.
Framework integration documentation: Providers that publish working code examples for your specific framework — not just generic setup guides — save meaningful setup time.

Proxy001 covers all of these for testing use cases: 100M+ residential IPs across 200+ regions, with geo-targeting options that let you specify the target region for each proxy endpoint — check proxy001.com for the current granularity options. Proxy001 also publishes integration documentation for major testing frameworks — verify current coverage at proxy001.com or your account dashboard after sign-up. They offer a trial option so you can run the geo-check spec above against your actual test targets before committing to a plan; visit proxy001.com to check current plans and sign-up terms.

Run Your Next Geo Test With Confidence

If you've been losing hours to false-negative CI failures caused by geo-blocking and datacenter IP reputation filters, the fix doesn't require framework changes or test rewrites — just routing your test traffic through IPs that match your users' actual locations.

Proxy001 makes that practical without enterprise overhead. Their 100M+ residential IP pool spans 200+ regions with real ISP assignments, supports both rotating and sticky sessions for login-heavy flows, and the integration snippets in this article are ready to use with their endpoints today.

Visit proxy001.com to explore plans and sign-up options. Drop your credentials into the playwright.config.ts template above, run the geo-check spec, and confirm the country_code assertion passes. If it does, your geo-blocking flakiness problem is solved.

Setting Up a Multi-Region Google SEO Rank Tracking System Using Residential Proxy IPs

Miller James — Wed, 08 Apr 2026 01:57:31 +0000

Disclosure: This article is produced by Proxy001's content team. Proxy001 is mentioned once in the Prerequisites section as a recommended provider based on verified product features. We recommend evaluating providers based on your specific regional requirements before committing to any plan.

Residential proxy IPs combined with a rank tracking tool can get you genuinely accurate, region-specific Google ranking data — this is standard practice for agencies and in-house teams managing SEO across multiple markets. Whether you're monitoring a US retail brand's city-level pack positions or tracking keyword movement across five countries simultaneously, the underlying architecture is the same.

The gap between "bought proxies" and "working system" is larger than most guides let on. Without the right geo-targeting configuration, IP rotation strategy, and request pacing, you'll either collect inaccurate data — because Google served results based on the proxy's ASN rather than the claimed region — or trigger rate protection that corrupts your dataset. This guide covers everything needed to build this system end to end: prerequisites, architecture, full code with SERP parsing, SEO PowerSuite setup, multi-region scaling, data verification, troubleshooting, and a realistic compliance assessment.

What Do You Need Before You Start?

Pick your tracking path first

There are two practical approaches. Commit to one before configuring anything — they have different proxy requirements and different maintenance burdens.

Path A — SEO PowerSuite Rank Tracker (GUI): Best if you need scheduling, a built-in reporting UI, and don't want to maintain code. Proxy rotation is built in, configuration takes under 10 minutes. Covered in full below.

Path B — Custom Python script: Best for teams that need programmatic control over keyword sets, output format, and multi-region parallelization. More setup overhead upfront, more flexibility once running. The full implementation — including SERP parsing and rank extraction — is in the integration section.

Both paths require a residential proxy account with geo-targeting support. If you're already using a commercial SERP API like DataForSEO or SerpAPI, those services handle geo-targeting and parsing internally; you don't need proxies at all. This guide is specifically for teams running their own tracking infrastructure.

Get a residential proxy account with geo-targeting support

Not every residential proxy service handles country- and city-level targeting with enough precision for rank tracking. You need a provider that supports geo-targeting at minimum to the country level (city-level for local SEO use cases), offers both rotating and sticky session modes, has meaningful IP pool depth in every region you're tracking, and explicitly supports SEO monitoring workloads.

For this setup, Proxy001 covers all of the above: 100M+ residential IPs across 200+ regions, country and city-level targeting, both rotating and static IP modes, and documented integration examples for Python, Scrapy, and Selenium. Their free trial is the most reliable way to validate IP pool depth in your specific target regions before committing to a plan — pool depth for city-level targeting varies significantly across providers and secondary markets, and you want to know about gaps before they appear as production data gaps.

Estimate your bandwidth before signing up

Residential proxies are billed by bandwidth, and multi-region rank tracking consumes more than most people expect. The HTML response for a Google SERP query — what your proxy request actually downloads — is the raw HTML document, not a fully rendered page with all assets. Plain informational SERPs typically run in the 20–40 KB range; rich SERPs with multiple ad blocks, featured snippets, and People Also Ask sections can push toward 60–80 KB.

You can measure your own target SERP sizes in under 5 minutes before committing to a bandwidth plan:

# Measure actual HTML response size for a sample keyword
curl -s -o /dev/null \
  -w "Size: %{size_download} bytes\n" \
  "https://www.google.com/search?q=seo+proxy+service&num=10"

Run this against 10–15 representative keywords from your tracking set and use the average for your estimate. Then apply this formula:

Monthly bandwidth (GB) = keywords × regions × daily_checks × avg_KB × 30 ÷ 1,000,000

Example: 500 keywords × 5 regions × 1 daily check × 40 KB × 30 ÷ 1,000,000 = 3 GB/month

Add 20–30% on top for retries and proxy verification requests. For city-level tracking (e.g., 10 cities per country instead of 1 national endpoint), multiply region count accordingly.

Lock in your region list before configuring anything

Define target countries and cities in advance and store them in a version-controlled config file. Switching from national to city-level targeting mid-run means reconfiguring every proxy endpoint and invalidating historical comparisons. The config structure is covered in the scaling section below.

How Does Multi-Region Rank Tracking Actually Work?

The system runs as a five-stage pipeline:

Your server / local machine
    ↓
Rank tracking tool or Python script
    ↓
Residential proxy endpoint (geo-targeted to target region)
    ↓
Google.com — sees an ISP-registered IP from the target region
    ↓
Region-specific SERP → parse rank → store with keyword + region + timestamp

The critical stage is what happens at Google. Organic rankings, local packs, and featured snippets all vary based on the requester's geographic location. Google determines that location by checking the requesting IP against its IP intelligence database, which maps addresses to their registered ISP and geography.

This is why datacenter proxies fail here. Datacenter IPs belong to ASNs registered to cloud providers — AWS, DigitalOcean, Vultr — and Google's systems recognize these ASNs. The result is either generic non-geo-personalized SERP results, a rate protection response, or an outright block. Residential IPs are registered to consumer ISPs (Comcast, BT, Deutsche Telekom) and carry the same trust profile as a real user's home connection. Proxyway's 2025 proxy market benchmark measured a median 94.3% success rate for residential proxies against Google specifically, compared to significantly lower rates for datacenter IPs. proxyway

The gl and hl URL parameters Google exposes for geo-targeting are a useful reinforcement layer when aligned with the proxy's IP location. Community-reported testing consistently shows better locale consistency when both the IP and URL parameters point at the same region — use both together as a safe default.

How Do You Set Up Geo-Targeting for Each Region?

Proxy endpoint formats

Most residential proxy providers expose geo-targeting through credential encoding. Two common patterns:

Username-embedded geo:

http://username-country-us-city-chicago:password@gate.provider.com:port

Country code shorthand:

http://username-cc-us:password@gate.provider.com:port

The exact format is provider-specific — check your provider's integration documentation for the credential syntax they support. Getting it wrong produces silently inaccurate geo-targeting: the request succeeds but the IP exits from the wrong region. The verification step in the Python code below catches this before it contaminates a full tracking run.

Country-level vs. city-level targeting

Country-level targeting is sufficient for national SERPs where Google's results don't vary significantly within a country. City-level becomes necessary for local pack rankings, "near me" queries, or any keyword where Google's local algorithm produces meaningfully different results by metro area — "emergency plumber" in Chicago and Houston return completely different results; a US national proxy gives you neither.

One practical consideration: IP pool depth for city-level targeting in secondary markets is documented by providers as notably smaller than for major US metropolitan areas. Ask your provider specifically about pool depth for each target city before configuring volume tracking at that granularity. Thinner pools mean each IP in the pool handles more requests, which increases per-IP exposure. For city-level tracking in secondary markets, keep request volume modest or spread jobs over longer time windows.

Rotating vs. sticky sessions: the actual decision logic

Use rotating proxies for production bulk runs. When running your full keyword set across all regions — say, a daily 1,000-keyword check — rotating mode distributes requests across the IP pool automatically. This keeps per-IP request counts low, which is the most reliable way to keep legitimate monitoring from triggering rate protection systems.

Use sticky sessions for spot-checks and troubleshooting. Rotating proxies can produce variance in repeated checks because different IPs within the same country pool may route through different regional sub-clusters, each serving slightly different local SERP compositions. What looks like a 2–3 position ranking change can actually be different IPs resolving to different Google edge nodes. Sticky sessions eliminate this variable when you need a reproducible result from a defined geographic point. Sticky sessions typically hold the same IP for 10–30 minutes depending on provider configuration.

Recommended operating mode: rotating proxies for all scheduled production runs, sticky session endpoints available for manual verification and anomaly investigation.

How Do You Connect the Proxy to Your Rank Tracker?

Path A: SEO PowerSuite Rank Tracker (GUI)

SEO PowerSuite's Rank Tracker has native proxy rotation support with direct credential entry. Steps from their official documentation: link-assistant

Open Rank Tracker → Preferences → Search Safety Settings
Click Proxy Rotation…
Check Enable proxy rotation
Click Add to enter proxy servers individually:
- Address: your proxy hostname
- Port: the assigned port number
- If using username/password auth: check Proxy requires authentication and enter credentials
To add in bulk: click Import and paste entries in hostname:port format, one per line
Set Number of Simultaneous Tasks to roughly one-third your proxy count — this ensures each active task has backup proxies available link-assistant
Disable "Look for new proxies" — this option searches for public proxies, which you don't want when using private endpoints link-assistant
For each geo-targeted region, add the corresponding regional endpoint from your provider

Verifying the proxy is actually working in Rank Tracker:

After completing configuration, run a manual single-keyword check before kicking off a full campaign:

Go to Preferences → Search Safety Settings → Proxy Rotation and click Check next to each proxy — Rank Tracker will test connectivity and display status (Alive / Dead) and response time
Run one keyword manually (right-click → Check Rankings) and open Logs from the bottom panel — successful proxy usage shows requests routed through the proxy host address rather than your direct IP
As a geo-accuracy sanity check: run a keyword you know has strong regional signal (e.g., a local business category) and verify the SERP results contain region-appropriate content. If a UK-targeted proxy returns predominantly US results, the geo-targeting configuration needs review

Path B: Custom Python tracker

Prerequisites: Python 3.8+, requests, beautifulsoup4

pip install requests beautifulsoup4

Step 1: Verify proxy geo-targeting before any data collection

Run this against every proxy endpoint before a tracking session. A misconfigured geo parameter produces silent data errors — this check takes under 5 seconds and catches the problem before it contaminates a full run.

import requests
from typing import Optional

def verify_proxy_location(proxy_url: str) -> dict:
    """
    Confirm a proxy endpoint routes through the expected geographic region.
    proxy_url format: 'http://username:password@gate.provider.com:port'
    Returns dict with ip, country, city, isp fields.
    Note: ip-api.com free tier allows 45 requests/minute — sufficient for
    pre-session verification of a small proxy set.
    """
    proxies = {"http": proxy_url, "https": proxy_url}
    try:
        response = requests.get(
            "http://ip-api.com/json",
            proxies=proxies,
            timeout=10
        )
        data = response.json()
        return {
            "ip": data.get("query"),
            "country": data.get("countryCode"),
            "city": data.get("city"),
            "isp": data.get("isp"),
        }
    except requests.exceptions.RequestException as e:
        return {"error": str(e)}

# Example usage
proxy_us = "http://username-country-us:password@gate.proxy001.com:7777"
location = verify_proxy_location(proxy_us)
print(location)
# Expected: {'ip': '...', 'country': 'US', 'city': 'Chicago', 'isp': 'Comcast Cable Communications'}

If country doesn't match your target, stop and fix the geo-targeting configuration before proceeding.

Step 2: Fetch a geo-targeted Google SERP

Both the proxy IP and the gl/hl URL parameters should point at the same region. The Accept-Language header must also match — a mismatch between the header and hl parameter can cause Google to return content in the wrong language. dev

import time
import random
from urllib.parse import quote_plus

def fetch_google_serp(
    keyword: str,
    country_code: str,     # ISO 3166-1 alpha-2, e.g. "us", "gb", "de"
    language_code: str,    # e.g. "en", "de", "fr"
    proxy_url: str,
    num_results: int = 10,
) -> Optional[requests.Response]:
    """
    Fetch a geo-targeted Google SERP through a residential proxy.
    Returns Response on success (HTTP 200, no rate-limit redirect), None on failure.
    """
    encoded_kw = quote_plus(keyword)
    url = (
        f"https://www.google.com/search"
        f"?q={encoded_kw}&gl={country_code}&hl={language_code}&num={num_results}"
    )
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/124.0.0.0 Safari/537.36"
        ),
        "Accept-Language": (
            f"{language_code}-{country_code.upper()},"
            f"{language_code};q=0.9,en;q=0.8"
        ),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
    }
    proxies = {"http": proxy_url, "https": proxy_url}
    try:
        response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
        if response.status_code == 200 and "sorry/index" not in response.url:
            return response
        return None
    except requests.exceptions.RequestException:
        return None

Step 3: Parse SERP HTML and extract rank

This is the step most tutorials skip. The proxy connection alone doesn't give you ranking data — you need to extract the position of your target domain from the returned HTML.

Important: Google's HTML structure changes periodically. The selectors below reflect Google's markup as of April 2026, based on community-documented SERP structure analysis. If extract_rank() unexpectedly returns None for keywords you know rank, run debug_serp_structure() to inspect the current element layout and update the selectors. blog.scrapeup

from bs4 import BeautifulSoup
from urllib.parse import urlparse

def extract_rank(html_content: str, target_domain: str) -> Optional[int]:
    """
    Parse Google SERP HTML to find the organic rank of target_domain.

    target_domain: domain to search for, e.g. "example.com"
                   (without scheme; www. prefix is normalized automatically)
    Returns: 1-based rank position (int), or None if not found in results.

    Primary container selector: div.Ww4FFb (organic result wrapper, April 2026)
    Update this selector if debug_serp_structure() shows 0 containers.
    """
    soup = BeautifulSoup(html_content, "html.parser")
    target_clean = target_domain.replace("www.", "").lower().rstrip("/")

    organic_containers = soup.select("div.Ww4FFb")

    rank = 0
    for container in organic_containers:
        link = container.select_one("a[href]")
        if not link:
            continue
        href = link.get("href", "")
        if not href.startswith("http"):
            continue
        rank += 1
        parsed_domain = urlparse(href).netloc.replace("www.", "").lower()
        if target_clean in parsed_domain:
            return rank

    return None  # target not found in the result set


def debug_serp_structure(html_content: str, n: int = 5) -> None:
    """
    Print the first N organic result URLs from a SERP response.
    Use this when extract_rank() returns unexpected None values to verify
    that selectors are still matching Google's current HTML layout.
    If 0 containers are found, Google has changed its structure —
    inspect raw HTML and update the selector in extract_rank().
    """
    soup = BeautifulSoup(html_content, "html.parser")
    containers = soup.select("div.Ww4FFb")
    print(f"Found {len(containers)} organic result containers")
    for i, c in enumerate(containers[:n]):
        a = c.select_one("a[href]")
        print(f"  Result {i + 1}: {a['href'][:80] if a else 'no link found'}")

Step 4: Scale to multiple regions in parallel

The threading model below assigns exactly one thread per region and processes that region's keywords sequentially within that thread. This is important: a flat task pool with max_workers=len(regions) doesn't prevent the scheduler from running multiple keywords from the same region concurrently, which can stack requests through the same proxy endpoint and defeat the per-request delay.

import concurrent.futures
from datetime import datetime, timezone

# Region config — keep this in a version-controlled YAML file in production
REGION_CONFIG = {
    "us": {
        "proxy": "http://user-country-us:pass@gate.proxy001.com:7777",
        "gl": "us",
        "hl": "en",
    },
    "gb": {
        "proxy": "http://user-country-gb:pass@gate.proxy001.com:7777",
        "gl": "gb",
        "hl": "en",
    },
    "de": {
        "proxy": "http://user-country-de:pass@gate.proxy001.com:7777",
        "gl": "de",
        "hl": "de",
    },
}

KEYWORDS = ["seo proxy service", "residential proxy", "best proxy providers"]
TARGET_DOMAIN = "yoursite.com"


def process_region(region_code: str, keywords: list) -> list:
    """
    Process all keywords for a single region sequentially.
    Sequential processing within a region ensures the per-request delay
    applies properly and requests don't stack on one proxy endpoint.
    """
    config = REGION_CONFIG[region_code]
    results = []
    for keyword in keywords:
        time.sleep(random.uniform(8, 15))
        response = fetch_google_serp(
            keyword=keyword,
            country_code=config["gl"],
            language_code=config["hl"],
            proxy_url=config["proxy"],
        )
        rank = None
        if response is not None:
            rank = extract_rank(response.text, TARGET_DOMAIN)
        results.append({
            "keyword": keyword,
            "region": region_code,
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "rank": rank,
            "success": response is not None,
        })
    return results


def run_multi_region_tracking(keywords: list) -> list:
    """
    Run rank tracking across all configured regions in parallel.
    One thread per region; keywords processed sequentially within each thread.
    Regions run concurrently with each other — total runtime ≈ single-region runtime.
    """
    all_results = []
    with concurrent.futures.ThreadPoolExecutor(
        max_workers=len(REGION_CONFIG)
    ) as executor:
        future_to_region = {
            executor.submit(process_region, region, keywords): region
            for region in REGION_CONFIG
        }
        for future in concurrent.futures.as_completed(future_to_region):
            all_results.extend(future.result())
    return all_results


if __name__ == "__main__":
    results = run_multi_region_tracking(KEYWORDS)
    for r in results:
        status = f"rank {r['rank']}" if r["rank"] else "not ranked / request failed"
        print(f"[{r['region'].upper()}] {r['keyword']} → {status} ({r['timestamp']})")

Request pacing: responsible configuration for legitimate monitoring

Google's automated rate protection activates on patterns that match bulk automated querying — the same mechanism that blocks abusive bot traffic can flag legitimate rank monitoring if configured too aggressively. Community-reported benchmarks and proxy provider documentation consistently identify similar thresholds for legitimate monitoring workflows. A key variable is predictability: a fixed interval carries a stronger automation signal than the same average rate with randomized timing, even if the requests-per-minute is identical. docs.google

Request pattern	Risk of triggering rate protection
> 1 request/second from same IP	Rate protection activates within minutes
Fixed 3–5 second interval, same IP	Matches automated patterns; elevated false-positive risk
Fixed 8 second interval, same IP	Reduced rate, but fixed interval remains a bot pattern
Randomized 8–15 seconds per request, same IP	Consistent with browsing patterns; low false-positive risk
Full IP rotation (new IP per request)	Distributes load across IP pool; lowest per-IP exposure

The randomization is what matters most. This keeps per-IP request frequency within the range that legitimate SEO monitoring consistently operates without triggering rate protection, while also avoiding the fixed-interval timing signature that automated systems recognize. wpseoai

How Do You Track Rankings Across 5+ Regions Without Chaos?

Use a config file, not hardcoded endpoints

Store region list, proxy credentials, and tracking parameters in a YAML file under version control. When a proxy credential rotates — which it will — you update one file, not the script. When you add a region, same process.

# regions.yaml
regions:
  - code: us
    proxy: "http://user-country-us:pass@gate.proxy001.com:7777"
    gl: us
    hl: en
    check_frequency: daily
  - code: gb
    proxy: "http://user-country-gb:pass@gate.proxy001.com:7777"
    gl: gb
    hl: en
    check_frequency: daily
  - code: de
    proxy: "http://user-country-de:pass@gate.proxy001.com:7777"
    gl: de
    hl: de
    check_frequency: weekly

Store results with three mandatory fields

Every tracking record needs keyword, region, and timestamp. Without all three, your data becomes ambiguous within 48 hours. If writing to a database, add a composite index on (keyword, region, timestamp) for efficient trend queries.

Differentiated check frequency saves bandwidth

Daily for core markets, weekly for secondary ones. Dropping two of five regions from daily to weekly cuts monthly requests by 40% at the same keyword set size. The check_frequency field in the YAML above is the implementation point — add a simple filter in your scheduling logic before calling run_multi_region_tracking().

How Do You Know the Ranking Data Is Actually Region-Accurate?

Verify IP location before every session. Run verify_proxy_location() against each proxy endpoint before starting a full tracking session. A misconfigured endpoint passes the connection test while quietly routing through the wrong region — catching it here is much cheaper than catching it in your data.

Check SERP language consistency. After fetching a SERP, verify the HTML contains content in the expected language. A hl=de request returning predominantly English results is a reliable indicator of geo-targeting misconfiguration, not a ranking problem.

Run a sticky-session consistency check. For a sample of 10–20 keywords per region, make three requests using sticky session endpoints within a 10-minute window. Rankings should be within ±1–2 positions. Larger variance suggests IP geo-instability — the same country-targeted endpoint may be cycling through IPs from different sub-regions on each request.

For Path A users (SEO PowerSuite): after a full tracking run completes, open Logs and spot-check 3–5 entries to confirm requests went through proxy hosts rather than your direct IP. Additionally, compare a handful of results against what you'd see browsing via a VPN from the same country — meaningful divergence is a signal to recheck your geo-targeting endpoint configuration.

Know your baseline success rate. For high-quality residential proxies against Google, expect 90–95% request success. Proxyway's 2025 research measured a median 94.3% for Google-specific targets. If your rate drops below 85%, investigate before scaling — common causes include IP pool exhaustion in a specific region, pacing that's too aggressive, or expired credentials. proxyway

Troubleshooting

Symptom	Most likely cause	Fix
Rate protection responses (HTTP 429 or redirect to sorry/index)	Per-IP request frequency above threshold	Increase per-request delay to 15–30s; switch to full rotating mode to distribute load across IP pool
Rankings inconsistent across repeated runs for same keyword	Geo-targeting drifting — proxy returning IPs from different sub-regions	Run `verify_proxy_location()` mid-session; check if provider is cycling geography alongside IP
One specific region has > 20% failure rate	That region's IP pool is thin or exhausted	Contact provider about pool depth for that location; fall back from city-level to country-level targeting
`extract_rank()` returns `None` for keywords you know rank	Google changed SERP HTML structure; selectors are stale	Run `debug_serp_structure()` on a live response; identify new organic result container class; update selector
Response times consistently > 8 seconds	Proxy exit node is geographically far from target Google data center	Ask provider about routing options for that region
SERP returns results in wrong language	`Accept-Language` header and `hl` parameter mismatch	Set both explicitly per region in `fetch_google_serp()` config
All regions stopped working simultaneously	Bandwidth cap hit or credentials expired	Check provider dashboard immediately; verify monthly usage against your plan limit

Is This Legal? What Are the Real Risks?

You need a realistic assessment, not reassurance.

Google's Terms of Service position is clear. Google's ToS explicitly prohibit using automated means to access or use their services. Automated rank tracking queries fall within that prohibition — there's no interpretation where this is ToS-compliant. policies.google

What that means in practice. Violating Google's ToS is a platform compliance issue, not a criminal matter. Google's enforcement tools are technical: IP blocks, rate protection responses, and in theory, restrictions on authenticated accounts. The legal landscape around automated access to publicly accessible data is genuinely contested — the 9th Circuit's ruling in hiQ v. LinkedIn held that automated access to publicly available data doesn't automatically constitute unauthorized computer access, though that ruling is specific to its facts and ongoing litigation across circuits has produced mixed outcomes. Rank tracking occupies a gray zone: it queries publicly accessible results but at automated scale in violation of platform terms. No documented enforcement action has targeted standard rank monitoring operations. eff

Risk tiers in plain terms:

Risk	Probability	Context
Proxy IP triggers rate protection	High without rotation; low with residential IP rotation	This is what proxy rotation is designed to manage
Google account restricted	Low — only relevant if querying while logged in	Don't run rank tracking through authenticated sessions
Civil legal action from Google	Very low	No documented precedent against standard monitoring operations
Criminal liability	Essentially zero	Not applicable to this use case

The industry context: Essentially every major SEO platform — Semrush, Ahrefs, Moz — runs large-scale data collection infrastructure that queries Google at scale. Rank monitoring via residential proxies is an accepted industry practice. Operating responsibly means controlling request rates, not overloading Google's infrastructure, and using collected data for your own analysis rather than commercially redistributing raw SERP data.

The ToS-compliant alternative: Google's Custom Search JSON API provides official programmatic search access. The free tier allows 100 queries/day; paid tiers scale further. It's not designed for rank tracking at production volume, but it's the right choice for organizations where ToS compliance is a hard requirement.

Start Tracking With a Free Trial

Start scoped: one region, 50 keywords, daily checks for a week. Use debug_serp_structure() to confirm your SERP parsing is working, spot-check 5–10 results against a VPN session from the same country, and verify your success rate is above 90% before scaling.

Proxy001 is built for this kind of seo proxy infrastructure. Their 100M+ residential IP pool spans 200+ regions with country and city-level geo-targeting — covering national SERPs and local pack tracking — alongside rotating and static IP modes for the mixed workflow this system requires. The Python, Scrapy, and Selenium integration documentation maps directly to the code in this guide. Their free trial lets you validate geo coverage and pool depth in your specific target markets before committing to a paid plan — particularly useful for secondary markets where depth varies significantly across providers. Start at proxy001.com.

How Rotating Residential Proxies Handle IP Assignment and Session Management Under the Hood

Miller James — Thu, 02 Apr 2026 03:16:55 +0000

You've been using rotating residential proxies long enough to know the basics: one entry point, a pool of IPs, requests get rotated. Then you build a multi-threaded scraper expecting 50 different IPs across 50 concurrent workers, and your logs show the same IP cycling through repeatedly. Or you configure a sticky session for a multi-step checkout flow and it breaks mid-way with no meaningful error. Both of those problems trace back to the same root cause — you're treating the proxy system as a black box when the actual mechanics are knowable and specific.

This article maps out exactly how backconnect gateways route requests, what triggers rotation at the protocol level, and how sticky sessions are implemented under the hood. The goal is that you leave here able to configure with precision, not trial-and-error.

How Does a Backconnect Gateway Actually Route Your Requests?

The gateway is the architectural linchpin of the entire system. Your script connects to a single fixed endpoint — one hostname, one port — and that's the only address it ever "sees." Every IP assignment, routing decision, and rotation event happens behind that endpoint, invisible to the client.

When a request arrives at the gateway, the routing layer selects an exit node from the pool and proxies traffic through it before a single byte reaches the target site. The full flow:

Your Script
    ↓
Gateway (single endpoint, e.g. gate.provider.com:7000)
    ↓
IP Pool Routing Layer
    ↓
Exit Node (Residential Device)
    ↓
Target Site
    ↓ (response travels back the same path)
Your Script

The target site sees only the exit node's IP. Your real IP never appears in the transaction. And critically, the gateway owns the full transaction — it authenticates with the exit node, manages the upstream TCP tunnel, and ensures that neither the exit node nor the target site ever resolves your client's identity.

This is why all rotation behavior is provider-side. You're not cycling through a list of IPs in your own code — you're making a new TCP connection to the gateway and relying on its routing layer to pick a different exit node. That architectural distinction becomes very important when debugging why rotation isn't working.

How Do Real Home IPs Enter the Pool?

Residential IPs come from actual consumer devices — home routers, laptops, mobile phones — whose owners have opted into a bandwidth-sharing arrangement. The typical mechanism is an SDK embedded in a free app or browser extension: the device owner installs it, agrees to the terms, and their idle bandwidth gets contributed to the proxy network.

The technical implication is that residential exit nodes are not servers. They don't have datacenter uptime SLAs. A device that was online during a health check might go offline two minutes later because the user closed their laptop or their ISP rebooted the router. That authenticity advantage is genuine, but it's paired with an availability trade-off you don't get with datacenter proxies. Exit node dropout — devices going offline mid-session — is the single most common cause of unexpected sticky session failures, and handling it gracefully is covered in the troubleshooting section.

It also explains why residential IPs carry higher trust scores with most target sites. The IP is assigned to a real ISP customer with a legitimate usage history. Traffic through it looks behaviorally like a real home user, not a server rack.

How Does the Gateway Pick an IP for Your Request?

The gateway's IP selection isn't random — it's a multi-factor routing decision that happens in milliseconds on every new connection.

Geographic filtering is applied first. When you specify targeting parameters (country, city, isp), the routing layer filters the active pool down to exit nodes whose geo attributes match. Reputable providers cross-check node location against multiple geolocation databases — MaxMind, IP2Location, and IPinfo — because geographic inconsistency between databases is itself a detection signal on the target side. Nodes that don't meet the geographic match threshold for your specified targeting don't enter the candidate set at all.

Node health status further narrows the pool. The gateway continuously probes exit nodes to verify they're online, responsive, and not on blocklists. IP pool operators typically run these health probes on rolling intervals — anywhere from every 15 minutes for high-priority pools to every 60 minutes across larger pool segments — automatically quarantining nodes where connection success rates fall below the provider's internal thresholds. A blocked or slow-responding exit node gets pulled from the active rotation queue and placed in a cooldown state. The gateway retests it periodically; if it recovers, it gets re-admitted.

Load distribution prevents over-concentration on popular exit nodes. The routing layer tracks active connection counts per exit node and avoids funneling too many requests through the same node — both because it degrades that node's performance and because unusually high request volume from a single residential IP is itself a detection pattern.

Your rotation strategy feeds into the final routing decision. Per-request, time-based, and sticky session modes all modify what the routing layer does with the candidate set. This is where your configuration choices actually land.

When and How Does the Proxy Switch IPs?

Three distinct rotation models, each triggering at a different point in the request lifecycle.

Per-request rotation swaps the exit node on every new TCP connection to the gateway — and that phrase "new TCP connection" is doing a lot of work. HTTP/1.1 keep-alive and connection pooling in most HTTP clients maintain a single persistent TCP connection to the proxy endpoint and multiplex multiple HTTP requests over it. If your Python requests.Session() object holds a keep-alive connection to the gateway, every subsequent request in that session rides through the same TCP tunnel — and therefore hits the same exit node, even with per-request rotation configured.

This is the most frequently reported rotation failure in production. Symptom: per-request rotation configured, same IP appearing repeatedly in logs. Cause: the HTTP client is reusing the TCP connection to the backconnect gateway. Per-request rotation can only trigger on a new connection. The fix is to ensure your code isn't inadvertently keeping connections alive — more on the specific mechanics in the troubleshooting section.

Time-based rotation runs entirely server-side. The gateway maintains a timer per client; when the configured interval expires, the next request from that client is routed through a new exit node regardless of the connection state. From your code's perspective, nothing changes — same endpoint, same credentials, same port. The IP changes when the gateway's timer fires. This mode is effectively "set and forget" — no connection management required on your end.

Forced rotation gives you client-side control over exactly when a switch happens. By changing the session parameter in the proxy username between requests, you signal to the gateway to treat this as a new session and assign a fresh exit node. This is different from sticky sessions (which hold an IP constant) — here you're using the same session parameter mechanism to actively force a change. The format is identical to what sticky sessions use, just with a different token per request rather than a constant one — that format is covered in detail in the next section.

Concurrent requests deserve a dedicated note. Each independent TCP connection to the gateway is its own routing event — the gateway assigns a different exit node per connection. Twenty concurrent connections should yield up to twenty distinct IPs. But if your HTTP client or thread pool multiplexes those 20 requests over fewer TCP connections (common with HTTP/2 or shared connection pools), you'll get fewer unique IPs than you expect. For maximum IP diversity in concurrent scraping, each worker needs its own isolated TCP connection to the gateway — not a shared pool.

Those mechanics explain when IP changes happen. The more interesting question for stateful workflows is how to prevent them — which is where session management comes in.

How Does Sticky Session Actually Work at the Protocol Level?

Sticky session works through a routing table the gateway maintains, keyed on a token you embed in the proxy authentication header.

Standard proxy authentication uses username:password. For sticky sessions, you extend the username field with a session identifier. The general format looks like this:

username-session-{your_token}:password@gateway.host:port

Providers implement this with their own field names. Decodo's residential proxy documentation, for example, uses user-username-session-example1-sessionduration-90:password where sessionduration sets how many minutes the session should persist. SOAX uses package-{id}-country-us-sessionid-rand3-sessionlength-600. The mechanism is the same across providers: the session token is embedded in the username string, parsed by the gateway's authentication layer, and used as a lookup key in the routing table.

When the gateway receives a request with a session token it recognizes, it skips the pool selection step entirely and routes directly to the exit node mapped to that token. When it sees a new or expired token, it runs the full selection process and creates a new mapping.

Session tokens have a TTL. Decodo's default is 10 minutes for residential proxies, with sessions configurable up to 24 hours depending on the endpoint type. When TTL expires, the mapping entry is dropped, and the next request with that token is treated as a new session. There's a second termination trigger that's less intuitive: if the exit node goes offline mid-session (the residential device disconnects), the gateway can't fulfill the routing promise. Some providers silently migrate you to a new exit node; others return a 503 error. If your sticky session breaks before the TTL expires with no clear error, the exit node going offline is the most likely explanation.

Running multiple concurrent sticky sessions works cleanly — different tokens map to different exit nodes in the routing table. You can maintain 50 parallel sticky sessions with 50 distinct tokens and get 50 persistent IPs simultaneously. The practical limit is your subscription's concurrent connection allowance.

Verifying Your Session Configuration

Before wiring any of this into production code, it's worth confirming your proxy handles both modes as expected. You need Python 3 with requests installed and valid credentials from your proxy provider.

import requests

# Your gateway endpoint and credentials — find these in your provider's dashboard
GATEWAY = "your.proxy-gateway.com:7000"
USER    = "your_username"
PASS    = "your_password"
CHECK   = "https://httpbin.org/ip"

def make_proxies(session_id=None):
    user = f"{USER}-session-{session_id}" if session_id else USER
    proxy_url = f"http://{user}:{PASS}@{GATEWAY}"
    return {"http": proxy_url, "https": proxy_url}

# Test 1: Per-request rotation — each call should return a different IP.
# Note: using requests.get() directly, NOT a Session object.
print("=== Per-request rotation ===")
for i in range(3):
    r = requests.get(CHECK, proxies=make_proxies(), timeout=15)
    print(f"  Request {i+1}: {r.json()['origin']}")

# Test 2: Sticky session — all three calls should return the same IP.
print("\n=== Sticky session (token: test-sticky-01) ===")
for i in range(3):
    r = requests.get(CHECK, proxies=make_proxies(session_id="test-sticky-01"), timeout=15)
    print(f"  Request {i+1}: {r.json()['origin']}")

# Test 3: New session token — should return a different IP than Test 2.
print("\n=== New session token (test-sticky-02) ===")
r = requests.get(CHECK, proxies=make_proxies(session_id="test-sticky-02"), timeout=15)
print(f"  Request 1: {r.json()['origin']}")

For Proxy001, the exact gateway endpoint and credential format are in your dashboard's connection guide — the credential generator outputs the correct session field name and separators for your account type.

Expected output: Test 1 should print three different IPs. Test 2 should print the same IP three times. Test 3 should print a different IP than any from Test 2.

If Test 1 shows the same IP repeating: Connection pooling is the culprit. Using requests.get() directly (as above) creates a fresh connection each time. If you're using a Session() object, add "Connection": "close" to your headers, or close and recreate the session per request. The urllib3 advanced usage documentation covers connection pool configuration in detail if you need finer-grained control.

If Test 2 shows different IPs: The session token isn't being parsed correctly. Check the exact field separator and parameter names in your provider's documentation — -session- is common, but providers are inconsistent on this. The credential generator in your provider's dashboard outputs the correct format.

A quick curl equivalent for Test 2:

# Run this twice — both should return the same IP
curl -s -U "username-session-test-sticky-01:password" \
     -x "your.proxy-gateway.com:7000" \
     https://httpbin.org/ip | python3 -m json.tool

If you're getting connection timeouts on Test 2, the assigned exit node may be temporarily offline. Retry with a different session token — the gateway will allocate a fresh node.

Compliant use: Residential proxies are a legitimate tool when the target is a publicly accessible resource and your activity doesn't violate the site's Terms of Service or applicable law. Don't route proxy traffic through authenticated systems you're not authorized to access. If your work involves collecting data that includes personal information — user reviews, public profiles, pricing tied to individual accounts — GDPR, CCPA, and equivalent laws govern what you do with that data, independent of how the proxy is configured. In commercial scraping operations, maintaining session logs and audit trails is standard practice.

Why Is My IP Rotating Unexpectedly (or Not at All)?

Four failure patterns cover the majority of rotation and session issues in practice. They break into two categories: connection-level problems, and target-side detection problems.

Connection-level failures are the more fixable of the two. If your sticky session drops before the TTL expires, the exit node went offline. Residential devices disconnect — users close laptops, ISPs recycle connections, the bandwidth-sharing app gets killed in the background. You can't prevent this, but you can handle it: catch requests.exceptions.ProxyError or a 503 response code, generate a new session token, and retry from the beginning of your multi-step flow. Logging the response code (or the error type on connection failure) helps distinguish node dropout from token expiry — the former tends to arrive as a connection reset, the latter as a clean expiry followed by a new IP assignment on retry.

If per-request rotation isn't rotating, the cause is almost always connection reuse. Python's requests.Session() object enables keep-alive via urllib3 by default, so all requests made through the session reuse the same TCP connection to the gateway. Use standalone requests.get() calls, or if you need a session for cookie handling, add session.headers.update({"Connection": "close"}) and call session.close() after each target domain transaction. In concurrent setups, verify your thread pool isn't sharing a connection pool: a Session object shared across threads maintains a shared urllib3 pool, so multiple workers may route through fewer actual TCP connections than you have threads. Give each worker its own independent connection management.

Target-side blocks that persist across IP changes are a different class of problem. When you're getting blocked even after rotating to a confirmed-fresh IP, the block isn't on the IP address. Target sites increasingly fingerprint on TLS client hello characteristics, HTTP/2 pseudo-header ordering, cookie residue from previous sessions, and request timing patterns. IP rotation won't fix any of these. If you've confirmed the new IP is clean by testing against httpbin.org/ip and a fresh request to the target in isolation, the issue is in your client configuration, not the proxy layer.

How to Configure These Mechanisms in Practice

A direct mapping from use case to the right mode and the parameters that matter:

Use Case	Rotation Mode	Configuration Priority
Stateless product / price scraping	Per-request	Disable keep-alive; fresh TCP connection per request
Multi-step login or checkout flow	Sticky session	TTL ≥ max expected session duration; handle ProxyError for node dropout
High-volume concurrent SERP crawling	Per-request, per-worker isolation	One connection per worker; no shared connection pool
Legitimate multi-account workflows (e.g., managing ad accounts or storefronts you own and are authorized to operate)	Multiple sticky sessions	Unique token per account; ensure all accounts are under your organization's authorization
Geo-targeted verification	Either mode + geo params	`country`, `city`, `isp` in username or API params

Geographic targeting and session parameters are both configured through the proxy username field in most backconnect providers — no separate API call needed for standard configurations. The exact field names (-country-, -city-, -session-, -sessionlength-) are provider-specific. Most backconnect providers expose both parameters through their dashboard credential generator — consult your provider's setup documentation for the exact field names and separators for your account type.

What to Do With This Now

Three concrete steps from here.

Run the verification script from the session section against your current proxy configuration. Confirm that per-request rotation and sticky session behavior match your expectations before wiring either into production code. The test takes under five minutes and saves hours of debugging later.

Audit concurrent setups for connection pooling. If your multi-worker scraper isn't achieving the IP diversity you expect, check whether your HTTP client is multiplexing worker threads onto shared connections. The fix is typically a one-line change — the problem is that it's invisible in logs unless you're actively printing the source IP per request.

Map your use cases to the rotation mode table. Sticky session and per-request rotation are both correct choices — for different jobs. Running per-request rotation on a stateful authenticated flow will break session continuity. Running sticky session on a high-volume stateless crawl wastes IP pool diversity. Neither failure is the proxy's fault; it's a configuration mismatch.

If you want to run these tests against a production-grade residential IP pool, Proxy001 offers a free trial with full access to sticky session and per-request rotation on the same endpoint. The pool spans 100M+ IPs across 200+ regions, with real ISP-assigned residential addresses — you get a live environment to verify everything covered in this article before committing to a plan. The dashboard credential generator outputs the exact username format for your session and geo parameters, so there's no manual string assembly. Request your free trial and run the verification script in this article within minutes.

DEV Community: Miller James

Building a Proxy Failover Layer With Circuit Breaker Logic

What circuit breaker logic adds over retry-and-rotate

Decide what actually counts as a proxy failure

Build the circuit breaker state machine

Put a failover selector in front of your residential proxy pool

Choose where the breaker lives: per-IP, per-endpoint, or per-provider

Tune the three thresholds

Layer fallback tiers for graceful degradation

Build your own failover layer or use a backconnect residential proxy network?

Mistakes that make a circuit breaker worse than none

Quick answers

Proxy-Aware HTTP Client in Python: Retries, Timeouts & Residential Proxy IPs

Building a Proxy-Aware HTTP Client in Python With Retries, Timeouts, and Sessions

Why residential proxy IPs need a retry-and-timeout strategy

Set up a requests Session with residential proxy IPs

Set timeouts for flaky residential routes

Add automatic retries with exponential backoff

Rotate residential IPs: gateway vs sticky session

Pair retries with rotation so you never hammer a dead IP

Run a proxy IP test to confirm rotation works

Handle proxy-specific errors cleanly

The complete proxy-aware client

Scaling up: async and concurrency

Wrap-up checklist

Integrating Residential Proxies Into Playwright Test Suites

What should your suite architecture look like?

How do you wire it into Playwright Test?

Step 1: Install and prepare Playwright

Step 2: Put the residential proxy in environment variables

Step 3: Configure the suite in playwright.config.ts

Step 4: Add a verification test before real suite traffic

How do you handle rotation, sticky sessions, and region-specific projects?

Use project-level routing for regional test suites

Use sticky sessions for stateful browser flows

Use test.use() for file-level specialization

What usually breaks, and how do you fix it fast?

Symptom: the proxy works in a standalone script but not in the test suite

Symptom: the browser loads, but the target flow becomes unstable after login

Symptom: SSL or certificate errors appear only on some residential routes

Symptom: CI installs fail before tests even start

CTA

Compliance note

Residential Proxies for Geo-Testing: A Practical Developer QA Workflow

Residential Proxies for Geo-Testing: A Practical Developer QA Workflow

What problem does geo-testing solve?

Safe use cases for residential proxies

The geo-testing workflow

Step 1: Write a specific test question

Step 2: Create a small test matrix

Step 3: Define pass/fail rules

Manual smoke test with curl

Build a minimal Node.js geo-test

1. Create the project

2. Create the test matrix

3. Create the test script

How to read failures

1. Proxy or network failure

2. Redirect failure

3. Cache variant failure

4. Localization failure

Add this to CI carefully

Compliance checklist

When not to use residential proxies

Final checklist

Need residential proxies for geo-testing?

How to Route Your Python Requests Through a Mobile Proxy

What Do You Need From Your Mobile Proxy Provider?

What Should a Mobile Proxy URL Look Like in Python Requests?

How Do You Use a Mobile Proxy With requests.get()?

How Can You Verify That Requests Is Really Using the Mobile Proxy?

Check 1: Confirm the proxy is active

Check 2: Confirm the exit IP is mobile

Should You Rotate IPs or Keep a Sticky Session?

Why Is My Mobile Proxy Not Working in Python Requests?

407 Proxy Authentication Required

requests.exceptions.ProxyError

The request hangs or times out

HTTPS fails with SSL errors

The request ignores the proxy

Step 3: Configure the suite in `playwright.config.ts`

Use `test.use()` for file-level specialization

How Do You Use a Mobile Proxy With `requests.get()`?

`407 Proxy Authentication Required`

`requests.exceptions.ProxyError`

`socks5` vs `socks5h`

Step 2 — Build the `RegionalRouter` Class With a Session Pool

1. `socks5://` Resolves DNS Locally — Use `socks5h://` Instead