Integrating Rotating Proxies into requests: Session-Level, Per-Request, or a Custom Adapter?

The requests library makes "use a proxy" look trivial: pass proxies={"https": "http://..."} and you're done. Then reality arrives. You need a different residential IP on every request. Or the same IP for a five-call login flow and a different one after. Or a retry that swaps to a fresh gateway when the current one times out. At that point the one-liner proxies dict starts to feel like it was designed for a corporate squid box, not a rotating pool of thousands of gateways. This article walks through the three real integration strategies — per-request, session-level, and a custom HTTPAdapter — and when each one is the right tool.

Strategy 1: Per-request proxies= (fine for simple rotation)

The most direct mapping to a rotating pool: build a fresh gateway string on every call. Providers that accept the "proxy city / session id" encoded in the username make this easy.

import random
import requests

def gateway(country=None, session=None):
    parts = ["user-XXX"]
    if country:
        parts.append(f"-COUNTRY-{country}")
    # A random session id forces a new exit IP on many gateway providers.
    parts.append(f"-SESSION-{random.randint(1, 10_000_000):08d}")
    return "http://" + "".join(parts) + ":[e-mail]:8000"


def get(url, country=None, **kw):
    gw = gateway(country=country)
    return requests.get(url, proxies={"http": gw, "https": gw}, timeout=20, **kw)

When it's right: stateless crawls where every page should look like a new visitor. When it breaks: you can't hold a session across calls, because each request hops IPs. Login flows, CSRF token chains, and cart-building all shatter. And you're constructing a string per call — no central place to add health tracking or retries.

Strategy 2: A sticky requests.Session (right for flows)

A Session reuses the underlying TCP connection and, crucially, cookies. Point the session at one gateway with a fixed session id and every request in that session exits the same residential IP — which is exactly what a multi-step flow needs.

import requests


def sticky_session(country="US"):
    s = requests.Session()
    gw = (
        f"http://user-XXX-COUNTRY-{country}"
        f"-SESSION-abc123:[e-mail]:8000"
    )
    s.proxies = {"http": gw, "https": gw}
    s.headers.update({"User-Agent": "collection-bot/1.0 ([e-mail])"})
    return s


def login_then_scrape():
    s = sticky_session("US")
    s.get("https://shop.example/csrf")          # cookie jar now holds the token
    s.post("https://shop.example/login", data={"u": "...", "p": "..."})
    return s.get("https://shop.example/account")  # same IP, still logged in

When it's right: anything requiring continuity — one identity, one IP, one cookie jar. The scaling question: one session = one account = one IP. To parallelize across 20 accounts, you need 20 independent Session objects, each with its own sticky gateway and cookie jar. Don't share a session across accounts; don't share cookies across sessions. That association discipline is the whole game for multi-account work.

Strategy 3: A custom HTTPAdapter (the pro move)

Both strategies above scatter proxy logic through your calling code. The clean design mounts a ProxyRotator onto the Session as an adapter, so callers just do session.get(url) and never think about gateways, retries, or per-request rotation again. requests already routes every call through HTTPAdapter.send, which makes it the ideal seam.

import itertools
import requests
from requests.adapters import HTTPAdapter


class ProxyRotator:
    def __init__(self, gateways):
        self._pool = itertools.cycle(gateways)

    def next(self):
        return next(self._pool)


class RotatingProxyAdapter(HTTPAdapter):
    """Rotates the exit IP per request and retries on gateway trouble."""

    def __init__(self, gateways, max_retries=3, timeout=20, **kw):
        self.rotator = ProxyRotator(gateways)
        self.max_retries = max_retries
        self.timeout = timeout
        super().__init__(**kw)

    def send(self, request, **kw):
        kw.setdefault("timeout", self.timeout)
        last_exc = None
        for _ in range(self.max_retries):
            gw = self.rotator.next()
            request.url = request.url  # keep as-is; set proxy on send
            kw["proxies"] = {"http": gw, "https": gw}
            try:
                resp = super().send(request, **kw)
                # Treat 407 (proxy auth) as a gateway failure worth rotating.
                if resp.status_code == 407:
                    continue
                return resp
            except (requests.ConnectionError, requests.Timeout) as e:
                last_exc = e  # swap gateway and retry
        if last_exc:
            raise last_exc
        return resp


def build_session(gateways):
    s = requests.Session()
    adapter = RotatingProxyAdapter(gateways, max_retries=3)
    s.mount("https://", adapter)   # mount for your target scheme(s)
    s.mount("http://", adapter)
    return s


if __name__ == "__main__":
    gates = [
        "http://user-x-sess-1:[e-mail]:8000",
        "http://user-x-sess-2:[e-mail]:8000",
        "http://user-x-sess-3:[e-mail]:8000",
    ]
    s = build_session(gates)
    for _ in range(5):
        r = s.get("https://httpbin.org/ip", timeout=20)
        print("origin sees:", r.json()["origin"])

Now your scraping code is beautifully boring: session.get(url). Rotation, retry, and proxy-auth handling all live in one testable class. When it's right: production pipelines where you want proxy policy centralized and swappable — and where you need retry-on-a-different-gateway, which the built-in urllib3.Retry can't do because it retries against the same proxy, not a fresh one.

Choosing, in one line each

• Per-request proxies= → stateless crawls, fastest to write, no continuity.
• Sticky Session → multi-step flows and per-account isolation; one identity, one IP, one cookie jar.
• Custom adapter → production rotation + per-gateway retries with zero proxy logic in your calling code.

The failure mode I see most: people pick per-request rotation (correct for stateless crawling) and then paste it into a login flow, silently breaking every session-continuity assumption. The proxy integration is the architecture decision — rotation versus stickiness — not a config detail.

Two integration gotchas worth pre-empting

Proxy env vars leak in. If the host machine has HTTP_PROXY/HTTPS_PROXY set, requests picks them up via trust_env and they can override or fight your explicit gateways. For predictable behavior set session.trust_env = False when you want only your configured proxies, and session.proxies = {} if you want direct.

SSL verification through MITM gateways. Some residential gateways terminate TLS. If you start seeing certificate errors, don't reach for verify=False reflexively — that silently disables a real security check for your data. Confirm with your provider whether interception is expected; if it is, verify the chain against the provider's CA bundle rather than turning verification off globally.

The takeaway

requests gives you three clean integration points for a rotating pool, and the right one depends entirely on whether your job needs continuity. Reach for a custom HTTPAdapter the moment your pipeline grows real retry logic — it lets your scraper code forget proxies exist, which is exactly what a well-integrated layer should do. Get rotation-versus-stickiness right first; everything else is plumbing.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele