Every scraping SDK grows a proxy layer eventually. Most grow it badly: a get_proxy() function that returns a random URL, then a global dict of "bad IPs" that everyone mutates from everywhere, then a session parameter grafted onto eleven function signatures as an afterthought. The result is a proxy layer that exists but has no design — its behavior is whatever the accumulation of patches implies.
This article is about doing it deliberately. Thesis: in a collection SDK, the proxy layer is not a networking detail — it is the resource scheduler of the whole system, and its three core concepts (rotation, affinity, failure domains) should be first-class in the API, composable, and visible in every stack trace and metric.
The three concepts, stated precisely
Vague words kill proxy layers, so let me pin down the vocabulary first.
Rotation is the policy for how long one logical identity rides one exit IP. It is a spectrum, not a boolean: per-request rotation, time-windowed rotation, and no rotation (static exit) are all points on the same axis — the "granularity" of exit reuse.
Affinity is the mapping from your logical identity (a SKU, an account, a city probe) to exits. Sticky-per-key, sticky-per-worker, one-exit-per-domain, and round-robin are affinity strategies. Rotation and affinity are often conflated into "sticky vs rotating," which is exactly the confusion that produces a fleet where every account shares one static IP for no reason anyone can articulate.
Failure domain is the blast radius you accept when something dies. A dead exit, a burned exit (alive but 403-ing one origin), an overloaded provider gateway, and a dead credential are four different failures with four different correct responses — and a designed proxy layer treats them as distinct, named conditions.
The API surface
Here is the core of a proxy layer I consider well-designed. The whole public surface fits on one screen:
from __future__ import annotations
import hashlib
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
class Granularity(Enum):
PER_REQUEST = "per_request" # new exit every time
WINDOWED = "windowed" # hold exit for N seconds
STATIC = "static" # one exit for the lifetime
@dataclass(frozen=True)
class LeaseRequest:
key: str # logical identity: sku, account, city...
granularity: Granularity
window_s: float = 600.0 # only meaningful for WINDOWED
origin: str = "default" # which site we're hitting
class ExitState(Enum):
HEALTHY = "healthy"
BURNED = "burned" # origin-refuses: 403/429 from this exit
DEAD = "dead" # infra-refuses: connect/timeout at proxy
QUARANTINED = "quarantined" # manual or provider-wide issue
@dataclass
class Exit:
proxy_url: str
state: ExitState = ExitState.HEALTHY
burned_origins: set[str] = field(default_factory=set)
cooldown_until: float = 0.0
leased_at: float = 0.0
class ExitCooldown(Exception):
"""Raised when the *provider layer* is degraded — not retriable per-exit."""
class ProxyScheduler:
def __init__(self, gateway_url: str, sticky_ttl_s: float = 600.0):
self.gateway = gateway_url
self.sticky_ttl = sticky_ttl_s
self._assigned: dict[tuple[str, str], Exit] = {} # (key, origin) -> exit
self._static_pool: dict[str, Exit] = {}
# -- public API ----------------------------------------------------
def lease(self, req: LeaseRequest) -> Exit:
"""Acquire the exit a request should travel through."""
if req.granularity is Granularity.STATIC:
return self._static_for(req)
if req.granularity is Granularity.WINDOWED:
exit_ = self._assigned.get((req.key, req.origin))
if exit_ and time.monotonic() - exit_.leased_at < self.sticky_ttl \
and self._usable(exit_, req.origin):
return exit_
exit_ = self._new_exit(req.key)
self._assigned[(req.key, req.origin)] = exit_
return exit_
return self._new_exit(req.key) # PER_REQUEST
def report(self, exit_: Exit, origin: str, outcome: str) -> None:
"""The ONLY mutation path. Outcomes name the failure domain."""
if outcome == "ok":
exit_.burned_origins.discard(origin)
elif outcome == "blocked": # origin said no
exit_.burned_origins.add(origin)
exit_.cooldown_until = time.monotonic() + 300.0
elif outcome == "proxy_error": # infra said no
exit_.state = ExitState.DEAD
elif outcome == "auth_error": # credential layer
raise ExitCooldown("provider rejected credentials — stop the fleet")
# -- internals ------------------------------------------------------
def _usable(self, exit_: Exit, origin: str) -> bool:
return (
exit_.state is ExitState.HEALTHY
and origin not in exit_.burned_origins
and time.monotonic() >= exit_.cooldown_until
)
def _new_exit(self, key: str) -> Exit:
# Provider-side rotation: a distinct session token yields a
# distinct exit from the gateway. Deterministic tokens make
# WINDOWED leases stable across workers.
token = hashlib.sha1(key.encode()).hexdigest()[:12]
user, rest = self.gateway.split("://", 1)[1].split("@", 1)
u, _, p = user.partition(":")
return Exit(f"http://{u}-sess-{token}:{p}@{rest}", leased_at=time.monotonic())
def _static_for(self, req: LeaseRequest) -> Exit:
if req.key not in self._static_pool:
self._static_pool[req.key] = self._new_exit("static-" + req.key)
return self._static_pool[req.key]
Why these specific choices
The lease is a noun, not a side effect. Callers ask for a lease with an explicit LeaseRequest; they don't call rotate() or poke a global. That makes every fetch's exit decision inspectable — you can log the request, replay the schedule deterministically in tests, and answer "which exit served this account yesterday?" from telemetry instead of memory.
Burned is per-origin, dead is global. This is the failure-domain separation that most homegrown layers miss. An exit that gets 403'd by one retailer is often perfectly healthy for every other origin — burning it globally throws away good capacity, while ignoring the burn gets you a loop of doomed retries. burned_origins is a set, so the same exit keeps serving other origins until evidence says otherwise.
report() is the only mutation path. All state changes flow through one method with a small vocabulary of named outcomes. This gives you a single place to attach metrics (see below) and makes the layer thread-safe by construction if you wrap lease/report pairs correctly — the classic pattern of acquiring a lock only around the two scheduler calls, never around network I/O.
Auth failure raises, it doesn't retry. A credential rejection is a fleet-level stop condition. Retrying it per-request just generates a thousand identical failures and one very confused on-call engineer. Making it a distinct exception type forces the caller's supervisor loop to treat it as "pause everything," which is the only correct response.
Using it from the fetch layer
The scheduler composes into a fetch loop cleanly because the loop only ever sees leases and reports:
import asyncio
import aiohttp
async def fetch(scheduler, url: str, key: str, origin: str,
granularity=Granularity.WINDOWED) -> str:
req = LeaseRequest(key=key, granularity=granularity, origin=origin)
for attempt in range(4):
exit_ = scheduler.lease(req)
try:
timeout = aiohttp.ClientTimeout(total=30)
async with aiohttp.ClientSession(timeout=timeout) as s:
async with s.get(url, proxy=exit_.proxy_url) as r:
if r.status in (403, 429):
scheduler.report(exit_, origin, "blocked")
await asyncio.sleep(min(2 ** attempt, 20))
continue
scheduler.report(exit_, origin, "ok")
return await r.text()
except (aiohttp.ClientProxyConnectionError, asyncio.TimeoutError):
scheduler.report(exit_, origin, "proxy_error")
await asyncio.sleep(0.5 * (attempt + 1))
raise RuntimeError(f"exhausted retries for {url}")
# A/B two granularities through the same scheduler:
# await fetch(s, product_url, key=sku, origin="retailer-x",
# granularity=Granularity.WINDOWED) # price checks
# await fetch(s, sitemap_url, key="sitemap", origin="retailer-x",
# granularity=Granularity.PER_REQUEST) # bulk crawl
Because granularity is per-call, one fleet can run bulk crawls on per-request rotation while authenticated price checks hold windowed leases — the two workloads that most teams end up splitting across two codebases, here sharing one scheduler, one metric stream, and one burn-list.
The observability you get for free
Since every state change passes through report() and every decision through lease(), two counters and one gauge describe the whole layer: reports by outcome (ok / blocked / proxy_error), and current distinct exits in use. A healthy layer shows a stable ok ratio per origin and a blocked rate that concentrates on specific exits — when blocked spreads uniformly across exits instead, the problem is your request pattern, not your pool, and no amount of IP churn will fix it. That single diagnostic distinction has saved me more debugging hours than any other signal in the stack.
Rotation, affinity, failure domains — name them, make them parameters, and your proxy layer stops being the scary part of the codebase and becomes the part you can reason about at 2 a.m.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)