Most multi-account proxy setups solve the same problem — "which exit IP should this account use?" — with one of two bad answers. Either they let the proxy gateway's rotation decide (random churn, and the account teleports between exits), or they keep a mutable mapping table in the database: account_id -> exit_ip, updated by whoever won the race to write it last. The second approach at least produces stability, but it is a distributed-systems headache dressed up as a config table: rows go stale, workers disagree about the current mapping, and every migration becomes a coordinated write.
There is a third answer that almost nobody in the scraping world uses, despite it being a solved problem in infrastructure engineering since 1997: consistent hashing. Map accounts to exits with a deterministic function instead of a mutable table, and the account–exit pairing becomes stable, stateless, and self-healing — no coordination, no stale rows, and a mathematically bounded amount of churn when the pool changes size.
My core claim: the account–exit assignment problem is a data-sharding problem, and the correct solution is a hash ring, not a database. If you are running more than a handful of accounts through a rotating pool, this one structural change removes an entire category of state-management bugs from your fleet.
Why the mutable mapping table hurts
The database approach starts innocent: one row per account, holding the assigned exit. Then reality arrives.
Stale assignments. An exit gets benched (reputation degraded, provider rotates it), but the row still points at it. Now every worker either trusts the row and fails, or ignores it — at which point why does the row exist?
Write races. Two workers simultaneously decide account A needs a new exit; both write; one wins; the other's requests are already in flight through the loser's exit. The mapping table says one thing, reality did another, and your session-IP coherence guarantees (the whole point of the mapping) are violated precisely during recovery, when you can least afford it.
Unbounded churn on pool resize. When the pool shrinks or grows, a naive remap — "hash accounts to pool index" — reassigns almost every account. Since exit stability is exactly what keeps session cookies alive, every pool change becomes a fleet-wide re-login event.
Consistent hashing addresses the third problem by construction and dissolves the first two entirely: there is no mutable state to go stale, because the assignment is computed, and there is nothing to race to write.
The hash ring, adapted to proxy pools
Quick recap for those who have not implemented one: place exits on a circle at positions determined by hashing their identifiers. To assign an account, hash the account ID to a position on the same circle and walk clockwise to the first exit. When you add an exit, only the accounts whose positions fall in the arc it now owns get reassigned — on average 1/N of them, instead of nearly all. When you remove one, only its arc's accounts move. Virtual nodes (multiple hash positions per exit) smooth out the arc-length variance so the load balances.
Two adaptations make it fit proxy fleets rather than cache clusters:
Exits need weights. In caching, nodes are roughly equal. In proxy fleets, exits differ — a mobile exit tolerates different load than a residential one, and your own capacity measurements (see below) should decide how many virtual nodes each exit gets. An exit you have measured safe for four concurrent identities gets four vnodes; one measured safe for one gets one. The ring becomes a weighted scheduler for free.
Benching must be fast and stateless. An exit does not get deleted from the database when it is benched; it is simply excluded from the input set the ring is built from, and the ring is rebuilt. The rebuild is pure computation — milliseconds — and thanks to consistency, only the benched exit's accounts move.
A compact implementation:
import bisect
import hashlib
from dataclasses import dataclass
def _h(key: str) -> int:
return int.from_bytes(hashlib.sha256(key.encode()).digest()[:8], "big")
@dataclass
class Exit:
exit_id: str # stable identifier, e.g. provider session token
ip: str
capacity: int = 1 # measured safe identities (vnode count)
class ConsistentExitRing:
def __init__(self, exits: list[Exit], vnode_salt: str = "vnode"):
self.ring: list[int] = [] # sorted hash positions
self.owner: dict[int, Exit] = {} # position -> exit
self.exits: dict[str, Exit] = {}
for ex in exits:
self.add_exit(ex, rebuild=False, vnode_salt=vnode_salt)
self._sort()
def add_exit(self, ex: Exit, rebuild: bool = True,
vnode_salt: str = "vnode") -> None:
self.exits[ex.exit_id] = ex
for v in range(max(1, ex.capacity)):
pos = _h(f"{vnode_salt}:{ex.exit_id}:{v}")
self.owner[pos] = ex
if rebuild:
self._sort()
def bench_exit(self, exit_id: str) -> list[str]:
"""Remove an exit; return the account keys that will move."""
if exit_id not in self.exits:
return []
moved = [k for k in list(getattr(self, "_cache", {}))
if self.assign(k).exit_id != exit_id] # see note below
del self.exits[exit_id]
self.owner = {p: e for p, e in self.owner.items()
if e.exit_id != exit_id}
self._sort()
return moved
def _sort(self) -> None:
self.ring = sorted(self.owner.keys())
self._cache = {}
def assign(self, account_id: str) -> Exit:
"""Deterministic: same account + same ring -> same exit, forever."""
cached = self._cache.get(account_id)
if cached is not None:
return cached
if not self.ring:
raise RuntimeError("empty exit pool")
pos = _h(account_id)
idx = bisect.bisect_right(self.ring, pos) % len(self.ring)
ex = self.owner[self.ring[idx]]
self._cache[account_id] = ex
return ex
# usage
ring = ConsistentExitRing([
Exit("sess-a1b2", "93.184.10.7", capacity=3),
Exit("sess-c3d4", "198.51.100.22", capacity=1),
Exit("sess-e5f6", "203.0.113.9", capacity=2),
])
acct_exit = ring.assign("account-3f9") # same result on every worker
The property this buys you is worth stating plainly: every worker, at the same moment, with the same exit set, computes the same assignment for every account — without communicating. No database row, no cache invalidation, no leader election. The shared state is reduced to the set of live exits (and their capacities), which you are already distributing for health reasons.
The cold question: what about new pairings?
The obvious objection: determinism means that when an account is first assigned, it lands on an exit it has never used — a fresh pairing. Earlier I would tell you fresh pairings are the riskiest moment in the fleet lifecycle, and a ring happily manufactures them at a controlled rate. These two designs are complements, not rivals:
- The ring guarantees the pairing is stable once formed — the property that matters for cookie survival.
- A warm-up governor (rate ramp for the first N minutes of a pairing's life) absorbs the risk of the pairing's birth.
Together they produce the trajectory you actually want: each account–exit pair is born slowly, lives as long as the exit stays healthy, and dies in a planned migration — when an exit is benched, the ring moves only its arc of accounts, and each moved account starts a fresh warm-up on its newly computed exit. Bounded churn, absorbed at birth.
One caveat on exit identity: hash on a stable exit identifier (a sticky session token or a static IP), never on a volatile label. If your provider's sticky token changes meaning mid-window, your "deterministic" assignment silently degrades into random churn. This is also why you should verify — as part of exit health checks — that the token still resolves to the same exit IP the ring believes it owns.
Capacity data closes the loop
The capacity field on each exit is where this design connects to measurement. You should not guess vnode counts. Track, per exit, the rolling 24h identity load and failure verdicts (the same telemetry any serious fleet already logs), and set capacity = measured_safe_identities, clamped to [1, 6]. Rebuild the ring when capacities drift materially — and because rebuilds are cheap and consistent, you can do this hourly without ceremony. An exit whose measured capacity fell to zero is benched; only its arc of accounts migrates; the fleet absorbs the change without a re-login storm.
The result is a fleet where assignment, load balancing, and failure handling are all the same mechanism — one ring, fed by one health signal — instead of three cooperating subsystems held together by hope and a database table.
The bigger point
The scraping stack has quietly inherited most of its hardest problems from distributed systems: state distribution, assignment stability, failure domains, bounded churn. It makes sense to also inherit the answers. Consistent hashing spent twenty-five years solving "which node owns this key" for caches and databases; "which exit owns this account" is the same question with higher stakes and flakier hardware. Reach for the old tools — they are old because they work.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)