Ask a scraping team how healthy their proxy pool is and you will get a number — "94% success this week." That number is worse than useless for routing decisions, because it averages over the two dimensions that actually matter. The same exit IP is a perfectly good citizen for a public news site and radioactive for a hardened checkout flow. The same origin will happily serve one exit from a residential ASN and 403 another from a range it burned last month. Health is not a property of the exit, and it is not a property of the origin. It is a property of the pair.
My argument: your pool health data belongs in a matrix — exits on one axis, origins on the other — and almost every routing mistake I debug in other people's pipelines traces back to collapsing that matrix into one number per exit or one number per origin. Building the matrix is an afternoon of work. Routing from it is trivial. And it changes decisions immediately, because it makes the two classic failures visible: burned pairs that keep receiving traffic, and healthy pairs nobody uses.
Two failures a single number cannot see
The burned pair. Exit 203.0.113.47 has a 91% success rate overall, so your round-robin keeps selecting it. But its last 200 requests to checkout.example all returned 403. Averaged into a week of 20,000 requests, 200 failures vanish. Meanwhile your scraper re-dials that pair, fails, retries through another exit, and burns time and money on a pairing that a five-second lookup would have forbidden.
The wasted healthy pair. Somewhere in your pool there is almost certainly an exit with a long unbroken success streak against your hardest origin — but because your rotation is uniform or your stickiness is configured per task and not per pair, that exit spends most of its capacity on origins that would have accepted anything. Premium affinity, allocated by accident instead of by evidence.
Both are allocation errors, and both are invisible without the pair-level view.
The matrix, with forgetting
Here is a pair-level health tracker in Python. The design choices that matter: an EWMA decay on outcomes (so a pair can recover, and so last Tuesday's block does not permanently poison today's routing), an explicit exploration floor (so a pair that failed early still gets occasionally re-tested), and a routing query that respects both the matrix and a fairness constraint so traffic does not stampede onto one golden exit:
import time
import random
from collections import defaultdict
from dataclasses import dataclass, field
CHALLENGE = {403, 429}
@dataclass
class PairStats:
ewma: float = 1.0 # 1 = last outcome was success, decays toward 0 on failure
alpha: float = 0.15
trials: int = 0
last_used: float = 0.0
cooldown_until: float = 0.0
def record(self, ok: bool) -> None:
self.trials += 1
self.ewma = self.alpha * (1.0 if ok else 0.0) + (1 - self.alpha) * self.ewma
if not ok:
self.cooldown_until = time.monotonic() + 300 # 5-min pair cooldown
class PairHealthMatrix:
def __init__(self, min_ewma: float = 0.70, explore_p: float = 0.05):
self.matrix: dict[tuple[str, str], PairStats] = defaultdict(PairStats)
self.min_ewma = min_ewma
self.explore_p = explore_p
def _key(self, exit_id: str, origin: str) -> tuple[str, str]:
return (exit_id, origin)
def eligible(self, exit_id: str, origin: str) -> bool:
s = self.matrix.get(self._key(exit_id, origin))
if s is None:
return True # unknown pair: eligible, worth testing
if time.monotonic() < s.cooldown_until:
return False
# low-traffic pairs get the benefit of the doubt (exploration floor)
if s.trials < 8:
return True
return s.ewma >= self.min_ewma
def record(self, exit_id: str, origin: str, status: int) -> None:
self.matrix[self._key(exit_id, origin)].record(status not in CHALLENGE and status < 500)
def rank_exits_for(self, origin: str, exits: list[str]) -> list[str]:
"""Order exits best-first for one origin, with an exploration twist."""
if random.random() < self.explore_p:
return random.sample(exits, len(exits)) # occasional full exploration
def score(e: str) -> float:
s = self.matrix.get(self._key(e, origin))
if s is None or s.trials == 0:
return 0.85 # optimistic prior for untested pairs
return s.ewma - 0.1 * min(s.trials / 50, 1.0) # mild load-spreading penalty
return sorted(exits, key=score, reverse=True)
# --- integration sketch ---
# exits are identified by whatever your gateway exposes: sticky session ids,
# a session token in the username, or the egress IP returned by a check endpoint.
matrix = PairHealthMatrix()
async def fetch(aiohttp_session, url: str, origin: str, candidate_exits: list[str]):
ranked = matrix.rank_exits_for(origin, candidate_exits)
for exit_id in ranked:
if not matrix.eligible(exit_id, origin):
continue
proxy = gateway_for(exit_id) # e.g. http://user-session-<exit_id>:pass@gate:7200
try:
async with aiohttp_session.get(url, proxy=proxy, timeout=aiohttp.ClientTimeout(total=25)) as r:
matrix.record(exit_id, origin, r.status)
if r.status < 400:
return await r.text()
except Exception:
matrix.record(exit_id, origin, 599) # network-layer failure counts against the pair
raise RuntimeError(f"no eligible exit for {origin}")
Notice the failure attribution in the exception handler: a proxy connection error or timeout is recorded against the pair, not skipped. Most pipelines only record HTTP status codes, which means their matrix is blind to the entire class of failures where the exit never even reached the origin. Log-layer errors still tell you something about the exit, even when they tell you nothing about the origin's opinion of it.
Reading the matrix: what it tells you that averages don't
Once the matrix has a few days of traffic, three queries become routine and each one changes an operational decision.
Per-origin exit entropy. How many exits actually carry traffic for your hardest origin? If one origin is served by two exits out of a hundred, you have a hidden single point of failure with a blast radius no pool-wide metric will ever show you. Spread deliberately: warm more pairs for that origin before you need them.
Burned-cell clusters. Sort the matrix by exit and look for rows with many failing cells concentrated in one ASN or one /24. That pattern usually means the origin is blocking by range, not by IP — and no amount of exit-level rotation inside the range will fix it. The matrix is the only view where range-level blocking is visible as geometry.
Recovery tracking. Because of the EWMA decay, a cell that was red last week and is pale green today is a pair that un-burned. Origins update blocklists, providers recycle hosts, and a recovering pair is free capacity. Routing code that kept a static blacklist would never revisit it; the matrix does, automatically.
The caveats
The matrix grows with exits × origins, and if you track egress IPs from a large rotating pool it can get large — hundreds of thousands of cells is fine in memory but noisy to reason about. The fix is aggregation on the exit axis: track health per exit class (ASN, gateway port, sticky-session cohort) rather than per individual IP for rotating traffic, and reserve per-IP granularity for static or sticky exits where identity persists. The matrix concept survives; only the row labels get coarser.
The second caveat is cold start: a new origin starts with an all-unknown column, and the optimistic prior sends it whatever exits are idle, which is fine. A new pool starts with all-unknown rows, which is also fine. The failure mode to avoid is migrating the matrix between environments — pair health learned against one origin's staging domain does not transfer, and copying stale matrices has cost people days of confused debugging. Start empty, let exploration fill it.
The pool-wide success number still belongs on the weekly dashboard — it is a decent regression signal. But routing decisions should be made against the matrix, because the matrix is where the truth has enough shape to act on. One afternoon of data structure work, and the two most expensive failure modes in proxy routing stop being invisible.
Top comments (0)