Every scraper team eventually builds a pool-health score. You log success rates, latency, and 429 counts per exit, roll them into a number, and route away from anything that dips. I have shipped that design and watched it fail in production, because a score is a snapshot and routing is a policy. The number tells you an exit looked bad at time T; it does not tell your workers what to do at T+1. What you actually need is a small state machine bolted onto every proxy exit, and the state that makes it work is the one most scorecards skip: half-open.
Treat each exit like a fuse. On a run of failures, timeouts, or 429s, trip it OPEN and stop routing traffic. Then, on a backoff timer, let a couple of cheap HALF_OPEN probes through to see whether the IP has cooled off. Close the breaker only when a probe succeeds. This beats both failure modes I see constantly: keep hammering a bad exit until the origin hardens on us, or quarantine a flaky IP forever and waste the capacity we paid for. A rate-capped residential exit often becomes usable again after minutes or hours. The circuit breaker is how you reheat it safely instead of burning it or freezing it.
Transient versus permanent is the whole game
A breaker is only as good as its definition of "failure." If you treat every non-200 as a strike, one origin-side 500 trips an innocent exit. If you treat everything as transient, you keep re-closing a dead proxy. Split the signals explicitly.
Transient (counts toward tripping, and is reheatable): connection timeouts, ConnectionError, dropped tunnels, HTTP 429 with a Retry-After, and soft-block challenge pages. These mean "this exit is temporarily out of favor or overloaded," which is exactly what half-open probing resolves.
Permanent (trip hard and park it far out): proxy authentication failures (HTTP 407 from the proxy, not the origin), connection refused to the gateway, and a CONNECT error that never reached the target. These mean "the exit is broken right now," so a 30-second reheat probe is wasted money. Park it for a long window and let a separate validator look.
The nuance that trips people up: a 429 is transient for the exit but it is often origin-specific. Which brings us to scope.
Breaker scope: per-exit is usually wrong
The classic circuit breaker guards one downstream service. In a proxy pool, your "downstream" is a pair: an exit plus an origin. An exit that gets you 429s from api.shop.example may be perfectly welcome on docs.news.example. If your breaker is keyed only on the exit, one angry origin poisons that IP across your whole workload and you throw away a healthy exit.
So I key breakers on (exit, origin). Yes, it is more objects; they are tiny dicts under a lock, not threads. The memory cost is trivial next to the capacity you stop wasting. Reserve a coarse per-exit breaker for genuinely global signals: the gateway refusing auth, or the exit failing DNS. For those, marking the exit bad everywhere is exactly right. In the code below I scope per (exit, origin) and note where a global trip belongs.
The CircuitBreaker
import threading
import time
import random
from enum import Enum
from typing import Optional, Dict, Any
class State(str, Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
class CircuitBreaker:
"""Guards one (exit, origin) pair. Thread-safe."""
def __init__(self, name: str, failure_threshold: int = 3,
base_timeout: float = 20.0, max_timeout: float = 900.0,
half_open_max_calls: int = 1):
self.name = name
self.failure_threshold = failure_threshold
self.base_timeout = base_timeout # first reheat window
self.max_timeout = max_timeout # ceiling for backoff / parking
self.half_open_max_calls = half_open_max_calls
self._lock = threading.Lock()
self._state = State.CLOSED
self._failures = 0
self._trip_count = 0 # grows the reheat backoff
self._next_probe_at = 0.0
self._half_open_inflight = 0
@property
def state(self) -> State:
with self._lock:
return self._state
def allow(self, now: Optional[float] = None) -> bool:
"""May this exit receive a request right now?"""
now = time.monotonic() if now is None else now
with self._lock:
if self._state is State.CLOSED:
return True
if self._state is State.OPEN:
if now >= self._next_probe_at:
self._state = State.HALF_OPEN
self._half_open_inflight = 0
else:
return False
# HALF_OPEN: admit a bounded number of probes.
if self._half_open_inflight >= self.half_open_max_calls:
return False
self._half_open_inflight += 1
return True
def record_success(self) -> None:
with self._lock:
if self._state is State.HALF_OPEN:
# The IP reheated cleanly: re-close and reset backoff.
self._state = State.CLOSED
self._trip_count = 0
self._failures = 0
self._half_open_inflight = max(0, self._half_open_inflight - 1)
def record_failure(self, permanent: bool = False) -> None:
now = time.monotonic()
with self._lock:
self._half_open_inflight = max(0, self._half_open_inflight - 1)
if self._state is State.HALF_OPEN or permanent:
self._open(now, permanent=permanent)
return
self._failures += 1
if self._failures >= self.failure_threshold:
self._open(now)
def _open(self, now: float, permanent: bool = False) -> None:
self._state = State.OPEN
self._failures = self.failure_threshold
if permanent:
# Dead proxy / auth failure: park it; do not waste probe budget.
self._next_probe_at = now + self.max_timeout
return
self._trip_count += 1
window = min(self.base_timeout * (2 ** (self._trip_count - 1)),
self.max_timeout)
window += random.uniform(0, self.base_timeout * 0.25) # jitter
self._next_probe_at = now + window
def seconds_until_probe(self) -> float:
now = time.monotonic()
with self._lock:
if self._state is not State.OPEN:
return 0.0
return max(0.0, self._next_probe_at - now)
def snapshot(self) -> Dict[str, Any]:
now = time.monotonic()
with self._lock:
return {
"name": self.name,
"state": self._state.value,
"failures": self._failures,
"trips": self._trip_count,
"secs_to_probe": max(0.0, self._next_probe_at - now),
}
Read the _open method again; that is the money function. Exponential backoff means the second time an exit trips, you wait roughly twice as long before trusting a probe. If a given IP keeps flipping OPEN, you probe it less often, not more, which is the opposite of what a naive retry loop does. The jitter is load-bearing: without it, a synchronized fleet of workers all schedules their first probe for the same instant.
The pool router
import requests
class NoHealthyExit(RuntimeError):
pass
class PoolRouter:
"""Holds a breaker per (exit, origin); picks ready exits; reheats on a timer."""
def __init__(self, exits: Dict[str, dict], origins):
# exits: {"exit-12": {"http": "...", "https": "..."}, ...}
self._exits = exits
self._breakers: Dict[tuple, CircuitBreaker] = {}
for name in exits:
for origin in origins:
self._breakers[(name, origin)] = CircuitBreaker(
name=f"{name}->{origin}",
failure_threshold=3,
base_timeout=20.0,
max_timeout=900.0,
)
self._rng = random.Random()
def next_for(self, origin: str):
"""Return (exit_name, proxies, breaker) or raise if all are OPEN."""
ready, waits = [], []
for (name, o), br in self._breakers.items():
if o != origin:
continue
if br.allow(): # allow() also promotes OPEN -> HALF_OPEN
ready.append((name, self._exits[name], br))
else:
waits.append(br.seconds_until_probe())
if not ready:
soonest = min(waits, default=0.0)
raise NoHealthyExit(
f"every exit for {origin} is OPEN; soonest reheat in {soonest:.0f}s")
return self._rng.choice(ready)
def fetch(self, url: str, origin: str, timeout: float = 10.0):
name, proxies, br = self.next_for(origin)
try:
resp = requests.get(url, proxies=proxies, timeout=timeout)
except (requests.Timeout, requests.ConnectionError):
br.record_failure(permanent=False) # transient, reheatable
raise
code = resp.status_code
if code == 429:
br.record_failure(permanent=False) # rate cap on THIS origin
raise RuntimeError(f"rate limited via {name}")
if code in (500, 502, 503, 504):
br.record_failure(permanent=False) # upstream noise; soften later
raise RuntimeError(f"upstream {code} via {name}")
if code == 407:
br.record_failure(permanent=True) # broken auth: park everywhere
raise RuntimeError(f"proxy rejected {name}")
br.record_success()
return resp
The random choice among ready exits matters. If you always take the first healthy exit, the moment it trips you dump every worker onto the second one, which then trips from the sudden load. Spread the traffic so no single breaker's failure count is dominated by your own herd.
Killing the thundering herd
Two mechanisms stop every worker from stampeding a just-opened exit. First, half_open_max_calls=1 plus the allow() lock means only one worker can hold the half-open slot: the second caller sees _half_open_inflight >= 1 and is refused, so it falls through to another exit instead of queueing on the probe. Second, the jittered exponential backoff guarantees different (exit, origin) pairs become probeable at different wall-clock times, so the fleet recovers in a stagger rather than a synchronized pulse that re-triggers the very rate limiter you are waiting out.
A nice side effect of allow() living inside the picker is that every next_for call is itself a chance to promote OPEN to HALF_OPEN. A busy router naturally reheats with real traffic; a background loop only needs to reheat exits nobody asked for.
A dedicated reheat loop
def reheat_loop(router: PoolRouter, probe_urls, stop, interval: float = 2.0):
"""Probe exits that idle workers are not exercising."""
while not stop.wait(interval):
for (name, origin), br in router._breakers.items():
if br.state is not State.OPEN or br.seconds_until_probe() > 0:
continue
if not br.allow(): # another worker grabbed the slot
continue
try:
r = requests.get(probe_urls[origin],
proxies=router._exits[name], timeout=8)
# 200/204 prove the tunnel works; a 404 still means the exit
# forwarded cleanly, so treat it as reachable.
if r.status_code in (200, 204, 404):
br.record_success()
elif r.status_code == 429:
br.record_failure(permanent=False)
else:
br.record_failure(permanent=(r.status_code == 407))
except Exception:
br.record_failure(permanent=False)
Point probes at a cheap, fast-flushing endpoint you control or a HEAD-able public page. A 404 from the target is a success for the breaker because it proves the exit carried the request end to end; you are testing the pipe, not the payload.
Observability: export the state, not just the count
A breaker's value is operational, so export its state as a gauge every scrape:
def emit_metrics(router: PoolRouter, push):
code = {"closed": 0, "open": 1, "half_open": 2}
for (name, origin), br in router._breakers.items():
s = br.snapshot()
push("proxy_breaker_state", code[s["state"]], {"exit": name, "origin": origin})
push("proxy_breaker_trips", s["trips"], {"exit": name, "origin": origin})
push("proxy_breaker_secs_to_probe", s["secs_to_probe"], {"exit": name, "origin": origin})
Now alerts come for free. A dashboard of proxy_breaker_state shows which exits are cooling; a rising trips counter on one exit tells you to replace it rather than keep reheating; and when next_for raises NoHealthyExit across a whole origin, that is pool exhaustion worth paging on. This is strictly more actionable than a blended health score, which hides why a route was refused.
Tuning, and where this breaks
Start with failure_threshold=3 and base_timeout=20s; for origins with 15-minute rate windows, raise base_timeout so you stop probing during the ban and let max_timeout be the parking lot for dead exits. The failure mode to watch is a flapping exit that keeps re-closing and re-tripping; if trips climbs while state oscillates, your threshold is too low or the probe endpoint is too expensive for that origin. A breaker is not a substitute for polite per-exit rate limiting either: it reacts after damage, so pair it with a token bucket that discourages the damage first. Still, for reusing cooled-off capacity safely, the half-open probe is the most honest tool I know. It gives every proxy exit a memory of how it failed and a disciplined path back to work.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)