One Exit, Fifty Workers: Stopping the Coordinated Retries That Turn One 429 Into Fifty

A 429 Too Many Requests is a polite message from an origin. It says "you're going too fast, back off, try again in a bit." Your scraper's job is to honor that message. Which is exactly why the most common way teams get permanently banned is not the first 429 — it's the fiftieth one, three seconds later, when fifty workers all obey the retry instructions at the same instant and through the same exit IP.

The core idea of this article: when many workers share one proxy exit, the origin rate-limits the exit, not the worker, so naive per-worker backoff produces a synchronized retry stampede that looks far more abusive than the original request wave — you need an exit-scoped coordination barrier plus decorrelated jitter to break the lockstep. This is a small, concrete piece of engineering most async scraping frameworks get wrong, and it's worth understanding precisely.

The setup: many workers, one shared exit

Modern Python scrapers fan out with asyncio. You might run one process with 50 concurrent tasks, or 10 processes each with 5, all pointed at the same pool of, say, 200 residential exits. Under a sticky-session policy you deliberately route many requests for the same logical session through the same exit — that's the whole point of stickiness. The problem is that a sticky exit is now a shared fate resource: a single 429 against https://origin.example/search on exit E-017 is a rate-limit verdict for the exit, and every one of your workers currently riding E-017 is subject to it.

Now imagine what happens with textbook per-worker backoff:

# The naive pattern, worker by worker.
async def fetch(session, url, exit_url):
    for attempt in range(8):
        resp = await session.get(url, proxy=exit_url)
        if resp.status == 429:
            retry_after = int(resp.headers.get("Retry-After", "1"))
            await asyncio.sleep(retry_after)   # <-- everyone sleeps the SAME seconds
            continue
        return resp
    raise Exception("gave up")

Fifty workers, one exit, one 429 with Retry-After: 3. Every worker sleeps 3 seconds, wakes up at the same moment, and fires 50 requests at the origin in one synchronized burst through the same exit. The origin sees a burst that is, from its perspective, more aggressive than the first wave, because it's tightly clustered in time. It rate-limits harder. The Retry-After grows. The stampede repeats, with worse timing alignment each round. This is a classic thundering herd, and a shared proxy exit is a thundering-herd amplifier: you have hidden 50 clients behind one identity.

Decorrelated jitter: the first fix, and why it's not enough

The standard remedy is exponential backoff with full jitter (the AWS Architecture Blog formulation): instead of sleeping a deterministic delay, each worker sleeps a random duration drawn uniformly from [0, min(cap, base * 2^attempt)]. This spreads the retries over the backoff window instead of clustering them at its start.

import random

async def fetch_with_jitter(session, url, exit_url, base=1.0, cap=60.0):
    for attempt in range(10):
        resp = await session.get(url, proxy=exit_url)
        if resp.status != 429:
            return resp
        exp = min(cap, base * (2 ** attempt))
        delay = random.uniform(0, exp)
        await asyncio.sleep(delay)
    raise Exception("gave up")

Better — but the distribution of wake-up times is still not what the origin cares about. The origin cares about the instantaneous request rate through exit E-017. With uniform jitter, the probability that several of your fifty workers draw a short delay and hit within the same 100ms bucket is high, and one unlucky co-occurrence is enough to trigger the next 429. Jitter alone decorrelates retries only in expectation, not at the granularity that a per-IP rate limiter measures. What you need is coordination.

The exit-scoped token bucket

The fix is to make the shared resource explicit in your code. If the origin limits per exit, your backoff should be owned per exit, not per worker. Concretely: a token bucket keyed by exit_url (not by origin URL, not by worker task ID) that every worker must acquire a token from before it sends — and that is drained the moment any worker observes a 429 on that exit.

import asyncio
import time
from dataclasses import dataclass, field

class ExitThrottle:
    """One shared throttle per exit across all workers on the machine.

    - Normal state: workers may send at a steady rate up to `rate`.
    - On 429: the exit's `open_until` is set to now + retry_after; every
      worker waiting for a token sleeps until exactly then, so the retry is
      serialized to the *earliest legally allowed* time, not staggered
      against the same origin limit that has just been hit.
    - A post-429 slow ramp: after the cooldown we halve the rate and creep
      back, so fifty workers don't resume at full speed at once.
    """
    def __init__(self, exit_url: str, rate: float = 5.0):
        self.exit_url = exit_url
        self.base_rate = rate
        self.rate = rate
        self._tokens = rate
        self._last_refill = time.monotonic()
        self._open_until = 0.0
        self._lock = asyncio.Lock()

    async def acquire(self):
        # Wait out any global cooldown first (this is the anti-stampede core).
        while True:
            async with self._lock:
                now = time.monotonic()
                if now >= self._open_until:
                    # token refill
                    elapsed = now - self._last_refill
                    self._tokens = min(self.base_rate, self._tokens + elapsed * self.rate)
                    self._last_refill = now
                    if self._tokens >= 1:
                        self._tokens -= 1
                        return  # allowed to send
                target = self._open_until
            await asyncio.sleep(max(0.05, target - time.monotonic()))

    async def penalize(self, retry_after: float):
        async with self._lock:
            now = time.monotonic()
            # The whole exit goes dark until the retry deadline.
            self._open_until = max(self._open_until, now + retry_after)
            # ...and resumes at half speed, then creeps back up.
            self.rate = max(0.5, self.rate * 0.5)

    async def reward(self):
        """Successful, un-throttled requests let the rate recover."""
        async with self._lock:
            self.rate = min(self.base_rate, self.rate * 1.1)

The essential difference from the naive code: the cooldown state lives on the exit, and the 50 workers observe the same state object. Only the first worker to actually receive the 429 calls penalize; the other 49 workers who were about to send are blocked at acquire() until the exit re-opens. Instead of 50 concurrent violations of the same limit, you get a single 429 and a serialized, rate-bounded recovery.

The caller looks like this:

# One throttle per exit, created and shared across all worker tasks.
THROTTLES: dict[str, ExitThrottle] = {}

def get_throttle(exit_url: str) -> ExitThrottle:
    t = THROTTLES.get(exit_url)
    if t is None:
        t = THROTTLES[exit_url] = ExitThrottle(exit_url)
    return t

async def fetch(session, url, exit_url):
    throttle = get_throttle(exit_url)
    for _ in range(20):
        await throttle.acquire()
        resp = await session.get(url, proxy=exit_url)
        if resp.status == 429:
            ra = parse_retry_after(resp.headers.get("Retry-After"))
            await throttle.penalize(ra)
            continue
        await throttle.reward()
        return resp
    raise Exception("gave up")

Parsing Retry-After properly (it's rarely an integer)

One detail that quietly breaks a lot of 429 handlers: RFC 9110 defines Retry-After as either a non-negative delta-seconds integer or an HTTP-date. Production APIs like GitHub, Stripe, and Cloudflare-fronted origins sometimes use the HTTP-date form, and naive int() raises. You also need to separate 429 from 503, which uses the same header but implies capacity problems you should treat differently — a 503 with a long Retry-After is often better served by switching exits than by waiting on one you've now penalized.

import email.utils, time

def parse_retry_after(value: str | None, default: float = 1.0) -> float:
    if value is None:
        return default
    try:
        return max(0.0, float(value))            # delta-seconds
    except ValueError:
        pass
    dt = email.utils.parsedate_to_datetime(value)  # HTTP-date
    if dt is not None:
        return max(0.0, dt.timestamp() - time.time())
    return default

The distributed problem: when "per exit" spans processes

If all fifty workers are tasks in one process, the shared dict above is enough — they're literally sharing a Python object. But if you're running fifty workers as separate processes or across five machines pointing at the same exit pool, the throttle is no longer shared, and you've reinvented the stampede at a higher scale. You need a distributed exit state.

The pragmatic implementation is a tiny shared store — Redis or DynamoDB — keyed by exit_url, holding open_until and a smoothed rate. Each worker, before sending, does a short GET; on a 429 it does an atomic SET open_until IF NOTALREADYLATER (a Lua compare-and-set so the earliest deadline wins, since multiple workers can observe 429 simultaneously). Because the cooldown is now visible across processes, the whole fleet sleeps as one unit. This is the same principle as the in-process throttle, only the state moved to a shared cache — which, not incidentally, is exactly where per-exit state should live if you already run a proxy pool with a gateway.

The metric that proves it's working

The signal to watch is not "did 429s go to zero" — you'll still occasionally take one. The signal is requests-per-exit-per-second distribution. Plot it with a shared exit in front of many workers and the naive backoff looks like a sawtooth: a burst, a cliff, a synchronized re-burst. The coordinated throttle looks like a smooth ceiling: one gentle bump at the first 429, then a controlled re-ramp. If your retry storm looks like a heartbeat monitor, you are still in lockstep. The moment it looks like a ramp, you've broken the herd — and the fifty requests that used to arrive as one violent thump now arrive as fifty separate, quiet, individually successful requests.

Backoff is often taught as a per-request concern: catch the 429, sleep, retry. But a shared proxy exit turns that into a distributed-systems problem. Respect the granularity the origin uses — the IP leaving your gateway, not the task waiting in your loop — and the coordinated retry storm stops being your fault.

---

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele