429s Belong to the Exit, Not the Worker: Per-IP Token Buckets for Rotating Proxy Pools

If you run a distributed scraper, you've probably built a rate limiter at some point. And if you run it behind a rotating proxy pool, I'd bet money the limiter is keyed to the wrong thing. Most implementations limit per worker, or globally per domain — both reasonable instincts, both wrong for the physics of the problem.

Here's the core observation: a 429 is not the server talking to your worker, and it's not the server talking to your job. It's the server talking to the exit IP that made the request. Rate limits on the server side are enforced against what the server can identify — and behind a proxy pool, that's the exit address (plus, increasingly, token buckets keyed on fingerprint and account, but the IP bucket is the one that trips first and loudest). Your limiter's key space should mirror the enforcement key space, or you'll be wrong in both directions at once.

Why the two common designs both fail

Global per-domain limiter, shared across all workers. Say your target allows roughly 30 requests/minute per IP, and you set a global 30/min limit while running 20 exits. You're using 1/20th of the capacity you're paying for. The crawl takes 20x longer than it should, and someone asks why the "expensive proxy plan" didn't make anything faster.

Per-worker limiter, workers drawing from a shared pool. Now each of 8 workers thinks it owns 30/min, but your pool rotation means they're landing on the same 20 exits. Effective per-exit load is somewhere between 30 and 240/min depending on rotation luck — and the 429s arrive exactly when rotation happens to collide. The failure looks random, which is what makes it maddening to debug. I've watched a team rewrite their entire parsing stack over this. It was never the parser.

The correct model: a token bucket per exit IP, replenished by time and corrected by the server's actual responses. The 429 with a Retry-After header is the server telling you the exact capacity of that exit's bucket — treat it as a calibration message, not an error.

The per-exit token bucket

import asyncio
import time
from dataclasses import dataclass, field

@dataclass
class ExitBucket:
    exit_id: str
    capacity: float = 30.0        # requests per window; conservative start
    tokens: float = 30.0
    refill_per_sec: float = 0.5   # capacity / window_seconds
    last_refill: float = field(default_factory=time.monotonic)
    penalty_until: float = 0.0    # Retry-After cooldown
    consecutive_429: int = 0

class ExitRateLimiter:
    """
    Token buckets keyed by exit identity. The exit_id should be whatever
    pins a unique upstream address: a sticky session ID, or the resolved
    exit IP if your provider exposes it back to you.
    """

    def __init__(self, default_capacity=30.0, window_seconds=60.0):
        self.buckets: dict[str, ExitBucket] = {}
        self.default_capacity = default_capacity
        self.refill_rate = default_capacity / window_seconds

    def _bucket(self, exit_id: str) -> ExitBucket:
        if exit_id not in self.buckets:
            self.buckets[exit_id] = ExitBucket(
                exit_id=exit_id,
                capacity=self.default_capacity,
                tokens=self.default_capacity,
                refill_per_sec=self.refill_rate,
            )
        return self.buckets[exit_id]

    async def acquire(self, exit_id: str, timeout: float = 60.0) -> bool:
        """Wait until the exit's bucket has a token. Returns False on
        timeout so the caller can re-route instead of blocking forever."""
        b = self._bucket(exit_id)
        deadline = time.monotonic() + timeout
        while True:
            now = time.monotonic()
            # refill
            elapsed = now - b.last_refill
            b.tokens = min(b.capacity, b.tokens + elapsed * b.refill_per_sec)
            b.last_refill = now
            # honor server-declared cooldown
            if now < b.penalty_until:
                wait = min(b.penalty_until - now, max(0.0, deadline - now))
                if wait <= 0:
                    return False
                await asyncio.sleep(min(wait, 1.0))
                continue
            if b.tokens >= 1.0:
                b.tokens -= 1.0
                return True
            if now >= deadline:
                return False
            await asyncio.sleep(min((1.0 - b.tokens) / b.refill_rate, 1.0))

    def report_success(self, exit_id: str):
        b = self._bucket(exit_id)
        b.consecutive_429 = 0

    def report_429(self, exit_id: str, retry_after: float | None):
        """A 429 is a capacity broadcast for THIS exit. Downgrade the
        bucket and honor Retry-After as a hard cooldown."""
        b = self._bucket(exit_id)
        b.consecutive_429 += 1
        if retry_after is not None:
            b.penalty_until = time.monotonic() + retry_after
        # Each 429 halves learned capacity, floor at 2 — the server is
        # telling us our assumed capacity is wrong, not that it's zero.
        b.capacity = max(2.0, b.capacity * 0.5)
        b.tokens = min(b.tokens, b.capacity)
        b.refill_per_sec = b.capacity / 60.0

And the crawler loop that ties it together — note that the exit is chosen before the limiter is consulted, because the bucket only means something once you know which address the request will come from:

import aiohttp
import random

async def fetch(session, url, exit_id, proxy_url, limiter):
    if not await limiter.acquire(exit_id):
        return {"status": "reroute", "exit": exit_id}   # bucket exhausted

    try:
        async with session.get(url, proxy=proxy_url, timeout=aiohttp.ClientTimeout(total=30)) as r:
            if r.status == 429:
                ra = r.headers.get("Retry-After")
                limiter.report_429(exit_id, float(ra) if ra and ra.isdigit() else None)
                return {"status": 429, "exit": exit_id}
            limiter.report_success(exit_id)
            return {"status": r.status, "body": await r.text()}
    except aiohttp.ClientError as e:
        return {"status": "error", "error": str(e), "exit": exit_id}

async def crawl(urls, exits, proxy_template):
    limiter = ExitRateLimiter(default_capacity=30.0, window_seconds=60.0)
    async with aiohttp.ClientSession() as session:
        for url in urls:
            exit_id = random.choice(exits)     # or session-affinity logic
            proxy = proxy_template.format(session=exit_id)
            result = await fetch(session, url, exit_id, proxy, limiter)
            if result["status"] == "reroute":
                # Bucket dry: pick a different exit and retry once,
                # rather than sleeping — another exit has tokens now.
                alt = random.choice([e for e in exits if e != exit_id])
                result = await fetch(session, url, alt,
                                     proxy_template.format(session=alt),
                                     limiter)
            print(result["status"], url)

if __name__ == "__main__":
    exits = [f"s{i:03d}" for i in range(10)]
    template = "http://user-session-{session}:[e-mail]:8080"
    urls = [f"https://httpbin.org/status/200?i={i}" for i in range(60)]
    asyncio.run(crawl(urls, exits, template))

The three details that make it production-grade

Key by real exit identity, not by session string, when you can. Session strings are a good proxy for exits within their sticky window, but if your provider rotates the underlying IP behind a long-lived session ID, your bucket is guarding an address that no longer exists while the new address runs ungoverned. Resolve your exit IP (many providers return it in a response header; otherwise hit a what-is-my-IP endpoint once per session) and key buckets on that.

Halve capacity, don't zero it. The capacity * 0.5 on 429 matters more than it looks. The server isn't saying "never come back" — it's saying "slower than that." A bucket that floors at 2/min keeps a burned exit crawling instead of dead, which is usually what you want for coverage tasks. For hard targets I let it floor at 1/min and rely on the cooldown for the rest.

Recovery has to go up as well as down. Buckets that only ever shrink converge to uselessness. On a sustained run of successes — say 100 consecutive — grow capacity back by 25% toward the default. Otherwise one bad hour permanently degrades an exit that the target's sliding window has long forgotten.

What this buys you

Once the limiter's key space matches the enforcement key space, a few things fall out for free. Your pool utilization stops being luck: every exit contributes up to its learned capacity, and reroute on bucket exhaustion means an exhausted exit costs you a millisecond, not a 429. Your 429 rate becomes a pool metric — if exits are getting 429s at capacity, your learned capacities are too high; if never, you're being too conservative and leaving throughput on the table. And when a specific exit starts drawing 429s at half its usual capacity, that's an early reputation signal — the exit's individual bucket on the server side got smaller, which usually precedes outright challenges.

The deeper point is about modeling. Rate limiting is not politeness theater bolted onto a crawler; it's reverse-engineering the server's enforcement data structure. When the key space of your limiter matches the key space of their limiter, the 429 stops being an error and becomes what it actually is: the server publishing its configuration to you, one response at a time. Read it where it's addressed — at the exit.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele