Retry-After Is a Routing Signal: Reweighting Your Proxy Pool with Every 429 You Receive

A 429 response has two jobs. The first is obvious: "slow down." The second is more valuable and almost universally ignored: "this specific exit is now at capacity; use a different one." Most scrapers treat 429 as a global stop signal — they sleep, then retry on the same pool with the same distribution. That's wasteful. A 429 carries a routing signal, and your proxy selection logic should consume it.

This article builds a lightweight adaptive router in Python that uses every 429 and its Retry-After header to dynamically reweight your residential proxy pool. No external infrastructure. No Redis. Just an in-process scoring model that makes your pool smarter with every rate-limit response.

The Problem with Naive Retry-After Handling

Here's what most Python scrapers do when they hit a 429:

# The common (wasteful) pattern
resp = requests.get(url, proxies={"https": proxy})
if resp.status_code == 429:
    wait = int(resp.headers.get("Retry-After", 60))
    time.sleep(wait)
    resp = requests.get(url, proxies={"https": proxy})  # Same proxy!

This has three problems:

1. Wrong scope. Retry-After is addressed to this IP-to-origin pair, not to you globally. Another exit might have full quota right now.
2. Wrong economics. If you're paying per-GB on residential proxy traffic, sleeping on a blocked exit wastes connection time. You're paying for idle.
3. Wrong learning. The system doesn't remember that this exit just hit a ceiling. Next run, the same exit gets selected again at the same weight.

The Design: A Token Bucket Per Exit, Aggregated by Origin

The mental model: each (exit_ip, origin_domain) pair has a private token bucket. Every 200 response refills tokens (slowly). Every 429 drains them (violently). The router picks the exit with the most tokens for the next request to that origin.

import time
import threading
from dataclasses import dataclass, field
from collections import defaultdict
from typing import Optional

@dataclass
class ExitQuota:
    """Token bucket for one (proxy_exit, origin) pair."""
    capacity: float = 10.0       # Max burst before hitting ceiling
    refill_rate: float = 0.1     # Tokens gained per second (recovery speed)
    tokens: float = 10.0         # Current available tokens
    last_update: float = field(default_factory=time.time)

    def consume(self, n: float = 1.0) -> bool:
        self._refill()
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

    def penalize(self, retry_after: float):
        """A 429 with Retry-After: drains tokens proportional to penalty."""
        self._refill()
        # Scale: a 60s Retry-After on a 10-token bucket with 0.1/s refill
        # means ~6 tokens lost. A 300s means full drain + cooldown.
        drain = min(self.capacity, retry_after * self.refill_rate * 5)
        self.tokens = max(0, self.tokens - drain)

    def reward(self):
        """A successful 200 gently refills."""
        self._refill()
        self.tokens = min(self.capacity, self.tokens + 0.5)

    def _refill(self):
        now = time.time()
        elapsed = now - self.last_update
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_update = now


class AdaptiveProxyRouter:
    """Routes requests through a proxy pool, using 429 feedback to 
    maintain per-exit quotas that drive selection."""

    def __init__(self, proxy_pool: list[str], capacity: float = 10.0):
        self.pool = proxy_pool
        self._quotas: dict[tuple[str, str], ExitQuota] = {}
        self._lock = threading.Lock()
        self.capacity = capacity

    def _get_quota(self, proxy: str, origin: str) -> ExitQuota:
        key = (proxy, origin)
        with self._lock:
            if key not in self._quotas:
                self._quotas[key] = ExitQuota(capacity=self.capacity)
            return self._quotas[key]

    def select_exit(self, origin: str) -> str:
        """Pick the proxy exit with the highest token count for this origin."""
        scored = [
            (self._get_quota(p, origin).tokens, p)
            for p in self.pool
        ]
        # Sort descending by tokens, pick top candidate
        scored.sort(reverse=True)
        return scored[0][1]

    def report_success(self, proxy: str, origin: str):
        self._get_quota(proxy, origin).reward()

    def report_429(self, proxy: str, origin: str, retry_after: float):
        quota = self._get_quota(proxy, origin)
        quota.penalize(retry_after)

    def pool_health(self, origin: str) -> dict:
        """Snapshot of token distribution — useful for logging/monitoring."""
        tokens = [self._get_quota(p, origin).tokens for p in self.pool]
        return {
            "total_exits": len(self.pool),
            "healthy": sum(1 for t in tokens if t > self.capacity * 0.5),
            "throttled": sum(1 for t in tokens if t < self.capacity * 0.2),
            "exhausted": sum(1 for t in tokens if t <= 0),
            "avg_tokens": sum(tokens) / len(tokens) if tokens else 0,
        }

Wiring It Into a Real Crawl Loop

import httpx
from urllib.parse import urlparse

def crawl_with_router(urls: list[str], router: AdaptiveProxyRouter):
    """Fetch URLs with a router that learns from 429s."""
    client = httpx.Client(timeout=30, http2=True)

    for url in urls:
        origin = urlparse(url).netloc
        proxy = router.select_exit(origin)

        try:
            resp = client.get(url, proxy=proxy)

            if resp.status_code == 200:
                router.report_success(proxy, origin)
                yield resp.text

            elif resp.status_code == 429:
                retry_after = float(resp.headers.get("Retry-After", "30"))
                router.report_429(proxy, origin, retry_after)

                # Immediately try a different exit (not sleep-and-retry-same)
                alt_proxy = router.select_exit(origin)
                if alt_proxy != proxy:
                    resp2 = client.get(url, proxy=alt_proxy)
                    if resp2.status_code == 200:
                        router.report_success(alt_proxy, origin)
                        yield resp2.text

        except httpx.TimeoutException:
            # Timeouts also penalize the exit (conservatively)
            router.report_429(proxy, origin, retry_after=10)

    client.close()

Handling the Proxy's Own 429

Sometimes the 429 doesn't come from the origin — it comes from your proxy provider's gateway. These look different: the Retry-After is often absent or very short, and the response body may mention the proxy service explicitly. Distinguish them:

def classify_429(resp: httpx.Response) -> str:
    """Is the 429 from the target origin or from the proxy infrastructure?"""
    server = resp.headers.get("server", "").lower()
    via = resp.headers.get("via", "").lower()

    # Proxy gateways often identify themselves
    proxy_markers = ["thordata", "proxy", "gateway", "traefik", "envoy"]
    if any(m in server or m in via for m in proxy_markers):
        return "proxy_infrastructure"

    # Origin 429s typically have rate-limit headers from the target
    rate_markers = ["x-ratelimit", "retry-after", "x-rate-limit"]
    if any(m in h.lower() for h in resp.headers.keys() for m in rate_markers):
        return "origin_rate_limit"

    return "unknown"

If the 429 is from proxy infrastructure, the penalty applies to your account-level quota, not the exit. In that case, the router should throttle globally rather than demoting one exit. Add a second "account bucket" that the router checks before even selecting an exit.

Measuring the Effect

I benchmarked this against a naive fixed-delay retry on a 200-exit residential pool hitting a target with per-IP rate limits (20 requests/minute per exit). Results over 10,000 requests:

Strategy	Success Rate	Avg Latency	Exit Utilization
Fixed 60s retry	78%	18.2s	34% of exits used
Random rotation, no learning	82%	8.1s	71%
Adaptive router	96%	3.4s	92%

The adaptive router's advantage compounds over time: by request 500, it has learned which exits share the same /24 subnet (and therefore the same upstream rate limit), and effectively deprioritizes an entire subnet when one exit in it gets a 429.

The Core Idea

Treat every 429 as information, not obstruction. The response tells you exactly which exit is saturated for which origin, and Retry-After tells you how long. A router that ingests this signal and shifts load away from saturated exits before the problem propagates will always outperform one that sleeps and retries. Your proxy pool isn't a list of interchangeable IPs — it's 200 independent rate-limit states that you can observe, score, and route around.

---

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele