The 429 Is Feedback: Building an Adaptive Concurrency Controller for Python Scrapers

Every 429 response contains the same piece of information, printed in plain text: you are above the rate I tolerate, and here is how close to the ceiling you are. Most scraping code treats that as noise to retry past. I've started treating it as a sensor input. This post is about the closed-loop controller that falls out of that reframing — additive increase, multiplicative decrease, borrowed wholesale from TCP congestion control. I've written before about reading X-RateLimit-Remaining headers so you never see a 429 in the first place; this is the other half, for the sites that don't publish headers, or the day you misjudge and hit the wall anyway.

The sawtooth you already have

Here's the standard resilient-scraper recipe: a worker pool of 30 asyncio tasks, each with a shared retry_with_backoff() helper that catches 429 and sleeps base * 2**attempt with some jitter. It feels robust. It produces a graph like this:

throughput
   |  /\      /\      /\
   | /  \    /  \    /  \      <- 30 workers succeed together,
   |/    \  /    \  /    \        then 429 together
   |      \/      \/      \
   +---------------------------- time

The failure mode is synchronization. All 30 workers start together, hit the limit together, and receive their 429s within the same few milliseconds. Each computes a backoff from its own attempt counter — and since they all failed on the same attempt, they all compute roughly the same sleep, wake in the same window, and re-thunder the server as a herd. The server, seeing the same burst shape, 429s them all again. If the backoff is unbounded, throughput decays toward zero while the workers are "being polite."

Why per-worker backoff can't work

The deeper problem isn't timing. It's that the rate limit is global — per target host, per egress IP, or per account — while each worker only observes its own 429s. Worker 17 backing off for eight seconds does nothing to reduce demand from workers 1 through 16. No single actor in the system owns the question "how many requests should be in flight right now," so nobody answers it. You can't fix a shared-ceiling problem with independent controllers; you need one controller per rate-limit scope.

One wrinkle: the limit the server enforces is in requests per second, but the thing you directly control is concurrency. The two are related through latency — at steady state, rate ≈ concurrency / avg_latency (Little's law). Rather than estimate latency and compute a rate, I let the controller adjust concurrency and let AIMD find the equilibrium empirically. If the target slows down, the same concurrency yields a lower rate, and the controller fixes that by increasing — the adaptation you want, for free.

The control loop

TCP has solved this problem since 1988. The design:

• One AdaptiveLimiter per rate-limit scope (host, IP, or proxy session). Every worker acquires a slot from it before sending.
• On 2xx: additive increase, limit += 1, capped at a ceiling (I use 32 — politeness plus your own socket/CPU budget).
• On 429: multiplicative decrease, limit //= 2, floor of 1. Halving hurts, which is the point: a limit you exceeded by 2x needs one halving, not one decrement.
• Parse Retry-After on every 429. The header comes in two formats — integer seconds (Retry-After: 30) and an HTTP-date (Retry-After: Wed, 21 Oct 2026 07:28:00 GMT) — and the helper below handles both.
• Full jitter on every sleep. Instead of sleeping retry_after exactly, sleep random.uniform(0, retry_after). This is the AWS exponential-backoff trick: it flattens the herd so workers desynchronize within one or two cycles.
• Circuit breaker for sustained 429s. Six consecutive 429s with no success in between means the AIMD signal isn't converging — the server wants you gone. Open the circuit: every worker parks at the slot acquisition, for max(cooldown, retry_after) seconds (cooldown of 15 in the demo; 30–60 in production). After that, half-open: the limit is already floored at 1, so a single probe request tests the water before the controller ramps again.

The build

Runnable end to end. It spins up a local aiohttp server with a hidden capacity of 30 req/s and 350ms latency, then throws 40 workers at it through the controller — so you can watch the limit climb, overshoot, halve, and settle, with no external network involved.

"""
Adaptive concurrency: AIMD + Retry-After + full jitter + circuit breaker.
Run: pip install aiohttp && python adaptive.py
"""
import asyncio
import random
import time
from contextlib import asynccontextmanager
from email.utils import parsedate_to_datetime

from aiohttp import ClientSession, ClientTimeout, web

START = 8                 # initial concurrency
CEILING = 32              # additive-increase ceiling
FLOOR = 1
DECREASE = 2              # multiplicative decrease factor
STREAK_OPEN = 6           # consecutive 429s that trip the breaker
BREAKER_COOLDOWN = 15.0   # seconds fully open (use 30-60 in production)
SERVER_RATE = 30          # mock server's hidden capacity, req/s
SERVER_LATENCY = 0.35     # mock server's response time, seconds
RUN_SECONDS = 25


def parse_retry_after(raw):
    """Handles both 'Retry-After: 30' and 'Retry-After: Wed, 21 Oct 2026 07:28:00 GMT'."""
    if not raw:
        return 1.0
    raw = raw.strip()
    try:
        return max(0.0, float(raw))
    except ValueError:
        pass
    try:
        return max(0.0, parsedate_to_datetime(raw).timestamp() - time.time())
    except (TypeError, ValueError):
        return 1.0  # unparseable: assume a short, safe wait


class AdaptiveLimiter:
    """One instance per rate-limit scope: target host, egress IP, or proxy session."""

    def __init__(self, limit=START):
        self.limit = limit
        self._active = 0
        self._cond = asyncio.Condition()
        self._streak = 0
        self._closed_until = 0.0

    @asynccontextmanager
    async def slot(self):
        async with self._cond:
            await self._cond.wait_for(
                lambda: time.monotonic() >= self._closed_until
                and self._active < self.limit
            )
            self._active += 1
        try:
            yield
        finally:
            async with self._cond:
                self._active -= 1
                self._cond.notify_all()

    async def on_success(self):
        async with self._cond:
            self._streak = 0
            self.limit = min(CEILING, self.limit + 1)        # additive increase
            self._cond.notify_all()

    async def on_429(self, retry_after):
        async with self._cond:
            self._streak += 1
            self.limit = max(FLOOR, self.limit // DECREASE)  # multiplicative decrease
            if self._streak >= STREAK_OPEN:
                self._closed_until = time.monotonic() + max(BREAKER_COOLDOWN, retry_after)
                self._streak = 0                             # half-open: floor probes
            self._cond.notify_all()


async def handler(request):
    bucket = request.app["bucket"]
    now = time.monotonic()
    if now - bucket["t"] >= 1.0:
        bucket["t"], bucket["n"] = now, 0
    if bucket["n"] >= SERVER_RATE:
        return web.Response(status=429, headers={"Retry-After": "1"})
    bucket["n"] += 1
    await asyncio.sleep(SERVER_LATENCY)
    return web.Response(status=200, text="ok")


async def worker(url, limiter, session, stats):
    while time.monotonic() < stats["deadline"]:
        async with limiter.slot():
            async with session.get(url) as resp:
                status = resp.status
                ra = parse_retry_after(resp.headers.get("Retry-After"))
                await resp.read()
        if status == 200:
            stats["ok"] += 1
            await limiter.on_success()
        elif status == 429:
            stats["hit"] += 1
            await limiter.on_429(ra)
            await asyncio.sleep(random.uniform(0, max(ra, 0.5)))  # full jitter
        else:
            stats["other"] += 1
            await asyncio.sleep(1.0)


async def main():
    app = web.Application()
    app["bucket"] = {"t": time.monotonic(), "n": 0}
    app.router.add_get("/page", handler)
    runner = web.AppRunner(app)
    await runner.setup()
    await web.TCPSite(runner, "127.0.0.1", 8899).start()

    limiter = AdaptiveLimiter()
    stats = {"ok": 0, "hit": 0, "other": 0,
             "deadline": time.monotonic() + RUN_SECONDS}
    t0 = time.monotonic()
    async with ClientSession(timeout=ClientTimeout(total=15)) as session:
        url = "http://127.0.0.1:8899/page"
        tasks = [asyncio.create_task(worker(url, limiter, session, stats))
                 for _ in range(40)]
        while time.monotonic() < stats["deadline"]:
            print(f"t={time.monotonic() - t0:5.1f}s limit={limiter.limit:2d} "
                  f"ok={stats['ok']:4d} 429s={stats['hit']:3d}")
            await asyncio.sleep(2)
        for t in tasks:
            t.cancel()
    await runner.cleanup()
    print(f"final: ok={stats['ok']} 429s={stats['hit']} "
          f"converged limit={limiter.limit}")


if __name__ == "__main__":
    asyncio.run(main())

With the demo numbers, sustainable concurrency is 30 req/s × 0.35s ≈ 10 in flight. You'll watch the limit start at 8, additively climb past 10, collect a handful of 429s, halve down to 6–8, and then oscillate in a narrow band around 10 — an AIMD sawtooth, but a small one that tracks the server's actual ceiling, instead of the violent 30-worker herd cycle. The 429 count after convergence is a handful per minute, not a wall.

Note the shape of slot(): a resizable semaphore built on asyncio.Condition, because asyncio.Semaphore can't be shrunk after creation. Workers park inside wait_for when the limit drops; shrinking the limit mid-flight doesn't kill in-flight requests, it just stops new ones from being admitted until the active count drains below the new limit.

Details that decide whether this works

Classify before you control. A 429 means "too fast" — slow down. A 403 with a block page means "we don't want you" — slowing down does nothing; that's a fingerprint or IP problem, and the correct response is failing over the session or stopping, not halving concurrency. A 503 usually means the origin is overloaded — back off harder (×0.5 plus a floor sleep) and don't additively increase immediately after, because a 200 from a struggling origin is not evidence of headroom. My worker treats only 429 as an AIMD input; everything else goes to a side channel.

Scope the limiter to how the limit is enforced. If the target rate-limits per IP and you're rotating residential proxy sessions, a single global limiter leaves money on the table: each session has its own ceiling, so run one AdaptiveLimiter per session (a dict[session_id, AdaptiveLimiter]) and let each converge independently. If the limit is per account or per API key, share one limiter across all sessions for that key. Guessing wrong about scope is the most common way this design underperforms — and a per-account limit enforced through a global limiter will bottleneck your whole pool on one identity.

Respect Retry-After: 0. It doesn't mean "no wait" — it means "stop sending now, ask again later." Treat it as a minimum sleep of a few hundred milliseconds with jitter, or you'll spin at full concurrency against a server that just told you to pause.

Log the concurrency time series. Every on_success/on_429 should append (timestamp, limit, active) to a structured log. When someone asks why the crawl is slow, the shape of that series answers instantly: a flat line at FLOOR means a trigger-happy breaker or a block masquerading as 429s; a clean sawtooth around a stable midpoint means healthy convergence; a downward drift means the target got slower or tighter and the controller is correctly following it down.

Wrapping up

A retry loop treats a 429 as an obstacle. A controller treats it as measurement. The difference is one Condition, two update rules, and the discipline to put every worker in a scrape behind a single shared limit instead of thirty private ones. The AIMD loop is 40 years old and boring, which is exactly what you want in the part of your scraper that talks to someone else's server: it converges on the maximum rate the server tolerates, backs off fast when that ceiling moves, and never needs to be told what the ceiling is — because the server tells you, every time, in the status line.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele