DEV Community

Cover image for Why Your Python asyncio Worker Still Hits 429 Rate Limits (And How Token-Bucket Semaphores Fix It)
Alex Mercer
Alex Mercer

Posted on

Why Your Python asyncio Worker Still Hits 429 Rate Limits (And How Token-Bucket Semaphores Fix It)

A common mistake when migrating a Python HTTP pipeline from synchronous requests to asyncio (aiohttp or httpx) is assuming that asyncio.Semaphore(10) solves API rate limiting.

It does not.

A semaphore only caps concurrent in-flight connections—it says nothing about requests per second (RPS). If an endpoint responds in 40ms, 10 concurrent workers will happily fire 250 requests per second, triggering immediate 429 Too Many Requests bans or WAF throttling.

Why asyncio.sleep() Loops Break Under Load

Naive throttling usually looks like adding await asyncio.sleep(0.2) inside each worker. In production, this fails for two reasons:

  1. Burst Spikes: When 20 tasks wake up simultaneously after an I/O pause, they hit the target server in a synchronized burst.
  2. Retry Storms: If failed requests retry immediately without jitter, they amplify server load right when the upstream API is struggling.

Combining Token-Bucket Throttling with Concurrency Semaphores

To keep throughput smooth and predictable, you need two separate controls working together:

  • asyncio.Semaphore: Caps maximum simultaneous open sockets so you never exhaust local file descriptors or overwhelm slow endpoints.
  • Monotonic Token Bucket: Refills tokens continuously using time.monotonic() (immune to system clock jumps) to enforce a strict average RPS while allowing controlled short bursts.

Here is the core token acquisition loop using an asyncio.Lock to prevent race conditions across coroutines:

async def _acquire_token(self) -> None:
    while True:
        async with self.lock:
            now = time.monotonic()
            elapsed = now - self.last_refill
            self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
            self.last_refill = now

            if self.tokens >= 1.0:
                self.tokens -= 1.0
                return

            wait_time = (1.0 - self.tokens) / self.rate

        await asyncio.sleep(wait_time)
Enter fullscreen mode Exit fullscreen mode

Adding Full-Jitter Exponential Backoff

When transient 429 or 503 errors still occur, fixed backoff delays (1s, 2s, 4s) cause synchronized retry waves. Using full jitter (random.uniform(0, base_delay * (2 ** attempt))) spreads retries evenly across time so workers recover gracefully without self-DDoSing the target server.


Get the Drop-In Module (Free & Zero Dependencies)

I packaged the complete AsyncTokenBucketLimiter class—ready to wrap around any aiohttp or httpx coroutine—as a standalone open-source module (direct download, no signup or redirects):

👉 Download rate_limiter.py Directly (OpenCode Mirror)

How do you handle strict upstream rate limits in your async Python pipelines—in-memory buckets or distributed Redis limiters? Let me know in the comments!

Top comments (0)