A common mistake when migrating a Python HTTP pipeline from synchronous requests to asyncio (aiohttp or httpx) is assuming that asyncio.Semaphore(10) solves API rate limiting.
It does not.
A semaphore only caps concurrent in-flight connections—it says nothing about requests per second (RPS). If an endpoint responds in 40ms, 10 concurrent workers will happily fire 250 requests per second, triggering immediate 429 Too Many Requests bans or WAF throttling.
Why asyncio.sleep() Loops Break Under Load
Naive throttling usually looks like adding await asyncio.sleep(0.2) inside each worker. In production, this fails for two reasons:
- Burst Spikes: When 20 tasks wake up simultaneously after an I/O pause, they hit the target server in a synchronized burst.
- Retry Storms: If failed requests retry immediately without jitter, they amplify server load right when the upstream API is struggling.
Combining Token-Bucket Throttling with Concurrency Semaphores
To keep throughput smooth and predictable, you need two separate controls working together:
-
asyncio.Semaphore: Caps maximum simultaneous open sockets so you never exhaust local file descriptors or overwhelm slow endpoints. -
Monotonic Token Bucket: Refills tokens continuously using
time.monotonic()(immune to system clock jumps) to enforce a strict average RPS while allowing controlled short bursts.
Here is the core token acquisition loop using an asyncio.Lock to prevent race conditions across coroutines:
async def _acquire_token(self) -> None:
while True:
async with self.lock:
now = time.monotonic()
elapsed = now - self.last_refill
self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
self.last_refill = now
if self.tokens >= 1.0:
self.tokens -= 1.0
return
wait_time = (1.0 - self.tokens) / self.rate
await asyncio.sleep(wait_time)
Adding Full-Jitter Exponential Backoff
When transient 429 or 503 errors still occur, fixed backoff delays (1s, 2s, 4s) cause synchronized retry waves. Using full jitter (random.uniform(0, base_delay * (2 ** attempt))) spreads retries evenly across time so workers recover gracefully without self-DDoSing the target server.
Get the Drop-In Module (Free & Zero Dependencies)
I packaged the complete AsyncTokenBucketLimiter class—ready to wrap around any aiohttp or httpx coroutine—as a standalone open-source module (direct download, no signup or redirects):
👉 Download rate_limiter.py Directly (OpenCode Mirror)
How do you handle strict upstream rate limits in your async Python pipelines—in-memory buckets or distributed Redis limiters? Let me know in the comments!
Top comments (0)