DEV Community

Ali Ahmed
Ali Ahmed

Posted on Originally published at aliahmed.si

Limiting concurrency in Python asyncio with a semaphore and per-task timeouts, in the right order

asyncio.gather(*(call(x) for x in items)) starts every call at once: with a thousand prompts, that's a thousand simultaneous requests to an LLM API, and an immediate wall of 429s. Cap it with asyncio.Semaphore(n): each task waits for a slot before it starts. Give each call a timeout with asyncio.timeout(), and pass return_exceptions=True so one failure doesn't cancel the rest. The order matters: the timeout must go inside the semaphore. Put it outside, and the clock starts while the task is still queueing, so tasks time out without ever running; in my test, 8 of 10 did. Here's the correct version in a dozen lines, tested on Python 3.11 and later.

The function

```python title="bounded.py"
import asyncio

async def map_limit(items, limit: int, worker, timeout: float | None = None):
"""Run worker(item) for every item, at most limit at once, each with its own timeout.
Returns results in input order; failures come back as exceptions instead of cancelling the rest."""
sem = asyncio.Semaphore(limit)

async def run(item):
    async with sem:  # wait for a free slot BEFORE starting the timer
        async with asyncio.timeout(timeout):
            return await worker(item)

return await asyncio.gather(*(run(i) for i in items), return_exceptions=True)
Enter fullscreen mode Exit fullscreen mode



* **`asyncio.Semaphore(limit)`** lets `limit` tasks hold it at once; the rest wait in line.
* **`asyncio.timeout()`** (Python 3.11+) cancels the work inside it when time runs out and raises `TimeoutError`. On older versions, `asyncio.wait_for(worker(item), timeout)` does the same job.
* **`return_exceptions=True`** returns each failure in its place in the results list. Without it, the first exception propagates from `gather`, and you lose every result that did succeed.

All the coroutines are still **created** up front; only their execution is limited. For hundreds of thousands of items, feed a fixed number of worker tasks from an `asyncio.Queue` instead, so you don't hold that many coroutine objects in memory.

## Does it work?

The test runs 20 fake LLM calls with a limit of 4 and a 0.2-second timeout. Call 7 takes 0.5 seconds, and call 13 raises `ValueError`:



```python title="Results"
results = await map_limit(range(20), 4, fake_llm, timeout=0.2)
# peak concurrency: 4
# [0, 1, 4, 9, 16, 25, 36, TimeoutError, 64, 81, 100, 121, 144, ValueError, 196, 225, 256, 289, 324, 361]
Enter fullscreen mode Exit fullscreen mode

Concurrency never exceeds 4, results stay in input order, the slow call becomes a TimeoutError and the bad one a ValueError, and the other 18 succeed. The whole batch takes about 0.23 seconds.

The wrong order

This version looks almost identical:

```python title="Timeout outside the semaphore (wrong)"
async def wrong(n):
async with asyncio.timeout(0.05):
async with sem: # Semaphore(2)
await asyncio.sleep(0.03) # the actual work: 30 ms
return n




Each task does 30 ms of work and has a 50 ms timeout, which should be plenty. But with 10 tasks and 2 slots, most tasks spend longer than 50 ms **waiting for a slot**, and the timeout counts that wait. In the test, **8 of 10 timed out**, and 6 of those 8 never started their work at all. The same work through `map_limit` with the same limit and timeout completes all 10.

This matters most when the timeout is your signal that a provider is slow. With the timeout outside, a long queue on your side looks like a slow provider, which can trigger retries and fallbacks that make the queue longer.

## Choosing the numbers

* **The limit** bounds concurrent requests, not their rate. LLM APIs limit requests and tokens per minute, so pair the semaphore with a rate limiter; the token-bucket design in [rate limiting LLM API calls](https://aliahmed.si/blog/llm-rate-limiter-token-bucket-nodejs/) translates directly to Python.
* **The timeout** should cover your slowest normal call. For streaming LLM responses, a time-to-first-token timeout is more useful than a total one.
* **Retries** go inside `worker`, so a retried call keeps its slot. Use [backoff with jitter](https://aliahmed.si/blog/llm-api-retries-backoff-fallbacks/), and don't retry timeouts on requests that change state unless they're idempotent.

The same pattern in JavaScript is [a 15-line promise pool](https://aliahmed.si/blog/limit-concurrent-promises-nodejs/).

## Summary

* `gather` alone starts everything at once; a `Semaphore` caps how many run.
* Put `asyncio.timeout()` **inside** `async with sem`, so queueing time doesn't count.
* Use `return_exceptions=True` to keep every result and see every failure.
* Pair the concurrency limit with a rate limiter for APIs with per-minute limits.

---

Written by Ali Ahmed ยท [Original article](https://aliahmed.si/blog/asyncio-semaphore-timeout-concurrency/)
Enter fullscreen mode Exit fullscreen mode

Top comments (0)