You have a thousand documents to send to a model. You set a semaphore to four and pass one coroutine per document to asyncio.gather. At most four calls can run at once. By the time they start, though, you already have a thousand tasks. Most of them are waiting for the semaphore.
With a thousand small inputs, that might go unnoticed. If the documents come from a database in batches and the model starts responding more slowly, I'd look at whether the code keeps reading ahead. By the time a task reaches the semaphore, it already exists and holds a reference to its document.
Where to put the limit
I'd give four workers a queue with eight slots. When the queue fills, the code reading the documents pauses at await queue.put(doc_id). It can continue as soon as a worker takes an item. If documents arrive slowly, idle workers wait at get(). That way, a slowdown at the model can reach the part of the program that reads new documents. See the Python documentation on asyncio.Queue with a positive maxsize.
A semaphore still fits a small batch where all you need to cap is concurrent calls. For a stream of unknown length, I'd also cap how much work we've accepted and have yet to start.
Two ways to process the same input
The example uses asyncio.sleep to stand in for the API. The first version creates a task per document and uses a semaphore around the call. The second creates four worker tasks and feeds them through a queue. Both versions use TaskGroup to wait for their tasks. The limits passed in this example are positive. It uses only the standard library and requires Python 3.11 or later.
import asyncio
from dataclasses import dataclass
@dataclass
class Counts:
tasks_created: int = 0
active: int = 0
peak_active: int = 0
peak_queued: int = 0
producer_waits: int = 0
processed: int = 0
async def fake_model(doc_id: int, counts: Counts) -> None:
counts.active += 1
counts.peak_active = max(counts.peak_active, counts.active)
try:
await asyncio.sleep(0.001 + (doc_id % 3) * 0.0002)
finally:
counts.active -= 1
counts.processed += 1
async def all_at_once(doc_ids: range, limit: int) -> Counts:
counts = Counts()
slots = asyncio.Semaphore(limit)
async def process(doc_id: int) -> None:
async with slots:
await fake_model(doc_id, counts)
async with asyncio.TaskGroup() as group:
for doc_id in doc_ids:
group.create_task(process(doc_id))
counts.tasks_created += 1
return counts
async def with_queue(doc_ids: range, workers: int, capacity: int) -> Counts:
counts = Counts()
queue: asyncio.Queue[int | None] = asyncio.Queue(maxsize=capacity)
async def worker() -> None:
while True:
doc_id = await queue.get()
try:
if doc_id is None:
return
await fake_model(doc_id, counts)
finally:
queue.task_done()
async with asyncio.TaskGroup() as group:
for _ in range(workers):
group.create_task(worker())
counts.tasks_created += 1
for doc_id in doc_ids:
if queue.full():
counts.producer_waits += 1
await queue.put(doc_id)
counts.peak_queued = max(counts.peak_queued, queue.qsize())
await queue.join()
for _ in range(workers):
await queue.put(None)
return counts
async def main() -> None:
doc_ids = range(1_000)
eager = await all_at_once(doc_ids, limit=4)
queued = await with_queue(doc_ids, workers=4, capacity=8)
assert eager.processed == queued.processed == len(doc_ids)
assert eager.peak_active <= 4 and queued.peak_active <= 4
assert queued.peak_queued <= 8 and queued.producer_waits > 0
print(f"Semaphore: tasks={eager.tasks_created}, peak_active={eager.peak_active}")
print(
f"Queue: tasks={queued.tasks_created}, peak_active={queued.peak_active}, "
f"peak_queued={queued.peak_queued}"
)
if __name__ == "__main__":
asyncio.run(main())
Here's what I got in a local run.
Semaphore: tasks=1000, peak_active=4
Queue: tasks=4, peak_active=4, peak_queued=8
Both versions reached four active requests. The other numbers tell the story. The first version created 1,000 tasks for the documents. The second ran four worker tasks and held at most eight items in the queue. The producer can hold one more ID while it waits at put(). With the queue full and all workers busy, that's eight waiting, four in progress, and one with the producer. That gives us a bound on how many items this part of the program holds at once.
The queue holds IDs in this example. If you put entire PDFs in it or collect every result in a list, memory usage will depend on those objects too. I'd read one document at a time or in small batches and save each result as it becomes available. Then the queue limit helps control how much data the pipeline keeps in memory.
When the queue fills up
A batch job can wait. For an incoming HTTP request, I'd set a limit on how long the client has to wait or reject the request explicitly so the client can try later. With asyncio.Queue, put_nowait() raises QueueFull when all slots are taken. If you create a new task for every queue.put() instead, the queue stays small and the waiting tasks pile up outside it.
Four workers cap concurrent calls. A provider quota of, say, 60 requests per minute sets a separate limit. With the same number of workers, faster responses let you start more calls per minute. Enforce the provider's quota before sending each request. The queue size still controls how many documents can wait for processing.
Finishing the batch
Once the input runs out, queue.join() waits for a task_done() call for each accepted item. Then we send one None per worker so all four can leave the loop. If a worker raises an exception other than CancelledError, TaskGroup cancels the other tasks and the batch fails. If the whole batch is canceled, the group waits for its tasks to finish shutting down. In the local tests, all four workers stopped in both cases, even with the producer waiting for a free slot. Python documentation on asyncio.TaskGroup
The queue lives in the process. On cancellation, some IDs can still be waiting in it while the active mock calls are interrupted. When the function exits, those pending items are lost with the queue. If a batch must resume after a restart, I'd store the IDs and mark each document complete after saving its result. On the next run, we can load the unfinished ones. If a real network call is canceled, check whether the remote service already accepted it before trying again.
If I were changing existing code that uses gather, I'd keep the function that calls the model and replace the part that creates one task per document. I'd give one producer responsibility for reading from the database and putting IDs into the queue, waiting at put() as needed. Keeping the same limit of four calls makes it easier to see what changed about the backlog.
Eight slots make the demo easy to inspect. In a real service, I'd measure how long documents wait and whether workers sit idle. If the queue stays full and wait times keep growing, more slots will let you accept more documents while processing continues at the same pace. That's when you need to decide which documents can wait and what to tell callers with new requests.
References
- Python documentation. Queues
- Python documentation. Coroutines and Tasks, Task Groups
- Python documentation. Synchronization Primitives, Semaphore
Top comments (0)