Linh Fish Dev

Posted on Nov 7 Originally published at linhfish.dev

Python: Call Multiple APIs in Parallel with asyncio.gather and Semaphore in 10 Minutes

#python #asyncio #httpx #tutorial

When you need to call dozens of APIs — syncing data, crawling prices, firing webhooks — a sequential for loop adds up the waiting time of every request: 20 requests × 1 second = 20 seconds. asyncio.gather runs many coroutines concurrently inside a single event loop, while asyncio.Semaphore limits how many requests fly at the same time so you don't overwhelm the partner API. This technique applies to every I/O-bound task (HTTP calls, file reads, database queries) and only needs the httpx library.

The Problem: a Sequential Loop Adds Up All the Waiting

Say you need to fetch info for 20 users from a partner API, each request taking about 1 second. The most "naive" code is a loop with the synchronous httpx.Client:

import time
import httpx

URLS = [f"https://api.example.com/users/{i}" for i in range(20)]

def fetch_all():
    results = []
    with httpx.Client(timeout=10) as client:
        for url in URLS:
            results.append(client.get(url).json())
    return results

t0 = time.perf_counter()
fetch_all()
print(f"finished in {time.perf_counter() - t0:.1f} seconds")

This code runs correctly, but the total time is ≈ 20 seconds because each request waits for the previous one to finish. The CPU sits idle almost the whole time — the bottleneck is I/O waiting, not processing power. With 200 records, you'd wait over 3 minutes.

The Solution: asyncio.gather + Semaphore

asyncio lets many coroutines "suspend" at I/O waiting points and hand the event loop to another coroutine. asyncio.gather(*coros) groups them and runs them concurrently, returning results in the same order you passed in. But don't fire 200 requests at once — the partner API may rate-limit you or block your IP. asyncio.Semaphore(n) is a "gate" that lets only n coroutines through at the same time:

import asyncio
import httpx

async def fetch_one(client, url, sem):
    async with sem:  # grab a ticket before calling HTTP
        resp = await client.get(url, timeout=10)
        resp.raise_for_status()
        return resp.json()

async def fetch_all_async(urls):
    sem = asyncio.Semaphore(5)  # at most 5 requests at once
    async with httpx.AsyncClient() as client:
        return await asyncio.gather(
            *(fetch_one(client, u, sem) for u in urls),
            return_exceptions=True,
        )

results = asyncio.run(fetch_all_async(URLS))

How it flows: each fetch_one grabs a "ticket" from the semaphore before calling HTTP; at most 5 requests fly together, and when one finishes another takes its place. With 20 requests × 1 second and Semaphore(5), total time drops to ≈ 4 seconds instead of 20. Measured on a local server (20 requests × 0.3 seconds): 6.19 seconds → 1.41 seconds.

Collect Errors Neatly with return_exceptions=True

By default, one failing coroutine makes gather raise the exception immediately, and you lose the results of the requests that already finished. Pass return_exceptions=True and the error comes back as a normal item in the result list — filter it afterwards:

ok = [r for r in results if not isinstance(r, Exception)]
failed = [r for r in results if isinstance(r, Exception)]
print(f"success: {len(ok)}/{len(results)}")
for e in failed:
    print("failed request:", repr(e))

This pattern shines in data-sync jobs: 98 of 100 records still get processed, and the 2 failures go to the log for a later retry — instead of the whole batch dying because of one bad request.

Apply It to Your Project

Start with Semaphore(5), then raise it gradually while watching response times and 429 codes from the partner API. Always set a timeout on every request — one request hanging forever holds its semaphore "ticket" forever and jams the whole batch. If your project already uses FastAPI, your async def endpoints can await fetch_all_async() directly without asyncio.run().

FAQ

What should I set Semaphore(n) to?

There's no magic number: start at 5–10, increase gradually, and watch for 429 codes or timeouts from the partner API. Many public APIs limit you to 5–10 requests per second.

Does gather keep the result order?

Yes. Results come back in the order you passed the coroutines in, not the order they finish. That's the difference from asyncio.as_completed().

Can one bad request kill the whole batch?

By default, yes — gather raises the first exception it meets. Use return_exceptions=True to collect errors into the list and handle them later, as shown above.

When should I NOT use asyncio?

For CPU-bound work (image processing, file compression, heavy math): the GIL prevents coroutines from truly running in parallel. Use multiprocessing or concurrent.futures.ProcessPoolExecutor for those.

💬 Have you tried this in your project? If you hit an error or know a cleaner way, leave a comment below — I read them all and will add the best ideas to this post.

References

Python docs — asyncio: https://docs.python.org/3/library/asyncio.html
httpx docs — Async API: https://www.python-httpx.org/async/

Originally published at linhfish.dev.

DEV Community