DEV Community

Dhanush Kumar Sivaji
Dhanush Kumar Sivaji

Posted on Edited on Originally published at dhanushkumarsivaji.substack.com

The path to AI FDE #04: Many calls at once, and a FastAPI service you can lean on

#ai #python #fastapi #programming

Originally published on my newsletter, The Path to AI FDE. Subscribe free to get each issue by email.

Issue #03 ended with tasks: the way async work actually overlaps. Real AI backends fan out, calling a model several times or fetching a few sources in parallel. So this issue starts with how to do that without losing errors or hanging forever. Then it turns to the FastAPI features that move an endpoint past hello-world. As always, every example is a toy one (a notes app), not my roadmap exercises.

Two panels. Part A, many calls at once: six cards on gather, TaskGroup, timeouts, semaphores and cancellation. Part B, FastAPI properly: six cards on request and response models, HTTPException, dependencies, def versus async def, and settings

Part A · Many calls at once

gather: three fetches finish in a different order than they were started, but gather returns results in argument order; by default the first error is raised and the others keep running, while return_exceptions=True puts errors in the list

1. gather returns results in the order you asked. a, b = await asyncio.gather(fetch(1), fetch(2)) runs both concurrently, and the result list matches the order of the arguments, not the order the calls finished in. It’s the closest thing to Promise.all.

2. Know what gather does with an error. By default, the first exception is raised straight to you, and the other calls are not cancelled: they keep running in the background. With return_exceptions=True, errors come back as items in the result list instead, which suits best-effort work.

TaskGroup: when one of three tasks fails the other two are cancelled, and the failures are raised together as an ExceptionGroup handled with except-star

3. TaskGroup is all-or-nothing. Inside async with asyncio.TaskGroup() as tg:, you start work with tg.create_task(...). The first failure cancels the rest, and when the block exits the failures are raised together as an ExceptionGroup, which you handle with except* SomeError:. Use it when partial results are useless. It also keeps task references for you (#03, point 11).

Every network wait needs a deadline: a slow call is cancelled at 10 seconds and TimeoutError is raised; catch it outside the async with asyncio.timeout block, not inside

4. Every network wait needs a deadline. async with asyncio.timeout(10): cancels whatever is inside after 10 seconds and raises TimeoutError, which you can only catch outside the block. For a single awaitable, await asyncio.wait_for(coro, timeout=10) does the same job. Without a deadline, one slow provider call can hold a request open indefinitely.

A semaphore caps how many run at once: items wait in a queue while at most five calls are in flight in Semaphore(5)'s slots

5. A semaphore caps how many run at once. sem = asyncio.Semaphore(5), then async with sem: around each call, lets at most five run concurrently while the rest wait their turn. That’s how you fan out over 200 items while keeping concurrency low enough to stay under a provider’s rate limit (the 429s from #01).

Don't swallow cancellation: timeout and TaskGroup cancel a task, CancelledError must travel up; catching BaseException and passing hides it, while catching specific errors with try/finally lets it through

6. Don’t swallow cancellation. TaskGroup and timeout() work by cancelling tasks, and the asyncio docs warn that they can misbehave if your code catches and hides asyncio.CancelledError. A bare except:, an except BaseException: or a re-raise you forgot quietly breaks your deadlines. (except Exception: doesn’t catch it: since Python 3.8, CancelledError is a BaseException.) Catch specific errors (#02, point 4).

Part B · FastAPI properly

FastAPI validates both directions: a toy notes app request passes NoteIn validation (invalid gives 422), the handler, and a NoteOut response model that drops owner_id; HTTPException returns a 404 for a missing note

7. Request models give you 422s for free. Declare a body parameter as a Pydantic model (note: NoteIn), and FastAPI validates the request, answering 422 with a list of what’s wrong before your function even runs.

8. Response models validate and filter what goes out. Set a return type or response_model=NoteOut, and FastAPI checks your output and drops any field the model doesn’t declare. That’s the docs’ security point: an internal field (an owner ID, a password hash) can’t leak just because it was on the object you returned. If you set both, response_model takes priority.

9. HTTPException for errors you mean. raise HTTPException(status_code=404, detail="Note not found") stops the handler and returns that status with a JSON body. Raise it for expected failures: a missing item, a bad state, an upstream provider that’s down.

Dependencies: get_db, get_current_user and get_settings feed an endpoint through an Annotated Depends alias; settings read from environment variables are validated by BaseSettings, cached with lru_cache and overridden in tests

10. Dependencies hold the shared plumbing. A dependency is a function FastAPI calls for you before your endpoint runs, declared as a parameter with Depends(get_db), usually through an Annotated alias so it’s written once. Shared logic, database sessions and auth live there instead of in every handler. Dependencies can be async def or plain def.

def vs async def endpoints: async def runs on the shared event loop and suits async clients; plain def runs in an external threadpool and suits synchronous libraries; avoid blocking calls inside async def

11. def versus async def endpoints. An async def endpoint runs on the event loop, so anything blocking inside it stalls every request (#03, point 12). A plain def endpoint runs in a threadpool instead, so blocking calls there are survivable. Rule of thumb: async def with async clients, plain def when your libraries are synchronous.

12. Settings come from the environment, validated. A pydantic-settings BaseSettings class reads environment variables (case-insensitively) and validates their types when it’s created. A missing API key or TIMEOUT_S=abc fails right there, not halfway through a request that uses it: at startup if you create the settings at import time, or on first use if you serve them the way the FastAPI docs do, from an @lru_cache‘d get_settings() dependency. That dependency also lets tests swap settings with app.dependency_overrides.

Try it (20 minutes, toy example)

Make async def fetch(i) that sleeps for a random 0–2 seconds and sometimes raises. Run 20 of them three ways: gather, gather(return_exceptions=True) and a TaskGroup. Then add Semaphore(5) and a 1-second timeout, and compare what you get back each time. For the FastAPI half, give a toy notes app a NoteOut model without an internal owner_id field, return an object that has one, and check that it doesn’t appear in the response.

Common beginner mistakes

  • Assuming gather stops the other calls when one fails.
  • Writing try/except TimeoutError inside async with asyncio.timeout().
  • Fanning out with no concurrency limit and getting rate-limited.
  • Returning a database object without a response model, and leaking fields.
  • Doing blocking calls in async def endpoints.

Takeaway

Fan out with a limit and a deadline. Let FastAPI validate both directions, and keep shared plumbing in dependencies.

Where this fits in my roadmap: this is the second half of my foundations block (async in practice, plus FastAPI and Pydantic). In my own study, I’m building that block’s small LLM endpoint myself, with real timeouts and error handling. So this post sticks to the concepts and toy examples, not my solution. Next comes testing (pytest, plus mocking the provider), and then structured outputs, where the Pydantic schema from #03 becomes the contract the model must follow.

Over to you: gather(return_exceptions=True) or TaskGroup: which do you reach for by default, and why?

Sources

  1. Python docs: Coroutines and Tasks (gather order, errors and return_exceptions; TaskGroup cancellation and ExceptionGroup; timeout() raising TimeoutError catchable only outside; wait_for; the warning against swallowing CancelledError). https://docs.python.org/3/library/asyncio-task.html
  2. Python tutorial 8.9: Raising and handling multiple unrelated exceptions (ExceptionGroup, except*). https://docs.python.org/3/tutorial/errors.html
  3. Python docs: Synchronization primitives (Semaphore). https://docs.python.org/3/library/asyncio-sync.html
  4. FastAPI: Request Body (Pydantic models; invalid data returns an error). https://fastapi.tiangolo.com/tutorial/body/
  5. FastAPI: Response Model (validates returned data; filters output to the declared type for security; response_model priority). https://fastapi.tiangolo.com/tutorial/response-model/
  6. FastAPI: Handling Errors (HTTPException; RequestValidationError and 422). https://fastapi.tiangolo.com/tutorial/handling-errors/
  7. FastAPI: Dependencies (Depends, shared logic, DB connections, auth; Annotated aliases; async def or def). https://fastapi.tiangolo.com/tutorial/dependencies/
  8. FastAPI: Concurrency and async/await (plain def path operations run in an external threadpool). https://fastapi.tiangolo.com/async/
  9. FastAPI: Settings and Environment Variables (BaseSettings, case-insensitive env vars, lru_cache, dependency_overrides, .env). https://fastapi.tiangolo.com/advanced/settings/
  10. Python docs: asyncio Exceptions (CancelledError is a BaseException since 3.8; must almost always be re-raised). https://docs.python.org/3/library/asyncio-exceptions.html

Sources checked Oct 18, 2026.

Originally published on The Path to AI FDE on Substack. Subscribe for the next issue: https://dhanushkumarsivaji.substack.com/subscribe

If this was useful, subscribe to The Path to AI FDE for the next issue.

Top comments (0)