FastAPI streams a response when the body is an async generator and nothing between you and the browser is buffering. Both halves of that sentence are load-bearing, and the second is where the afternoon goes.
The endpoint
pip install fastapi uvicorn httpx. The whole server is one file. The client is built once at startup and shared, because a connection pool per request throws away keep-alive and adds a TLS handshake to every user’s time-to-first-token.
# server.py
import asyncio
import json
import os
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
BASE_URL = os.environ["LLM_BASE_URL"].rstrip("/")
API_KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ.get("LLM_MODEL", "openai/gpt-4o-mini")
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.client = httpx.AsyncClient(
base_url=BASE_URL,
headers={"Authorization": f"Bearer {API_KEY}"},
timeout=httpx.Timeout(connect=5.0, read=120.0, write=10.0, pool=10.0),
limits=httpx.Limits(max_connections=50, max_keepalive_connections=20),
)
yield
await app.state.client.aclose()
app = FastAPI(lifespan=lifespan)
class ChatRequest(BaseModel):
prompt: str
def sse(event: str, data: dict) -> str:
return f"event: {event}\ndata: {json.dumps(data)}\n\n"
async def generate(client: httpx.AsyncClient, request: Request,
prompt: str) -> AsyncIterator[str]:
payload = {
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"stream": True,
}
try:
async with client.stream("POST", "/chat/completions", json=payload) as upstream:
if upstream.status_code >= 400:
await upstream.aread()
yield sse("error", {"status": upstream.status_code,
"detail": upstream.text[:500]})
return
async for line in upstream.aiter_lines():
if await request.is_disconnected():
return # closes the upstream stream too
line = line.strip()
if not line.startswith("data:"):
continue
data = line[5:].strip()
if data == "[DONE]":
break
try:
frame = json.loads(data)
except json.JSONDecodeError:
continue
for choice in frame.get("choices", []):
text = choice.get("delta", {}).get("content")
if text:
yield sse("token", {"text": text})
yield sse("done", {})
except asyncio.CancelledError:
raise # let the server tear it down
except httpx.HTTPError as exc:
yield sse("error", {"detail": f"{type(exc).__name__}: {exc}"})
@app.post("/chat")
async def chat(body: ChatRequest, request: Request) -> StreamingResponse:
return StreamingResponse(
generate(request.app.state.client, request, body.prompt),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
)
uvicorn server:app --reload --port 8000
curl -N -X POST http://localhost:8000/chat \
-H 'content-type: application/json' \
-d '{"prompt":"Count to ten slowly."}'
curl -N is the first thing to run. If tokens trickle out of curl, your Python is correct and any remaining problem is infrastructure.
The event: lines are the part worth copying. A raw token stream with no event names leaves the browser unable to distinguish text from an error from completion, so every client ends up sniffing the payload. Three named events — token, error, done — cost nothing and make the client trivial.
The browser side
The EventSource API is GET-only, so a POST endpoint needs fetch with a reader. That is a few more lines and it is what most real chat UIs do:
const response = await fetch("/chat", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ prompt }),
signal: controller.signal, // an AbortController — see below
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
// frames are separated by a blank line; the last piece may be incomplete
const frames = buffer.split("\n\n");
buffer = frames.pop() ?? "";
for (const frame of frames) {
const event = /^event: (.*)$/m.exec(frame)?.[1] ?? "message";
const data = /^data: (.*)$/m.exec(frame)?.[1];
if (!data) continue;
if (event === "token") appendToken(JSON.parse(data).text);
if (event === "error") showError(JSON.parse(data));
}
}
Keeping the trailing partial frame in buffer is the bug every hand-written SSE client has once: a chunk boundary can land in the middle of a frame, and splitting without a remainder silently drops it.
When the client disappears
A user closes the tab thirty tokens into a two-thousand-token answer. If nothing stops the upstream call, you keep receiving — and paying for — the rest, and the connection stays occupied.
The check is await request.is_disconnected(), called inside the loop. Because the loop only continues while the generator is being consumed, returning from it exits the async with client.stream(...) block, which closes the upstream connection. Starlette also cancels the task when the client goes away, which surfaces as asyncio.CancelledError inside the generator — re-raise it rather than swallowing it, or the server cannot complete the teardown.
Cancelling the HTTP connection stops the bytes reaching you. Whether the provider stops generating — and therefore stops charging — on a dropped connection is a per-provider behaviour, and not one to assume in either direction. If cost control matters more than answer length, cap max_tokens as well; denial of wallet is about why an uncapped endpoint is a liability.
On the browser side, the matching half is an AbortController aborted in a cleanup function, so navigating away actually closes the socket instead of leaving it to a timeout.
Backpressure, and what it is not
Backpressure is the slow consumer telling the fast producer to wait. In this endpoint it is mostly free: an async generator produces one chunk per yield and is not resumed until the server has handed the previous chunk to the transport, so a slow client naturally slows the reads from upstream. There is no unbounded queue in the middle unless you build one.
You build one by accumulating. A common mistake is collecting fragments into a list to log or persist the full answer at the end; on a long answer with many concurrent users that is a growing per-connection buffer. If you need the full text, write it out as it arrives, or persist it from the token events rather than holding the whole thing. Backpressure covers the general shape of the problem.
Two smaller things worth doing. Emit a comment line — ": keep-alive\n\n" — every fifteen seconds or so if the model can be silent for a long time, because idle proxies close connections at thirty or sixty seconds. And set a wall-clock cap in the generator with time.monotonic(), since an httpx read timeout resets on each chunk and so never fires on a slow-but-steady stream.
Limiting concurrency on your own endpoint
A streaming endpoint holds a connection open for the whole answer, so fifty simultaneous users mean fifty concurrent upstream calls that last for tens of seconds each. Two limits bind — your connection pool and your provider’s rate limit — and neither of them fails in a way that tells the fifty-first user anything useful.
Without an explicit cap, the fifty-first request waits on the httpx pool until pool timeout expires and then fails, having consumed a worker slot for the whole wait. A semaphore acquired without blocking turns that into an immediate, honest 503:
import asyncio
from fastapi import HTTPException
MAX_IN_FLIGHT = 40
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.client = httpx.AsyncClient(...) # as before
app.state.slots = asyncio.Semaphore(MAX_IN_FLIGHT)
yield
await app.state.client.aclose()
@app.post("/chat")
async def chat(body: ChatRequest, request: Request) -> StreamingResponse:
slots: asyncio.Semaphore = request.app.state.slots
if slots.locked():
raise HTTPException(
status_code=503,
detail="at capacity, retry shortly",
headers={"Retry-After": "5"},
)
await slots.acquire()
async def guarded():
try:
async for chunk in generate(request.app.state.client, request, body.prompt):
yield chunk
finally:
slots.release() # runs on completion AND on cancel
return StreamingResponse(guarded(), media_type="text/event-stream",
headers={"Cache-Control": "no-cache",
"X-Accel-Buffering": "no"})
The finally is the whole trick and the easiest thing to get wrong. When a client disconnects, Starlette cancels the task, which raises CancelledError inside the generator — a release placed after the loop would never run, and the semaphore would leak one slot per abandoned request until the server stopped accepting anything. A finally runs on both paths.
Set MAX_IN_FLIGHT below the httpx pool’s max_connections, so the semaphore is the binding constraint and the pool never becomes a hidden queue — the same reasoning as in forty requests at once with asyncio. And remember it is per worker: four uvicorn workers at 40 each is 160 concurrent upstream calls, which is the number that has to fit inside your provider’s limit.
Deploying it without a buffer in the way
| Layer | Description |
|---|---|
| uvicorn | Streams correctly by default. Run it with --workers N in production rather than --reload, and remember each worker holds its own client and connection pool, so max_connections is per worker. |
| nginx | Buffers proxied responses by default. Either honour X-Accel-Buffering: no (it does, for proxied responses) or set proxy_buffering off and proxy_read_timeout above your longest answer. |
| GZip middleware | Starlette’s GZipMiddleware compresses the whole response, which defeats streaming. Exclude the route, or drop the middleware — SSE token frames are small and compress badly anyway. |
| CDN or platform edge | Many buffer responses under some size, and some have a hard response-duration cap measured in seconds. Test the deployed URL with curl -N; this is not something to infer from documentation. |
Errors after the headers have gone
Once StreamingResponse has started, the status code is 200 and cannot be changed. An exception raised inside the generator after that point does not become a 500 — it closes the connection mid-stream, and the browser sees a truncated answer with no indication that anything went wrong.
- Validate before you start streaming. Anything that can be checked — the model id, the prompt length, the user’s quota — is checked in the route handler, before
StreamingResponseis constructed, so it can still be a 400 or a 429. - Send failures as events, not exceptions. The
errorevent above is why: the client can render “the answer was cut short” instead of showing half a sentence as if it were complete. - Always emit a terminal event. A client that has seen
doneknows the answer is whole. A client that has seen the socket close knows nothing. - Log the outcome, not the request. The interesting fields are known at the end: tokens delivered, whether the client disconnected, total duration. Logging every model call has the record to write.
Top comments (0)