Streaming LLM answers from FastAPI with SSE: sources at 52 ms, and one time.sleep that pushed it to 3.3 s

#fastapi #python #llm #rag

A RAG answer takes seconds to finish. Users judge the wait by when something appears. So: what does streaming change in a FastAPI endpoint, and what breaks it?

Short answer: send the retrieved sources as the first SSE event, then the tokens. In my test the first byte arrived after 52 ms instead of 1,664 ms, and the first token after 453 ms. The total time stayed the same (1,673 ms vs 1,664 ms). The most common way to break it is a blocking call inside an async generator: with 5 clients the median first byte rose to 3,280 ms.

Tested on a 2 vCPU GitHub runner (CPU only), Python 3.12, fastapi 0.161.1, starlette 1.9.0, uvicorn 0.54.0, httpx 0.28.1, real uvicorn server on localhost. Retrieval (50 ms), LLM first token (400 ms) and token rate (60 tokens x 20 ms) are simulated with asyncio.sleep, so absolute numbers are not what a real model gives.

Setup

The endpoint is a plain StreamingResponse with media_type="text/event-stream" and hand-formatted events. Recent FastAPI versions also ship a dedicated EventSourceResponse in fastapi.sse; I did not test it, and the manual version works the same way on older releases.

The stream has three kinds of event:

sources: one event with the retrieved chunks (sent right after retrieval).
token: one event per token.
done: a final marker so the client knows the stream ended cleanly.

The comparison endpoint does the same work and returns one JSON body at the end. A client script (httpx) times the first byte, the sources event, the first token event and the end. Each case ran in a real uvicorn process.

Results

All times in milliseconds (median over runs; last column is the worst first byte).

Case	First byte	Sources	First token	Total	Worst first byte
Non-streaming JSON, 1 client	1664	-	-	1664	1666
SSE sources-first, 1 client	52	52	453	1673	53
SSE sources-first, 20 concurrent	71	71	472	1715	86
SSE, 5 concurrent (control)	54	54	455	1685	56
SSE + blocking `time.sleep`, 5 concurrent	3280	3280	3681	4894	6510
SSE + `GZipMiddleware`, 1 client	52	52	453	1673	53

What the numbers mean

Streaming moves the first visible byte, not the finish. First byte improved by 1,612 ms. The first token showed up 401 ms after the sources, which is the simulated model delay. Total time did not improve.
Sources first is nearly free. Retrieval is done before generation starts, so the user sees the evidence 52 ms in while the model is still thinking.
20 concurrent streams were fine (71 ms median first byte), as long as the event loop is not blocked.
Blocking the loop is the real hazard. The time.sleep variant does the same simulated work with a blocking call. With 5 clients the median first byte was 3,280 ms, about 61 times the control (54 ms), and the worst was 6,510 ms. One client's stream stalls everyone else's. In real code the culprit is a sync SDK call, a CPU-heavy rerank or a blocking DB driver inside async def. Use the async client, or run sync work in a thread (run_in_threadpool, asyncio.to_thread).
GZip did not buffer the stream here. Starlette's GZipMiddleware leaves text/event-stream alone, and all 62 events arrived as separate chunks, with first byte unchanged at 52 ms. Other middleware, or a proxy, may behave differently.

Caveats

The model and retrieval are fakes. A real provider adds network jitter and a longer, variable first-token time; the shape of the gain (first byte early, total unchanged) should hold, the numbers will not.
Localhost has no proxy. nginx buffers responses by default; the X-Accel-Buffering: no header below asks it not to, but I did not test it.
I did not test client disconnects. In production, stop the upstream LLM call when the client leaves, or you pay for tokens nobody reads.
The blocking case is deliberately extreme (every token blocks). Shorter blocks hurt less but still add to every stream's latency.

Copyable code

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

def sse(event: str, data) -> bytes:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n".encode()

@app.get("/ask")
async def ask(q: str):
    async def events():
        chunks = await retrieve(q)             # async vector search
        yield sse("sources", chunks)           # evidence first
        async for token in llm_stream(q, chunks):  # async client, never blocking
            yield sse("token", token)
        yield sse("done", {})
    return StreamingResponse(
        events(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

retrieve and llm_stream are your own functions. The full test server and benchmark script are on the original post.

Rule of thumb

Send sources as the first event, then tokens, then an explicit done.
Everything inside the generator must be await-able or pushed to a thread.
Measure first byte under 5-20 concurrent clients, not just one, and put that number in CI.
Test through your real proxy before shipping; localhost results do not cover buffering.

If you want to check the sources the model cites afterwards, I wrote up validating LLM citations in FastAPI.

DEV Community