Streaming LLM Output to the Browser Through Your Own Backend: FastAPI, Server-Sent Events and the Buffering Traps

I'm Aman Kumar. I build an OpenAI-compatible gateway, and one of the most common support questions I see isn't about models at all. It's "streaming works in my terminal, but in the browser the whole answer shows up at once." The model is streaming fine. Something between your backend and the browser is holding the bytes.

This post is the setup I use for relaying a streamed chat completion from a Python backend to a web page, plus the specific places where buffering sneaks in.

Why not call the API from the browser directly?

Because the API key would ship to every visitor. Any key in front-end JavaScript is public. The normal pattern is:

1. The browser sends the user's message to your backend.
2. Your backend calls the LLM API with stream=True, using a key stored in an environment variable.
3. Your backend forwards each text chunk to the browser as it arrives.

Your backend also becomes the place to enforce per-user limits, log usage and pick the model, which you want anyway.

The backend: FastAPI plus the OpenAI SDK

The official openai package works against any OpenAI-compatible endpoint as long as you set base_url. I use the async client so one slow stream doesn't block other requests.

import os, json
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url=os.environ["LLM_BASE_URL"],   # any OpenAI-compatible /v1 URL
    api_key=os.environ["LLM_API_KEY"],
)
MODEL = os.environ["LLM_MODEL"]            # copy the exact ID from your provider's model list

app = FastAPI()

def sse(event: dict) -> str:
    return f"data: {json.dumps(event)}\n\n"

@app.post("/chat")
async def chat(request: Request):
    body = await request.json()
    messages = body["messages"]

    async def gen():
        try:
            stream = await client.chat.completions.create(
                model=MODEL, messages=messages, stream=True,
            )
            async for chunk in stream:
                if await request.is_disconnected():
                    await stream.close()   # stop paying for tokens nobody reads
                    return
                if not chunk.choices:
                    continue
                delta = chunk.choices[0].delta.content
                if delta:
                    yield sse({"type": "text", "text": delta})
            yield sse({"type": "done"})
        except Exception as e:
            yield sse({"type": "error", "message": str(e)[:300]})

    return StreamingResponse(
        gen(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

A few details in there matter more than they look:

• Wrap each chunk as JSON. Model output contains newlines, and a raw newline inside an SSE data: line breaks the framing. Encoding each piece as JSON sidesteps that completely.
• Skip chunks with empty choices. Some providers send a final chunk with only usage data and no choices. Indexing [0] on it throws.
• Check for disconnects. If the user closes the tab, the upstream request keeps generating unless you close it. On metered APIs that's money; on request-limited plans it's quota.
• Send errors as events. Once the response has started with a 200, you can't change the status code. An explicit error event lets the front end show something useful instead of hanging.

The front end: fetch, not EventSource

EventSource only does GET requests and can't send a JSON body, so for a chat endpoint I read the stream with fetch:

async function send(messages, onText) {
  const res = await fetch("/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });
  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buf = "";
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buf += decoder.decode(value, { stream: true });
    let i;
    while ((i = buf.indexOf("\n\n")) !== -1) {
      const line = buf.slice(0, i); buf = buf.slice(i + 2);
      if (!line.startsWith("data: ")) continue;
      const ev = JSON.parse(line.slice(6));
      if (ev.type === "text") onText(ev.text);
      if (ev.type === "error") throw new Error(ev.message);
    }
  }
}

The important part is the buffer. A network read can end in the middle of an event, or contain three events at once. Splitting on the blank line between events and keeping the leftover is what makes this reliable.

The buffering traps

When streaming "doesn't work" in production but works locally, it's almost always one of these.

Nginx proxy buffering. By default Nginx buffers upstream responses. Either send the X-Accel-Buffering: no header (as above) or set proxy_buffering off; for that location. Also raise proxy_read_timeout so long answers aren't cut at 60 seconds.

Compression middleware. Gzip middleware often waits to fill a block before sending anything. Exclude text/event-stream from compression, or don't add GZipMiddleware to the streaming route.

Serverless and some PaaS platforms. Some hosting setups collect the full response before returning it. If your platform doesn't document streaming support for your runtime, test with a tiny endpoint that yields a timestamp every second before blaming your code.

Cloudflare and other CDNs. Streaming generally passes through, but caching rules or response transformations can interfere. Cache-Control: no-cache helps, and make sure the route isn't matched by a "cache everything" rule.

Your own logging. I've seen a middleware that read the full response body to log it, which turned the stream back into one blob. Anything that touches the body needs to be stream-aware.

A quick check from the command line tells you which side the problem is on:

curl -N -X POST https://your-app.example/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Count to 20 slowly"}]}'

-N turns off curl's own buffering. If the text trickles in here but not in the browser, the problem is front-end parsing. If it arrives all at once here too, look at the server and proxies.

Rendering without jank

Appending raw text to a <div> on every token is fine. Re-rendering Markdown on every token gets slow for long answers. What I do is append plain text while streaming and render Markdown once on done, or re-render at most every 100 ms or so. Also, never insert model output as raw HTML; treat it as untrusted.

Per-user limits belong here

Since every request passes through your backend, this is where a simple limiter goes: a counter per user per minute in Redis or memory, checked before you open the upstream stream. It protects you from one user (or one buggy front-end loop) burning through your entire allowance, whether you pay per token or have a fixed monthly request budget.

Checklist

• Key lives only on the server.
• Each SSE event is JSON-encoded and ends with a blank line.
• Empty choices chunks are skipped.
• Client disconnects close the upstream stream.
• Errors go out as events.
• Proxy buffering and compression are off for the route.
• Tested with curl -N before debugging the browser.

Disclosure: I build APIClaw, an OpenAI-compatible gateway with flat monthly pricing. The code above works the same with any provider that speaks the Chat Completions format; just set the base URL, key and model ID.