Streaming an LLM response looks like the easiest feature in your product. The SDK gives you an iterator, you print tokens, it works on your laptop in under a minute.
Then you deploy it, and one of these happens: the response arrives all at once after 40 seconds instead of token by token. Or the connection drops mid-answer with no error anywhere in your logs. Or a user closes the tab and you keep paying for 60,000 tokens nobody will ever read.
None of those are model problems. They're the six things that break in the space between api.anthropic.com and a browser tab — a proxy, a load balancer, an HTTP client, and a JavaScript API that each have opinions about long-lived responses.
I teach AI engineering at Cursuri-AI.ro, an AI education platform in Eastern Europe, and this is the failure surface I see most often in production reviews — because every layer of it looks fine in isolation. Here's the whole path, one break at a time.
First: streaming is not a UX nicety anymore
It's worth being clear about why you're doing this, because it changes how you treat failures.
The perceived-latency argument is the famous one, and it's real. But on current models, streaming is also a correctness requirement for a growing share of requests. Claude Opus 5, Sonnet 5, and the 4.6/4.7/4.8 family support up to 128K output tokens, and Anthropic's SDKs will refuse a non-streaming request they estimate will exceed the connection's tolerance — the Python SDK raises a ValueError rather than let you build something that hangs and drops. The default client timeout is 10 minutes (note the units differ by SDK: seconds in Python and Ruby, milliseconds in TypeScript), and with thinking on by default on Opus 5, a hard task can spend minutes generating before the first visible character.
So: any request with a large max_tokens, a long input, or a reasoning-heavy prompt is a streaming request. Not for the animation — for the connection.
import anthropic
client = anthropic.Anthropic()
with client.messages.stream(
model="claude-opus-5",
max_tokens=64000,
thinking={"type": "adaptive", "display": "summarized"},
messages=[{"role": "user", "content": "Analyze this incident report..."}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
final = stream.get_final_message()
One thing in that snippet is not cosmetic. On Opus 5, Opus 4.8/4.7, Fable 5, and Sonnet 5, the thinking display default is "omitted" — the thinking block opens, emits a single signature_delta, and closes, with no thinking_delta events. If you stream reasoning to users and leave the default in place, your UI shows a long dead pause and then a wall of text. display: "summarized" is what gives you something to render during the think. It costs nothing extra: thinking happens and is billed identically under every display setting.
Break #1: your proxy is holding the tokens hostage
This is the number-one "streaming doesn't work in production" bug, and the tell is unmistakable: locally you see tokens appear one by one; deployed, the whole response lands at once at the end.
Nothing in your app is wrong. nginx is buffering. From the nginx documentation, proxy_buffering defaults to on, and when buffering is enabled nginx reads the response from the upstream into its own buffers before passing it along. When it's off, "the response is passed to a client synchronously, immediately as it is received."
There are two ways to fix it, and you want the second one.
location /api/chat/stream {
proxy_pass http://app;
proxy_buffering off;
proxy_read_timeout 300s; # default is 60s
}
That works, but it puts a per-route infrastructure rule in a file your application team doesn't own. nginx also honors a response header: "Buffering can also be enabled or disabled by passing yes or no in the X-Accel-Buffering response header field." So the streaming endpoint can turn buffering off for itself:
from fastapi.responses import StreamingResponse
def sse_response(generator):
return StreamingResponse(
generator,
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no", # the important one
},
)
The header travels with the endpoint. Add a route, get correct behavior. No config drift, no "works in staging" incident six months later when someone rebuilds the ingress.
While you're there, note proxy_read_timeout — it defaults to 60s in nginx. A model thinking hard for 90 seconds before its first token will be cut off by your own proxy, and the error you'll see is a generic upstream timeout with no mention of streaming at all.
Break #2: the load balancer kills you during the pause
Same class of bug, one layer out, and it bites hardest exactly on the requests you care most about.
An AWS Application Load Balancer's idle_timeout.timeout_seconds attribute has a valid range of 1–4000 seconds and a default of 60 (see LoadBalancerAttribute in the ELB API reference). "Idle" means no bytes in either direction. An LLM that thinks for 75 seconds before emitting its first token produces exactly that: a silent, live, perfectly healthy TCP connection that your load balancer decides is dead.
You can raise the timeout, and you probably should. But raising it alone is a fragile fix, because it only pushes the cliff further out. The robust fix is to make sure the connection is never actually idle — which SSE has a purpose-built mechanism for.
The SSE wire format treats a line beginning with a colon as a comment. MDN puts it plainly: "A colon as the first character of a line is in essence a comment, and is ignored," and "The comment line can be used to prevent connections from timing out; a server can send a comment periodically to keep the connection alive."
import asyncio
async def sse_stream(request, params):
queue: asyncio.Queue = asyncio.Queue()
async def heartbeat():
while True:
await asyncio.sleep(15)
await queue.put(": keepalive\n\n") # ignored by every SSE client
# ... producer task pushes real events onto the same queue ...
Fifteen seconds is a good default — comfortably under a 60-second idle timeout, cheap enough that nobody notices. The bytes are discarded by the client and keep every hop on the path convinced the connection is alive.
Anthropic does the same thing on its side, by the way: "Event streams may also include any number of ping events." Your parser needs to expect them and ignore them, which leads directly to the next break.
Break #3: the errors arrive with HTTP 200
Here's the part that catches teams who have otherwise done everything right.
Once the stream has started, the HTTP status code is already sent. It's 200. It will stay 200 no matter what happens next. Failures after that point arrive as events inside the body, and if your parser only handles the happy path, they vanish silently.
The Messages API documents this directly: "The API may occasionally send errors in the event stream. For example, during periods of high usage, you may receive an overloaded_error, which would normally correspond to an HTTP 529 in a non-streaming context":
event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}
If you're using the SDK helpers this is handled for you. If you're parsing raw SSE — which you are, at the browser end, and often at the server end too — you need to branch on it explicitly.
Two more things belong in the same parser:
Unknown event types must not throw. The docs are explicit: "In accordance with the versioning policy, new event types may be added, and your code should handle unknown event types gracefully." A parser with an else: raise in it is a scheduled outage with an unknown date. Log and skip.
stop_reason is not always end_turn. On Opus 4.7 and later, safety classifiers can decline a request with HTTP 200 and stop_reason: "refusal", with a category in stop_details. Check stop_reason before you read content. (stop_details is populated only for refusals and is null for every other stop reason, so guard before reading it.) On Opus 5 and Fable 5 you can also opt into server-side fallbacks — betas: ["server-side-fallback-2026-07-01"] with fallbacks: "default" — which routes by refusal category. If you do, expect a fallback content block in the stream at each model boundary: a content_block_start / content_block_stop pair with no deltas between them. A parser that assumes every block has deltas will choke on it.
Here's the shape of a parser that survives all of it:
for event in raw_events:
t = event.get("type")
if t == "content_block_delta":
d = event["delta"]
if d["type"] == "text_delta":
yield sse("token", {"text": d["text"]})
elif d["type"] == "thinking_delta":
yield sse("thinking", {"text": d["thinking"]})
# input_json_delta / signature_delta: accumulate, don't render
elif t == "message_delta":
# NOTE: usage counts in message_delta are CUMULATIVE, not incremental
usage = event.get("usage") or {}
output_tokens = usage.get("output_tokens", output_tokens)
stop_reason = event["delta"].get("stop_reason", stop_reason)
elif t == "error":
yield sse("error", {"code": event["error"]["type"]})
return
elif t in ("ping", "content_block_start", "content_block_stop", "message_start", "message_stop"):
pass
else:
log.info("unknown stream event, ignoring", extra={"event_type": t})
That comment about cumulative usage is worth internalizing — the docs flag it with a warning box. Adding up message_delta usage across events gives you a token count that grows quadratically and a cost dashboard that is confidently wrong.
Break #4: the user left and you're still paying
A user asks a question, gets three sentences in, sees the answer isn't what they wanted, and closes the tab.
What happens to the 60,000-token generation you started? In a lot of production apps: it runs to completion, bills in full, and writes its result to a database row nobody will read. Multiply by your bounce rate.
Cancellation has to be propagated explicitly, at every hop. Server side, Starlette (and therefore FastAPI) exposes the disconnect signal:
async def generate(request, params):
with client.messages.stream(**params) as stream:
for text in stream.text_stream:
if await request.is_disconnected():
break # exiting the with-block closes the upstream connection
yield sse("token", {"text": text})
Exiting the context manager is what actually matters — it closes the HTTP connection to Anthropic, and generation stops. A break without the context manager, or a background task that owns the stream and outlives the request, keeps burning tokens.
Browser side, fetch cancellation goes through AbortController:
const controller = new AbortController();
stopButton.onclick = () => controller.abort();
const res = await fetch("/api/chat/stream", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt }),
signal: controller.signal,
});
Aborting closes the TCP connection, which is what your is_disconnected() check observes. The chain only works if every link is present.
Break #5: EventSource can't do what your chat app needs
The browser's native SSE client is EventSource, and reaching for it is the obvious move. It's also, for most LLM chat UIs, the wrong one.
The constructor takes a URL and an options object whose only meaningful member is withCredentials. There is no place to put an HTTP method, a request body, or headers. That rules out sending a prompt as a POST body and rules out an Authorization header. And its built-in auto-reconnect — "By default, if the connection between the client and server closes, the connection is restarted" — is actively hostile for LLM generation: a dropped connection silently fires a brand-new generation, at full cost, with no memory of the tokens already delivered. One flaky connection becomes a duplicate bill.
Use fetch with a ReadableStream reader instead. You get POST, headers, AbortController, and — crucially — no reconnect you didn't ask for:
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buf = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buf += value;
const frames = buf.split("\n\n");
buf = frames.pop(); // keep the incomplete tail
for (const frame of frames) {
const dataLines = frame
.split("\n")
.filter((l) => l.startsWith("data:")) // ':' comment lines fall out here
.map((l) => l.slice(5).trimStart());
if (!dataLines.length) continue; // heartbeat frame
handle(JSON.parse(dataLines.join("\n")));
}
}
Two details that cause real bugs. First, TCP does not deliver your frames whole — reader.read() will hand you half an event, and the buf.split("\n\n") / buf.pop() pattern is what keeps you from parsing a truncated JSON object. Second, multiple consecutive data: lines in one frame are concatenated with newlines between them, per the spec; joining them with "" corrupts any payload containing a newline.
Break #6: partial output is a state you have to design for
Every stream can end three ways: complete, cancelled, failed mid-flight. Most codebases only persist the first.
That produces two ugly symptoms. A user refreshes after a network blip and their half-written answer is simply gone — the connection is stateless, so the tokens are too. And your cost tracking under-reports, because you only record usage on clean completions, while every abandoned generation was billed in full.
The fix is not resumable streams — that's a hard, mostly unnecessary feature. It's persisting the assistant message as it's produced, with a status:
- Insert the message row with
status = "streaming"before the first token. - Flush accumulated text to it periodically (every ~50 tokens or every second — not every token, unless you enjoy write amplification).
- On
message_stop, setstatus = "complete"and write the final usage fromget_final_message(). - On disconnect or error, set
status = "partial"or"failed"and record whatever usage you observed.
Now a refresh renders the partial answer with an honest "generation was interrupted" marker, and your cost dashboard counts abandoned generations — which is the number that tells you whether Break #4 is costing you real money. Wiring persistence, usage accounting, and streaming state together is exactly the kind of plumbing we build end to end in our course on shipping a production AI SaaS, because it's where "demo works" and "product works" actually diverge.
The checklist
Everything above, compressed into things you can go verify this afternoon:
| # | Check | Where |
|---|---|---|
| 1 |
proxy_buffering off or X-Accel-Buffering: no on streaming routes |
nginx / app |
| 2 | Proxy read timeout and LB idle timeout raised above your p99 time-to-first-token | infra |
| 3 | Heartbeat comment frames every ~15s | app |
| 4 | Parser handles ping, error, and unknown event types without throwing |
app + client |
| 5 |
stop_reason checked before reading content; stop_details guarded |
app |
| 6 |
message_delta usage treated as cumulative, not incremental |
app |
| 7 |
thinking.display: "summarized" if you render reasoning |
app |
| 8 | Client disconnect propagated to the upstream stream (context manager exits) | app |
| 9 |
AbortController wired to a visible stop control |
client |
| 10 | Partial messages persisted with a status; abandoned usage recorded | app |
If you want to know whether any of this is actually costing you, the honest test is a load test with a 30% mid-stream abandonment rate, run against staging, with token usage measured at both ends. The gap between what your dashboard reports and what your Anthropic console reports is the size of the problem.
Streaming is where a lot of AI products quietly lose both money and trust — and it's a plumbing problem, not a prompting one. If you want the full path from a first API call to an application that holds up under real traffic, that's the arc of our Advanced LLM Integration in Production course; the failure-mode discipline behind it — knowing whether a change actually helped — comes from LLM Evaluation and Testing. And if your next feature is voice, every one of these six breaks gets harder, which is its own topic in Voice AI and Realtime Multimodal Agents.
Sources: Streaming Messages — Claude API docs · Using server-sent events — MDN · ngx_http_proxy_module — nginx · LoadBalancerAttribute — AWS ELB API Reference
Top comments (0)