A language model produces its answer one token at a time. You can either buffer — wait for the whole response, then show it — or stream — push each token to the client the moment it's generated and render it immediately. Both finish at the exact same total time. But one of them feels 5–10x faster, and I built a side-by-side race to show exactly why, plus the client reader loop that makes real streaming work. Here's the whole thing.
Two latencies, not one
Streaming is only interesting because there are two latencies. Time-to-first-token (TTFT) is how long until the first content appears; time-to-complete is how long until the last. Buffering makes them equal — you see nothing until the end. Streaming decouples them: TTFT drops to a few hundred milliseconds while total stays the same.
buffered: TTFT = total = 3000 ms (blank, then a wall of text)
streamed: TTFT = 300 ms, total = 3000 ms (types out live)
Perceived latency is what you're optimizing
Humans don't measure throughput; they measure how long until something happened. A screen that starts filling at 300 ms feels responsive even if it keeps writing for three seconds; a blank screen for three seconds feels broken. So felt speed tracks TTFT, not total — and 3000/300 reads as roughly 10x faster for zero change in the actual work done. When TTFT is already most of the total (a very slow model) the multiplier shrinks, but you almost never regret streaming.
The transport: SSE, and how providers frame it
The plumbing is Server-Sent Events — one long-lived HTTP response whose body is a series of small data: frames, each ended by a blank line. The server writes-and-flushes a frame per token; the one detail that bites everyone is the flush, because a proxy or framework buffer will silently hold the whole body and un-stream your stream.
res.writeHead(200, { "Content-Type": "text/event-stream" });
for await (const token of model.generate(prompt)) {
res.write("data: " + JSON.stringify({ token }) + "\n\n");
res.flush?.(); // flush! or a proxy buffers it all
}
res.write("data: [DONE]\n\n");
Providers layer their own shape on top — OpenAI-style data: {...} lines ending in data: [DONE], Anthropic's typed content_block_delta events with no [DONE] sentinel — but it's the same idea: many small chunks instead of one big body. Match your reader to the provider.
The reader loop, and the bug everyone hits
On the client you fetch the endpoint and read res.body as a ReadableStream. Each read() gives you whatever bytes arrived — not aligned to token or line boundaries — so you decode into a running buffer and pull complete frames out of it.
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
let idx;
while ((idx = buffer.indexOf("\n\n")) !== -1) { // one complete frame
const frame = buffer.slice(0, idx);
buffer = buffer.slice(idx + 2); // keep the remainder!
const line = frame.replace(/^data: /, "");
if (line === "[DONE]") return finish();
onToken(JSON.parse(line).token); // one line == one COMPLETE JSON
}
}
The classic first bug is parsing a chunk directly. Chunks split mid-line, so you scan for the blank-line delimiter, process each complete frame, and leave the unfinished tail in the buffer for the next read(). The fix is a buffer, not a bigger regex.
Render partial structured output safely
For prose, rendering is trivial — append each token. For JSON it's a trap: the accumulated buffer is invalid for almost the entire stream, so calling JSON.parse every tick throws until the final byte.
// DON'T JSON.parse the partial each tick — it throws until the last byte.
// Render only fields that have FULLY arrived:
function renderPartial(buf) {
for (const [k, v] of completedFields(buf)) setField(k, v);
}
Extract complete "key":value pairs (or feed the buffer to a tolerant streaming parser) so a half-arrived field just waits its turn instead of crashing the UI.
Cancellation is the underrated win
Because you hold the connection open, you can abort it. controller.abort() tears down the fetch; the server sees the client drop and stops generating, so you stop paying for tokens. And whatever streamed already stays on screen and is still useful — the opposite of a buffered request, where aborting leaves you with nothing.
const controller = new AbortController();
fetch(url, { signal: controller.signal });
stopBtn.onclick = () => controller.abort(); // server halts; partial text stays
Skip streaming when you need one validated object, for batch jobs, for tool-only turns the user never reads, or for tiny responses where buffering is already instant.
Run the streamed-vs-buffered race and watch which one feels fast:
https://dev48v.infy.uk/prompt/day41-streaming.html
Top comments (0)