Why Do AI Models Stream Their Responses?

AI models stream their responses because they generate text one small piece at a time, and waiting for the whole answer would leave you staring at a blank screen.

AI is now part of daily work. We use it to write code, research topics, and analyse data. Whether you use ChatGPT, Claude, or Gemini, one thing looks the same: the answer appears word by word, as if someone is typing it live.

That is not a design gimmick. It is a direct result of how these models produce text. To see why, let's follow a single prompt from the moment you press Enter to the moment the answer shows up on your screen.

Our example prompt: "Why do cats purr?"

What happens when you press Enter

Before the model writes a single word, it has to turn your text into numbers and read through all of it.

Inference is the name for this whole process. Training is when a model learns its weights (the numbers that store what it knows). Inference is when it uses those fixed weights to answer you. It does not learn from your question; it just runs.

Step 1: Tokenization. Models do not read words. They read tokens, which are chunks of text. A token can be a whole word, part of a word, or punctuation. Our prompt might be split like this:

["Why", " do", " cats", " purr", "?"]

Each token maps to an integer ID in the model's vocabulary. The exact split depends on the tokenizer, so two models can split the same sentence differently. This is also why API pricing, context limits, and generation speed are all measured in tokens.

Step 2: Prefill. The model now runs all the prompt tokens through its layers to build an understanding of the question. Because every prompt token already exists, the GPU can process them in parallel. This step is fast compared to what comes next.

During prefill, the model also saves some intermediate results for each token, called keys and values, in a memory area called the KV cache. Keep that in mind; it matters in a moment.

At the end of prefill, the model scores every token in its vocabulary and picks the first word of its answer, for example "Cats".

Decode: one token at a time

The answer is written one token per loop, and each loop has to wait for the one before it.

This stage is called decode. The model cannot produce a full sentence in one go. It picks one token, adds it to the sequence, then uses the updated sequence to pick the next one:

Cats
Cats purr
Cats purr when
Cats purr when they

This is called autoregressive generation: every new token depends on all the tokens before it. The loop looks like this:

1. Run the newest token through the model (a forward pass).
2. Score every possible next token and pick one.
3. Add it to the sequence.
4. Stop if it is an end-of-sequence token or the output limit is reached. Otherwise, repeat.

The KV cache from prefill saves work here. Instead of recomputing keys and values for every earlier token on every loop, the model reuses them and only computes them for the newest token.

Even so, decode is the slow part. Prefill can process the prompt in parallel because all of it already exists. Decode cannot, because token 10 does not exist until token 9 has been chosen. On top of that, every loop has to read the model's huge set of weights from GPU memory, and that memory traffic, not raw computation, is often the bottleneck.

The waiting problem

Without streaming, you would wait for the very last token before seeing the first one.

Let's put rough numbers on it. Say a model generates about 50 tokens per second, and a typical answer is around 600 tokens (roughly 450 words). That is about 12 seconds of decoding. Add network time and time waiting in a queue for a free GPU, and you could be looking at a blank screen for 13 to 15 seconds.

Most people assume something broke long before then.

This is why AI products track a metric called Time to First Token (TTFT): the time between sending a request and receiving the first piece of the response. It covers network travel, queueing, and prefill. With streaming, TTFT is what the user actually feels, and it is usually well under a second or two.

	Without streaming	With streaming
First text appears after	~13 s (whole answer)	~0.5 to 1 s (first token)
Total generation time	~12 s	~12 s
What the user sees meanwhile	Blank screen or spinner	Answer forming as they read

Figures are illustrative; real speeds depend on the model, hardware, and load.

What streaming actually changes

Streaming does not make the model faster. It changes when you receive the output.

Streaming means the server sends each piece of the answer to your app as soon as it is generated, instead of holding everything until the end. Your browser renders "Cats purr" while the model is still deciding what comes after "when".

That gives you three real benefits:

• It feels fast. You start reading within a second, even if the full answer takes 12.
• You can bail early. If the model misunderstood you, you see it in the first line and hit Stop, instead of waiting for (and paying for) a useless answer.
• It matches reading speed. People read at roughly 4 to 5 words per second. A model producing 30 to 40 words per second stays comfortably ahead, so the text never feels like it is stalling.

How the stream reaches your screen

Most AI chat apps use Server-Sent Events, a plain HTTP connection that stays open while the server pushes chunks.

Server-Sent Events (SSE) keep one HTTP response open and let the server send a series of small events down it. Traffic flows one way, server to client, which fits chat well: you send one prompt, then the server does all the talking. On the wire, it looks like this:

data: {"text":"The"}

data: {"text":" sky"}

data: {"text":" appears"}

data: [DONE]

Each event starts with data: and ends with a blank line.

WebSockets open a two-way connection where both sides can send at any time. That is overkill for text chat, but useful for things like real-time voice assistants, where you keep sending audio while the model keeps answering.

Here is a minimal TypeScript client that reads an SSE stream with fetch and renders text as it arrives. The { text } payload shape is a simplified example; each provider's SDK has its own event format.

interface StreamChunk {
  text: string;
}

interface StreamCompletionOptions {
  url: string;
  prompt: string;
  onToken: (text: string) => void;
  signal?: AbortSignal;
}

const SSE_DATA_PREFIX = "data: ";
const STREAM_DONE_MARKER = "[DONE]";

function parseSseEvent(rawEvent: string): StreamChunk | null {
  const dataLine = rawEvent
    .split("\n")
    .find((line) => line.startsWith(SSE_DATA_PREFIX));

  if (!dataLine) return null;

  const payload = dataLine.slice(SSE_DATA_PREFIX.length);
  if (payload === STREAM_DONE_MARKER) return null;

  return JSON.parse(payload) as StreamChunk;
}

export async function streamCompletion(
  options: StreamCompletionOptions,
): Promise<void> {
  const { url, prompt, onToken, signal } = options;

  const response = await fetch(url, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt, stream: true }),
    signal,
  });

  if (!response.ok || !response.body) {
    throw new Error(`Stream request failed with status ${response.status}`);
  }

  const reader = response.body
    .pipeThrough(new TextDecoderStream())
    .getReader();
  let pendingText = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;

    pendingText += value;
    const completeEvents = pendingText.split("\n\n");
    pendingText = completeEvents.pop() ?? ""; // keep any partial event

    for (const rawEvent of completeEvents) {
      const chunk = parseSseEvent(rawEvent);
      if (chunk) onToken(chunk.text);
    }
  }
}

Two details worth noticing. Network packets do not line up with events, so the client keeps the leftover partial event in pendingText until the rest arrives. And passing an AbortSignal is what powers the Stop button: aborting closes the connection, and well-built servers stop generating when the client disconnects.

Buffering is the last knob. The server does not have to send every token as its own event. It can group a few tokens per chunk:

• Smaller, more frequent chunks make the text appear smoother.
• Larger chunks cut the overhead of sending many tiny messages.

So there is a tradeoff between how responsive the stream feels and how much network overhead it costs.

When you should not stream

Streaming is for humans watching a screen; when no human is watching, it often just adds complexity.

• Structured output. If your code needs a complete JSON object to parse, half a JSON object is useless. Wait for the full response (or use a parser built for partial JSON).
• Background and batch jobs. Summarising 10,000 support tickets overnight has no one waiting on the first token. A single complete response is simpler to handle, retry, and log.
• Very short answers. A yes/no classification finishes in a fraction of a second. The extra connection handling buys nothing.
• Moderation before display. If every answer must pass a safety or compliance check before a user sees it, you need the full text first, or a pipeline that checks chunks as they arrive.

Wrapping up

AI models stream because decode is a sequential, token-by-token loop, and streaming turns that unavoidable wait into something you can read through.

Here is the full journey of "Why do cats purr?":

Text → Tokens → Prefill → Decode → Stream → Your screen

The model's speed does not change. What changes is that you see progress within a second instead of staring at a spinner for ten. Next time you watch an answer type itself out, you will know that is the decode loop, running live.

---

Further reading:What is LLM Inference