Streaming Responses Have a Silent Gap: A Three-Myth FAQ

I was watching a chat UI print one character, stall, print two, stall again. The first guess is a dead server. The second guess is a broken proxy. Both guesses miss.

The stream is alive. The tokens just arrive in the wrong rhythm.

This FAQ busts three streaming myths I keep seeing in free-tier integrations. MonkeyCode has a free model access path and a free server option, which makes an experiment like this cheap to run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You do not need MonkeyCode to run the probe. Any endpoint that streams SSE events works.

Myth 1: Streaming always makes the response faster

Streaming moves the first byte earlier. It does not promise a shorter total trip. A stream can take longer than a buffered response and still feel faster.

Measure two numbers, not one:

• First-token latency: time from request until the first real SSE data line.
• Inter-token gap: time between consecutive SSE data lines.

Both matter. Chat UIs care about the first. Linters and validators care about the second. A whole paragraph can arrive, then silence. That silence is part of the latency budget.

Correct mental model: streaming fast means low first-token latency. Total duration belongs to the buffered path. Do not call it faster until you measure both.

Myth 2: The end of the stream means the answer is complete

An SSE stream ends when the server closes the connection. That is not the same as a finished model response. Proxies, CDNs, and free-tier gateways can close long-lived streams.

Many chat-completions providers use a [DONE] sentinel as their end-of-stream marker. Check for it before you treat the response as complete. The stream may also end inside a JSON fragment. Count the closing brace, or prefer a client that understands event boundaries.

Correct mental model: HTTP response complete is not model output complete. A stream without its sentinel is an interrupted candidate, not a result.

Myth 3: A quiet stream is a slow one, so wait longer

A quiet stream can be a blocked stream. The socket is open, but no token is coming. The server might be scheduling work, GCing, waiting on a queue, or stuck behind a hidden buffer.

The fix is an idle timeout, not a total timeout. Track the time since the last SSE event. If that gap passes your threshold, abort and plan a retry. A total timeout kills busy streams that still make progress.

Try this rule: if one inter-token gap is more than five times the median gap, your client should record it. The mean hides it. The max catches it too late.

Correct mental model: silence is a metric, not a verdict. Record the gap length, abort deliberately, and classify the failure before retrying.

The probe

I wrote a small Node script that watches a streaming chat completion. It parses SSE data lines, records inter-token gaps, and prints the distribution. Run it against an endpoint you own.

// stream-probe.mjs
// ENV: ENDPOINT, TOKEN, MODEL (optional), IDLE_TIMEOUT_MS (optional), MAX_MS (optional)
import { performance } from 'node:perf_hooks';

const {
  ENDPOINT,
  TOKEN,
  MODEL = 'placeholder-model',
  IDLE_TIMEOUT_MS = 5000,
  MAX_MS = 120000
} = process.env;

if (!ENDPOINT || !TOKEN) {
  console.error('Set ENDPOINT and TOKEN.');
  process.exit(1);
}

const controller = new AbortController();
const started = performance.now();

let bytes = 0;
let firstEventAt = null;
let lastEventAt = null;
let doneSeen = false;
const gaps = [];
let buffer = '';
let idleTimer;
let timer = setTimeout(() => abort(`stream longer than ${MAX_MS}ms`), Number(MAX_MS));

function abort(reason) {
  if (!controller.signal.aborted) controller.abort(new Error(reason));
}

function resetIdleTimer() {
  clearTimeout(idleTimer);
  idleTimer = setTimeout(() => abort(`no SSE event within ${IDLE_TIMEOUT_MS}ms`), Number(IDLE_TIMEOUT_MS));
}

function handleSse(raw) {
  buffer += raw;
  let sep;
  while ((sep = buffer.indexOf('\n\n')) !== -1) {
    const event = buffer.slice(0, sep);
    buffer = buffer.slice(sep + 2);
    const data = event.split('\n').find((line) => line.startsWith('data:'))?.slice(5).trimStart();
    if (!data) continue;
    const now = performance.now();
    if (firstEventAt === null) firstEventAt = now - started;
    gaps.push(now - (lastEventAt ?? started));
    lastEventAt = now;
    if (data === '[DONE]') doneSeen = true;
    resetIdleTimer();
  }
}

const response = await fetch(ENDPOINT, {
  method: 'POST',
  headers: {
    'content-type': 'application/json',
    authorization: 'Bearer ' + TOKEN
  },
  body: JSON.stringify({
    model: MODEL,
    messages: [{ role: 'user', content: 'Write one short paragraph about measuring streams.' }],
    stream: true
  }),
  signal: controller.signal
});

if (!response.ok) throw new Error('HTTP ' + response.status + ' ' + response.statusText);

const reader = response.body.getReader();
const decoder = new TextDecoder();

try {
  resetIdleTimer();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    bytes += value.byteLength;
    handleSse(decoder.decode(value, { stream: true }));
  }
} catch (err) {
  if (err.name !== 'AbortError') throw err;
  console.error('aborted:', err.message);
} finally {
  handleSse('\n\n');
  clearTimeout(idleTimer);
  clearTimeout(timer);
}

const durationMs = performance.now() - started;
const sorted = [...gaps].sort((a, b) => a - b);
const n = sorted.length;
const median = n ? n % 2 ? sorted[Math.floor(n / 2)] : (sorted[n / 2 - 1] + sorted[n / 2]) / 2 : 0;
const p90 = n ? sorted[Math.min(n - 1, Math.floor(n * 0.9))] : 0;

console.log(JSON.stringify({
  status: response.status,
  durationMs: Math.round(durationMs),
  firstEventMs: firstEventAt === null ? null : Math.round(firstEventAt),
  bytes,
  complete: doneSeen,
  gapsMs: {
    n,
    min: Math.round(sorted[0] ?? 0),
    median: Math.round(median),
    p90: Math.round(p90),
    max: Math.round(sorted[n - 1] ?? 0)
  }
}, null, 2));

Reading the output

The script prints JSON. The shape looks like this:

{
  "status": 200,
  "durationMs": 9120,
  "firstEventMs": 312,
  "bytes": 4102,
  "complete": true,
  "gapsMs": {
    "n": 74,
    "min": 8,
    "median": 12,
    "p90": 35,
    "max": 4200
  }
}

Treat those numbers as a shape, not a benchmark. Your provider will show a different story. Look at the relationship between the fields:

• High firstEventMs means the server queued the whole response before sending.
• One big max is a scheduling hiccup. Record it, keep going.
• Many high inter-token gaps mean a stalling stream.
• complete: false means the sentinel never arrived, even if HTTP says 200.

Decision table

You see	Likely cause	What helps
High first-event latency	Network or server queue	Warm up, then stream again
One huge gap	Server scheduling or GC	Log it; do not retry blindly
Repeated large gaps	Proxy buffering or read backpressure	Shorten prompt, read chunks faster
complete: false	Interrupted stream	Idle timeout, surface interrupted state

Limitations

This probe measures what your client receives, not what the model generates. A CDN that batches tokens will inflate your gaps. That is still useful for UX, but it is not a server benchmark.

Do not use it as a load test. One request at a time. Free tiers will not enjoy a hammering client.

Do not use streaming when you are writing output to a file. Buffer the whole JSON and validate it. Streaming earns its complexity in chat, typeahead, and long-running generation.

Bottom line

Before you file an outage ticket, ask for the inter-token gap histogram. A stream can be alive and still be unusable. Measure the rhythm, set an idle timeout, and treat an ending without a sentinel as incomplete.

The myth is that a stream either works or fails. The reality is that a stream degrades between pixels. That is where you debug next.