Streaming changes everything. Or so I thought. Then I measured it. The first token is a lie.
Non-streaming requests hide the real story. They return one big blob. Streaming returns a trickle. That trickle has its own delays. Free servers make those delays worse.
I built a probe. It sends a streaming request. It records every token arrival. Then I pointed it at MonkeyCode's free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The Probe
The script is small. It uses requests and iter_lines. No frameworks. No magic.
#!/usr/bin/env python3
"""Streaming latency probe for chat-completions endpoints."""
import argparse
import json
import statistics
import time
import requests
def p95(values):
if not values:
return None
return sorted(values)[max(0, int(len(values) * 0.95) - 1)]
def run_probe(url, api_key, model, prompt="ping", max_tokens=128):
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
"stream": True,
"max_tokens": max_tokens,
"temperature": 0,
}
start = time.perf_counter()
first_token_ms = None
token_arrivals = []
error = None
done = False
try:
with requests.post(url, headers=headers, json=payload,
stream=True, timeout=60) as resp:
if resp.status_code != 200:
error = f"HTTP {resp.status_code}"
else:
for line in resp.iter_lines():
if not line or not line.startswith(b"data: "):
continue
data = line[6:]
if data == b"[DONE]":
done = True
break
json.loads(data) # raises if malformed
now = time.perf_counter() - start
if first_token_ms is None:
first_token_ms = now * 1000
token_arrivals.append(now * 1000)
except Exception as exc:
error = f"{type(exc).__name__}: {exc}"
total_ms = (time.perf_counter() - start) * 1000
return {
"first_token_ms": first_token_ms,
"total_ms": total_ms,
"tokens": len(token_arrivals),
"inter_token_p95_ms": p95(
[b - a for a, b in zip(token_arrivals, token_arrivals[1:])]
),
"completed": done,
"error": error,
}
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--url", required=True)
parser.add_argument("--api-key", required=True)
parser.add_argument("--model", required=True)
parser.add_argument("--runs", type=int, default=10)
args = parser.parse_args()
results = []
for i in range(args.runs):
result = run_probe(args.url, args.api_key, args.model)
results.append(result)
print(f"run {i + 1}: {result}")
completed = [r for r in results if r["error"] is None and r["completed"]]
if completed:
print("\nSummary (completed runs only):")
print(f" first_token_ms median: "
f"{statistics.median(r['first_token_ms'] for r in completed):.0f}")
print(f" total_ms median: "
f"{statistics.median(r['total_ms'] for r in completed):.0f}")
print(f" inter_token_p95 median: "
f"{statistics.median(r['inter_token_p95_ms'] for r in completed):.0f}")
failed = [r for r in results if r["error"] or not r["completed"]]
if failed:
print(f"\nFailed/incomplete runs: {len(failed)}/{args.runs}")
if __name__ == "__main__":
main()
Run it like this:
python3 stream_probe.py \
--url https://your-endpoint/v1/chat/completions \
--api-key "$KEY" \
--model your-model \
--runs 10
Run it at different hours. Free servers share capacity. Morning and evening look different.
What I Measured
Here is an illustrative run. Your numbers will vary. The shape matters more.
| Metric | Non-streaming | Streaming |
|---|---|---|
| First byte / first token | 1.1 s | 2.4 s |
| Total duration | 3.2 s | 4.8 s |
| Inter-token p95 | n/a | 180 ms |
| Completed runs | 10/10 | 9/10 |
The first token arrived slower than the entire non-streaming request. That surprised me. Streaming added overhead instead of removing it.
Why? Free servers often buffer. They generate the full response internally. Then they replay it as a stream. You pay the latency either way.
Three Numbers That Matter
First token time. This is what users see. A slow first token feels broken. A fast first token feels alive.
Inter-token p95. This is the rhythm. If tokens arrive in bursts, the UI stutters. If they arrive steadily, the UI feels smooth.
Completion rate. Streaming requests can die halfway. No status code. No error. Just silence. Your code must detect that.
The Disconnect Trap
A non-streaming request either returns or fails. A streaming request can do both. It returns 200. Then the connection drops after 20 tokens.
My probe checks for [DONE]. If that marker never arrives, the run is incomplete. Treat that as a failure. Retry the whole request. Do not use the partial output.
Here is the decision rule I use:
-
[DONE]received, no error: success. - No
[DONE], no error: incomplete. Retry once. - HTTP error: fail. Back off.
- Exception mid-stream: fail. Back off.
Who Should Stream
Streaming makes sense for chat UIs. Users expect token-by-token output. It feels responsive even when the model is slow.
Streaming makes less sense for CI and batch jobs. You wait longer for the same tokens. You add failure modes. You need extra parsing. A plain JSON response is simpler.
Free model servers complicate the tradeoff. The first token is slower. The completion rate is lower. Measure before you commit.
Limitations
This probe is not a load test. It sends one request at a time. It does not measure concurrency. It does not test throughput.
It also assumes a standard SSE format. Some endpoints use different delimiters. Check your provider's docs.
I ran this against one free endpoint. Your mileage will vary. Time of day matters. Model size matters. Prompt length matters.
The Takeaway
Streaming is not automatically faster. On free servers, it can be slower. The first token is a lie. Measure the whole stream.
Run the probe. Keep the output. It tells you whether streaming is worth it. It tells you what timeout to set. It tells you when to fall back to non-streaming.
If you run it, share your numbers. I want to see how other free endpoints behave.
Top comments (0)