A Timeout Is Not a Slow Model: Five Latency Myths and a 40-Line Attribution Script

Your request timed out. So the model is slow, right?

Maybe. But that sentence hides four other suspects. I see this in code review all the time: a 504 appears, someone bumps the timeout, ticket closed. Nobody measured anything.

Here is the workflow I reach for instead. It costs one script and about ten minutes. Swap the endpoint for whatever you actually call.

Myth 1: A timeout means the model is slow

A single wall-clock number is a sum, not a diagnosis. It contains at least five things:

1. DNS lookup
2. TCP connect
3. TLS handshake
4. upload plus queue wait
5. generation

Only item five belongs to the model. curl can split the first four for free.

#!/usr/bin/env bash
# timeout-attribution.sh — split total wall clock into phases
set -euo pipefail
URL="${1:?usage: timeout-attribution.sh <url> [runs]}"
N="${2:-5}"
FMT='dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code}\n'
for i in $(seq 1 "$N"); do
  printf 'run=%s ' "$i"
  curl -sS -o /dev/null -w "$FMT" --max-time 60 "$URL" || echo "curl_exit=$?"
  sleep 1
done

Now subtract. Handshake is tls - tcp. Pre-generation wait is ttfb - tls. Generation is total - ttfb.

If ttfb - tls dominates, you are waiting in a queue. You are not waiting in a decoder. Raising the client timeout hides that fact. It does not fix it.

Myth 2: Time to first byte is time to first token

TTFB means the first byte of the response body. With text/event-stream, that is often a header frame or a keepalive comment. The first useful token may land seconds later.

Measure the gaps between body chunks instead. Chunks are also not tokens, and I will come back to that.

// chunk-gaps.mjs — measure body chunk arrivals, not tokens
const url = process.env.STREAM_URL;
if (!url) throw new Error('set STREAM_URL to your streaming endpoint');

const t0 = performance.now();
const res = await fetch(url, {
  method: 'POST',
  headers: { 'content-type': 'application/json', accept: 'text/event-stream' },
  body: process.env.STREAM_BODY ?? '{}',
});
const tHeaders = performance.now();
if (!res.body) throw new Error(`no body, status ${res.status}`);

const marks = [];
const reader = res.body.getReader();
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  marks.push({ at: performance.now(), bytes: value.byteLength });
}
const tEnd = performance.now();

const gaps = marks.slice(1).map((m, i) => m.at - marks[i].at);
const q = (arr, p) => {
  if (!arr.length) return 0;
  const s = [...arr].sort((a, b) => a - b);
  return s[Math.min(s.length - 1, Math.floor(p * s.length))];
};

console.log({
  headers_ms: +(tHeaders - t0).toFixed(1),
  first_chunk_ms: marks.length ? +(marks[0].at - t0).toFixed(1) : null,
  chunks: marks.length,
  bytes: marks.reduce((n, m) => n + m.bytes, 0),
  gap_p50_ms: +q(gaps, 0.5).toFixed(1),
  gap_p95_ms: +q(gaps, 0.95).toFixed(1),
  total_ms: +(tEnd - t0).toFixed(1),
});

One caution, and it matters. Proxies coalesce and split frames. So report bytes and gap shape, not tokens per second. A "tokens/sec" number derived from chunk counts is fiction.

Myth 3: Retrying from a faster box fixes it

Often the path is the problem, not the CPU. A laptop on hotel Wi-Fi and a server in another region take different routes to the same endpoint.

That is why I run the same probe from two vantage points. Same payload, same minute, two machines. Then compare phase by phase.

This is also where MonkeyCode showed up for me. It offers free model access and a free server option, both as operator-supplied availability rather than a spec sheet. I have no quota, hardware, or region details to hand you. Treat those as unknowns you measure, not constants you assume.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# run the same probe from the second box
scp timeout-attribution.sh box:/tmp/
ssh box 'STREAM_URL=... bash /tmp/timeout-attribution.sh "$STREAM_URL" 5'

Read the result like this:

• Laptop slow, server fast: suspect your network path.
• Both slow at the same phase: suspect the provider side.
• Both fine alone, bad under load: suspect your client.

No measurement, no myth busting. Just guesses wearing a lab coat.

Myth 4: The average latency is the user experience

A mean hides the stalls that users actually feel. Streaming UI dies on the worst gap, not the middle one.

So report three numbers, not one: p50, p95, and the count of runs that never finished. That last number is the one people forget to log.

Here is the decision table I keep next to the script.

Symptom	Where the clock went	Wrong fix	Right next step
ttfb - tls is large	queue or upstream wait	raise client timeout	measure at a second vantage point
tls - tcp is large	handshake, cold connection	add more retries	reuse connections, check path
high gap_p95_ms	generation or proxy buffering	blame the model	check for buffering proxies first
many curl_exit=28	client deadline too tight	infinite timeout	size the deadline to measured p95
fast alone, slow in parallel	client-side concurrency	bigger server	cap concurrency, re-measure

Myth 5: Retrying after a timeout is safe and free

A timeout is your client giving up. It is not the server stopping. The work may have finished after you walked away.

So a blind retry can double the work and double the consumption. On any metered free tier that is a real cost, not a rounding error.

Use an idempotency key if the API supports one. Otherwise read back state before retrying. If you cannot do either, log the ambiguity and move on.

Limitations and who should skip this

This probe is honest about what it cannot see. Keep the list short and read it anyway.

• It measures one vantage point at a time, so clock skew never fully cancels.
• TCP retransmits are invisible to curl timings.
• Shared free servers have noisy neighbours you cannot isolate.
• Provider-side queueing shows up only as unexplained TTFB.

Skip this if you need billing-grade or SLA-grade numbers. Skip it if you cannot fix the test payload across runs. Skip it if your plan is to retry blindly anyway, because the data will not change that habit.

The corrected mental model

A timeout is a client-side verdict about a shared clock. Attribute the phases first, then argue about the model.

If you want to try the two-vantage-point version, MonkeyCode's free model access plus its free server option is enough to run the same script twice. Log both runs side by side before you conclude anything.