Letter to Pre-Benchmark-Me: Three Timing Mistakes That Cost Me a Day

Last Tuesday I lost a full day to a stream that was mostly healthy.
Forty-one runs failed my own latency check.
Not one of them was a slow model. The ruler was broken.

Here is the letter I needed before run one.
It is specific, it is short, and it takes forty minutes to follow.

The goal was a boring probe

I wanted a repeatable measurement for an endpoint I did not control.
That endpoint sat behind a free server option, so cold starts were plausible.
I needed first-token time and total time as separate numbers.

I also needed to know when a retry is safe.
That question turned out to be the expensive one.

Setup, and what is claimed

MonkeyCode is described by its operator as an open-source project with free model access and a free server option. The same operator states the free allowance includes ten million tokens.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I have not audited the repository, the network, or the quota terms. Treat those figures as vendor claims and verify them yourself.

Everything below works with any OpenAI-compatible streaming endpoint.
The script is the artifact. Swap the base URL and keep the method.

Mistake 1: One clock for the whole request

My first script recorded a single number. Wall-clock time until the last byte arrived.
On a cold server that number mixes three stages. Connect, prefill, and decode.
I labelled the sum "model latency". That label sent me hunting in the wrong place for hours.

The fix is to split the clock at the first content delta.

async def one_run(client, prompt, state):
    body = {'model': MODEL, 'stream': True,
            'messages': [{'role': 'user', 'content': prompt}]}
    t0 = time.perf_counter()
    t_first = None
    chars = 0
    async with client.stream('POST', f'{BASE_URL}/chat/completions',
                             json=body, headers=headers()) as r:
        r.raise_for_status()
        async for line in r.aiter_lines():
            if not line.startswith('data:'):
                continue
            payload = line[5:].strip()
            if payload == '[DONE]':
                break
            delta = json.loads(payload)['choices'][0]['delta'].get('content') or ''
            if delta and t_first is None:
                t_first = time.perf_counter()
                state['first_token'] = True
            chars += len(delta)
    t_end = time.perf_counter()
    start = t_first or t_end
    return {'ttft_s': round(start - t0, 3),
            'decode_s': round(t_end - start, 3),
            'total_s': round(t_end - t0, 3),
            'chars': chars}

Ratio beats raw numbers when you are triaging.
If first-token time is most of the total, suspect queueing or startup.
If decode dominates, suspect output length or throughput. Different fixes, different owners.

Mistake 2: Retrying after the first token landed

I retried every timeout in a tight loop. Two costs followed.
First, a retry after content had streamed was a new generation, so I paid twice.
Second, the loop amplified failures into bursts and hid the real pattern.

A retry gate needs one boolean and one budget. Nothing more.

RETRYABLE = (httpx.ConnectError, httpx.ReadTimeout, httpx.RemoteProtocolError)

async def run_with_gate(client, prompt, attempts=3):
    state = {'first_token': False}
    for attempt in range(1, attempts + 1):
        try:
            return await one_run(client, prompt, state), attempt
        except RETRYABLE:
            if state['first_token'] or attempt == attempts:
                raise
            await asyncio.sleep(0.5 * 2 ** attempt)

A stream you have already consumed is not resumable by default.
Unless the call is side-effect free, stop after the first delta and surface the error.
That single rule would have saved me half the morning.

Mistake 3: Comparing runs with different prompts

My prompt drifted between runs. I edited it, then compared columns.
My baseline was also always cold, so "before" and "after" measured my own edits.

Freeze the prompt, hash it, and warm the connection first.

async def warmup(client, rounds=2):
    state = {'first_token': False}
    for _ in range(rounds):
        await one_run(client, 'ok', state)

PROMPT = open('prompt.txt', encoding='utf-8').read()
PROMPT_SHA = hashlib.sha256(PROMPT.encode()).hexdigest()[:12]

Two warmup rounds were enough for me to separate startup cost from steady state. That is a local observation, not a guarantee for your server.
Write prompt_sha into every row. Then a mismatch is visible, not inferred.

The forty-minute rerun

1. Save the prompt to prompt.txt. Do not edit it during the sweep.
2. Create an isolated environment. python -m venv .venv && source .venv/bin/activate
3. Install the only dependency. pip install httpx
4. Export three variables. BASE_URL, API_KEY, and MODEL.
5. Run twenty probes. python probe_ttft.py --runs 20 --out runs.csv
6. Print the median first-token time. python -c "import csv,statistics as s; r=list(csv.DictReader(open('runs.csv'))); print(s.median(float(x['ttft_s']) for x in r if x['ttft_s']))"

Here is the runnable script behind those steps.

# probe_ttft.py
import argparse, asyncio, csv, hashlib, json, os, time
import httpx

BASE_URL = os.environ['BASE_URL'].rstrip('/')
API_KEY = os.environ['API_KEY']
MODEL = os.environ['MODEL']
PROMPT = open('prompt.txt', encoding='utf-8').read()
PROMPT_SHA = hashlib.sha256(PROMPT.encode()).hexdigest()[:12]

def headers():
    return {'Authorization': f'Bearer {API_KEY}', 'Content-Type': 'application/json'}

async def main(runs, out_path):
    async with httpx.AsyncClient(timeout=httpx.Timeout(60.0, connect=10.0)) as client:
        await warmup(client)
        rows = []
        for i in range(1, runs + 1):
            try:
                m, attempt = await run_with_gate(client, PROMPT)
                rows.append({'run': i, 'prompt_sha': PROMPT_SHA,
                             'retry': attempt - 1, **m, 'err': ''})
            except Exception as exc:
                rows.append({'run': i, 'prompt_sha': PROMPT_SHA, 'retry': -1,
                             'ttft_s': '', 'decode_s': '', 'total_s': '',
                             'chars': '', 'err': type(exc).__name__})
            await asyncio.sleep(0.7)
    with open(out_path, 'w', newline='', encoding='utf-8') as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader()
        w.writerows(rows)

if __name__ == '__main__':
    p = argparse.ArgumentParser()
    p.add_argument('--runs', type=int, default=20)
    p.add_argument('--out', default='runs.csv')
    a = p.parse_args()
    asyncio.run(main(a.runs, a.out))

Assumption to check first: the stream uses data: SSE lines with an OpenAI-style delta.content. Adjust the parser if your provider differs.

Reading the output

Five patterns show up often. The table maps each to one action.

Signal in runs.csv	Likely cause	Action
High ttft_s on run 1 only	Cold start on a free server	Keep warmup rounds, re-measure
High ttft_s on every run	Queueing or routing	Shorten the prompt, retest, then consider another route
High decode_s, small ttft_s	Long output, healthy stream	Cap the output length, measure again
retry above 0, empty err	Transient connect or read timeout	Keep the gate, track retry rate over time
Many timeout rows	Endpoint or network, not the model	Stop sweeping, check status page first

The chars column is a cheap sanity check. Two runs with the same prompt should return similar volume.
If volume swings wildly, your comparison is invalid before any timing argument starts.

Limitations, and who should skip this

• This is a probe, not a load test. Concurrency changes the picture.
• A free server is shared. Your neighbour's traffic is part of your sample.
• Network path matters. Run twice from the same region before trusting a delta.
• Quota and availability are vendor claims. Re-read the current terms before planning.
• If you need a hard p99 with contractual backing, buy that. Do not infer it from twenty runs.
• If your calls mutate state, a retry gate is not enough. Add idempotency keys first.

Skip this workflow if you only need one answer today.
Write the call, read the answer, move on.

Letter to past me

Start with the clock, not the model.
Split first token from total, then gate the retry.
Freeze the prompt before you compare anything.

If you want a cheap place to run the sweep, the free allowance on MonkeyCode covered mine as part of that outreach; paste your own endpoint into the script and keep the method either way.