Six open models, one question: 1.3 s for the fastest answer, 16 s for the slowest, and hidden thinking made the difference

#llm #benchmark #python #ai

Written by the Runcrate team, using our own API.

We wanted a model for a small internal helper: a teammate pastes a PyTorch question, gets a short answer. Leaderboards tell you which model is smartest. They don't tell you how long you'll stare at a spinner. So we timed it.

The setup. Six open models served behind one OpenAI-compatible endpoint (Runcrate's inference API at https://api.runcrate.ai/v1; open models are free with your GitHub account). One prompt, temperature=0, streamed, 8 requests per model in a row, then 4 and 16 requests at once on the fastest one. Run on 2026-10-24 between 21:43 and 21:50 UTC from a laptop in the UK.

The prompt:

A teammate asks: why does my PyTorch training loop get slower when I call .item() on the loss every step? Answer in about 150 words, plain prose, no lists.

All six answered correctly (it's the GPU-to-CPU sync). The interesting part was the clock.

Results

Medians of 8 sequential requests per model. "First token" is the first streamed token of any kind, "first text" the first token you would show a user.

Model	First token	First text	Total	Output tok/s	Output tokens	of which reasoning	Cost of 8 answers
openai/gpt-oss-120b	0.64 s	0.85 s	1.30 s	481	321	100	$0.00025
deepseek-ai/DeepSeek-V4.1-Flash-1017	0.56 s	1.22 s	1.80 s	252	296	125	$0.00040
deepseek-ai/DeepSeek-V4.1-Flash	0.92 s	1.50 s	2.00 s	424	366	210	$0.00064
zai-org/GLM-5.3-Flash	3.63 s	4.07 s	6.21 s	120	236	11	$0.00105
zai-org/GLM-5.3	1.24 s	11.73 s	14.61 s	38	834	652	$0.01119
Qwen/Qwen3.8-Flash	1.27 s	14.86 s	16.32 s	71	1016	844	$0.00174

Output tok/s counts every generated token (reasoning included) after the first one arrives.

Three things we didn't expect:

1. "First token" lies for reasoning models. Qwen3.8-Flash started streaming in 1.27 s, but those were reasoning tokens. The first word a user could read came at 14.86 s. If your UI shows a spinner until the first delta.content, this is the number that matters, and it's 12x worse than the one most benchmarks report.

2. Hidden reasoning is most of the bill. For a 150-word answer, Qwen3.8-Flash generated a median 1,016 tokens, 844 of them reasoning you never see. GLM-5.3 spent 652 of 834. gpt-oss-120b spent 100 of 321. Same question, same visible answer length, roughly 3x the tokens.

3. Price per token is not price per answer. GLM-5.3 cost $0.0112 for 8 answers; gpt-oss-120b cost $0.00025 for the same 8. That's 45x. The list prices explain part of it (output tokens: $2.50 per million for GLM-5.3, $0.17 for gpt-oss-120b, from GET /v1/models on 2026-10-24 at 22:13 UTC), and the reasoning tokens from point 2 explain the rest: GLM-5.3 wrote 2.6 times as many tokens per answer.

Under load

The fastest model, gpt-oss-120b, with concurrent requests:

Concurrent requests	Requests	Wall time	Median total per request	Aggregate output tok/s
1	8	9.81 s	1.23 s	262
4	8	2.62 s	1.25 s	978
16	32	4.68 s	1.41 s	2,189

Going from 1 to 16 concurrent users moved the median answer from 1.23 s to 1.41 s. No errors in any of the 96 requests in this run.

One thing that went wrong

Our first pass hung. One GLM-5.3-Flash stream stayed open for over five minutes without sending a single token, and the OpenAI client's timeout=180 never fired, because a stream that keeps the connection alive doesn't trip a read timeout. We killed it, added a hard deadline that closes the stream after 90 s, and re-ran everything from scratch. In the re-run no request hit the deadline; the slowest took 47.2 s. If you stream from any provider in production, put a wall-clock deadline on it:

import threading, time
from openai import OpenAI

api = OpenAI(base_url="https://api.runcrate.ai/v1", api_key=os.environ["RUNCRATE_API_KEY"], max_retries=0)

def ask(model, prompt, deadline=90):
    t0 = time.perf_counter(); first_any = first_text = None; usage = None; text = []
    stream = api.chat.completions.create(
        model=model, messages=[{"role": "user", "content": prompt}],
        max_tokens=1200, temperature=0, stream=True, stream_options={"include_usage": True})
    timer = threading.Timer(deadline, stream.close); timer.start()  # read timeouts don't catch a silent stream
    try:
        for ch in stream:
            now = time.perf_counter() - t0
            if ch.usage:
                usage = ch.usage
            if ch.choices:
                d = ch.choices[0].delta
                extra = d.model_extra or {}
                if first_any is None and (d.content or extra.get("reasoning_content") or extra.get("reasoning")):
                    first_any = now
                if d.content:
                    first_text = first_text or now
                    text.append(d.content)
    finally:
        timer.cancel()
    return {"first_any": first_any, "first_text": first_text, "total": time.perf_counter() - t0,
            "tokens": usage.completion_tokens if usage else None,
            "reasoning": (usage.completion_tokens_details.reasoning_tokens if usage and usage.completion_tokens_details else None),
            "cost": getattr(usage, "cost", None), "text": "".join(text)}

The cost field is what the API returns in usage; the reasoning count is in completion_tokens_details.reasoning_tokens. The full script (sequential runs, the concurrency sweep, raw JSONL per request) is about 90 lines.

What we'd pick

For a chat helper where people wait for the answer: gpt-oss-120b. 0.85 s to the first readable word, 1.3 s total, and the smallest bill of the six. For harder questions where you want the model to think, the reasoning models earn their tokens, but show the user something while they think, and budget for 3x the tokens you see.

Caveats

One prompt, one time of day, one client location, 8 runs per model. Treat it as a measurement of this workload, not a ranking of the models.
Open models are free with your GitHub account. Our test account isn't linked to GitHub, so these calls were billed: the whole run, including the hung first pass, cost $0.018.
Medians hide spread: GLM-5.3's total ranged from 8.7 s to 47.2 s across its 8 runs.

We kept the full script and the raw per-request JSONL; ask in the comments if you want them. The endpoint and its streaming options are documented in the chat completions reference.

Written with AI assistance. Every number above comes from the run on 2026-10-24 and was checked against the raw data.

DEV Community