I Wish I Benchmarked Model Speed Sooner — Here's the Full Breakdown

#deepseek #python #tutorial #machinelearning

I gotta say, i Wish I Benchmarked Model Speed Sooner — Here's the Full Breakdown

Six months ago I shipped what I thought was a perfectly good AI feature. It worked. It gave users the right answers. My conversion metrics were fine.

Then a user emailed me. They said the chat felt "weird and sluggish," and they weren't sure they'd come back. I checked my logs. TTFT was sitting around 800ms. I had no idea.

That's when I started running real benchmarks. Not the marketing claims on vendor websites — actual numbers from my own infrastructure, with my own prompts, at scale. What follows is everything I learned the hard way, so you don't have to.

Why Latency Matters More Than Most CTOs Admit

The first thing every founder optimizes for is cost. Then quality. Then maybe they think about latency. That's the wrong order, and I'll tell you why.

Latency is the only metric that kills you before the user even sees your output. A $0.50/M model that responds in 150ms will outperform a $0.10/M model that takes 1200ms on any user-facing flow. People don't wait. They'll close the tab, blame your product, and never tell you why.

When I plotted our churn against p95 TTFT, the curve was brutal. Anything above 400ms and our 7-day retention started dropping. Above 800ms it fell off a cliff. We were running a reasoning model at 800ms because we thought it was "smarter." The reasoning was killing us.

So I went looking for honest benchmarks. Global API had already published a solid test suite, and since I route most of my traffic through their gateway anyway to avoid vendor lock-in, I re-ran everything myself to verify. The numbers in this post come from my own repeated runs against their https://global-apis.com/v1 endpoint.

My Test Setup

I won't pretend this was some academic study. I built a quick Python script that hammered 15 models with the same prompt — "Explain recursion in 200 words" — and measured TTFT and sustained tokens per second. Ten iterations per model, averaged. Streaming via SSE.

Here's the core of the harness, which you can copy if you want to run your own numbers:

import time
import requests
import statistics

API_URL = "https://global-apis.com/v1/chat/completions"
HEADERS = {
    "Authorization": "Bearer YOUR_GLOBAL_API_KEY",
    "Content-Type": "application/json",
}

def benchmark(model: str, prompt: str, runs: int = 10) -> dict:
    ttfts = []
    tps_list = []

    for _ in range(runs):
        payload = {
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": True,
            "max_tokens": 150,
        }

        start = time.perf_counter()
        first_token_time = None
        token_count = 0

        with requests.post(API_URL, headers=HEADERS, json=payload, stream=True) as r:
            r.raise_for_status()
            for chunk in r.iter_lines():
                if not chunk:
                    continue
                if first_token_time is None:
                    first_token_time = time.perf_counter()
                token_count += 1

        if first_token_time:
            ttfts.append((first_token_time - start) * 1000)
            elapsed = time.perf_counter() - first_token_time
            tps_list.append(token_count / elapsed if elapsed > 0 else 0)

    return {
        "model": model,
        "ttft_ms": round(statistics.mean(ttfts), 1),
        "tokens_per_sec": round(statistics.mean(tps_list), 1),
    }

Test date: May 20, 2026. Regions: US East (Ohio) and Asia (Singapore). Output: roughly 150 tokens per run.

The Numbers, Ranked by Speed

After running everything, here's what came out. Top to bottom, fastest TTFT first:

Rank	Model	TTFT (ms)	Tokens/sec	Provider	$/M Output
🥇	Step-3.5-Flash	120	80	StepFun	$0.15
🥈	Qwen3-8B	150	70	Qwen	$0.01
🥉	DeepSeek V4 Flash	180	60	DeepSeek	$0.25
4	Hunyuan-TurboS	200	55	Tencent	$0.28
5	Doubao-Seed-Lite	220	50	ByteDance	$0.40
6	Qwen3-32B	250	45	Qwen	$0.28
7	Hunyuan-Turbo	280	42	Tencent	$0.57
8	GLM-4-32B	300	38	Zhipu	$0.56
9	Qwen3.5-27B	350	35	Qwen	$0.19
10	DeepSeek V4 Pro	400	30	DeepSeek	$0.78
11	MiniMax M2.5	450	28	MiniMax	$1.15
12	GLM-5	500	25	Zhipu	$1.92
13	Kimi K2.5	600	20	Moonshot	$3.00
14	DeepSeek-R1	800	15	DeepSeek	$2.50
15	Qwen3.5-397B	1200	10	Qwen	$2.34

One footnote worth mentioning: the reasoning-style models (R1, K2.5, and similar) eat their own thinking time before they ever emit a visible token. That 800ms TTFT on DeepSeek-R1 includes a long internal monologue. If you need a snappy product, those models aren't the answer regardless of how smart they look on benchmarks.

The Real Question: Cost vs Speed vs Quality

Ranking models by raw speed is a fun exercise, but it's not how you make architecture decisions. What I actually needed to know was: which model gives me the best ROI for a given user experience target?

Here's how I sliced it after staring at the data for a week.

Ultra-budget tier (< $0.15/M output):

Qwen3-8B at 70 tok/s and $0.01/M
Step-3.5-Flash at 80 tok/s and $0.15/M

Qwen3-8B is the most absurd value I found anywhere. Seventy tokens per second for a tenth of a cent per million? For classification, extraction, simple Q&A, or pre-processing, this thing is a cheat code. I now route about 30% of my traffic through it.

Budget tier ($0.15-$0.30/M output):

DeepSeek V4 Flash at 60 tok/s and $0.25/M
Hunyuan-TurboS at 55 tok/s and $0.28/M
Qwen3-32B at 45 tok/s and $0.28/M

DeepSeek V4 Flash is what I'd call the sweet spot. You're paying roughly $0.25/M and getting GPT-4o-class quality at a TTFT most users perceive as instant. If I could only pick one model for a general-purpose chat feature, this would be it.

Mid-range ($0.30-$0.80/M output):

Doubao-Seed-Lite at 50 tok/s and $0.40/M
Hunyuan-Turbo at 42 tok/s and $0.57/M
GLM-4-32B at 38 tok/s and $0.56/M
DeepSeek V4 Pro at 30 tok/s and $0.78/M

This is where you trade speed for capability. The V4 Pro is noticeably better at multi-step reasoning, but 30 tok/s means your streaming UX starts feeling like a typewriter. Reserve these for batch jobs or back-end pipelines, not user-facing chat.

Premium ($0.80+/M output):

MiniMax M2.5 at 28 tok/s and $1.15/M
GLM-5 at 25 tok/s and $1.92/M
Kimi K2.5 at 20 tok/s and $3.00/M

I treat these like specialized consultants. I only call them when correctness genuinely matters and latency doesn't. Code generation that has to compile. Legal text that has to be precise. Things where a wrong answer costs more than waiting a second.

Geographic Latency: Where Your Users Are Matters

I ran the same suite from Singapore to see how server location shifted the numbers:

Model	US East TTFT	Asia TTFT	Diff
DeepSeek V4 Flash	180ms	150ms	-30ms
Qwen3-32B	250ms	210ms	-40ms
GLM-5	500ms	420ms	-80ms
Kimi K2.5	600ms	480ms	-120ms

The pattern is obvious but worth stating plainly: Asian-built models (Qwen, GLM, Kimi) sit on infrastructure closer to Asia, so my Singapore TTFT dropped 16-20% on those. DeepSeek is well-distributed globally and barely budges — which is part of why it's become my default.

If you're shipping a product for a global audience, this is the case for running your inference through a routing layer. I use Global API specifically because it lets me hot-swap providers without rewriting my service code. That's the kind of vendor lock-in avoidance that pays off when your traffic shifts from US to APAC overnight.

What "Fast" Actually Means to Users

I've seen enough A/B tests now to have strong opinions about how users perceive latency. Here are the buckets I use when I'm designing a feature:

Under 200ms TTFT: feels instant. Users assume the system is "just working."
200-400ms: feels fast. Acceptable for any interactive chat.
400-800ms: noticeable delay. Power users tolerate it, casual users start to squirm.
Over 800ms: people bounce. They'll close the tab and open a competitor.

Anything above 400ms TTFT needs a strong reason. I'm not saying never use a slow model — sometimes you have to. But it should be a deliberate decision, not an accident.

A Production-Ready Pattern

Here's the second code snippet, which is closer to what I actually run in production. It's a tiered router that picks a model based on the request type, with automatic fallback if the primary is slow or down:

import time
import requests

API_URL = "https://global-apis.com/v1/chat/completions"
HEADERS = {
    "Authorization": "Bearer YOUR_GLOBAL_API_KEY",
    "Content-Type": "application/json",
}

TIERS = {
    "simple":   {"model": "Qwen3-8B",         "max_ttft_ms": 200},
    "default":  {"model": "DeepSeek V4 Flash", "max_ttft_ms": 300},
    "premium":  {"model": "GLM-5",            "max_ttft_ms": 600},
}

def call_with_tier(prompt: str, tier: str, fallback: bool = True):
    config = TIERS[tier]
    payload = {
        "model": config["model"],
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
    }

    start = time.perf_counter()
    response = requests.post(API_URL, headers=HEADERS, json=payload, stream=True, timeout=10)

    if fallback and (time.perf_counter() - start) * 1000 > config["max_ttft_ms"]:
        fallback_tier = "simple" if tier == "default" else "default"
        return call_with_tier(prompt, fallback_tier, fallback=False)

    return response