P99 Latency on 15 LLMs: An Engineer's Field Notes

P99 Latency on 15 LLMs: An Engineer's Field Notes

I've been running production AI workloads long enough to know that p50 numbers lie. Anyone can hit a great median latency once. What separates a toy demo from a 99.9% uptime production system is what happens at p99, when the autoscaler is cold-starting a new pod at 3 AM and your customer's CEO is typing into your chatbot. So I spent two weeks hammering fifteen different LLMs through Global API's edge to see which ones actually hold up under real conditions. Here's what I learned, what broke, and which models I'd stake my SLA on.

Why I Stopped Trusting the Marketing Pages

Every model card claims to be the fastest. Every provider has a benchmark that conveniently puts them at the top. The problem with those numbers is they're usually measured once, from a warm connection, on the provider's own fleet, with a prompt they cherry-picked. None of that matters when you're serving traffic from São Paulo to Seoul.

In my world, latency isn't a number on a dashboard. It's the budget between your customer's click and the first pixel of meaningful response. If your TTFT drifts above 400ms at p99, your chat product starts feeling like a 2014-era customer support widget. People tolerate a slow database query. They do not tolerate a slow AI.

So I set up a proper test harness through Global API at https://global-apis.com/v1, pointed it at two regions, and ran each model ten times. Not three. Not five. Ten runs, average reported, with streaming enabled the whole way. I wanted sustained throughput, not a single lucky request.

My Test Rig

Here's the setup I used. Nothing exotic, just the kind of thing any platform team could replicate:

Parameter	Value
Test Date	May 20, 2026
Test Region (Primary)	US East (Ohio)
Test Region (Secondary)	Asia (Singapore)
Test Prompt	"Explain recursion in 200 words"
Output Tokens	~150 per request
Iterations	10 runs, average recorded
Streaming	Yes (SSE)
Endpoint	https://global-apis.com/v1

The prompt is intentionally boring. I don't care about creativity here. I care about tokens-per-second, TTFT, and how those numbers move when the network is the bottleneck instead of the model. A 200-word recursion explanation lands right around 150 tokens of output, which is enough to measure sustained throughput without padding the response with noise.

The Headline Results

After crunching all ten iterations per model, here's how the field shakes out. Sorted by raw speed, not by quality or cost:

Rank	Model	TTFT (ms)	Tokens/sec	Provider	$/M Output
1	Step-3.5-Flash	120	80	StepFun	$0.15
2	Qwen3-8B	150	70	Qwen	$0.01
3	DeepSeek V4 Flash	180	60	DeepSeek	$0.25
4	Hunyuan-TurboS	200	55	Tencent	$0.28
5	Doubao-Seed-Lite	220	50	ByteDance	$0.40
6	Qwen3-32B	250	45	Qwen	$0.28
7	Hunyuan-Turbo	280	42	Tencent	$0.57
8	GLM-4-32B	300	38	Zhipu	$0.56
9	Qwen3.5-27B	350	35	Qwen	$0.19
10	DeepSeek V4 Pro	400	30	DeepSeek	$0.78
11	MiniMax M2.5	450	28	MiniMax	$1.15
12	GLM-5	500	25	Zhipu	$1.92
13	Kimi K2.5	600	20	Moonshot	$3.00
14	DeepSeek-R1	800	15	DeepSeek	$2.50
15	Qwen3.5-397B	1200	10	Qwen	$2.34

One quick note: the reasoning-style models (R1, K2.5, and friends) have a TTFT that includes internal "thinking" time. The first visible token to your user is delayed because the model is quietly reasoning first. If you're building a customer-facing product, that hidden thinking time is the latency you actually feel.

Where I Land on Auto-Scaling Strategy

Most teams I talk to pick one model and call it done. That's a recipe for a brittle system. My rule of thumb after running these tests: keep at least three models in rotation, each for a different tier of traffic.

Tier 1: The Firehose (ultra-cheap, fast enough to be invisible)

Model	tok/s	$/M Output
Qwen3-8B	70	$0.01
Step-3.5-Flash	80	$0.15

Qwen3-8B at $0.01 per million output tokens is, frankly, absurd. Seventy tokens per second means your users see a steady stream of words scrolling in, and at a tenth of a cent per million tokens you can auto-scale horizontally without ever opening a finance ticket. I use this tier for spam classification, intent detection, simple autocomplete, and the first line of any agentic pipeline.

Tier 2: The Workhorse (balanced quality and speed)

Model	tok/s	$/M Output
DeepSeek V4 Flash	60	$0.25
Hunyuan-TurboS	55	$0.28
Qwen3-32B	45	$0.28

This is the band where most production chat traffic should live. DeepSeek V4 Flash is my default pick — 60 tok/s sustained, 180ms TTFT, and the output quality is in GPT-4o territory for anything short of a PhD-level reasoning question. At $0.25 per million tokens you can serve tens of millions of completions a day without breaking your margin.

Tier 3: The Specialist (slower, smarter, route by intent)

Model	tok/s	$/M Output
Doubao-Seed-Lite	50	$0.40
GLM-4-32B	38	$0.56
Hunyuan-Turbo	42	$0.57
DeepSeek V4 Pro	30	$0.78

Once you cross into the mid-range, throughput drops because you're paying for larger context windows and better instruction following. DeepSeek V4 Pro at 30 tok/s is roughly half the speed of V4 Flash, but if your task involves long-document analysis or multi-step planning, the quality delta is worth it.

Tier 4: The Heavy Hitter (quality above all, accept the latency)

Model	tok/s	$/M Output
MiniMax M2.5	28	$1.15
GLM-5	25	$1.92
Kimi K2.5	20	$3.00
DeepSeek-R1	15	$2.50
Qwen3.5-397B	10	$2.34

These models live at the bottom of the speed chart because they're not trying to be fast. They exist for the queries where correctness matters more than user-perceived snappiness — code review, legal summarization, anything where a hallucination costs real money. I keep them as a fallback in my routing layer, not as the default.

Multi-Region Matters More Than You'd Think

The single biggest surprise from this round of testing was how much the geography shifted things. I ran the same benchmark suite from US East and from Singapore. Here's what came back:

Model	US East TTFT	Asia TTFT	Delta
DeepSeek V4 Flash	180ms	150ms	-30ms
Qwen3-32B	250ms	210ms	-40ms
GLM-5	500ms	420ms	-80ms
Kimi K2.5	600ms	480ms	-120ms

The Asian-originated models (Qwen, GLM, Kimi) get a 16-20% latency reduction when called from Asia because their inference fleets are physically closer. That's not a small optimization — at scale, 80ms of TTFT can flip a model from "feels fast" to "feels slow" on the user-perception curve.

The interesting case is DeepSeek. Their infrastructure is genuinely well-distributed globally. The delta from US to Asia for V4 Flash was only 30ms, which tells me they've done the CDN work that most providers skip. If you're serving a multi-region product and you can only pick one model for the default tier, DeepSeek V4 Flash is the safest bet because it doesn't punish you for being far from the inference cluster.

P99 Is Where Dreams Go to Die

Here's the part of the post where I get a little salty. The numbers above are averages. Averages hide the tail. The difference between a model with 180ms average TTFT and 380ms p99 TTFT is the difference between "this is fast" and "this is occasionally awful." I won't publish my exact p99 numbers in a public post (security through obscurity, my friends), but I will tell you that the models I trust are the ones whose p99 stays under 1.5x their median. Anything worse, and you'll start seeing timeout cascades under load.

If you're targeting a 99.9% SLA — and you should be, because that's table stakes for any B2B product — your budget per request is roughly 8.7 hours of downtime per year. Sound like a lot? It isn't. A single cold-start spike that takes out 0.5% of your traffic for an hour will burn through your entire annual SLA budget. Pick models whose p99 you trust, not models whose median looks good on a slide.

The Routing Layer I Actually Use

Let me share the routing code I have running in production. It's nothing fancy — just a tier-based router that picks a model based on the request type, with a timeout budget that fails over to a faster (but dumber) model if the primary one drags. This is the kind of code that survives a 3 AM page.

python
import os
import time
import requests
from typing import Optional

BASE_URL = "https://global-apis.com/v1"
API_KEY = os.environ["GLOBAL_API_KEY"]

TIER_CONFIG = {
    "firehose": ("Qwen3-8B", 250, 0.01),
    "workhorse": ("DeepSeek V4 Flash", 350, 0.25),
    "specialist": ("DeepSeek V4 Pro", 700, 0.78),
    "heavy": ("GLM-5", 900, 1.92),
}

def classify_request(prompt: str) -> str:
    """Route based on simple heuristics. Real version uses a classifier."""
    if len(prompt) < 200 and "summarize" not in prompt.lower():
        return "firehose"
    if "code" in prompt.lower() or "analyze" in prompt.lower():
        return "specialist"
    if len(prompt) > 4000:
        return "heavy"
    return "workhorse"

def call_with_failover(prompt: str, tier: str, timeout_ms: int =