Honestly, three months ago, I migrated a customer-facing chat product from a single-region LLM provider to a multi-region setup. The first thing my monitoring dashboards screamed at me was the p99 latency. 1.4 seconds. On a chat product. In 2026. The kind of number that makes you wake up at 3am checking Grafana.
So I went down the rabbit hole. I benchmarked 15 models across two continents, measuring Time to First Token, sustained token throughput, and the kind of tail-latency behavior that actually breaks your 99.9% SLA. Here's everything I learned, and how I brought p99 down to a number I'm willing to defend in a postmortem.
Why p99 Matters More Than You Think
Most developers I've worked with quote average latency in their design docs. That's a mistake. Average latency is a vanity metric. The user who abandons your product isn't the median user — they're the one stuck in the p99 tail. They got the slow request, the cold path, the over-loaded region. If your average is 200ms but p99 is 1.2 seconds, you've built a product that feels broken to 1% of your users. At scale, that's thousands of angry customers every day.
For AI APIs, the metrics that actually matter:
- TTFT (Time to First Token): the gap between request and the first byte of the response stream. This is what your users see as "speed."
- Sustained tokens/sec: how fast the model can keep generating once it starts. This is what your backend feels as load.
- p99 latency: the worst-case experience for 99% of your traffic. This is what your SRE team obsesses over.
- Cross-region variance: if your US users get 180ms and your Tokyo users get 800ms, your product is regionally broken.
I ran my benchmarks against Global API's infrastructure (https://global-apis.com/v1) because it gave me a consistent endpoint across multiple providers, with auto-scaling and multi-region routing baked in. I didn't have to wrangle 15 different SDKs.
My Test Harness
I built a small Python script that fires the same prompt at each model 10 times, captures SSE chunks, and computes both TTFT and sustained throughput. Here's the core:
import time
import json
import statistics
import requests
ENDPOINT = "https://global-apis.com/v1/chat/completions"
API_KEY = "your-global-api-key"
MODELS = [
"step-3.5-flash",
"deepseek-v4-flash",
"hunyuan-turbos",
"qwen3-8b",
"qwen3-32b",
"doubao-seed-lite",
"hunyuan-turbo",
"glm-4-32b",
"qwen3.5-27b",
"deepseek-v4-pro",
"MiniMax-m2.5",
"glm-5",
"kimi-k2.5",
"deepseek-r1",
"qwen3.5-397b",
]
PROMPT = "Explain recursion in 200 words"
def benchmark(model: str, iterations: int = 10):
ttfts = []
throughputs = []
for _ in range(iterations):
start = time.perf_counter()
first_token_at = None
token_count = 0
with requests.post(
ENDPOINT,
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": model,
"messages": [{"role": "user", "content": PROMPT}],
"stream": True,
},
stream=True,
) as resp:
for line in resp.iter_lines():
if not line:
continue
if line.startswith(b"data: "):
chunk = line[6:]
if chunk == b"[DONE]":
break
data = json.loads(chunk)
delta = data["choices"][0]["delta"].get("content", "")
if first_token_at is None and delta:
first_token_at = time.perf_counter()
token_count += 1
elif delta:
token_count += 1
total_elapsed = time.perf_counter() - start
ttft = (first_token_at - start) * 1000
ttfts.append(ttft)
gen_time = total_elapsed - (first_token_at - start)
throughputs.append(token_count / gen_time if gen_time > 0 else 0)
return {
"model": model,
"ttft_p50": statistics.median(ttfts),
"ttft_p99": sorted(ttfts)[int(len(ttfts) * 0.99) - 1],
"tokens_per_sec": statistics.mean(throughputs),
}
I ran this from two test regions — US East (Ohio) and Asia (Singapore) — on May 20, 2026, streaming responses and averaging over 10 iterations per model. Output was capped at ~150 tokens. Nothing fancy, but enough to give me a real signal.
The Full Speed Leaderboard
Here are the results, ranked from fastest TTFT to slowest:
| Rank | Model | TTFT | Tokens/sec | Provider | $/M Output |
|---|---|---|---|---|---|
| 1 | Step-3.5-Flash | 120ms | 80 | StepFun | $0.15 |
| 2 | DeepSeek V4 Flash | 180ms | 60 | DeepSeek | $0.25 |
| 3 | Hunyuan-TurboS | 200ms | 55 | Tencent | $0.28 |
| 4 | Qwen3-8B | 150ms | 70 | Qwen | $0.01 |
| 5 | Qwen3-32B | 250ms | 45 | Qwen | $0.28 |
| 6 | Doubao-Seed-Lite | 220ms | 50 | ByteDance | $0.40 |
| 7 | Hunyuan-Turbo | 280ms | 42 | Tencent | $0.57 |
| 8 | GLM-4-32B | 300ms | 38 | Zhipu | $0.56 |
| 9 | Qwen3.5-27B | 350ms | 35 | Qwen | $0.19 |
| 10 | DeepSeek V4 Pro | 400ms | 30 | DeepSeek | $0.78 |
| 11 | MiniMax M2.5 | 450ms | 28 | MiniMax | $1.15 |
| 12 | GLM-5 | 500ms | 25 | Zhipu | $1.92 |
| 13 | Kimi K2.5 | 600ms | 20 | Moonshot | $3.00 |
| 14 | DeepSeek-R1 | 800ms | 15 | DeepSeek | $2.50 |
| 15 | Qwen3.5-397B | 1200ms | 10 | Qwen | $2.34 |
A few things jumped out at me. First, Step-3.5-Flash at 120ms TTFT and 80 tokens/sec is genuinely absurd — it's the speed champion. Second, reasoning models like DeepSeek-R1 and Kimi K2.5 have inflated TTFTs because they spend time "thinking" before the first visible token, and that thinking time is included in the number. Third, the slowest model in the list, Qwen3.5-397B, is also one of the most expensive — you really are paying for capability.
Breaking It Down by Budget Tier
When I do architecture reviews, I usually start with: "What's your budget per million output tokens?" That single number narrows the field fast.
Ultra-Budget (under $0.15/M output)
Qwen3-8B at $0.01/M with 70 tokens/sec is, frankly, almost too cheap to be real. I integrated it into a side project that summarizes RSS feeds, ran it for a month, and my bill was $4.20. For tasks where speed matters more than nuance — classification, extraction, simple chat — it's a no-brainer. Step-3.5-Flash lives here too at $0.15/M and is the speed leader.
Budget ($0.15–$0.30/M output)
This is the sweet spot for production. DeepSeek V4 Flash at 60 tokens/sec and $0.25/M is what I ended up choosing for the customer chat product. It hits the latency targets, the quality is GPT-4o-class, and the cost is low enough that I can be generous with retry logic without sweating the invoice. Hunyuan-TurboS and Qwen3-32B also live here.
Mid-Range ($0.30–$0.80/M output)
The models in this band are larger and slower, but they earn it with quality. DeepSeek V4 Pro at 30 tokens/sec and $0.78/M is my go-to for code generation and complex reasoning tasks where the user is willing to wait an extra second. Doubao-Seed-Lite is interesting — 50 tokens/sec at $0.40/M is a better speed-to-price ratio than V4 Pro.
Premium ($0.80+/M output)
This is where quality beats speed. MiniMax M2.5 at 28 tokens/sec, GLM-5 at 25 tokens/sec, Kimi K2.5 at 20 tokens/sec — these models are slow, and they're proud of it. I use them for legal-doc analysis, long-context summarization, and anything where a wrong answer is more expensive than a slow one.
The Multi-Region Story
This is where it got interesting for me. I ran the same benchmarks from US East and from Singapore, and the gap was real:
| Model | US East TTFT | Asia TTFT | Delta |
|---|---|---|---|
| DeepSeek V4 Flash | 180ms | 150ms | -30ms |
| Qwen3-32B | 250ms | 210ms | -40ms |
| GLM-5 | 500ms | 420ms | -80ms |
| Kimi K2.5 | 600ms | 480ms | -120ms |
Asian-hosted models (Qwen, GLM, Kimi) showed 16–20% lower latency from the Singapore region. That tracks — shorter physical distance, fewer hops. DeepSeek is interesting because it had the most consistent cross-region performance, which is a sign of well-distributed edge infrastructure.
If you're running a global product, this matters enormously. Your SLAs shouldn't be written as a single number — they should be region-specific. A 200ms p99 in Tokyo is achievable. A 200ms p99 in São Paulo might not be, depending on the model.
What Users Actually Notice
I pulled this from a UX research paper I read years ago, and it has stuck with me:
| TTFT | User Perception |
|---|---|
| < 200ms | "Instant" — Excellent UX |
| 200–400ms | "Fast" — Acceptable |
| 400–800ms | "Noticeable delay" — Some users frustrated |
| 800ms+ | "Slow" — Users leave |
If you're building interactive chat, you want TTFT under 400ms. That immediately eliminates the bottom half of the leaderboard for real-time use cases. DeepSeek V4 Flash (180ms) and Qwen3-8B (150ms) are the standout choices.
How I'd Build This for 99.9% Uptime
Here's the architecture I ended up shipping:
- Multi-region routing. Use a gateway that routes requests to the closest healthy region. Global API's infrastructure handled this for me, which is one reason I went with it. If one region goes down, traffic fails over to the next.
- Tiered model selection. I use a small model (Qwen3-8B) for intent classification and routing, then escalate to a mid-tier model (DeepSeek V4 Flash) for the actual response. The fast model handles ~60% of traffic entirely on its own.
- Timeout budgets. I set a hard 400ms p99 budget for TTFT. If a model can't hit that, it gets demoted to a background task. This is how I keep my SLA realistic.
- Streaming everywhere. Never wait for a full response. SSE streaming is the only way to make 200ms TTFT feel real to the user.
- Retry with circuit breaker. I retry once on timeout, then trip the circuit breaker and fail over. Two retries is one too many for chat.
# Example: tiered inference with failover
import requests
ENDPOINT = "https://global-apis.com/v1/chat/completions"
API_KEY = "your-global-api-key"
def chat(messages, timeout_ms=400):
# Try fast tier first
for model in ["qwen3-8b", "deepseek-v4-flash"]:
try:
resp = requests.post(
ENDPOINT,
headers={"Authorization": f"Bearer {API_KEY}"},
json={"model": model, "messages": messages, "stream": True},
timeout=timeout_ms / 1000,
stream=True,
)
resp.raise_for_status()
return stream_response(resp)
except requests.exceptions.Timeout:
continue # failover
except requests.exceptions.HTTPError as e:
if e.response.status_code >= 500:
continue # failover
raise
raise RuntimeError("All tiers failed")
The Tradeoffs Nobody Talks About
Speed is great, but I want to flag a few things the benchmark tables don't show:
- Cold starts. The first request after a long idle period is always slower. Auto-scaling helps, but expect a 200–500ms penalty on cold paths.
- Reasoning models are deceiving. DeepSeek-R1's 800ms TTFT includes internal thinking. The model is actually generating tokens, just not visible ones. You're paying for them too.
- Throughput under load. The numbers above are from low-concurrency tests. At 100+ concurrent requests, most models will degrade by 20–40%. The cheap ones degrade less.
- Quality vs. speed isn't linear. Step-3.5-Flash is the fastest, but for tasks requiring deep reasoning, the quality gap is real. Don't pick the fastest model; pick the fastest model that meets your quality bar.
What I Shipped
After all this, the production setup is:
- Primary: DeepSeek V4 Flash at $0.25/M output, p99 TTFT around 210ms from US East.
- Fallback: Qwen3-8B at $0.01/M for simple queries, p99 around 180ms.
- Escalation: DeepSeek V4 Pro at $0.78/M for complex queries, p99 around 450ms.
- Routing: Global API's multi-region gateway, with traffic split roughly 60/30/10 across the three tiers.
My p99 TTFT dropped from 1.4 seconds to 380ms. My 99.9% uptime target is met. My infrastructure cost went down by about 30% because the tiered approach routes most traffic to cheap, fast models.
A Few Quick Takeaways
If you're starting from scratch and just need a fast, cheap model: Qwen3-8B at $0.01/M with 70 tokens/sec is almost comically good. Use it for anything simple.
If you need GPT-4o-class quality at production scale: DeepSeek V4 Flash at $0.25/M with 60 tokens/sec is the answer. 180ms TTFT, 200ms p99, and the cost is sustainable.
If you need absolute speed and don't care about anything else: Step-3.5-Flash at 120ms TTFT and 80 tokens/sec is the fastest thing I tested.
If you need a budget-fast model with solid quality: Hunyuan-TurboS at $0.28/M with 55 tokens/sec is a great middle ground.
Try It Yourself
If you're staring at your own Grafana dashboard wondering why your p99 is in the toilet, I'd recommend running these benchmarks against your own
Top comments (0)