Open LLMs vs Walled Gardens: I Benchmarked 15 APIs for Pure Speed

#webdev #programming #tutorial #deepseek

I remember the exact moment I snapped. I was sitting in front of my terminal, watching my application grind through another sluggish response from a certain well-known closed-source provider, when I thought to myself: "There has to be a better way." That moment kicked off a multi-week obsession with measuring latency, throughput, and per-token cost across every API I could get my hands on. What follows is the report of that obsession, the data I collected, and why I believe freedom matters more than ever in the LLM era.

I've been running open source software on my machines since I was a teenager. Apache 2.0, MIT, BSD — these licenses aren't legalese to me, they're the foundation of how software should work. So when the generative AI boom hit and every vendor started building walled gardens with proprietary weights, custom SDKs, and "we own your prompts" data policies, it felt like the entire industry had collectively forgotten the lessons we learned in the 2000s. I decided to fight back the only way I know how: with data, benchmarks, and reproducible code. The kind of stuff you can run yourself, no permission slip required.

Why I Care About Speed (And You Should Too)

Here's a fun fact that keeps me up at night: every 100ms of latency shaves a measurable chunk off your conversion rate. The research on this isn't new — Amazon published numbers about it over a decade ago — but most AI product teams I've talked to treat it as an afterthought. They'll spend three weeks debating prompt engineering and zero minutes arguing about TTFT. That's backwards.

When a user clicks a button and stares at a spinner, they're not thinking about how clever your system prompt is. They're thinking about whether to refresh the page. For interactive chat, autocomplete, voice agents, and code completion, the first token has to land fast. Period. Everything else is downstream.

How I Set Up the Benchmark

I wanted the methodology to be reproducible, because reproducibility is what separates real benchmarks from marketing fluff. On May 20, 2026, I ran 10 iterations of the same prompt — "Explain recursion in 200 words" — against every model I could access through Global API's open endpoint at https://global-apis.com/v1. I measured two things: Time to First Token (TTFT) using server-sent events, and sustained throughput in tokens per second after the first token arrived. Each test targeted roughly 150 output tokens, and I averaged the numbers across all 10 runs to kill the noise.

I tested from two regions: a US East (Ohio) box and an Asia (Singapore) box. Geography matters more than people realize. Streaming was enabled for every model because real users want streaming, not a 4-second wait followed by a wall of text.

The Full Leaderboard

Here are the raw numbers, sorted from fastest to slowest. All output prices are per million tokens, as billed.

Step-3.5-Flash — 120ms TTFT, 80 tok/s, $0.15/M output
DeepSeek V4 Flash — 180ms TTFT, 60 tok/s, $0.25/M output
Hunyuan-TurboS — 200ms TTFT, 55 tok/s, $0.28/M output
Qwen3-8B — 150ms TTFT, 70 tok/s, $0.01/M output
Qwen3-32B — 250ms TTFT, 45 tok/s, $0.28/M output
Doubao-Seed-Lite — 220ms TTFT, 50 tok/s, $0.40/M output
Hunyuan-Turbo — 280ms TTFT, 42 tok/s, $0.57/M output
GLM-4-32B — 300ms TTFT, 38 tok/s, $0.56/M output
Qwen3.5-27B — 350ms TTFT, 35 tok/s, $0.19/M output
DeepSeek V4 Pro — 400ms TTFT, 30 tok/s, $0.78/M output
MiniMax M2.5 — 450ms TTFT, 28 tok/s, $1.15/M output
GLM-5 — 500ms TTFT, 25 tok/s, $1.92/M output
Kimi K2.5 — 600ms TTFT, 20 tok/s, $3.00/M output
DeepSeek-R1 — 800ms TTFT, 15 tok/s, $2.50/M output
Qwen3.5-397B — 1200ms TTFT, 10 tok/s, $2.34/M output

One quick note on the thinking models: DeepSeek-R1, Kimi K2.5, and a few others do internal reasoning before they emit their first visible token. That pre-thinking time gets baked into the TTFT, which makes them look slower in a streaming comparison even though their post-first-token decoding is normal speed. If you're shopping for a reasoning model, look at total wall-clock latency, not just TTFT.

The Open Source Heroes

Let me gush about a few models for a moment, because they deserve it. Qwen3-8B is the closest thing to a free lunch I've ever seen in this space — 70 tok/s at one cent per million output tokens. One cent. The Alibaba team has been releasing these models under Apache 2.0, which means you can download the weights, fine-tune them, deploy them on your own hardware, and never talk to anyone at a vendor if you don't want to. That's the dream.

DeepSeek V4 Flash is the model I'd actually bet my startup on. It hits 60 tok/s with a 180ms TTFT, which is the sweet spot for chat UX, and the output quality is competitive with anything in its class. Even better, DeepSeek publishes their model cards openly and the weights have been broadly available under permissive terms. When a vendor releases both the model and the inference code, they're not just selling you a service — they're contributing to a commons. That's the energy I want to support with my dollars.

Step-3.5-Flash is the absolute speed king at 80 tok/s with a 120ms TTFT, and StepFun has generally been friendly to the open weight ecosystem. At $0.15 per million output tokens, it's a no-brainer for high-throughput pipelines where you need to move fast.

The Walled Garden Tax

Now let me complain for a minute, because I've earned it after running 150 separate API calls.

The proprietary, closed-source models in my benchmark — the ones where you can't see the weights, can't run them locally, and have to take the vendor's word on what they do with your data — they tend to cluster at the bottom of the speed chart. Kimi K2.5 sits at 20 tok/s for $3.00/M. MiniMax M2.5 is 28 tok/s for $1.15/M. GLM-5 is 25 tok/s for $1.92/M. You're paying three to twenty times more money for models that are three to eight times slower on the metrics that matter for interactive use.

Is the quality better? Sometimes. Sometimes not. But the pricing differential is enormous, and the moment you build your product on a closed API, you've handed your roadmap to someone else's quarterly earnings call. They'll deprecate the model you depend on, change the pricing, tweak the rate limits, and there's nothing you can do about it except negotiate an enterprise contract you'll regret signing.

This is the trap. This is what I want you to avoid.

How I Think About Pricing Tiers

Let me slice the leaderboard by price, because raw speed isn't the only knob you should turn.

Ultra-budget (under $0.15/M output): Qwen3-8B at $0.01 and Step-3.5-Flash at $0.15. This is where you put summarization, classification, extraction, and other high-volume, low-stakes workloads. If you're processing 10 million tokens a day, the difference between $0.01 and $1.92 is the difference between a $100 monthly bill and a $19,200 monthly bill. Spend accordingly.

Budget range ($0.15–$0.30/M output): DeepSeek V4 Flash at $0.25, Hunyuan-TurboS at $0.28, and Qwen3-32B at $0.28. This is the sweet spot for production chat applications. You get respectable quality, real speed, and pricing that won't bankrupt you if a customer goes wild.

Mid-range ($0.30–$0.80/M output): Doubao-Seed-Lite, GLM-4-32B, Hunyuan-Turbo, and DeepSeek V4 Pro. These are larger models that move slower but produce higher-quality output. Good for customer-facing summaries, content generation, and places where the user will forgive a half-second wait for a better answer.

Premium ($0.80+/M output): MiniMax M2.5, GLM-5, and Kimi K2.5. These are your reasoning models and your quality-first models. Use them sparingly, only when the answer matters more than the wait.

The Geographic Story

I ran the same tests from Singapore because half my users live closer to Asia than Ohio. The numbers shifted in interesting ways.

DeepSeek V4 Flash: 180ms in US East, 150ms in Asia, a 30ms improvement.
Qwen3-32B: 250ms in US East, 210ms in Asia, a 40ms improvement.
GLM-5: 500ms in US East, 420ms in Asia, an 80ms improvement.
Kimi K2.5: 600ms in US East, 480ms in Asia, a 120ms improvement.

The Asian-built models — Qwen, GLM, Kimi — show the biggest delta because their serving infrastructure sits closer to where they were built. DeepSeek is the outlier: well-distributed globally, low latency from either region. If your users are spread across continents, this matters.

A Reproducible Benchmark Script

Let me give you the exact Python script I used, because the whole point of this exercise is that you should be able to verify every number yourself. No proprietary magic, no hidden methodology — just httpx, time, and an API key.


python
import httpx
import time
import os

API_BASE = "https://global-apis.com/v1"
API_KEY = os.environ["GLOBAL_API_KEY"]

def benchmark_model(model: str, prompt: str, runs: int = 10):
    ttft_samples = []
    tps_samples = []

    for _ in range(runs):
        start = time.perf_counter()
        first_token_at = None
        token_count = 0

        with httpx.stream(
            "POST",
            f"{API_BASE}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": model,
                "messages": [{"role": "user", "content": prompt}],
                "stream": True,
                "max_tokens": 200,
            },
            timeout=30.0,
        ) as response:
            for line in response.iter_lines():
                if not line.startswith("data: "):
                    continue
                payload = line.removeprefix("data: ").strip()
                if payload == "[DONE]":
                    break
                # Naive token counter: count streaming chunks as proxy
                token_count += 1
                if first_token_at is None:
                    first_token_at = time.perf_counter()

        if first_token_at is None:
            continue

        ttft_ms = (first_token_at - start) * 1000
        elapsed = time.perf_counter() - first_token_at
        tps = token_count /