I Spent Two Weeks Benchmarking AI APIs So You Don't Have To

I Spent Two Weeks Benchmarking AI APIs So You Don't Have To

honestly, I never thought I'd care this much about latency. like, I've been shipping AI products for a while now and speed was always that thing I knew mattered but never actually measured properly. until one of my apps started tanking in retention and I had to figure out WHY.

turns out it was the API. users were waiting too long, getting impatient, and bailing. pretty much every indie hacker I know has hit this wall at some point.

so I did what any unhinged builder would do — I grabbed every model I could get my hands on through Global API and started timing them. like obsessively. for two weeks. I ran thousands of requests. I had a spreadsheet that looked like something out of a NASA mission control room.

heres what I learned.

Why I Was Wrong About Latency

heres the thing nobody tells you when you're building AI products — your users FEEL every millisecond. I'm not exaggerating. there's research showing you lose conversions after like 100ms of delay, and I brushed it off as marketing fluff. until I actually instrumented my own app and watched real users in real time.

I had a chatbot feature. response times were averaging around 1.2 seconds (yeah, embarrassing). my activation rate for that feature? 18%. I shipped a swap to a faster model and suddenly it was 200ms-400ms range. new activation rate? 41%.

thats not a typo. 41%.

so yeah, I gotta say, speed is NOT optional. its basically the difference between a product people use and a product they forget about.

I ended up settling on Global API for most of my routing because they let me access every model I needed through one endpoint. one bill, one auth flow. we're talking DeepSeek, Qwen, GLM, Kimi, Hunyuan — all the chinese models plus the bigger names. and importantly: their infrastructure is actually fast.

How I Actually Ran These Tests

ok so methodology time. I'm gonna be real with you, I'm an indie dev not a research lab. but I tried to be rigorous.

I built a Python script that hit each model 10 times with the same prompt: "Explain recursion in 200 words." output is roughly 150 tokens. I timed both TTFT (time to first token, basically how long until the model starts spitting out) and sustained tokens/sec (how fast it streams after that).

I tested from two regions — US East (Ohio) and Singapore — to see how geography matters.

I ran all this on May 20, 2026, streaming enabled, using Global API's /v1/chat/completions endpoint. if you wanna replicate it, I'll show you the code in a sec.

The Winners (and The Losers)

ok lets just rip the bandaid off. here's the full leaderboard:

Rank	Model	TTFT	Tokens/sec	$/M Output
1	Step-3.5-Flash	120ms	80	$0.15
2	DeepSeek V4 Flash	180ms	60	$0.25
3	Hunyuan-TurboS	200ms	55	$0.28
4	Qwen3-8B	150ms	70	$0.01
5	Qwen3-32B	250ms	45	$0.28
6	Doubao-Seed-Lite	220ms	50	$0.40
7	Hunyuan-Turbo	280ms	42	$0.57
8	GLM-4-32B	300ms	38	$0.56
9	Qwen3.5-27B	350ms	35	$0.19
10	DeepSeek V4 Pro	400ms	30	$0.78
11	MiniMax M2.5	450ms	28	$1.15
12	GLM-5	500ms	25	$1.92
13	Kimi K2.5	600ms	20	$3.00
14	DeepSeek-R1	800ms	15	$2.50
15	Qwen3.5-397B	1200ms	10	$2.34

couple things stand out. first — Step-3.5-Flash is FAST. 120ms TTFT and a sustained 80 tokens per second. thats insane. second — the reasoning models (R1, K2.5) are slow AF because they think before they speak. like, you can literally watch them "think out loud" which is cool but also adds 800ms+ before you see anything useful.

I should mention: R1 at 15 tok/s and 800ms TTFT sounds bad but its because the model spends compute doing internal reasoning. its not necessarily "slow" — its doing more work. so context matters here.

The Cheap Speedsters That Blew My Mind

lets talk about the budget tier because honestly this is where indie hackers live and die.

Qwen3-8B at $0.01/M output is the most absurd value I've ever seen. seventy tokens per second. one. penny. per. million. tokens. for any simple task — classification, extraction, short responses, basic chat — its basically free and lightning fast. I use it for stuff like tagging support tickets and routing user intents. its not smart enough for complex reasoning but thats not what its for.

Step-3.5-Flash is also a budget pick at $0.15/M and it tops the speed charts. 80 tok/s is genuinely hard to beat. if I'm being honest, this is my new default for any user-facing chat experience where I just need fast responses.

The Sweet Spot (Where I Live)

heres where I spend most of my API budget now — the $0.25-$0.30 range:

• DeepSeek V4 Flash — 60 tok/s at $0.25/M
• Hunyuan-TurboS — 55 tok/s at $0.28/M
• Qwen3-32B — 45 tok/s at $0.28/M

DeepSeek V4 Flash is my workhorse now. honestly, its the best balance I've found. its got GPT-4o-class output quality (I'm not gonna claim its better, but its close enough for 95% of what I build), its fast as hell at 180ms TTFT, and at $0.25/M my margins are intact.

I tested it on my coding assistant side project and the perceived snappiness went through the roof compared to the bigger models.

The Big Boys (When You Need Them)

sometimes you need quality over speed. like when I'm doing complex multi-step reasoning, code generation for senior engineers, or anything where being wrong costs more than being slow. heres the premium tier:

Model	tok/s	$/M
DeepSeek V4 Pro	30	$0.78
MiniMax M2.5	28	$1.15
GLM-5	25	$1.92
Kimi K2.5	20	$3.00

these are slower because they're thinking harder. GLM-5 at 500ms TTFT and 25 tok/s feels sluggish for chat but for a backend task that runs async? its fine. its actually incredible quality.

Geography Matters More Than I Thought

I didn't expect geography to be this significant. but heres what I found when I tested the same models from different regions:

Model	US East	Asia	Diff
DeepSeek V4 Flash	180ms	150ms	-30ms
Qwen3-32B	250ms	210ms	-40ms
GLM-5	500ms	420ms	-80ms
Kimi K2.5	600ms	480ms	-120ms

Kimi K2.5 gets a 120ms boost just from being closer to its servers. thats huge. if your user base is mostly in Asia, the chinese-origin models are gonna FEEL way faster to them than they do to my US-based test box.

DeepSeek seems pretty well distributed globally — only 30ms difference between regions. thats pretty much negligible for any practical use case.

moral of the story: pick your model based on where your users actually are. I made this mistake for months routing everyone through US servers when half my users were in Singapore.

What Actually Feels Fast To Users

I went through like 200 user sessions and timed when people hit the back button. heres what I learned about user perception:

TTFT	What Users Say
< 200ms	"Instant" — feels like real-time chat
200-400ms	"Fast" — totally fine
400-800ms	"Noticeable delay" — some frustration creeps in
800ms+	"Slow" — people bounce

the magic line for me is 200ms. anything faster than that and users start feeling like its "real-time" — like they're chatting with a person, not a machine. DeepSeek V4 Flash at 180ms hits this. Qwen3-8B at 150ms absolutely crushes this.

if I have to ship something with TTFT over 400ms, I'll add a "thinking..." indicator, fake typing dots, something. because users need to feel like SOMETHING is happening even when the model is slow.

My Decision Framework (Steal This)

heres what I actually do now when picking a model for a new feature:

1. Is it user-facing chat? → Qwen3-8B or Step-3.5-Flash. period. under 200ms TTFT or users bounce.
2. Is it a backend task with some complexity? → DeepSeek V4 Flash. sweet spot of quality + speed + price.
3. Is quality critical and latency doesn't matter? → GLM-5 or MiniMax M2.5. yeah its $1.92/M but you get what you pay for.
4. Is it a simple classification/extraction task? → Qwen3-8B. its $0.01/M. stop overthinking it.

I've been running this framework across three different products and its saved me a ton of money while keeping users happy.

The Actual Code I Use

ok since I'm a dev and you're probably a dev, lemme show you the benchmarking script. I run this against Global API because they expose everything through one endpoint:

import time
import requests
import statistics

API_KEY = "your-global-api-key"
BASE_URL = "https://global-apis.com/v1"

def benchmark_model(model_name, iterations=10):
    ttft_list = []
    tps_list = []

    for i in range(iterations):
        start = time.perf_counter()
        first_token_time = None
        token_count = 0

        response = requests.post(
            f"{BASE_URL}/chat/completions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={
                "model": model_name,
                "messages": [{"role": "user", 
                              "content": "Explain recursion in 200 words"}],
                "stream": True,
                "max_tokens": 200
            },
            stream=True
        )

        for chunk in response.iter_lines():
            if chunk:
                elapsed = time.perf_counter() - start
                if first_token_time is None:
                    first_token_time = elapsed
                token_count += 1

        ttft_list.append(first_token_time * 1000)
        tps = token_count / (time.perf_counter() - start - first_token_time)
        tps_list.append(tps)

    return {
        "model": model_name,
        "avg_ttft_ms": round(statistics.mean(ttft_list), 1),
        "avg_tps": round(statistics.mean(tps_list), 1)
    }

for model in ["deepseek-v4-flash", "qwen3-8b", "step-3.5-flash"]:
    print(benchmark_model(model))

I also use this bad boy for production routing when I need to A/B test:

def smart_route(prompt, user_region="US"):
    # pick model based on latency budget + cost
    if is_simple_task(prompt):  # your own classifier here
        return "qwen3-8b"  # $0.01/M, 150ms TTFT
    elif user_region == "ASIA":
        return "kimi-k2.5"  # better from asia
    else:
        return "deepseek-v4-flash"  # safe default

run it through the same /v1/chat/completions endpoint, no special setup. pretty much plug and play.

The Stuff I Didn't Expect

couple weird findings I wanna flag:

Reasoning models are deceiving. DeepSeek-R1 at 800ms TTFT LOOKS terrible on the leaderboard. but its because its thinking. for math, logic, coding puzzles — its actually faster end-to-end than a non-reasoning model that gets the wrong answer. think of TTFT for these models as "time to first thought," not "time to first answer."

Tiny models are criminally underrated. Qwen3-8B at $0.01/M is