I Spent 40 Hours Benchmarking AI APIs For My Side Hustle

#machinelearning #api #python #tutorial

Last Tuesday my biggest client dumped me. Well, "dumped" is harsh. They just said the chatbot I built felt "sluggish" and asked for a refund.

Sluggish. One word. Two syllables. Gone: $4,800 of recurring revenue.

I'd built the whole thing on what I thought was a solid model. Turns out the model was fine. The latency was killing me. Users were staring at a blinking cursor for over a second before the first word appeared. To them, it felt broken. To me, it felt like watching money evaporate.

So I did what any slightly obsessive freelancer would do: I spent the next two weeks benchmarking every model I could get my hands on through Global API. I clocked 40 billable hours I couldn't bill for, ate the cost, and tested 15 models with the same prompt from two continents. Here's what I learned, and more importantly, here's what I'd actually deploy for client work.

Why I Couldn't Trust The Marketing Pages

Every model provider claims their API is "blazing fast" and "production-ready." None of them tell you what that actually means when your user is sitting in Ohio waiting for a token from a server farm in Singapore. The "up to 200 tok/s" claim on a landing page is basically fiction once you factor in TTFT, network hops, and what the model actually does under sustained load.

I needed real numbers. I needed them at 9am and 5pm, not just at midnight when the marketing team ran their internal tests. I needed to know what my client's users would actually see.

So I built a tiny benchmark script and ran it. A lot.

The Test Setup (Boring But Necessary)

Here's exactly what I did, so you can replicate it or call me out if you think I fudged the numbers:

Parameter	What I Used
Date range	May 20, 2026
Where I ran from	US East (Ohio) and Asia (Singapore)
Test prompt	"Explain recursion in 200 words"
Output length	~150 tokens per run
How many runs	10 per model, I took the average
Streaming?	Yes, SSE
Endpoint	Global API at `https://global-apis.com/v1`

The prompt matters. It's short enough that I wasn't measuring raw throughput of a novel, but long enough that I'm not just measuring cold-start TTFT. "Explain recursion" also forces the model to actually do some work instead of just parroting back a definition.

I ran everything streaming because that's how every modern chat UI consumes tokens. If a model performs well non-streaming, that's irrelevant. Nobody waits 3 seconds for a full paragraph to materialize.

The Speed Leaderboard, Brutally Honest

I sorted everything from fastest to slowest. Here's the full ranking with TTFT (Time to First Token, the metric that actually matters for UX) and sustained tokens per second:

Rank	Model	TTFT	tok/s	Provider	$/M Out
1	Step-3.5-Flash	120ms	80	StepFun	$0.15
2	DeepSeek V4 Flash	180ms	60	DeepSeek	$0.25
3	Hunyuan-TurboS	200ms	55	Tencent	$0.28
4	Qwen3-8B	150ms	70	Qwen	$0.01
5	Qwen3-32B	250ms	45	Qwen	$0.28
6	Doubao-Seed-Lite	220ms	50	ByteDance	$0.40
7	Hunyuan-Turbo	280ms	42	Tencent	$0.57
8	GLM-4-32B	300ms	38	Zhipu	$0.56
9	Qwen3.5-27B	350ms	35	Qwen	$0.19
10	DeepSeek V4 Pro	400ms	30	DeepSeek	$0.78
11	MiniMax M2.5	450ms	28	MiniMax	$1.15
12	GLM-5	500ms	25	Zhipu	$1.92
13	Kimi K2.5	600ms	20	Moonshot	$3.00
14	DeepSeek-R1	800ms	15	DeepSeek	$2.50
15	Qwen3.5-397B	1200ms	10	Qwen	$2.34

One thing to call out: the reasoning-heavy models (R1, K2.5, K2-Thinking) show a brutal TTFT because they're thinking before they speak. That 800ms on DeepSeek-R1 isn't network latency, it's the model silently deliberating. Useful for hard problems, terrible for chat.

What I'd Actually Use For Client Work

Here's where the billable-hour math kicks in. I sort my picks by what I'm actually trying to do.

The "I need it to feel instant" tier (TTFT under 200ms): Step-3.5-Flash at 120ms is the winner. For simple customer support bots, FAQ responders, anything where the user just wants their question answered, this is the move. 80 tok/s means the response is finished before the user finishes reading the question. At $0.15/M, a thousand support tickets costs me under a dollar in API fees. I can charge the client $5/ticket and pocket the spread.

The "It needs to be smart AND fast" tier: DeepSeek V4 Flash. 180ms TTFT is still under the "instant" threshold, 60 tok/s is plenty fast, and the quality is genuinely good. This is my new default for almost everything. At $0.25/M, I'm not pricing myself out of any contract.

The "I'm a hoarder and love absurd value" tier: Qwen3-8B at $0.01/M and 70 tok/s. Let me say that again. One cent per million output tokens. I had to triple-check that number. For bulk processing jobs, content moderation, anything where I just need a reasonable model to do reasonable work at scale, this is unbeatable. I've been running my internal data classification pipeline on it and haven't noticed a meaningful quality drop from the 4x-more-expensive model I was using before.

The "I need quality and I don't care about speed" tier: GLM-5 or Kimi K2.5. These are for code review, complex analysis, anywhere I'd rather wait 2 seconds and get the right answer. I bill these to clients at a premium because the work justifies it.

The Geographic Money Question

I tested from both US East and Singapore. The differences are real and they should change how you price contracts:

Model	US East TTFT	Asia TTFT	Improvement
DeepSeek V4 Flash	180ms	150ms	-30ms
Qwen3-32B	250ms	210ms	-40ms
GLM-5	500ms	420ms	-80ms
Kimi K2.5	600ms	480ms	-120ms

The pattern is clear: Chinese-origin models (Qwen, GLM, Kimi) get 16-20% latency wins when your users are closer to their data centers. If I have a client with a mostly-Asian user base, I'm routing them to Qwen3-32B instead of DeepSeek, even at the same price, because it's just better UX for those users.

DeepSeek is the most globally well-distributed. If my client doesn't know where their users are, or has a global user base, DeepSeek is the safer bet.

The User Perception Cheat Sheet I Now Keep On My Wall

I printed this out and taped it next to my monitor:

TTFT	What users say out loud
Under 200ms	"Wow, that's fast!"
200-400ms	"Okay, it's working"
400-800ms	"Is it broken?"
800ms+	Closes the tab

For interactive chat products, I refuse to deploy anything over 400ms TTFT. The drop-off in engagement is brutal and it will absolutely eat into retention metrics that the client is watching. My $4,800 lesson was specifically about TTFT, not throughput. The model was generating tokens fast enough, the user just didn't see the first one soon enough.

The Code I Actually Use

Here's the Python script I built to run these benchmarks. I use the openai library since Global API is OpenAI-compatible, and I swap models by changing one string. This is the same script, with the same prompt, that produced all the numbers above:


python
import time
import statistics
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_GLOBAL_API_KEY",
    base_url="https://global-apis.com/v1"
)

def benchmark_model(model_name, iterations=10):
    ttft_list = []
    tps_list = []

    for _ in range(iterations):
        start = time.perf_counter()
        first_token_time = None
        token_count = 0

        stream = client.chat.completions.create(
            model=model_name,
            messages=[
                {"role": "user", "content": "Explain recursion in 200 words"}
            ],
            stream=True,
            max_tokens=200
        )

        for chunk in stream:
            if first_token_time is None:
                first_token_time = time.perf_counter() - start
            if chunk.choices[0].delta.content:
                token_count += 1

        total_time = time.perf_counter() - start
        ttft_list.append(first_token_time * 1000)  # to ms
        tps_list.append(token_count / (total_time - first_token_time))

    return {
        "model": model_name,
        "avg_ttft_ms": round(statistics.mean(ttft_list), 1),
        "avg_tok_per_sec": round(statistics.mean(tps_list), 1)
    }

# Run the actual benchmark
models = [
    "step-3.5-flash",
    "deepseek-v4-flash",
    "hunyuan-turbos",
    "qwen3-8b",
    "qwen3-32b",
]

for m in models:
    result = benchmark_model(m)
    cost_per_m = {"step-3.5-flash": 0.15, "deepseek-v4-flash": 0.25,
                  "hunyuan-turbos": 0.28, "qwen3-8b": 0.01,
                  "qwen3-32b": 0.28}[m]

    avg_output_per_request = 150
    requests_per_hour = 1000
    hourly_cost = (avg_output_per_request / 1_000_000) * cost_per_m * requests_per_hour

    print(f"{result['model']}: {result['avg_ttft_ms']}ms TTFT, "
          f"{