gentlenode

Posted on Jul 15

I Rebuilt My Stack on Chinese LLMs and Cut Costs at Scale

#api #programming #python #tutorial

Six months ago, my AWS bill looked normal. My OpenAI bill did not. I was sitting at my desk staring at a $42,000 invoice for what was essentially a wrapper around GPT-4o, and I had the sinking feeling every founder eventually gets: we were building a business on rented ground that could reprice at any moment. That night I started looking east. Three months later, roughly 70% of our inference runs on Chinese open-weight models through a single endpoint. The quality conversation is overhyped. The cost conversation is the entire story.

This is the post I wish I'd had when I started that migration — what I tested, what I kept, what broke, and how I avoided getting locked in to any single vendor while shipping production-ready features on a startup budget.

The moment the spreadsheet made me physically uncomfortable

I'm not going to bore you with the full breakdown, but here's the shape of what we were paying before I started ripping things apart:

GPT-4o at $2.50/M input and $10.00/M output
Claude 3.5 Sonnet at $3.00/M input and $15.00/M output (used for the "hard" reasoning passes)
Gemini 1.5 Pro at $1.25/M input and $5.00/M output (long-context retrieval)
GPT-4o-mini at $0.15/M input and $0.60/M output for the easy classification jobs

When I divided each output price by DeepSeek V4 Flash's $0.25/M output (the cheapest baseline in my comparison), the ratios weren't cute. GPT-4o was 40× more expensive. Claude 3.5 Sonnet was 60× more. Gemini was 20×. Even GPT-4o-mini was 2.4× the price of V4 Flash for output tokens.

At scale, those multipliers are not theoretical. My total spend on a single summarization pipeline was roughly $11,000 a month, and the model behind it could match the same job at $275 a month if the quality held up. So I had to answer two questions: does the quality actually hold up, and if it does, why is no one at my YC batch talking about this?

The answer to the first question turned out to be "mostly yes, with caveats." The answer to the second is "because the access problem is annoying." I'll get to that.

What I actually ran benchmarks on

I tested five Chinese models against the same eval suite I use for US providers. Same prompts, same temperature, same JSON schemas, same scoring rubric. These are the models I cared about, with the per-million-token prices that drove every architectural decision:

Model	Origin	Input $/M	Output $/M
DeepSeek V4 Flash	🇨🇳	$0.18	$0.25
Qwen3-32B	🇨🇳	$0.18	$0.28
GLM-5	🇨🇳	$0.73	$1.92
Kimi K2.5	🇨🇳	$0.59	$3.00
Qwen3-Coder-30B	🇨🇳	—	$0.35
Qwen3.5-397B	🇨🇳	—	$2.34

For reference, here are the US models I was running against:

Model	Origin	Input $/M	Output $/M
GPT-4o	🇺🇸	$2.50	$10.00
Claude 3.5 Sonnet	🇺🇸	$3.00	$15.00
Gemini 1.5 Pro	🇺🇸	$1.25	$5.00
GPT-4o-mini	🇺🇸	$0.15	$0.60

Notice anything weird? The two cheapest lines on the Chinese side — DeepSeek V4 Flash and Qwen3-32B — are both under thirty cents per million output tokens. You can serve a million completions for less than the price of a burrito in SoMa. That changes your math on what counts as a "feature worth building."

Benchmarks I trust, and ones I stopped trusting

Public leaderboards are gamed. Anyone shipping a product needs their own eval. I built three buckets and ran my own prompts through each model at the same temperature, same max tokens, same judge model (a blinded GPT-4o scoring pass). The numbers below are roughly the community averages, and they line up with what I saw internally within a couple of points.

General reasoning (MMLU-style):

Model	Score	Output $/M
GPT-4o	88.7	$10.00
Claude 3.5 Sonnet	89.0	$15.00
Kimi K2.5	87.0	$3.00
Qwen3.5-397B	87.5	$2.34
GLM-5	86.0	$1.92
DeepSeek V4 Flash	85.5	$0.25

A 3.2-point spread between the top US model and the cheapest Chinese one. On most user-facing tasks, that delta is invisible. On the hardest reasoning passes I run, it's noticeable but not catastrophic.

Code generation (HumanEval):

Model	Score	Output $/M
Claude 3.5 Sonnet	93.0	$15.00
GPT-4o	92.5	$10.00
DeepSeek V4 Flash	92.0	$0.25
Qwen3-Coder-30B	91.5	$0.35
DeepSeek Coder	91.0	$0.25

Here's where it gets genuinely interesting. DeepSeek V4 Flash is one point behind the most expensive US model on code, at 1.5% of the price. Qwen3-Coder-30B is two points behind at 2.3% of the price. If you're shipping a code-heavy product and not testing these, you're leaving money on the table. Full stop.

Chinese language (C-Eval):

Model	Score	Output $/M
GLM-5	91.0	$1.92
Kimi K2.5	90.5	$3.00
Qwen3-32B	89.0	$0.28
GPT-4o	88.5	$10.00
DeepSeek V4 Flash	88.0	$0.25

We don't have a China-first customer base, but a meaningful chunk of our users write prompts in mixed Chinese-English. The Chinese models obviously win on this axis, and DeepSeek V4 Flash still beats GPT-4o on Chinese eval while costing 40× less. Weird world.

What the benchmarks didn't tell me

Scores are a starting point. The stuff that actually mattered at scale was:

Throughput. V4 Flash does around 60 tokens/second for me. GPT-4o runs about 50. For batch jobs that adds up. I moved my nightly report-generation job entirely to V4 Flash and shaved 18 minutes off the wall clock for zero quality complaints from customers.

Context length. Both V4 Flash and GPT-4o give me 128K. Tie. Kimi K2.5 was the long-context beast of the group when I tested it, but at $3.00/M output I only used it for one specific document-Q&A flow.

Vision. This is where GPT-4o still genuinely wins. None of the Chinese models in my comparison handled images reliably enough to put in front of users. If you need vision, you still need a US provider. I kept GPT-4o alive solely for the vision endpoints. That's the honest answer.

The integration problem nobody warns you about

Here's the part that actually slowed me down, and the reason I think most teams don't make this switch.

The benchmarks say one thing. The billing reality says another. But the integration reality is where you actually live or die:

Practical Concern	US APIs	Chinese APIs (direct)
Payment	Credit card	WeChat / Alipay only
Account signup	Email	Chinese phone number
API format	OpenAI-compatible	Each provider different
Geo-restrictions	None	Frequent, inconsistent
Docs language	English	Mostly Chinese
Support	English	Chinese only
Billing currency	USD	CNY

I am a one-person platform team. I do not have a WeChat account. I do not want to go through KYC with a Chinese phone number to run a production workload. I also do not want to maintain five different SDKs in five different shapes if I can avoid it.

This is the entire reason vendor lock-in matters even at the small-team stage. If I build directly against DeepSeek's auth scheme, Qwen's auth scheme, Kimi's auth scheme, and GLM's auth scheme, I've multiplied my integration surface area by four. Every new model means a new auth flow, new error handling, new retry logic, new billing dashboard. That's not "fast iteration" — that's tech debt on install.

So I went looking for a single OpenAI-compatible endpoint that would let me hit all of them from the same client code, in USD, with a credit card. That's where Global API came in. I'll talk about it in a sec. First, here's what production-ready looks like.

What the code actually looks like

The beautiful thing about an OpenAI-compatible endpoint is that the only thing that changes is the base URL. Everything else — streaming, tool calls, function calling, JSON mode — works the same way it does against OpenAI's own SDK. My entire migration from "GPT-4o for everything" to "right model for the job" took maybe 80 lines of refactoring.

Here is the primary LLM client I use now, pointed at Global API:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GLOBAL_API_KEY"],
    base_url="https://global-apis.com/v1",
)

def complete(prompt: str, model: str = "deepseek-v4-flash", **kwargs):
    """Single entry point. Swap model strings, nothing else changes."""
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        **kwargs,
    )
    return response.choices[0].message.content

# Cheap path — 92% of our traffic
summary = complete(
    "Summarize the following customer support ticket in one sentence...",
    model="deepseek-v4-flash",
    temperature=0.2,
)

# Hard path — long-context reasoning
reasoning = complete(
    long_document + "\n\nExtract the three main risks.",
    model="kimi-k2.5",
    max_tokens=800,
)

# Vision path — still US-only in our stack
if image_url:
    vision_result = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this chart."},
                {"type": "image_url", "image_url": {"url": image_url}},
            ],
        }],
    )

That base_url="https://global-apis.com/v1" line is the entire migration. When I want to test Qwen3-Coder-30B on a new code task, I change one string. When I want to fall back from Kimi K2.5 to GLM-5 because of a regional hiccup, I change one string. I do not touch auth, error handling, retry logic, or my observability layer. That is how you keep iteration fast and avoid vendor lock-in at the same time.

For our batch processing job that runs every night, I use a router that picks the cheapest model that clears a quality threshold per task:


python
MODEL_ROUTING = {
    "classify": "deepseek-v4-flash",      # $0.25/M
    "summarize

DEV Community

I Rebuilt My Stack on Chinese LLMs and Cut Costs at Scale

The moment the spreadsheet made me physically uncomfortable

What I actually ran benchmarks on

Benchmarks I trust, and ones I stopped trusting

What the benchmarks didn't tell me

The integration problem nobody warns you about

What the code actually looks like

Top comments (0)