I built an OpenAI-compatible gateway for 6 Chinese LLMs — what the happy path hides

page6668 — Wed, 14 Oct 2026 08:03:28 +0000

Two months of full-time work went into this. The part that nearly sank it was not
the routing layer, the Kafka pipeline, or the admin console. It was token
accounting.

I built He-API: a self-hosted gateway that puts six Chinese model providers
(DeepSeek, Kimi, Qwen, Doubao, ERNIE, GLM) behind a single OpenAI-compatible
endpoint. The thesis is boringly practical — Chinese models run 80–90% cheaper
than GPT-4/Claude-class Western models for most workloads, but if you're a
developer outside China, actually using them is painful: docs in Chinese,
payment methods that don't work internationally, inconsistent auth, no shared
format.

So: one endpoint, one key, one request shape. Here's the architecture, and then
the parts that cost me the most time.

The shape of it

Client (any OpenAI SDK)
  → Nginx + Let's Encrypt
  → api-gateway (Go)
      → entitlement resolver   (Redis)
      → rate limiter           (Redis + Lua)
      → model router
      → provider adapter       (kimi / deepseek / qwen / doubao / ernie / glm)
      → upstream provider
  → Kafka → ClickHouse   (usage events)
  → PostgreSQL           (billing, models, providers)

15 services on systemd — all Go except a Next.js admin console. Postgres,
Kafka, Redis, ClickHouse. 22 migrations, all idempotent.

That's the diagram everyone draws. Here's what actually bit.

1. "OpenAI-compatible" is 90% edge cases

The happy path — a non-streaming completion — took an afternoon. Everything
after that was tails.

Streaming is where it starts. Providers don't agree on chunk shape. Some send
data: [DONE], some just close the connection. Some include a usage block in
the final chunk; some omit it entirely on streams and only report usage on
non-streaming calls. If you're building a gateway, you discover that
"compatible" means "compatible for the paths you happened to test."

Then error shapes. A 400 from one provider is a JSON body with an error.message
field; from another it's a bare string; from a third it's a 200 with the error
inside the body. Your gateway has to normalize all of that into something that
doesn't break the client's existing error handling — otherwise "zero code
changes" is a lie the first time a provider sneezes.

2. Usage accounting is a schema problem, not a code problem

This is the one that nearly sank it.

If you bill per token, the token count is money, and money is never a float.
Prices land around ¥0.1–¥2 per million tokens, which means per-request amounts
are genuinely small — and if you do the arithmetic in float64 you will drift,
and your drift will compound across millions of rows.

So: NUMERIC in Postgres, integer micro-units in the hot path, and a deliberate
decision about where rounding happens. Then, on top of that, the reconciliation
problem — because providers report usage inconsistently (see above), you can't
just trust the upstream number. The policy I settled on is stricter than estimating: if the stream never delivers a usage chunk, nothing is charged at all. That keeps estimates out of the money path — at the cost of the ledger being unable to tell you about the attempts it declined to bill, which is a gap I'd close with an explicit usage_source column and a record of unmetered attempts

The lesson: decide your money representation before you write the first billing
line, not after you find a 0.3% discrepancy you can't explain.

3. Hot reload beats redeploys

Provider keys used to require a restart. That meant a redeploy — and therefore a
window where requests fail — every time a key rotated.

The fix is unglamorous and it removed an entire class of operations pain: the
gateway polls an internal active providers endpoint every 30 seconds and swaps
the in-memory provider set atomically. Keys are encrypted at rest with
AES-256-GCM, decrypted only in memory. Rotating a key is now "save in the admin
console, wait 30 seconds, done."

If you're building anything that talks to multiple third-party APIs: build the
reload path before you need it. Ops pain compounds faster than feature requests.

4. Rate limiting needs two axes, not one

My first limiter was a single per-key QPS number. It failed in a way that took
me a while to see: internal sub-requests were consuming the same budget as the
user-facing request, so a free-tier account with QPS=1 rate-limited itself.

The version that works has two axes — a steady per-key rate plus a burst bucket,
with burst size scaled by account age and spend. New accounts get a tiny burst;
accounts with history get a generous one. It's crude, but it kills almost all
automated abuse without touching legitimate power users.

Gate the counter itself carefully, too. A read-then-write in application code
races under load; that check-and-increment belongs in a single atomic Lua script
against Redis.

5. The `base_url` trap

Small, embarrassing, worth writing down. Most OpenAI SDKs concatenate paths onto
your base_url, so if you document https://api.example.com and the SDK appends
/v1/chat/completions, you 404. Some SDKs want the host, some want it with
/v1. Get this wrong and your very first integration attempt fails for a reason
that has nothing to do with your actual product.

What I'd do differently

Write the billing correctness tests first. Not because billing is interesting, but because everything else is cheap to change and that isn't.
Assume every upstream field is optional. Design the normalization layer around the worst reporter, then optimize for the good ones.
Don't self-host ClickHouse for a project this size. It's excellent and it was the right call on paper. For a two-person team it's one more thing to keep alive at 2am.
Ask the distribution question on day one. More below.

Where it is now

Honest status: it works, it's verified end-to-end (real upstream call → real
model response → HTTP 200, captured in the repo), and it has zero users and
zero revenue.

I built it. I did not distribute it. Those are different skills and I'm only
good at one of them — which is exactly the thing nobody tells you when they say
"just build it and they will come."

The code is public, and the notes above are the parts I'd have wanted someone to
hand me:

→ https://github.com/page6668/He-API

If you're building something similar, I'm happy to answer questions in the
comments — especially on the usage-accounting and hot-reload parts, which are
where I'd do it differently a second time.

And if you're based in Southeast Asia or the Middle East and want to run an API
reselling business on top of this cost advantage, the whole thing — source,
live environment, domain, ops scripts, handover docs — is available. Details by
email; it's in my profile.

DEV Community: page6668