Two months of full-time work went into this. The part that nearly sank it was not
the routing layer, the Kafka pipeline, or the admin console. It was token
accounting.
I built He-API: a self-hosted gateway that puts six Chinese model providers
(DeepSeek, Kimi, Qwen, Doubao, ERNIE, GLM) behind a single OpenAI-compatible
endpoint. The thesis is boringly practical — Chinese models run 80–90% cheaper
than GPT-4/Claude-class Western models for most workloads, but if you're a
developer outside China, actually using them is painful: docs in Chinese,
payment methods that don't work internationally, inconsistent auth, no shared
format.
So: one endpoint, one key, one request shape. Here's the architecture, and then
the parts that cost me the most time.
The shape of it
Client (any OpenAI SDK)
→ Nginx + Let's Encrypt
→ api-gateway (Go)
→ entitlement resolver (Redis)
→ rate limiter (Redis + Lua)
→ model router
→ provider adapter (kimi / deepseek / qwen / doubao / ernie / glm)
→ upstream provider
→ Kafka → ClickHouse (usage events)
→ PostgreSQL (billing, models, providers)
15 services on systemd — all Go except a Next.js admin console. Postgres,
Kafka, Redis, ClickHouse. 22 migrations, all idempotent.
That's the diagram everyone draws. Here's what actually bit.
1. "OpenAI-compatible" is 90% edge cases
The happy path — a non-streaming completion — took an afternoon. Everything
after that was tails.
Streaming is where it starts. Providers don't agree on chunk shape. Some send
data: [DONE], some just close the connection. Some include a usage block in
the final chunk; some omit it entirely on streams and only report usage on
non-streaming calls. If you're building a gateway, you discover that
"compatible" means "compatible for the paths you happened to test."
Then error shapes. A 400 from one provider is a JSON body with an error.message
field; from another it's a bare string; from a third it's a 200 with the error
inside the body. Your gateway has to normalize all of that into something that
doesn't break the client's existing error handling — otherwise "zero code
changes" is a lie the first time a provider sneezes.
2. Usage accounting is a schema problem, not a code problem
This is the one that nearly sank it.
If you bill per token, the token count is money, and money is never a float.
Prices land around ¥0.1–¥2 per million tokens, which means per-request amounts
are genuinely small — and if you do the arithmetic in float64 you will drift,
and your drift will compound across millions of rows.
So: NUMERIC in Postgres, integer micro-units in the hot path, and a deliberate
decision about where rounding happens. Then, on top of that, the reconciliation
problem — because providers report usage inconsistently (see above), you can't
just trust the upstream number. The policy I settled on is stricter than estimating: if the stream never delivers a usage chunk, nothing is charged at all. That keeps estimates out of the money path — at the cost of the ledger being unable to tell you about the attempts it declined to bill, which is a gap I'd close with an explicit usage_source column and a record of unmetered attempts
The lesson: decide your money representation before you write the first billing
line, not after you find a 0.3% discrepancy you can't explain.
3. Hot reload beats redeploys
Provider keys used to require a restart. That meant a redeploy — and therefore a
window where requests fail — every time a key rotated.
The fix is unglamorous and it removed an entire class of operations pain: the
gateway polls an internal active providers endpoint every 30 seconds and swaps
the in-memory provider set atomically. Keys are encrypted at rest with
AES-256-GCM, decrypted only in memory. Rotating a key is now "save in the admin
console, wait 30 seconds, done."
If you're building anything that talks to multiple third-party APIs: build the
reload path before you need it. Ops pain compounds faster than feature requests.
4. Rate limiting needs two axes, not one
My first limiter was a single per-key QPS number. It failed in a way that took
me a while to see: internal sub-requests were consuming the same budget as the
user-facing request, so a free-tier account with QPS=1 rate-limited itself.
The version that works has two axes — a steady per-key rate plus a burst bucket,
with burst size scaled by account age and spend. New accounts get a tiny burst;
accounts with history get a generous one. It's crude, but it kills almost all
automated abuse without touching legitimate power users.
Gate the counter itself carefully, too. A read-then-write in application code
races under load; that check-and-increment belongs in a single atomic Lua script
against Redis.
5. The base_url trap
Small, embarrassing, worth writing down. Most OpenAI SDKs concatenate paths onto
your base_url, so if you document https://api.example.com and the SDK appends
/v1/chat/completions, you 404. Some SDKs want the host, some want it with
/v1. Get this wrong and your very first integration attempt fails for a reason
that has nothing to do with your actual product.
What I'd do differently
- Write the billing correctness tests first. Not because billing is interesting, but because everything else is cheap to change and that isn't.
- Assume every upstream field is optional. Design the normalization layer around the worst reporter, then optimize for the good ones.
- Don't self-host ClickHouse for a project this size. It's excellent and it was the right call on paper. For a two-person team it's one more thing to keep alive at 2am.
- Ask the distribution question on day one. More below.
Where it is now
Honest status: it works, it's verified end-to-end (real upstream call → real
model response → HTTP 200, captured in the repo), and it has zero users and
zero revenue.
I built it. I did not distribute it. Those are different skills and I'm only
good at one of them — which is exactly the thing nobody tells you when they say
"just build it and they will come."
The code is public, and the notes above are the parts I'd have wanted someone to
hand me:
→ https://github.com/page6668/He-API
If you're building something similar, I'm happy to answer questions in the
comments — especially on the usage-accounting and hot-reload parts, which are
where I'd do it differently a second time.
And if you're based in Southeast Asia or the Middle East and want to run an API
reselling business on top of this cost advantage, the whole thing — source,
live environment, domain, ops scripts, handover docs — is available. Details by
email; it's in my profile.
Top comments (3)
For the local estimation fallback, can an invoice distinguish estimated usage from provider-reported usage? I'd test a stream that emits some text, then disconnects before its usage chunk, followed by a client retry that gets a full response. The ledger should keep both attempts under the logical request, show which counts are estimates, and make the charging policy explicit rather than silently replacing an estimate or counting one attempt twice. Integer micro-units fix arithmetic drift; that fixture tests whether the measured thing is the thing you intended to bill.
no — and you've found a real gap, not a rounding detail.
Right now the ledger physically cannot distinguish the two, because there
are no estimates in it. The settled policy went the other way from what the
post implied:
usage.recordedis emitted ONLY on a clean completion with a tail usage chunk present. A partial or client-disconnected stream emits no billing event at all — not an estimate, nothing.Your fixture, walked through on the current code:
So no double-count — but you're right that the ledger goes silent about
attempt 1. To answer "what did we serve and not bill?" today you'd have to
diff ClickHouse against the ledger, which is exactly the assertion a fixture
should make, and there isn't one.
On "both attempts under the logical request": there's no client-supplied
idempotency key and no attempt linkage. ledger_key dedups Kafka redelivery
of the same event; it does not correlate a retry with its predecessor. I'd
argue the correct default is that attempt 2 IS a separate charge — the
upstream billed us twice too. So the fixture should assert visibility, not
merging.
Two things I'd add on a second pass, both basically your list:
One asymmetry worth knowing, since it looks like a bug from outside: the
rate-limit axis and the money axis deliberately disagree. On a disconnect
after the tail usage chunk, the Redis TPM counter deducts prompt tokens —
money is still only charged on clean completion. Two axes, two policies.
And your closing line is the sharpest version of the point: integer
micro-units fix the arithmetic, which was always the easy half. Whether the
measured thing is the thing you intended to bill is a semantics question
with no test — and the line in the post about a "local estimation fallback"
was loose of me. What's actually there is "no tail usage → no charge",
which is a worse sentence and a better policy.
The 'cannot tell you about the requests it declined to bill' line is the more interesting half of that gap, and I would put a field on it even if the billing policy stays exactly as it is - otherwise the un-billed served work is invisible to precisely the people who would want to change the policy. One field that costs almost nothing: record a completion status on every request, including the ones that emit no usage event, so the ledger can report served N and billed M as two numbers rather than one number and an absence. Disclosure: I work on jzstoken.com, a prepaid AI API option - prepaid balance shows the same asymmetry from the other side, because un-billed served work appears as spend that never depleted.