I built an OpenAI-compatible gateway in one Python file. Here's the architecture.

I built an OpenAI-compatible gateway in one Python file. Here's the architecture.

Not a framework. Not a library. One gateway.py file that proxies
requests from one OpenAI-compatible endpoint to 15 Chinese AI models
across 4 providers.

Here's how it works, and what I'd do differently.

The core: one endpoint, many upstreams

The whole thing is a FastAPI app. Every request hits the same path:

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    caller = verify_key(request)
    body = await request.json()
    model = body["model"]

    # Resolve the model string to an upstream provider
    cfg = resolve_upstream(model)   # {"base_url": ..., "api_key": ..., "model": ...}

    # Forward the request, swapping in the real upstream model name
    body["model"] = cfg["model"]
    resp = await client.post(
        f"{cfg['base_url']}/chat/completions",
        headers={"Authorization": f"Bearer {cfg['api_key']}"},
        json=body,
    )
    return resp.json()

The UPSTREAMS dict is the heart of it:

UPSTREAMS = {
    "deepseek-chat":  {"base_url": "https://api.deepseek.com/v1", "api_key": os.getenv("DEEPSEEK_API_KEY"), "model": "deepseek-chat"},
    "qwen-max":       {"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1", "api_key": os.getenv("QWEN_API_KEY"), "model": "qwen-max"},
    "glm-4-plus":     {"base_url": "https://open.bigmodel.cn/api/paas/v4", "api_key": os.getenv("ZHIPU_API_KEY"), "model": "glm-4-plus"},
    "kimi-k3":        {"base_url": "https://api.moonshot.cn/v1", "api_key": os.getenv("MOONSHOT_API_KEY"), "model": "kimi-k3"},
    # ... 11 more
}

Adding a model is a config entry, not a code change. When Kimi K3
launched, adding it took 20 minutes — three lines in the dict, one in
a display-name map, one in a provider map, plus a models.json entry.

The layers around the core

A gateway that just forwards requests is trivial. The hard parts are
the layers:

Auth — API keys hashed with SHA-256. The caller_id in every log
is sha256(api_key)[:8], so the key itself never touches a log file.

Rate limiting — a sliding window per key, in memory. 60 RPM default.

Quota — the tricky one. Free users get 500K tokens/month. The check
and the deduction have to be atomic, or concurrent requests slip
through. I learned this the hard way — 19 users pushed past 500K
before I moved the check inside the _USERS_LOCK critical section.

Streaming — SSE passthrough. Tokens flow through as they're
generated. The first_token_ms metric tells you how fast the upstream
model actually starts responding, which matters more than total
latency for UX.

Caching — one header. X-Cache-TTL: 3600. Same prompt within the
hour returns from cache. Zero tokens. Zero upstream cost. 200ms
instead of 1200ms.

Observability — an in-memory ring buffer (200 entries) plus a
JSONL file (5000 entries) for persistence. Every request logged with
request_id, caller, model, latency, tokens, status.

The mistake I'd fix

The ring buffer loads 200 entries from disk on startup, and the admin
dashboard reads from memory. For a while the dashboard showed zero
logs because the filter was excluding type="response" entries — the
most common type — leaving nothing to display.

The fix was a fallback: prefer detailed logs, fall back to basic ones.
Never return zero when the buffer has 200 entries. Sounds obvious.
Wasn't.

What the gateway taught me

The forwarding logic is 50 lines. The production-grade stuff — auth,
quota, rate limiting, streaming, caching, observability, error
handling — is 5000 lines. The gap between "works on my machine" and
"works for paying users" is always bigger than you think, and it's
always in the layers, not the core.

Want to try it? The gateway is live:

→ aibridge-api.com/playground.html (15 models, no signup)
→ aibridge-api.com/prompts.html (24 prompts, no signup)

Free tier: 500K tokens/month. No credit card.