I built a 200-line OpenAI-compatible gateway with automatic dual-upstream failover

Harvey He — Mon, 31 Aug 2026 16:51:27 +0000

Last week DeepSeek's API went flaky for ~20 minutes at 3am and my batch job went red. That was the last straw — I stopped trusting a single provider.

If you call DeepSeek or Qwen through the OpenAI SDK, the cheapest failover isn't rewriting your app. It's putting an OpenAI-compatible gateway in front: change base_url, zero changes to business code.

Here's the ~200-line FastAPI version I run:

One /v1/chat/completions entry that proxies to upstreams
Two upstreams: SiliconFlow (primary) + DeepSeek official (fallback)
On 8s timeout or 5xx, auto-switch
Per-user quota + logs in local SQLite

The switch is ~20 lines:

async def chat(request, user):
    for upstream in [PRIMARY, FALLBACK]:
        try:
            return await forward(upstream, request, timeout=8)
        except (TimeoutError, Upstream5xx):
            log(f"switch {upstream} -> next")
            continue
    raise HTTPException(502)

Gotchas:

Upstreams return usage under different field names; normalize to OpenAI format or your tokenizer miscounts
Streaming is SSE; when switching, don't truncate already-sent chunks
Store quota as integer tokens, not float, or concurrency gives you negative balances

If you don't want to run this yourself, keheai.com does it for you — dual-upstream failover, 330k free tokens to try, self-serve keys. I built it for my own use and opened it up.

Want the load-test script? I can post it.

用 200 行 Python 搭一个 OpenAI 兼容网关（双上游自动容灾）

Harvey He — Mon, 31 Aug 2026 16:51:26 +0000

上周 DeepSeek 官方接口凌晨抖了 20 分钟，我一个跑批任务全红了。那天之后我决定不再把命系在单个供应商上。

如果你也用 OpenAI SDK 调 DeepSeek / Qwen，最省事的容灾方案不是重写代码，而是前面挡一层 OpenAI 兼容网关：换 base_url 就行，业务代码一行不动。

我是这么搭的（约 200 行 Python + FastAPI）：

统一 /v1/chat/completions 入口，把请求转给上游
配两个上游：硅基流动（主）+ DeepSeek 官方（备）
主上游超时 8 秒或返回 5xx，自动切备
每用户配额和日志落在本地 SQLite

核心切换就 20 行：

async def chat(request, user):
    for upstream in [PRIMARY, FALLBACK]:
        try:
            return await forward(upstream, request, timeout=8)
        except (TimeoutError, Upstream5xx):
            log(f"switch {upstream} -> next")
            continue
    raise HTTPException(502)

踩过的坑：

上游返回的 usage 字段名不一样，要统一成 OpenAI 格式，不然你的 tokenizer 算错
流式为 SSE，切上游时要保证已发出的 chunk 不被截断
配额别用 float 存，用整数 token 数，否则并发下会出现负额度

如果你不想自己维护这套，也可以用现成的 keheai.com —— 双上游容灾、免费 33 万 tokens 试用、自助发 key。我搭它就是为了自己用，顺手开放出来。

你想自己搭的话，上面这套够跑中小流量了。要压测脚本我也可以发。

DEV Community: Harvey He

I built a 200-line OpenAI-compatible gateway with automatic dual-upstream failover

用 200 行 Python 搭一个 OpenAI 兼容网关（双上游自动容灾）