DEV Community

Harvey He
Harvey He

Posted on

I built a 200-line OpenAI-compatible gateway with automatic dual-upstream failover

#api #python #openai #llm

Last week DeepSeek's API went flaky for ~20 minutes at 3am and my batch job went red. That was the last straw — I stopped trusting a single provider.

If you call DeepSeek or Qwen through the OpenAI SDK, the cheapest failover isn't rewriting your app. It's putting an OpenAI-compatible gateway in front: change base_url, zero changes to business code.

Here's the ~200-line FastAPI version I run:

  1. One /v1/chat/completions entry that proxies to upstreams
  2. Two upstreams: SiliconFlow (primary) + DeepSeek official (fallback)
  3. On 8s timeout or 5xx, auto-switch
  4. Per-user quota + logs in local SQLite

The switch is ~20 lines:

async def chat(request, user):
    for upstream in [PRIMARY, FALLBACK]:
        try:
            return await forward(upstream, request, timeout=8)
        except (TimeoutError, Upstream5xx):
            log(f"switch {upstream} -> next")
            continue
    raise HTTPException(502)
Enter fullscreen mode Exit fullscreen mode

Gotchas:

  • Upstreams return usage under different field names; normalize to OpenAI format or your tokenizer miscounts
  • Streaming is SSE; when switching, don't truncate already-sent chunks
  • Store quota as integer tokens, not float, or concurrency gives you negative balances

If you don't want to run this yourself, keheai.com does it for you — dual-upstream failover, 330k free tokens to try, self-serve keys. I built it for my own use and opened it up.

Want the load-test script? I can post it.

Top comments (0)