I Built a Multi-Provider AI API Gateway — One Key for 18 Models, Auto-Failover, Time-of-Day Pricing

#ai #llm #api #backend

If you've built anything on top of LLM APIs, you've probably hit the same wall I did: every provider wants its own account, its own top-up, its own API key. DeepSeek for coding, GLM for chat, Kimi for long context, Qwen for multilingual — suddenly your code is full of if provider == 'deepseek' branches, and when one provider goes down, your whole service goes with it.

Here's how I solved it: a single OpenAI-compatible endpoint in front of 18+ models, with smart routing, automatic failover, and pricing that follows upstream peak/valley windows. No vendor lock-in, no 3am pager duty.

1. One OpenAI-compatible layer

The client only changes base_url. Model names are passed as-is; the gateway routes internally.

from openai import OpenAI

client = OpenAI(
    base_url="https://yingsuan.top/v1",
    api_key="YOUR_KEY"
)

resp = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "write a quicksort"}]
)
print(resp.choices[0].message.content)

Swapping providers = changing a config, not your codebase.

2. Smart triage — let the gateway pick the model

Pass model=auto and the gateway profiles the request: simple tasks go to free/cheap models, complex reasoning goes to flagship. It also downgrades one tier during peak hours to save cost, and keeps flagship during off-peak.

The profiling is pure rules — no extra LLM call:

function profileRequest(body) {
  const content = body.messages.map(m => m.content).join(' ');
  const hasCode = /```
{% endraw %}
|def |function |SELECT /.test(content);
  const hasComplex = /analyze|reason|prove|optimize|architect|refactor/.test(content);
  const estTokens = estimateTokens(content);

  if (hasComplex || estTokens > COMPLEX_THRESHOLD) return 'complex';  // → flagship (threshold tuned per workload)
  if (hasCode || estTokens > STANDARD_THRESHOLD)  return 'standard';    // → standard (threshold tuned per workload)
  return 'simple';                                        // → free
}
{% raw %}

Most daily traffic (translation, summarization, simple Q&A) never needs a flagship model. Routing it to cheaper tiers cuts cost visibly.

3. Multi-provider failover — zero-perception switching

The critical part. Same model, multiple providers. On 429/5xx/timeout from the primary, automatically retry on the backup provider.

Three design calls worth sharing:

Passive health tracking, no active ping. Many gateways ping all providers on a timer (real cost). I only mark a provider "unhealthy for a while" when a failover attempt fails. Zero probe overhead.
Backoff to prevent cascading failure. Exponential backoff, increasing the wait between retries — don't take down the backup too.
Multi-dimensional provider ranking: health, time-slot weight, latency. Not random, not hardcoded priority.

Real production log (anonymized, early-stage recording):


console
[2026-08-17 17:15:03] FAILOVER deepseek-v4-flash primary 429 → SiliconFlow → ok
[2026-08-17 17:16:21] FAILOVER glm-4-flash primary timeout → Zhipu → ok
[2026-08-17 17:22:40] FAILOVER deepseek-v4-flash primary 503 → Volcengine → ok
... (8 switches, 8 succeeded)

8 switches, 8 succeeded, users noticed nothing. Not a "guaranteed uptime" claim — real logs.

4. Time-of-day pricing

Upstream providers (e.g. DeepSeek) already have peak/valley pricing — expensive during upstream peak windows, cheap off-peak. A gateway with fixed markup wastes the off-peak advantage. Mine senses the current time slot and adjusts the downstream markup to follow upstream peaks and valleys: slightly higher at peak to cover cost, lower at off-peak. Users don't think about the clock — same model is just cheaper off-peak.

Try it

I turned this into a working gateway. Free tier: 100 API calls + 20 calls on DeepSeek V4-Flash (flagship, 1M context). Email signup, no credit card:

👉 https://yingsuan.top/payment.html#free-trial

API docs with Python/Node examples: https://yingsuan.top/api.html

Happy to compare notes if you're building something similar.

DEV Community