Building an OpenAI-Compatible API Gateway: What I Learned About Multi-Model Routing
I've been building ModelBridge — an OpenAI-compatible API gateway that unifies GPT, Claude, Gemini, DeepSeek, Qwen, and more behind a single SDK.
Here are a few things I wish I knew before I started.
1. "OpenAI-compatible" means different things to different providers
Most providers accept the same request shape. But behavior differs significantly:
- System prompts — some providers treat them as strict instructions, others as suggestions.
- Structured output — some enforce JSON Schema strictly, others silently fall back to plain text.
- Streaming — chunk formats vary. Finish reasons differ. Error handling is inconsistent.
The takeaway: "compatible" is a spectrum, not a binary.
2. Routing is not "choose a model"
Real users don't ask for "many models." They ask for a model that fits a constraint:
- Lower latency
- Better reasoning
- Lower cost
- Longer context
- Better at code
So routing becomes a policy engine, not a switch statement. You need to know what each model can do, how much it costs, how it behaves under load, and which tasks it's appropriate for.
3. Latency variance matters more than average speed
A model that usually responds in 1s but sometimes takes 12s feels unreliable, even if the average looks fine.
We started tracking p95 and p99 latency for every provider. That changed how we think about fallback and failover.
4. Streaming is where the abstraction breaks
Normal request-response is easy to normalize. Streaming is where you see the real differences:
- Delta semantics
- Finish reasons
- Heartbeat behavior
- Partial failures
- Connection endings
Building a stream normalization layer was the most humbling part of the project.
5. Usage accounting is closer to distributed state than billing
In a multi-model gateway, usage is part of runtime truth. You need to track:
- Received → admitted → forwarded → partially streamed → completed → failed → compensated
Not a single "token in / token out" record.
What I'm building
ModelBridge — one OpenAI-compatible API, multiple models, pay-as-you-go.
No vendor lock-in. No complex contracts. Just build.
What's next
I'm continuing to improve routing intelligence, cost optimization, and reliability. If you're building in this space too, I'd love to hear what challenges you've run into.
Top comments (0)