Most "one API for every model" services still route you to a single upstream vendor per model. The harder problem is what happens when that vendor throttles you at 3 a.m.: retries stack up, latency explodes, and your bill keeps growing for requests that never returned a token.
I have been building an OpenAI-compatible gateway that answers a different question: what if the supply side were a marketplace instead of a fixed vendor list? Here are the engineering notes from that build, the parts that were genuinely hard, and the trade-offs I would not make again.
1. One endpoint, many sellers
Sellers list spare model capacity with their own prices. A request arrives at POST /v1/chat/completions, the gateway picks a seller whose quote matches the model and the request shape, forwards it, and settles the difference afterwards. Clients keep the OpenAI SDK - only base_url and api_key change.
Keeping it protocol-compatible sounds trivial until you hit the small divergences: stream_options, tool-call deltas, logprobs, response_format, and providers that quietly drop max_completion_tokens. Every adapter needs a conformance test against the OpenAI schema, otherwise clients break in ways users cannot debug.
2. Failover is a scheduling problem, not a retry loop
The naive fix for 429s is exponential backoff. That makes p99 worse, because every client retries into the same saturated seller. What worked better:
- Track per-seller health separately for each model, not per seller overall.
- Treat 429, 5xx and mid-stream disconnects as routing signals, not errors to hide. Re-route to a healthy seller on the first token; if tokens have already streamed, finish the stream and mark the seller degraded.
- Cap concurrent in-flight requests per seller so you saturate but do not stampede.
- Never bill a request that produced no tokens. Billing on delivery, not on attempt, removes the incentive to silently retry.
Across roughly 51 upstream providers, this moved rate-limit failures from a few percent of calls to a rounding error in daily dashboards.
3. Transparent pricing beats cheap pricing
Discounts are easy to advertise and hard to operate. What developers actually ask for is a per-request price they can see before they send it, and a line item afterwards that matches. That means the gateway has to price the request (tokens, model, region, cache) before routing, then reconcile with the delivered usage - including the annoying case where a seller reports different usage than the tokenizer predicted.
4. Agents need a different surface than chat
Half of the traffic now comes from agents and IDEs rather than chat UIs. They want model discovery and tool-calling with schemas, not a wall of text. Shipping an MCP server next to the HTTP API turned out to be the cheapest integration path: the agent lists models, calls chat completions, and never needs a bespoke SDK.
What I would do differently
- Build the conformance suite before the second adapter. Retrofitting nine adapters is unpleasant.
- Store every upstream response body for 24 hours. Debugging "the model answered differently through your gateway" is impossible without the raw payload.
- Decide early whether you optimize for latency or for price. Trying to win both at once produces a router nobody can reason about.
If you want to poke at the result, it is running at LONG AI - 100+ live LLM and video models, up to 78% below list price, OpenAI SDK compatible. The MCP server is open source, and I am happy to answer questions about the routing or settlement design in the comments.
Top comments (0)