The Hardest Part of a Multi-Model AI API Isn’t Routing Requests
At first glance, building a unified AI API seems straightforward:
- Receive a request
- Choose a provider
- Forward the request
- Return the response
But once an application starts using multiple AI models in production, the real challenge becomes clear.
The difficult part is not routing HTTP requests.
The difficult part is handling the differences between models without hiding information that developers still need.
I’ve been thinking about this problem while building ApiHub, a unified API platform for accessing multiple AI models.
Here are some of the most important lessons I’ve learned so far.
One endpoint does not automatically mean compatibility
Many AI providers now offer APIs that look similar to the OpenAI API.
That makes the first integration easier, but similar endpoints do not always mean identical behavior.
Two models may both accept a request like this:
{
"model": "model-name",
"messages": [
{
"role": "user",
"content": "Summarize this document."
}
],
"temperature": 0.7,
"stream": true
}
However, the models may behave differently when the request includes:
- Tool calls
- JSON output
- System messages
- Image input
- Large context windows
- Structured response formats
- Reasoning parameters
- Unsupported sampling parameters
A request may succeed with one model and fail with another, even when both are described as OpenAI-compatible.
This means model switching is not always as simple as changing the model name.
Normalize what is common, expose what is different
A unified API should provide a consistent interface for common functionality.
For example:
- Authentication
- Basic chat completion requests
- Streaming events
- Usage records
- Error structures
- Request identifiers
But it should not pretend that every model has exactly the same capabilities.
If a platform hides too many differences, developers may only discover them after something breaks in production.
A better approach is:
Normalize the common behavior, but clearly expose model-specific capabilities and limitations.
This gives developers convenience without creating false compatibility.
Model capability metadata is essential
A model name alone does not tell developers enough.
Before sending a request, an application may need to know:
- Does this model support streaming?
- Does it support tool calling?
- Can it process images?
- Does it support JSON mode?
- What input formats are accepted?
- What is the context limit?
- Which parameters are ignored or rejected?
- Is the model currently available?
A unified platform could provide metadata such as:
{
"id": "example-model",
"capabilities": {
"streaming": true,
"tool_calls": false,
"json_mode": true,
"vision": false
},
"input_modalities": [
"text"
],
"output_modalities": [
"text"
],
"context_window": 128000,
"status": "available"
}
With this information, developers can validate requests before sending them.
It also makes model routing more reliable.
For example, an application should not route a vision request to a text-only model simply because that model is currently cheaper.
Error normalization matters more than expected
Different providers return different status codes, error messages, and response formats.
One provider may return:
{
"error": {
"message": "Insufficient balance"
}
}
Another may return:
{
"code": "ACCOUNT_QUOTA_EXCEEDED",
"detail": "Your available quota is insufficient."
}
Another provider may return a generic HTTP 500 response.
For an application using several providers, these differences make error handling difficult.
A unified error format could look like this:
{
"error": {
"type": "insufficient_balance",
"message": "The request could not be completed because the available balance is insufficient.",
"provider": "example-provider",
"provider_error_code": "ACCOUNT_QUOTA_EXCEEDED",
"request_id": "req_123456"
}
}
The normalized type allows applications to handle the error consistently.
The original provider information remains available for debugging.
This is important because normalization should improve clarity, not remove useful details.
Streaming is rarely completely identical
Streaming responses appear simple because most APIs use Server-Sent Events.
But providers may still differ in how they send:
- Initial role information
- Empty content chunks
- Reasoning content
- Tool call arguments
- Finish reasons
- Usage statistics
- Error events
- The final termination message
Applications that depend on a specific event order may work with one model and fail with another.
A unified API needs to decide:
- Which events should be normalized?
- Should empty chunks be preserved?
- How should reasoning content be represented?
- How should interrupted streams report errors?
- Should token usage be included in the final event?
These details are easy to ignore during a basic demo, but they become important in production.
Billing needs both consistency and transparency
Different providers calculate usage in different ways.
Pricing may depend on:
- Input tokens
- Output tokens
- Cached input
- Reasoning tokens
- Context length
- Model tier
- Batch requests
- Provider-specific discounts
A unified platform should make billing easier to understand, but it should not reduce everything to one unexplained number.
A useful usage record might include:
{
"usage": {
"input_tokens": 1250,
"output_tokens": 420,
"cached_input_tokens": 800,
"total_tokens": 1670
},
"billing": {
"currency": "USD",
"input_cost": 0.0012,
"output_cost": 0.0021,
"total_cost": 0.0033
}
}
Developers should be able to understand how the final cost was calculated.
This becomes especially important when an application switches between providers automatically.
Model routing requires more than price comparison
A basic router might select the cheapest available model.
But production routing usually needs to consider more than cost:
- Required capabilities
- Current availability
- Latency
- Context size
- Tool-calling support
- Output quality
- Rate limits
- Regional availability
- Historical error rates
For example, the cheapest model is not useful when it does not support the required input format.
A better routing process might look like this:
Request received
↓
Validate required capabilities
↓
Filter unavailable models
↓
Apply latency, quality, and cost rules
↓
Select model
↓
Send request
↓
Fallback if the failure is retryable
Fallback also needs clear rules.
An invalid prompt should not automatically be retried across five providers.
A temporary upstream timeout might be a valid reason to try another model.
A unified API should not make models look identical
Developers want simpler integrations, but they also need predictable behavior.
The goal should not be to make every model appear interchangeable.
The goal should be to reduce unnecessary integration work while preserving important differences.
That means a unified API should provide:
- A consistent interface for common operations
- Clear capability metadata
- Predictable error categories
- Transparent usage and billing records
- Stable streaming behavior
- Access to provider-specific information when needed
This is the direction I’m exploring while building ApiHub.
The routing layer is only the beginning.
The more difficult work is creating a consistent developer experience without hiding the differences that affect application behavior.
When you switch between AI models, what usually breaks first?
Prompt behavior, tool calls, streaming, error handling, or billing?
Disclosure: I’m building ApiHub. This article shares some of the design problems I’m exploring while working on a multi-model AI API platform.
Top comments (1)
Routing is the visible part; contract stability is the production problem. If two models return different uncertainty, refusal, streaming, or tool-call behavior, a common endpoint can hide the difference until the product starts making different decisions.