DEV Community

Cover image for Building an AI Gateway: Model Routing, Fallbacks, Budgets and Provider Abstraction
Serif COLAKEL
Serif COLAKEL

Posted on Edited on

Building an AI Gateway: Model Routing, Fallbacks, Budgets and Provider Abstraction

Designing a centralized infrastructure layer to manage multiple LLM providers, route requests intelligently, handle failures, and enforce usage policies.

1. Why Do We Need an AI Gateway?

As AI adoption grows, applications increasingly depend on multiple model providers and deployment environments.

One application may use OpenAI for general reasoning, Anthropic for code analysis, Google for multimodal tasks, and internally hosted models for specific workloads.

Initially, integrating each provider directly into an application seems straightforward.

We initialize an SDK, configure an API key, select a model, and start making requests.

However, as the number of applications and teams increases, managing these integrations independently introduces several problems:

  • Every application manages its own provider credentials.
  • Model configuration becomes duplicated.
  • Rate limits and retry policies are implemented differently.
  • Switching providers requires application changes.
  • Usage and spending become difficult to attribute.
  • Provider failures affect applications independently.
  • Security and data-handling policies become inconsistent.

Consider an organization with several AI-powered applications:

  • An internal developer assistant.
  • A document analysis service.
  • A financial research agent.
  • A customer support assistant.
  • A multi-agent orchestration platform.

Each application may use different models, but many of their infrastructure requirements are identical.

Instead of implementing provider management separately in every application, we can introduce a centralized infrastructure layer.

An AI Gateway acts as a unified entry point between applications and model providers.

It centralizes model access, provider routing, authentication, usage policies, and failure handling.

The objective is not simply to reduce the number of API integrations.

It is to make model access a shared, governed, and independently manageable platform capability.

2. What Is an AI Gateway?

An AI Gateway is an infrastructure component that sits between AI applications and model providers.

Applications send model requests through a common gateway endpoint. The gateway handles provider-specific integrations and applies the configured routing and access policies.

A simplified architecture:

                 AI Applications
                        |
          +-------------+-------------+
          |             |             |
          v             v             v
       Chat App     Agent Runtime   API Service
          |             |             |
          +-------------+-------------+
                        |
                        v
                   AI Gateway
                        |
          +-------------+-------------+
          |             |             |
          v             v             v
       Identity      Model Router   Policies
       Management       Engine      & Quotas
          |             |             |
          +-------------+-------------+
                        |
              +---------+---------+
              |         |         |
              v         v         v
           OpenAI   Anthropic   Google
              |         |         |
              +---------+---------+
                        |
                        v
                Model Response
                        |
                        v
                   Application
Enter fullscreen mode Exit fullscreen mode

A gateway can provide several capabilities:

Capability Responsibility
Provider abstraction Common interface for multiple providers
Model routing Select the appropriate model or deployment
Provider routing Select an upstream provider serving a model
Load balancing Distribute traffic across eligible deployments
Failover Switch to an alternative after eligible failures
Authentication Centralize application authentication
Authorization Control access to providers and models
Rate limiting Enforce request and token quotas
Budget management Track and restrict usage
Observability Record request, latency, and usage metadata
Policy enforcement Apply organizational model-access rules

Not every AI Gateway implements all these capabilities in the same way.

Some focus on request routing and provider normalization, while others offer more comprehensive enterprise governance features.

It is also important to distinguish an AI Gateway from an agent runtime.

The gateway manages model access. The agent runtime manages planning, tool execution, memory, and agent orchestration.

These layers can work together without taking responsibility for each other's operations.

3. AI Gateway vs. Model Provider vs. AI SDK

These technologies are complementary.

Model Provider

A model provider hosts or exposes the actual inference service.

Examples include OpenAI, Anthropic, Google, Amazon Bedrock, Azure AI, and self-hosted inference endpoints.

AI SDK

An AI SDK provides application-level APIs for interacting with models.

For example, Vercel AI SDK provides generateText, streamText, structured output, and tool-calling abstractions.

AI Gateway

An AI Gateway provides infrastructure-level management of model requests, including routing, provider selection, authentication, and usage policies.

Layer Primary responsibility
AI SDK Application development
AI Gateway Model access infrastructure
Model Provider Model inference

For example, an application can use Vercel AI SDK for its agent implementation while routing all model calls through an AI Gateway.

The application still owns its business logic. The gateway manages the infrastructure surrounding model access.

4. Model Routing: Selecting the Right Model for the Right Task

Model routing is one of the most important capabilities in a multi-provider architecture.

However, the term model routing can describe several different decisions.

We should distinguish them before designing the routing layer.

4.1. Task-Based Model Selection

This determines which model is appropriate for a particular task.

For example:

Task Model requirements
Text classification Low latency and cost
Document extraction Structured output
Complex reasoning Strong reasoning capabilities
Code review Coding performance and tool calling
Image understanding Vision support
Embedding generation Compatible embedding dimensions

Using an expensive reasoning model for every classification request may increase cost unnecessarily.

Likewise, routing every complex analysis task to the cheapest available model may compromise quality.

This decision is generally owned by application-level routing logic or a dedicated classification layer.

It should not be confused with provider failover.

4.2. Provider Routing

After selecting a model, we may have multiple providers or deployments capable of serving it.

For example:

Requested Model: Claude Sonnet
                 |
                 v
              Gateway
                 |
        +--------+--------+
        |        |        |
        v        v        v
     Anthropic  Vertex   Bedrock
        |        |        |
        +--------+--------+
                 |
                 v
              Response
Enter fullscreen mode Exit fullscreen mode

The gateway decides which eligible provider should receive the request.

Routing policies can consider:

  • Provider availability.
  • Recent latency.
  • Throughput.
  • Cost.
  • Region.
  • Data-retention requirements.
  • Provider-specific capabilities.

For example, Vercel AI Gateway can choose among providers based on recent uptime and latency, with additional controls for provider ordering and filtering.

4.3. Deployment Load Balancing

A single logical model may have multiple deployments.

A gateway can distribute traffic across them using strategies such as:

  • Round robin.
  • Random selection.
  • Weighted distribution.
  • Least loaded deployment.
  • Latency-aware selection.

The appropriate strategy depends on traffic characteristics and provider capabilities.

4.4. Semantic or Complexity-Based Routing

A more advanced approach is to classify incoming requests based on complexity.

For example:

Incoming Request
       |
       v
Task Classifier
       |
       +----> Simple Task
       |          |
       |          v
       |      Fast Model
       |
       +----> Complex Task
       |          |
       |          v
       |     Reasoning Model
       |
       +----> Code Task
                  |
                  v
              Coding Model
Enter fullscreen mode Exit fullscreen mode

This can reduce inference costs when a large proportion of requests are simple.

However, this classifier is often an application-level component or a dedicated routing service.

We should not assume that every AI Gateway automatically understands task complexity and selects an appropriate model.

The key architectural distinction: Task selection chooses the model. Provider routing chooses where that model runs. Failover handles failures in either layer.

References: Vercel AI Gateway — Provider Options · OpenRouter — Model Routing

5. Provider Abstraction: One Interface, Multiple Providers

One of the most useful properties of an AI Gateway is provider abstraction.

Instead of making application code depend on specific provider endpoints, we can introduce logical model identifiers.

For example:

Application
    |
    v
general-reasoning
    |
    v
AI Gateway
    |
    v
Provider Resolution
    |
    +----> OpenAI
    |
    +----> Anthropic
    |
    +----> Azure AI
    |
    +----> Self-hosted Model
Enter fullscreen mode Exit fullscreen mode

The application requests general-reasoning, while the gateway resolves that identifier using its model catalog and routing policy.

This allows the underlying deployment to change without requiring every consuming application to change its integration code.

OpenAI-Compatible APIs

Many gateways expose OpenAI-compatible endpoints.

Common examples include:

POST /v1/chat/completions
POST /v1/embeddings
Enter fullscreen mode Exit fullscreen mode

This makes migration relatively straightforward for applications already using compatible clients.

For example:

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: process.env.AI_GATEWAY_URL,
  apiKey: process.env.AI_GATEWAY_API_KEY,
});

const response = await client.chat.completions.create({
  model: 'general-reasoning',

  messages: [
    {
      role: 'user',
      content: 'Explain the purpose of an AI Gateway.',
    },
  ],
});

console.log(response.choices[0].message.content);
Enter fullscreen mode Exit fullscreen mode

The application does not need to know which provider serves general-reasoning.

However, there is an important limitation.

OpenAI compatibility does not guarantee complete model interoperability.

Different providers have different support for:

  • Tool calling.
  • Structured output.
  • JSON Schema.
  • Streaming.
  • Reasoning parameters.
  • Multimodal input.
  • Prompt caching.
  • Response metadata.
  • Cancellation behavior.

A gateway can normalize common request and response structures, but it cannot guarantee identical model behavior.

Provider abstraction should therefore be based on explicit capabilities, not the assumption that every model is interchangeable.

Define a Capability Contract

A model registry can maintain the capabilities that matter to the application:

type ModelCapabilities = {
  streaming: boolean;
  toolCalling: boolean;
  structuredOutput: boolean;
  vision: boolean;
  audio: boolean;
  embeddings: boolean;
  maxContextTokens?: number;
};

type ModelDefinition = {
  id: string;
  provider: string;
  model: string;
  capabilities: ModelCapabilities;
};
Enter fullscreen mode Exit fullscreen mode

A routing policy can then exclude models that do not satisfy the requirements of a request.

For example, a workflow that requires tool calling and structured output should not silently fall back to a model that supports neither.

This is especially important in agentic applications, where a model change can affect the behavior of an entire execution loop.

6. Model Fallbacks: Handling Provider Failures

Provider outages, rate limits, and network failures are inevitable in distributed systems.

A multi-provider architecture allows us to introduce fallback strategies.

Consider an application that requests a model from a primary provider.

The provider becomes unavailable and returns a temporary error.

Without failover, the application has to handle the failure itself.

With a configured gateway fallback, the request can be routed to an alternative deployment or model.

Two Different Fallback Strategies

It is important to distinguish between provider fallback and model fallback.

Provider fallback

The requested model remains the same, but the gateway tries another provider that serves it.

Requested Model
       |
       v
Provider A
       |
     Failed
       |
       v
Provider B
       |
       v
Same Model Response
Enter fullscreen mode Exit fullscreen mode

Model fallback

The primary model fails, so the gateway tries a different model.

Primary Model
       |
     Failed
       |
       v
Fallback Model 1
       |
     Failed
       |
       v
Fallback Model 2
       |
       v
Response
Enter fullscreen mode Exit fullscreen mode

These strategies solve different availability problems.

Example Failure Policy

const fallbackPolicy = {
  primary: 'primary-model',

  providerFallbacks: [
    'approved-provider-a',
    'approved-provider-b',
  ],

  modelFallbacks: [
    'compatible-model-a',
    'compatible-model-b',
  ],

  maxAttempts: 3,
};
Enter fullscreen mode Exit fullscreen mode

This is an illustrative application-level policy, not a universal gateway configuration format.

Actual routing and fallback behavior depends on the gateway.

Retry Is Not the Same as Fallback

A retry attempts the same logical operation again.

A fallback changes the eligible provider or model according to a policy.

For example:

Failure Possible strategy
HTTP 429 Respect retry guidance or use another deployment
HTTP 500 / 503 Bounded retry or failover
Network timeout Retry within an execution deadline
HTTP 400 Do not blindly retry
Authentication failure Correct credentials or configuration
Context window exceeded Reduce input or use a compatible larger-context model
Unsupported capability Select a model that supports the required capability
Content policy rejection Do not automatically bypass the policy

Avoid Retry Storms

If multiple application instances retry simultaneously during an outage, they can increase the load on an already degraded provider.

Use:

  • Exponential backoff.
  • Randomized jitter.
  • Maximum retry attempts.
  • Request deadlines.
  • Provider cooldowns.
  • Circuit-breaker patterns.
  • Retry-After handling.

There should also be one clearly defined retry owner.

If both the application and gateway independently retry without coordinated limits, one logical request can generate many more upstream attempts than expected.

Streaming Requires a Separate Strategy

Streaming introduces another important limitation.

If a provider fails before sending any content, a gateway may be able to route the request to another provider.

But if partial output has already been delivered to the user, switching providers is no longer transparent.

The application may receive duplicate, contradictory, or incomplete content.

For that reason, streaming systems should define how to handle mid-stream failures.

Possible approaches include:

  • Retry before the first emitted content.
  • Terminate the stream and expose a recoverable error.
  • Restart using an explicit new attempt.
  • Use a continuation strategy where supported.

Never assume that every fallback is invisible to the user.

Model Compatibility Matters

A successful HTTP response does not necessarily mean the fallback result is equivalent to the primary model's result.

A fallback might:

  • Generate a different output structure.
  • Use different tool-calling semantics.
  • Have a smaller context window.
  • Interpret system instructions differently.
  • Produce less reliable structured output.

Every fallback should pass the application's capability and quality requirements.

References: Vercel — Model Fallbacks · LiteLLM — Routing and Reliability

7. Authentication, Authorization and Key Management

In enterprise environments, provider credentials quickly become a governance problem.

If each application stores its own provider API keys, the organization loses centralized control.

This can create several risks:

  • Credentials accidentally exposed in application environments.
  • Unauthorized provider usage.
  • Inconsistent key rotation.
  • Shared administrative credentials.
  • Difficult usage attribution.
  • Inconsistent access policies.

An AI Gateway provides an opportunity to centralize provider credentials and application identities.

Application-to-Gateway Authentication

The application authenticates to the gateway with a gateway-specific credential.

The gateway uses its configured provider credentials to access the upstream inference service.

Application
    |
    | Gateway Credential
    v
AI Gateway
    |
    | Provider Credential
    v
LLM Provider
Enter fullscreen mode Exit fullscreen mode

The application does not need direct access to upstream provider credentials.

Virtual Keys

Some gateways provide virtual keys that identify applications, users, teams, or projects.

For example:

Key Purpose Access
frontend-assistant Chat application Approved chat models
research-service Research workflow Reasoning and embeddings
document-worker Document processing Extraction models
development-agent Coding assistant Approved coding models

Virtual keys can enforce model access, usage limits, and spending policies.

However, the exact capabilities depend on the gateway and deployment model.

Identity Propagation

Consider a shared backend serving multiple authenticated users.

The gateway should receive trusted application or user identity metadata derived from the server-side authentication context.

It should not trust arbitrary user identifiers sent directly from a browser.

This matters for:

  • Per-user budgets.
  • Tenant isolation.
  • Audit logs.
  • Access control.
  • Cost attribution.
  • Investigating unexpected usage.

Vercel AI Gateway Authentication

Vercel AI Gateway supports API key authentication and OIDC authentication.

For applications deployed on Vercel, OIDC tokens can be used without maintaining a static gateway key in the application.

For other environments, gateway API keys can be configured through environment variables.

It also supports Bring Your Own Key (BYOK), allowing teams to use their own provider credentials.

An important operational detail: when BYOK credentials fail, the configured gateway fallback behavior may use Vercel-managed credentials. Teams should understand the resulting billing behavior rather than assuming that every request is always billed directly to their provider account.

Reference: Vercel — Authentication and BYOK

8. Budgets, Quotas and Rate Limiting

Centralizing model access also makes it possible to manage usage consistently.

Without a shared gateway, individual applications may implement different rate-limit and budget mechanisms.

A gateway can provide controls at different scopes:

  • Organization.
  • Team.
  • Project.
  • Application.
  • API key.
  • User.
  • Model.

RPM and TPM

Two important rate-limit dimensions are:

RPM — Requests Per Minute

Controls the number of requests in a defined time window.

TPM — Tokens Per Minute

Controls token consumption within a defined time window.

These limits are useful for protecting upstream provider quotas and preventing individual applications from monopolizing shared capacity.

Budget Management

Rate limits control request volume.

Budgets control spending.

For example:

AI Platform Budget

Monthly Limit: $1,000

Development Team: $300
Research Team:    $400
Automation Team:  $300
Enter fullscreen mode Exit fullscreen mode

The gateway can track consumption and enforce configured spending policies.

However, budget behavior is not identical across platforms.

Vercel AI Gateway

Vercel supports team, project, API key, and user budget scopes.

These budgets are soft caps, not exact hard spending limits.

The check occurs when a request starts. The request that crosses the threshold can still finish, and the actual total may exceed the configured limit slightly.

BYOK billing also requires separate consideration because usage billed directly through provider credentials may not be covered by the same gateway spending boundary.

LiteLLM

LiteLLM provides key-, user-, team-, and other scope-based usage controls.

Its budget and virtual-key functionality depends on database-backed usage tracking. Running only the configuration-based proxy without a database does not provide the same budget enforcement.

This is an important deployment distinction.

Budget Enforcement Under Concurrency

Imagine ten concurrent requests arriving when a team has only a small amount of budget remaining.

Depending on the implementation, multiple requests may pass a check before the usage records are fully updated.

For strict spending controls, consider:

  • Atomic budget reservation.
  • Shared quota state.
  • Concurrency limits.
  • Request admission control.
  • Maximum estimated request cost.
  • Reconciliation against actual provider usage.

A gateway budget is an important control, but it should not be confused with a guarantee that an entire distributed AI workflow cannot exceed a fixed monetary amount.

What Should Happen When a Budget Is Exhausted?

Possible policies include:

  1. Reject new requests.
  2. Require additional authorization.
  3. Route to an approved lower-cost model.
  4. Route to an approved self-hosted model.
  5. Reduce usage for non-critical workloads.

The correct behavior depends on business requirements.

A critical workflow should not silently switch to a less capable model simply because its budget has been reached.

References: Vercel — Budgets and Spend Limits · LiteLLM — Budgets and Rate Limits

9. Integrating Vercel AI Gateway with AI SDK v7

One of the advantages of Vercel AI Gateway is its direct integration with the AI SDK.

An important correction to many older examples:

The Gateway provider is now part of the ai package.

You do not need to install a separate provider package for the standard integration.

Step 1: Install the dependency

pnpm add ai
Enter fullscreen mode Exit fullscreen mode

Step 2: Configure authentication

For local development or external deployments:

AI_GATEWAY_API_KEY=your_gateway_api_key
Enter fullscreen mode Exit fullscreen mode

On supported Vercel deployments, OIDC authentication can be used automatically.

Step 3: Make a request

import { gateway, generateText } from 'ai';

const result = await generateText({
  model: gateway('openai/gpt-6-astra'),

  prompt: `
    Analyze this application architecture
    and identify its main scalability risks.
  `,
});

console.log(result.text);
Enter fullscreen mode Exit fullscreen mode

The model identifier follows the provider/model-name format.

The gateway resolves the model and manages the upstream provider request.

Step 4: Configure Model Fallbacks

We can specify alternative models through providerOptions.gateway.models.

import { gateway, generateText } from 'ai';

const result = await generateText({
  model: gateway('openai/gpt-6-astra'),

  prompt: 'Analyze the architectural risks of this system.',

  providerOptions: {
    gateway: {
      models: [
        'anthropic/claude-sonnet-5.5',
        'google/gemini-4-hydrogen',
      ],
    },
  },
});

console.log(result.text);
Enter fullscreen mode Exit fullscreen mode

The gateway tries the primary model first.

If it fails, it proceeds through the configured fallback model list and returns the first successful result.

The model identifiers above are examples from the current AI Gateway catalog documentation. Always verify actual model availability and capability compatibility before deployment.

Step 5: Control Provider Ordering

Model fallback and provider ordering are separate controls.

We can specify a preferred provider order for a particular model:

import { gateway, generateText } from 'ai';

const result = await generateText({
  model: gateway('anthropic/claude-sonnet-5.5'),

  prompt: 'Explain distributed systems.',

  providerOptions: {
    gateway: {
      order: [
        'vertex',
        'anthropic',
      ],
    },
  },
});

console.log(result.text);
Enter fullscreen mode Exit fullscreen mode

The order option controls provider preference.

The only option restricts routing to an explicit provider allowlist.

For example:

providerOptions: {
  gateway: {
    only: ['vertex', 'anthropic'],
  },
}
Enter fullscreen mode Exit fullscreen mode

This distinction is useful when a request is required to use specific provider infrastructure.

The Gateway provider also exposes model discovery and usage reporting capabilities through the AI SDK interface.

For example, available models can be inspected programmatically rather than relying on hardcoded model lists.

References: AI SDK — AI Gateway Provider · Vercel — Provider Options

10. Building a Self-Hosted AI Gateway with LiteLLM

A managed gateway is not always the right solution.

Some organizations need to operate the model access layer inside their own infrastructure because of:

  • Network restrictions.
  • Internal model endpoints.
  • Data residency requirements.
  • Private deployments.
  • Custom authentication.
  • Infrastructure ownership.
  • Provider-specific policies.

LiteLLM provides an open-source gateway with a unified interface for multiple model providers.

It supports model routing, retries, fallbacks, virtual keys, budgets, rate limiting, and usage tracking.

Step 1: Define Logical Models

A simplified configuration:

model_list:
  - model_name: fast-chat
    litellm_params:
      model: openai/gpt-6.1-sol
      api_key: os.environ/OPENAI_API_KEY

  - model_name: reasoning-chat
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  num_retries: 2

  fallbacks:
    - fast-chat: ["reasoning-chat"]
Enter fullscreen mode Exit fullscreen mode

Here:

  • fast-chat is the logical model name exposed by the gateway.
  • The OpenAI model is the primary deployment.
  • reasoning-chat is an alternative model group.
  • num_retries controls retry attempts within the configured routing behavior.
  • fallbacks defines a cross-model fallback relationship.

The model names and provider configuration should be validated against the installed LiteLLM version and actual provider availability.

Step 2: Deploy with PostgreSQL

For a real deployment with virtual keys and budget enforcement, use a database-backed setup.

LiteLLM's official Docker Compose quickstart starts the proxy together with PostgreSQL.

The official setup also requires:

  • LITELLM_MASTER_KEY for administration and management API authentication.
  • LITELLM_SALT_KEY for encrypting provider credentials stored in the gateway database.

The official quickstart is a good starting point:

curl -sSLO \
  https://github.com/BerriAI/litellm/raw/main/litellm/deploy/docker/docker-compose.quickstart.yml

printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' \
  "$(openssl rand -hex 32)" \
  "$(openssl rand -hex 32)" > .env

docker compose -f docker-compose.quickstart.yml up -d
Enter fullscreen mode Exit fullscreen mode

Models and virtual keys can then be configured using the Admin UI or version-controlled configuration.

Important: A standalone LiteLLM proxy running without a database can still provide basic model routing, but virtual keys and database-backed spend tracking and budget enforcement will not work. Do not treat a configuration-only Docker container as a complete budget-managed gateway.

Also, do not use a floating latest image tag for a controlled production deployment. Pin a tested version or image digest and review dependency updates.

Step 3: Connect an application

LiteLLM exposes an OpenAI-compatible API.

An existing OpenAI SDK client can communicate with the gateway:

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: process.env.LITELLM_BASE_URL,
  apiKey: process.env.LITELLM_VIRTUAL_KEY!,
});

const response = await client.chat.completions.create({
  model: 'fast-chat',

  messages: [
    {
      role: 'user',
      content: 'Explain the benefits of model routing.',
    },
  ],
});

console.log(response.choices[0].message.content);
Enter fullscreen mode Exit fullscreen mode

The application uses the logical model name fast-chat.

It does not need to know whether the request is handled by OpenAI or another configured provider.

For production, give each application or workload a dedicated virtual key with the minimum required model access and usage limits.

References: LiteLLM Quickstart · LiteLLM Router Configuration · LiteLLM Fallbacks

11. Choosing an AI Gateway

Several platforms are worth evaluating, but they do not all have the same deployment model or intended use case.

Vercel AI Gateway

A managed model gateway with close integration into the Vercel AI SDK.

Its capabilities include:

  • Multi-provider model access.
  • Provider ordering and filtering.
  • Model fallbacks.
  • BYOK.
  • OIDC authentication.
  • Usage reporting.
  • Team, project, key, and user budgets.

It is attractive when teams want a managed gateway without operating the gateway infrastructure themselves.

LiteLLM

An open-source Python-based gateway with self-hosting and managed deployment options.

Its capabilities include:

  • Broad provider integrations.
  • Model aliases.
  • Routing and load balancing.
  • Retries and fallbacks.
  • Virtual keys.
  • Team and key budgets.
  • Rate limiting.
  • Usage tracking.

It is particularly useful when an organization needs control over its model access infrastructure.

The database requirement for features such as virtual keys and spend tracking should be included in capacity and operations planning.

Portkey

An AI gateway platform with a self-hostable open-source gateway and managed offerings.

Its gateway capabilities include:

  • Provider routing.
  • Fallbacks and retries.
  • Load balancing.
  • Guardrails.
  • Request management.

Teams should evaluate which advanced logging, retention, and compliance capabilities are available in their chosen deployment and plan.

Envoy AI Gateway

An open-source gateway built around Envoy.

It is particularly relevant to teams already operating Kubernetes, Envoy Proxy, or service mesh infrastructure.

Its focus includes model-aware routing, upstream authorization, failover, and gateway-level traffic policies.

It can be a natural choice when AI traffic management needs to become part of an existing platform networking architecture.

OpenRouter

OpenRouter is best understood as a managed model-routing and provider aggregation platform.

It provides access to models through a unified interface and supports provider ordering, filtering, price-oriented routing, latency and throughput preferences, and fallback controls.

It is useful when broad model access and managed provider routing are the primary requirements.

It is not the same deployment model as a self-hosted gateway such as LiteLLM or Envoy AI Gateway.

High-Level Comparison

Platform Deployment model Main strength Important consideration
Vercel AI Gateway Managed AI SDK integration and managed operations Provider and platform dependency
LiteLLM Open source / managed Self-hosting, provider coverage, virtual keys Database and operational ownership
Portkey Open source / managed Routing and guardrails Hosted versus self-hosted feature boundaries
Envoy AI Gateway Self-hosted open source Kubernetes and Envoy integration More infrastructure to operate
OpenRouter Managed Broad model and provider routing Less control over the gateway infrastructure itself

This is a deployment-oriented comparison rather than a complete feature checklist.

Feature availability, licensing, pricing, and hosted-plan boundaries change over time. Validate them against the current documentation and your own requirements.

I would also benchmark candidate gateways against representative traffic instead of relying on published performance claims.

References: Vercel — Open Source AI Gateways Compared · Portkey Documentation · Envoy AI Gateway · OpenRouter Routing

12. Why an OpenAI-Compatible Interface Is Not Enough

An OpenAI-compatible API is useful, but compatibility should not be confused with behavioral equivalence.

Different models and providers may implement the same feature differently.

For example:

  • Tool-calling arguments may be represented differently.
  • Structured output constraints may vary.
  • Streaming events may contain different metadata.
  • Reasoning parameters may not map directly.
  • Some providers may not support a required input modality.
  • Context windows may differ.
  • Cancellation and timeout behavior may differ.

A gateway can normalize the common protocol, but application behavior must still account for provider differences.

Keep Model Identity Visible

Even when the application uses a logical model alias, it should remain possible to inspect:

  • Requested model.
  • Actual model selected.
  • Upstream provider.
  • Fallback attempts.
  • Provider-specific metadata.
  • Usage and pricing information.

This is essential for debugging, performance analysis, evaluation, and reproducibility.

Test Fallback Compatibility

For every important logical model group, define a compatibility test suite.

For example:

Test Why it matters
Text generation Basic response compatibility
Streaming Event and completion behavior
Tool calling Agent execution compatibility
Structured output Schema compliance
Multimodal input Required input support
Usage reporting Cost attribution
Cancellation Correct failure handling

A fallback that produces a successful HTTP response but cannot execute the next step of an agent workflow is not a useful fallback.

The right abstraction hides integration complexity without hiding operational reality.

13. Cost-Aware Routing and Usage Attribution

An AI Gateway provides a central point for understanding where model consumption originates.

However, usage tracking alone does not optimize costs.

We need to associate usage with the workload that generated it.

For example:

AI Platform Usage
|
+-- Development Assistant
|   |
|   +-- Code Review
|   +-- Code Generation
|   +-- Test Generation
|
+-- Research Platform
|   |
|   +-- Document Analysis
|   +-- Market Research
|
+-- Automation
    |
    +-- Data Extraction
    +-- Classification
Enter fullscreen mode Exit fullscreen mode

A useful usage model includes:

  • Application.
  • Team.
  • User or service identity.
  • Feature.
  • Model.
  • Provider.
  • Input tokens.
  • Output tokens.
  • Cached tokens.
  • Request count.
  • Total cost.

Basic Model Cost Calculation

A simplified model cost formula:

Model cost = Σ (Input tokens × Input price + Output tokens × Output price)

Where:

  • (T) represents token usage.
  • (P) represents the applicable price per token.
  • (i) represents an individual model operation.

A production implementation should also account for cached input, multimodal usage, provider-specific charges, and other billable operations.

For an agent workflow, we should aggregate usage across all model calls rather than measuring only the final response.

Cost Per Successful Task

A model that costs less per request is not necessarily the better option.

Consider:

Metric Model A Model B
Cost per attempt $0.04 $0.07
Successful completion rate 90% 98%
Cost per successful task ~$0.044 ~$0.071

The cheaper model is more economical in this example, but this does not account for differences in quality, latency, or the business impact of failed tasks.

A useful metric is:

Cost per successful task = Total execution cost / Successful tasks

This should include failed attempts that consumed resources.

Cost Optimization Strategies

  • Use smaller models for predictable, low-complexity tasks.
  • Use stronger models when the task requires them.
  • Avoid unnecessary model calls.
  • Measure fallback frequency.
  • Use prompt caching where appropriate.
  • Define per-workload quotas.
  • Introduce evaluation-based escalation when appropriate.

Cost-aware routing should never become a reason to silently downgrade critical operations.

Reference: Vercel — Cost-Aware Model Routing

14. Security and Governance

An AI Gateway becomes a sensitive infrastructure component because it controls access to external model providers and processes potentially confidential application data.

Several security requirements should be considered.

Centralized Credential Management

Applications should authenticate to the gateway using dedicated application credentials.

Upstream provider keys should be stored in an approved secret-management system or the gateway's secure credential store.

Application-Specific Identities

Avoid sharing one unrestricted administrator credential across every application.

Use separate service identities, virtual keys, or other supported workload credentials.

Model Allowlists

Applications should only access approved models and providers.

A development assistant should not automatically have access to every production model deployment.

Data Residency and Provider Policies

Provider selection must consider more than cost and latency.

Some workloads may require:

  • Specific regions.
  • Approved providers.
  • Zero Data Retention arrangements.
  • Restrictions on provider data usage.
  • Internal-only inference endpoints.

Where supported, apply these constraints at the gateway layer.

However, a gateway's routing policy does not replace verification of the provider's contractual and data-handling commitments.

Protect the Gateway Itself

A gateway should have:

  • Authenticated administrative endpoints.
  • Restricted network access.
  • Secure credential storage.
  • Request validation.
  • Rate limits.
  • Availability monitoring.
  • Backup and recovery procedures.
  • Controlled configuration changes.

Since multiple applications depend on it, the gateway is itself a critical infrastructure service.

15. Operational Considerations: Avoid Creating a New Bottleneck

Centralizing model access simplifies application integrations, but it also introduces a shared dependency.

Every application now relies on the gateway to communicate with providers.

Several operational considerations follow.

High Availability

For critical workloads, use multiple gateway instances behind an appropriate load balancer.

Avoid relying on one process or in-memory state for shared configuration and quota enforcement.

Shared Rate-Limit State

If multiple gateway instances enforce the same team or application quota, they must share consistent usage state.

Otherwise, each instance may independently allow requests and exceed the intended limit.

Provider Health Management

Monitor upstream providers and deployments.

Temporarily exclude unhealthy deployments when appropriate, but avoid aggressive health-check behavior that creates additional provider load.

Configuration Management

Model aliases, provider priorities, budgets, and access policies are production configuration.

Version and review changes where possible.

A provider change should be treated with the same care as a change to an external service dependency.

Gateway Observability

Track:

  • Gateway request latency.
  • Upstream provider latency.
  • Provider failure rate.
  • Fallback frequency.
  • Model selection distribution.
  • Budget rejections.
  • Queue and connection utilization.

This helps distinguish gateway overhead from upstream inference latency.

Understand the Additional Failure Domain

A gateway adds another service to operate.

Teams should evaluate its:

  • Availability.
  • Throughput.
  • Network overhead.
  • Connection limits.
  • Capacity requirements.
  • Failure isolation.
  • Scaling behavior.

The gateway should simplify model operations without becoming a single point of failure.

16. What Can an AI Chapter Lead Contribute?

When multiple teams use different models and providers independently, infrastructure complexity grows quickly.

Each team may implement its own credentials, model configurations, retries, usage reporting, and provider policies.

This creates duplicated engineering effort and inconsistent governance.

An AI Chapter Lead can turn model access into a reusable platform capability.

1. Establish a Shared Model Access Layer

Define a common gateway endpoint for AI-powered applications.

It should standardize:

  • Provider integration.
  • Model aliases.
  • Routing.
  • Fallback policies.
  • Authentication.
  • Usage tracking.
  • Access control.

Feature teams should focus on application behavior rather than managing provider-specific infrastructure.

2. Build an Internal Model Catalog

Maintain an approved registry of model deployments.

For example:

Logical model Purpose Capabilities
fast-chat Classification and simple tasks Text, structured output
reasoning-chat Complex analysis Text, tool calling
coding-model Code generation and review Text, tool calling
vision-model Image analysis Multimodal
embedding-model Semantic retrieval Embeddings

Each definition should also contain provider information, model version, supported capabilities, ownership, and access restrictions.

3. Define Shared Routing Policies

Provide reusable routing profiles:

  • Cost-optimized.
  • Latency-sensitive.
  • Reasoning-focused.
  • Coding-focused.
  • Multimodal.
  • Region-restricted.

Teams can select an approved routing profile rather than independently implementing provider-specific decisions.

4. Centralize Access Governance

Introduce application-specific credentials and model allowlists.

Teams should only access the models and providers approved for their use cases.

This also creates a foundation for usage attribution and auditing.

5. Standardize Model Compatibility Testing

Build automated tests that verify:

  • Streaming behavior.
  • Tool calling.
  • Structured output.
  • Multimodal capabilities.
  • Cancellation.
  • Usage reporting.
  • Failure handling.

This becomes particularly important when introducing new models or changing fallback configurations.

6. Introduce a Model Promotion Process

A new model should not automatically become the default for every application.

A promotion process could look like:

New Model Available
        |
        v
Capability Validation
        |
        v
Offline Evaluation
        |
        v
Cost / Latency Benchmark
        |
        v
Security Review
        |
        v
Limited Traffic
        |
        v
Production Approval
        |
        v
Model Catalog Update
Enter fullscreen mode Exit fullscreen mode

This allows the organization to introduce new models without uncontrolled changes to existing applications.

7. Separate Platform Ownership from Application Ownership

The platform team should own:

  • Gateway infrastructure.
  • Provider integration.
  • Credentials.
  • Routing engine.
  • Global access policies.
  • Platform availability.

Application teams should own:

  • Business logic.
  • Prompt and agent behavior.
  • Model requirements.
  • Application-level evaluation.
  • Workflow-specific constraints.

This creates a clearer responsibility boundary.

17. When Should We Build Our Own AI Gateway?

Not every organization needs a custom gateway.

For one application using one provider, adding a separate infrastructure component may not provide enough value.

A direct provider integration or managed gateway may be simpler.

A dedicated gateway becomes more attractive when:

  • Multiple applications use different model providers.
  • Different teams need centralized access control.
  • Provider credentials need centralized management.
  • Cost attribution is important.
  • Fallback and availability are business requirements.
  • Internal model deployments coexist with external providers.
  • Data governance requires provider restrictions.
  • Model routing policies need to evolve independently of applications.

There is also an important distinction between customizing an existing gateway and building one from scratch.

I would start with an existing gateway, validate its capabilities, and identify the remaining requirements.

Build custom infrastructure only when there is a clear requirement that existing solutions cannot satisfy.

Conclusion

As AI adoption grows, managing model access independently in every application becomes increasingly difficult.

An AI Gateway provides a shared infrastructure layer for provider abstraction, model routing, authentication, fallback handling, usage policies, and cost management.

Combined with an application-level SDK such as Vercel AI SDK, it allows development teams to focus on building AI-powered features while the platform manages model connectivity.

But a gateway should not create the illusion that all models are interchangeable.

Provider differences, capability compatibility, streaming behavior, data governance, and failure semantics still matter.

For engineering teams, the opportunity is to turn model access into a standardized internal platform capability.

My recommendation: Start with a small model catalog, introduce application-specific identities, and define explicit routing and fallback policies. Then validate provider compatibility and budget behavior before expanding the gateway across teams.

The goal is not to create another proxy.

It is to make model access predictable, governed, and independent of individual applications.


References and Further Reading

AI Gateway and AI SDK

  1. AI SDK — AI Gateway Provider — Current provider integration, model discovery, routing, and fallbacks.
  2. Vercel AI Gateway — Managed model access infrastructure.
  3. Provider Options — Provider ordering, filtering, and routing.
  4. Model Fallbacks — Model-level failover.
  5. Authentication and BYOK — API keys, OIDC, and provider credentials.
  6. Budgets and Spend Limits — Team, project, API key, and user budgets.
  7. Cost-Aware Model Routing — Model selection based on task complexity and cost.

LiteLLM

  1. LiteLLM Documentation — Gateway architecture and provider support.
  2. LiteLLM Docker Quickstart — PostgreSQL-backed gateway setup.
  3. Router Configuration — Model groups, retries, and fallbacks.
  4. Fallbacks and Reliability — Provider failover and timeout handling.
  5. Budgets and Rate Limits — Virtual keys, team budgets, and usage limits.

Other Gateway Platforms

  1. Portkey Documentation — Routing, reliability, and gateway configuration.
  2. Envoy AI Gateway — Open-source gateway architecture for Envoy-based infrastructure.
  3. OpenRouter — Model Routing — Provider selection, routing, and failover.
  4. OpenRouter Documentation — Model and provider integration.
  5. Open Source AI Gateways Compared — A comparative overview of LiteLLM, Portkey, Envoy AI Gateway, and Bifrost. This is a Vercel-published comparison and should be treated as one input, not an independent benchmark.

Top comments (1)

Collapse
 
__d03cc26214f3 profile image
איתמר וענונו •

Great breakdown. One thing I'd add from running a small app in production: latency is a failure mode too, not only outages. We started on our own open model on a serverless GPU; quality was close to a small hosted model on our eval, but cold starts of 2-3 minutes made it unusable as the primary. Now the hosted model is primary and the open model is the fallback. For a user-facing app, I'd put a per-request latency budget right next to the cost budget in the gateway.