DEV Community

Alex Morgan
Alex Morgan

Posted on • Originally published at saaswithalex.pages.dev

AI Gateway vs API Gateway: The Token-Request Divergence

AI adoption reached 78% of organizations in 2024, up from 55% the previous year, according to Stanford HAI's 2025 AI Index Report. Gartner projects worldwide generative AI spending will reach $644 billion in 2025, with total AI spending tracking toward $1.5 trillion. That money is flowing through infrastructure layers that were never designed for token-metered, semantically variable, streaming-heavy LLM traffic. The result is a persistent capability gap that's forcing a real architectural decision: do you extend your existing API gateway to handle AI traffic, or do you add a separate, token-native control plane?

The answer depends on your constraints. Here's the analytical breakdown.

The Fundamental Mismatch: Requests vs. Tokens

An API gateway manages HTTP traffic between services—handling authentication, routing, rate limiting, and load balancing for REST and gRPC—with the HTTP request as its unit of operation, and it does not inspect the request body. An AI gateway manages LLM workloads, providing token-based rate limiting, model routing, cost attribution, semantic caching, and guardrails. The distinction sounds subtle until you see the billing implications.

API gateways meter by request and treat the request body as opaque, while AI gateways meter by tokens, cache by semantic meaning, and read the payload. This is what I call the Token-Request Divergence: a single LLM call with a 50,000-token prompt counts as one request in an API gateway's rate limiter—the same as a 200-token call. Your cost per request varies by 250x, but your gateway sees them as identical.

Traditional API gateways are insufficient for AI workloads because their request-based metering, exact-match caching, and opaque payload inspection cannot address token-level cost attribution, semantic caching, or LLM-specific governance requirements. This isn't a missing feature you can patch. It's a different unit of computation entirely.

An AI gateway is also broader than an LLM gateway; it governs model calls, agent workflows, MCP server access, tool calls, API traffic, policies, audit logs, and AI cost controls. AI gateways classify into three types: Model/LLM gateway (targeting models), MCP gateway (targeting tools), and Agent gateway (targeting other agents). Most production teams eventually need both an API gateway and an AI gateway, sitting at different layers.

What Breaks When You Retrofit

One team using Kong for LLM traffic for six months lacked token-level cost visibility, budget caps, token-aware rate limiting, and provider failover, requiring a switch to an AI-native gateway. Their requests got through. Their billing was blind.

This anecdote illustrates a broader pattern. When you route LLM traffic through a traditional API gateway, you get four specific gaps:

  • No token-level cost attribution. You can't answer "which team spent what on tokens last month?" because the gateway counts requests, not tokens.
  • No semantic caching. Two prompts asking the same question in different words both hit the provider. An AI gateway recognizes semantic equivalence and returns a cached response, directly reducing token spend.
  • No provider failover. When OpenAI has a partial outage, a traditional gateway serves errors. An AI gateway can automatically retry with Anthropic or another configured fallback.
  • No prompt-level guardrails. PII redaction, prompt injection defense, and content moderation require reading the payload. API gateways don't.

None of this is a criticism of API gateways. They're doing exactly what they were designed to do. The problem is expecting a request-layer proxy to handle a token-layer problem.

Pricing: The Per-Request Trap at Scale

Here's where the Token-Request Divergence hits your budget. Traditional API gateway vendors price per request. AI-native gateways price per token or per log volume. For high-volume LLM workloads, the pricing models produce wildly different cost profiles.

Gateway Pricing Model Starting Price Best For
Kong Konnect Plus Per-service per-request $105/month per gateway service + $200/additional million requests Teams with existing Kong investment and moderate AI traffic
AWS API Gateway Per-request $1.00/million HTTP API requests to $3.50/million REST API requests AWS-native teams with simple, low-volume AI routing
LiteLLM (self-hosted) Open-source + infrastructure costs Free (MIT license), $500–$2,000/month in ops costs Teams with DevOps capacity and cost-sensitive, high-volume LLM workloads
Portkey Pro Log-based billing $49/month Teams wanting turnkey managed observability without infrastructure overhead

The Kong pricing example makes the per-request trap concrete. A 50-service deployment on Kong Konnect costs $63,000 per year in base subscription fees (50 services × $105/month × 12 months), before any traffic overage charges. Each additional million requests costs $200. For a high-volume API handling 50 million calls per month, that's nearly $10,000 in monthly overage charges on traffic alone.

Token-native AI gateways charge based on token volume or log volume, which is far lower for typical LLM workloads with high average token counts per request. The pricing structure aligns with the actual cost driver.

But don't assume open-source is always cheaper. LiteLLM is free and open-source (MIT licensed), but self-hosting it in production requires PostgreSQL, Redis, and Docker, with operational costs ranging from $500 to $2,000 per month for a mid-size deployment handling 100–500 RPS. Portkey's Pro tier starts at $49 per month, with log-based billing that can balloon at scale. The "free" option has the highest ops tax. The "affordable" managed option gets expensive at high throughput. There's no universally cheap option.

Performance: Throughput and Latency Considerations

Kong achieves approximately 138,000 RPS in official benchmarks (without plugins), while Envoy operates near bare-metal performance and AWS API Gateway defaults to a 10,000 RPS throttle. These numbers matter for raw HTTP throughput, but they tell you nothing about AI-specific performance.

LLM traffic has different performance characteristics than traditional API traffic. Latency is measured in seconds, not milliseconds. Responses stream via Server-Sent Events. A single user request may trigger dozens of model calls in an agentic workflow. The gateway's overhead is a smaller fraction of total latency, but caching and routing intelligence have outsized impact on cost.

This is where semantic caching changes the math. When two callers submit semantically equivalent prompts, an AI gateway can return a cached response rather than making a redundant API call. For applications with repetitive query patterns—customer support bots, internal knowledge tools—this directly reduces token spend without any application code change. An API gateway's exact-match cache can't do this because it doesn't read the payload.

The Convergence Trend: Unified Control Planes in 2026

Enterprise teams are consolidating API and AI traffic to a single control plane, though traditional API gateway vendors' per-request pricing structures often make them uncompetitive for high-volume LLM workloads compared to token-native AI gateways. The convergence is real, but the pricing mismatch persists.

Several major vendors shipped unified platforms in 2026:

MCP activity grew from 11% late last year to 41.4% by mid-2026, and monthly AI transaction volume grew twelve-fold over the same six months, according to Palo Alto Networks telemetry. The AI footprint you can govern today is the smallest it will ever be.

The tension here is genuine. Vendors market unified API/AI gateways as the future of enterprise traffic control, but pricing and capability data shows retrofitted API gateways are uncompetitive for high-volume LLM workloads. MuleSoft's Omni Gateway and WSO2's unified API Platform are built explicitly to extend existing API management investments to cover AI traffic. Forbes reports that enterprises are actively seeking consolidated control planes to avoid stitching together point solutions. But the metering unit mismatch means a "unified" gateway that prices per request will always overcharge for token-heavy LLM traffic.

The Open-Source Hidden Cost Paradox

Open-source AI gateways are promoted as cost-effective alternatives to managed tools, but their hidden operational costs often exceed the price of managed offerings for mid-size teams. LiteLLM is free and ranked the top self-hosted AI gateway for overall flexibility in 2026, but analysis of its production costs shows it requires PostgreSQL, Redis, and dedicated DevOps staff to run reliably, costing $500–$2,000 monthly for 100–500 RPS. That's higher than managed Portkey's $49 monthly Pro tier for the same throughput.

The tradeoff is clear: open-source gives you data sovereignty, no vendor lock-in, and full control over configuration. Managed gives you zero operational overhead and turnkey observability. The right choice depends on whether you have DevOps headcount or budget—and which one is more constrained.

For teams that already have Kubernetes infrastructure and platform engineering capacity, self-hosting an open-source AI gateway is the cost-efficient path. For teams where the nearest platform engineer is also handling three other priorities, managed is cheaper despite the per-log billing risk. If you're exploring this path further, our analysis of AI gateway architecture and the 2026 control plane covers the open-source vs. managed decision in more depth.

Decision Framework: Which Gateway for Which Traffic

For any team running more than two LLM providers or spending over $1,000 monthly on LLM tokens, a standalone token-native AI gateway is non-negotiable. Retrofitting an existing API gateway for AI traffic will cost more in unoptimized token spend and operational gaps than the overhead of managing a separate layer.

Here's the decision matrix:

  1. Single LLM provider, low token volume (<$1,000/month): Your existing API gateway is fine. Add provider failover and cost tracking when you add a second provider or when token spend becomes visible enough to require budget caps.

  2. Multiple LLM providers, moderate token volume ($1,000–$10,000/month): Add a token-native AI gateway alongside your API gateway. The AI gateway handles model routing, semantic caching, token-based rate limiting, and provider failover. The API gateway continues handling your microservice traffic. This is the two-layer architecture most production teams converge on.

  3. High token volume (>$10,000/month) or agentic workflows with MCP/A2A traffic: You need a full AI gateway with agent governance capabilities. The MCP metadata tax on enterprise deployments is a real cost factor for common workflows, so gateway-level governance of MCP traffic is critical for cost control.

  4. Regulated enterprise with data sovereignty requirements: Self-hosted open-source (LiteLLM, Envoy AI Gateway) or vendor-hosted-in-your-VPC options. The AWS Claude Apps Gateway is a notable example of a self-hosted control plane that handles identity, policy, telemetry, routing, and spend caps within your own infrastructure.

  5. Team already standardized on Kong, Apigee, or MuleSoft: Evaluate their AI gateway extensions first, but run the pricing math against a token-native alternative. A 50-service Kong deployment costs $63,000/year before overages. Compare that to what a token-native gateway would cost for your actual token volume.

The industry guidance that API and AI gateways should remain separate layers is correct architecturally but increasingly out of step with procurement reality. Enterprise procurement teams prioritize consolidated vendors to reduce tooling sprawl. The vendors that win will be the ones that offer a unified control plane with token-native pricing—not per-request pricing retrofitted for AI traffic.

The Bottom Line

The AI gateway vs. API gateway question isn't "which one?" It's "which layer handles which traffic, and what does each layer cost at your scale?" The Token-Request Divergence means a gateway that meters by requests will never give you accurate token cost attribution, semantic caching, or token-aware rate limiting. You can approximate some of these with plugins and custom code, but you're building a partial AI gateway inside your API gateway—and paying per-request pricing for the privilege.

The specific question worth asking before you choose: what's your monthly token spend across all providers, and can your current gateway tell you which team consumed which tokens? If the answer is "we don't know" or "we can't break it down by team," you're already losing money to unoptimized routing and uncached redundant calls. That's the gap a token-native AI gateway closes—and the per-request pricing model of a retrofitted API gateway can't.

Originally published at SaaS with Alex

Top comments (0)