37% of enterprises are running five or more models in production by 2026, turning the infrastructure layer between applications and model providers into a critical control plane for cost, latency, and availability. AI load balancing is the practice of distributing AI requests across multiple API keys, models, and providers so no single dependency becomes a bottleneck or point of failure, according to a Maxim AI overview. The LLM gateway market that delivers this capability is projected to hit $7.21 billion by 2030, and the growth is driven by a hard operational reality: traditional load balancing strategies are actively destructive to LLM inference economics.
Here's why that matters. Your standard round-robin load balancer assumes requests are fast, uniform, and stateless. LLM inference breaks all three assumptions simultaneously. A 128-token prompt and a 4,096-token prompt look identical at the network layer but impose wildly different compute loads, making traditional load balancing inadequate for LLM inference, per an MLflow infrastructure guide. The result is that the infrastructure layer meant to distribute work efficiently is instead destroying the state that makes efficient inference possible.
Why Does Naive Round-Robin Destroy LLM Cache Hit Rates?
The core problem is cache locality. LLM serving systems maintain key-value (KV) caches for active sequences, and routing a returning user's request to a different backend than the one holding their KV cache forces a cold start, destroying throughput gains, as explained in the same MLflow analysis. Prompt caching typically cuts input token costs by 50-90% and can reduce Time to First Token (TTFT) latency by up to 80%, per DigitalOcean's LLM serving guide. Those gains assume your request lands on the replica that already has the relevant prefix cached.
Under naive round-robin load balancing across N replicas, the probability of a cache hit is 1/N, causing cache hit rates to degrade almost linearly as the fleet grows, according to the same DigitalOcean analysis. If you have four replicas serving a chatbot and every user's request starts with the same system prompt, round-robin computes that prompt four separate times. With intelligent scheduling, it gets computed once and reused.
This is what I call the Competence Tax: the most marketed gateway features—routing, visibility, dashboards—obscure the hardest requirements. Cache-aware scheduling has inverted from an optimization to a fundamental requirement. Traditional load balancers are actively destructive to LLM inference economics because naive round-robin degrades cache hit rates linearly. The infrastructure meant to distribute work is instead destroying the state that makes efficient inference possible.
How Does Cache-Aware Routing Actually Work?
Cache-aware routing solves this by adding a scheduling layer that inspects request prefixes before forwarding them. The open-source llm-d project implements this with an endpoint picker that routes each inference request to the GPU that already has the relevant data cached, factoring in queue depth and real-time load. In upstream benchmarks on shared-prefix workloads, llm-d cut TTFT by more than 99% and more than doubled throughput without changing hardware.
Google's GKE Inference Gateway takes a similar approach at the Kubernetes layer. It reads incoming request prefixes and matches them to the specific pods that already hold that data in memory. According to an independent benchmark report, the GKE Inference Gateway outperforms the next leading managed Kubernetes service with 15.7% higher throughput, 92.8% shorter wait times, and 62.6% lower inter-token latency.
The anecdotal evidence from production deployments is compelling. Snap reported prefix cache hit rates ranging up to 75-80% using prefix-cache-aware routing with llm-d in production, according to a Google Cloud case study. That's not a marginal optimization. It's a fundamental shift in how inference infrastructure needs to work.
When Does QPS Fail as a Load Balancing Metric?
For real-time AI agents with long-lived bidirectional streams, QPS is insufficient and session-aware load balancing is required because CPU utilization does not capture the committed workload of active sessions, per a Google Developers Blog analysis. Consider two backend tasks: Task A handles 100 short requests finishing in 50 milliseconds each. Task B accepts 5 requests, but each turns into a 20-minute session. By request rate, Task B appears less busy. By committed workload, it's shouldering significantly more.
This matters because real-time AI systems manage continuous, live bidirectional streams—a constant flow of audio chunks, transcripts, model outputs, and synthesized speech. When a user interrupts, the server must halt current generation, update context, trigger a new tool, and draft a different response without dropping the connection. QPS tracks arrival volume but fails to capture the number of live conversations a server is already managing.
CPU utilization is equally deceptive here. A voice runtime hosting 20 silent sessions looks underutilized because there's no active speech processing. The moment those 20 users start speaking simultaneously, CPU usage spikes. Active session counts reveal the work the backend has already promised to handle, while CPU metrics reflect only immediate processing load. For real-time AI, you need both signals. This connects to broader AI gateway architecture decisions where sub-millisecond overhead and session governance define production readiness.
Which Gateway Matches Your Spend Tier and Operational Competence?
An AI gateway becomes mandatory when monthly LLM spend crosses approximately $30,000, a second provider is onboarded, or compliance requires per-team cost attribution, per a Particula.Tech decision framework. Below that threshold with a single provider, a DIY router is honest engineering. The expensive mistake is skipping the maturity tier that matches your actual operational competence.
Here's where the cost-capability inversion gets painful. LiteLLM is the open-source default for mid-range spend, but it carries a hard performance ceiling. LiteLLM P99 latency hits 28 seconds at 500 RPS and crashes at 1,000+ RPS. That ceiling is precisely what growing teams hit when they need reliable gateway infrastructure most. Meanwhile, self-hosted open-source gateways incur hidden operational costs including DevOps headcount, infrastructure, monitoring, and security that often exceed managed service costs at scale, according to a TURION.AI pricing comparison.
Portkey is positioned as the governance solution for high-spend enterprises, but its log-based pricing model creates a direct conflict with the high-throughput workloads that generate the governance risks it addresses. The high-throughput workloads that create governance exposure are exactly what make log-based billing prohibitively expensive. Kong AI Gateway is positioned as the enterprise choice for large deployments, but its value proposition depends on pre-existing Kong operational expertise that contradicts its positioning as an upgrade path.
| Gateway | Pricing | Key Tradeoff | Target Audience |
|---|---|---|---|
| LiteLLM (Self-Hosted) | ~$500–$2,000/mo ops costs per TURION.AI | P99 hits 28s at 500 RPS, crashes at 1,000+ RPS per Zylos Research | Teams with DevOps capacity, cost-sensitive |
| Portkey | Pro starts at $49/mo per TURION.AI | Log-based billing balloons at scale | Teams wanting turnkey governance + observability |
| Kong AI Gateway | $25/mo per control plane per TURION.AI; Konnect Plus ~$105/mo per gateway service per Zuplo | Requires Kong operational knowledge | Teams already on Kong, multi-cloud enterprises |
What Are the Hidden Costs That Break the Sticker Price?
The pricing patterns across these gateways reveal a systematic pattern: the lowest upfront price points correlate with the highest operational taxes. LiteLLM's MIT license is free, but running it in production requires PostgreSQL, Redis, Docker, and a load balancer in front of proxy replicas. For a mid-size deployment handling 100–500 RPS, that's roughly $200–$600/month in cloud compute alone, before you factor in the people cost of configuring model groups, managing virtual keys, and monitoring proxy health.
Kong's pricing structure illustrates the same inversion at the enterprise tier. Kong Konnect Plus costs approximately $105 per month per gateway service, including one million requests; each additional million requests costs $200, per a Zuplo pricing comparison. A high-volume API handling 50 million calls per month can trigger nearly $10,000 in monthly overage charges on traffic alone. The per-service model penalizes teams with many microservices or API endpoints—exactly the architecture pattern that production AI agent designs tend to produce.
Self-hosted open-source gateways give you full control and zero licensing fees, but the DevOps headcount and operational complexity often exceed managed service costs at scale. Rich observability dashboards and routing visibility create false governance coverage that increases risk by conflating visibility with actual control. Broad provider coverage and flexibility come with hard performance ceilings that become bottlenecks exactly when scaling demands them. Every feature beyond basic routing, failover, and cost attribution adds operational complexity that compounds faster than its marginal value.
How Should You Actually Decide?
A 50-developer team cost cannot be projected from the available research: no per-seat pricing exists for the AI load balancing tools evaluated. This gap itself is telling. Gateway pricing is structured around infrastructure consumption (control planes, log volume, request counts) rather than team size, which means your cost projection depends entirely on your traffic patterns and operational choices, not your headcount.
The decision framework I'd use is simple. Start with your spend tier and compliance requirements. Below approximately $10K monthly spend with a single provider, a DIY router is honest engineering. Between $10K and $50K, LiteLLM covers 100+ providers with fallback and budgets without adding a vendor relationship—but only if you have the DevOps capacity to absorb the operational tax. Above $50K with governance requirements, Portkey's observability pays for itself until log-based billing becomes the bottleneck.
The question worth asking isn't which gateway is best. It's which gateway matches your operational competence without skipping a tier. The most expensive mistake is buying enterprise infrastructure your team can't operate, or shipping homegrown code at a spend level where a managed gateway would have paid for itself in the first month of token economics alone.
Originally published at SaaS with Alex
Top comments (0)