your ai gateway is a cache scheduler now

Google published a GKE Inference Gateway benchmark that looks, at first, like normal cloud-provider performance marketing.

You know the genre. A chart goes up. Another chart goes down. Someone says "up to" with a straight face. We all pretend not to have seen this movie before.

But this one is more interesting than the usual "our managed service is faster than another managed service" post.

The useful part is not only the numbers. Google says GKE Inference Gateway, using prefix-cache-aware routing, delivered 15.7% higher throughput, 92.8% shorter wait time to first token, and 62.6% lower inter-token latency on a shared-prefix Llama 3.1 8B workload, compared with conventional round-robin HTTP load balancing on the same A100 hardware.

The interesting part is what those numbers are really saying:

For AI systems, the gateway is no longer just forwarding requests.

It is deciding where memory already exists.

the old load balancer was too innocent

Traditional load balancing has a nice, simple mental model.

You have many backends. A request arrives. The load balancer picks one based on round robin, least connections, latency, weights, locality, health, or some other policy that mostly treats requests as independent traffic.

That model worked well enough for a lot of web systems.

A request for /checkout can usually go to any healthy checkout pod. The pod might hit a cache, a database, a queue, or another service, but the load balancer itself does not need to understand the semantic shape of the request.

LLM inference is less polite.

Two requests can look similar at the HTTP layer and be completely different at the compute layer. One request may include a small prompt. Another may include the same 10,000-token documentation prefix that the model already processed five seconds ago on one specific pod. If the gateway sends that second request to a random backend, the system may recompute work it could have reused.

That is not just latency. That is money wearing a fake mustache.

prefix caching turns memory into placement

Prefix caching is the trick here.

LLM servers can keep the KV cache for repeated prompt prefixes in memory. If many requests share the same system prompt, product documentation, policy text, support script, codebase context, or RAG-provided reference block, the model should not have to process the same prefix from scratch every time.

The catch is that the useful cache is local to where it lives.

If pod A has the warm cache for the big static prefix, sending the next matching request to pod B may lose the benefit. Pod B is healthy. Pod B has capacity. Pod B is also the wrong place.

This is why the gateway starts looking like a scheduler.

It is not asking only "which backend can serve this request?"

It is asking "which backend already paid part of the bill for this request?"

That is a different job.

ai traffic has shape

The web trained us to talk about traffic as volume.

Requests per second. P95 latency. Error rate. Bytes transferred. CPU. Memory. Saturation. All useful, all still relevant, please do not replace your dashboards with vibes in YAML.

But AI traffic also has shape.

There are static prefixes and dynamic suffixes. There are repeated agent instructions, long product docs, compliance policies, customer-service personas, codebase summaries, tool definitions, and giant prompts that change one small question at the end.

If the platform cannot see that shape, it will schedule badly.

This is the same pattern we keep rediscovering in infrastructure. Generic abstractions work until the expensive detail refuses to stay hidden.

Kubernetes did this with hardware. "Give me a GPU" became too vague once topology, accelerator type, networking, and sharing behavior started to matter.

Now AI gateways are doing a similar thing for inference.

"Send this request to a healthy backend" is becoming too vague when the backend's memory state changes the cost and latency of the answer.

rag makes this very real

The obvious use case is retrieval-augmented generation.

Imagine an internal developer assistant. Every request includes a large, mostly static pile of API documentation, service ownership rules, deployment conventions, and "please do not suggest deleting production" instructions. Then the user asks one small question:

How do I handle a 429 in this SDK?

The dumb path sends the whole prompt anywhere. The model reprocesses the reference block. The user waits. Finance gets another tiny paper cut.

The smarter path routes the request to a backend that already has that documentation prefix warm in memory. The model focuses on the new part.

That is not magic. It is cache locality.

But it changes where architecture lives.

Suddenly the prompt layout matters. The gateway policy matters. The runtime matters. The way you chunk and pin documentation matters. Your "AI feature" has an infrastructure design hiding inside it, because of course it does.

this is why ai platforms are becoming boring

There is a funny thing happening in AI infrastructure.

The demo layer keeps trying to be dramatic. Agents! Reasoning! Autonomous workflows! Digital coworkers! The usual fireworks.

The production layer keeps becoming more like every other serious platform.

It needs routing, caches, admission control, quota, observability, private networking, rollback, capacity planning, policy, ownership, and incident response.

The GKE Inference Gateway story fits that pattern perfectly.

The model is important, yes. But the difference between an expensive, sluggish feature and a usable one may be a routing decision that sends the request to the pod with the right warm memory.

That is deeply unromantic. It is also where a lot of real engineering value lives.

what i would watch first

If I were reviewing an AI platform design right now, I would ask boring questions before debating model taste.

Are repeated prompt prefixes actually stable, or are we changing them on every request with timestamps, random IDs, user-specific noise, or formatting churn?

Can the runtime reuse KV cache effectively, or are we assuming a cache exists because a diagram had the word "cache" in it?

Does the gateway understand prefix locality, or are we still spraying inference requests around like stateless JSON calls?

Can we measure cache hit rate, time to first token, inter-token latency, tokens per second, and cost per useful answer?

And, my favorite platform question:

Who owns this after the launch blog post is forgotten?

Because someone will. When users complain that the assistant feels slow, it will not be enough to say "the model is large." When the cloud bill spikes, it will not be enough to say "AI is expensive."

gateways are becoming part of the model serving contract

The practical lesson is simple: AI gateways are becoming part of the serving contract, not just network furniture.

For normal HTTP systems, you can often swap one load balancer for another and mostly keep the application model intact. There are details, obviously, because infrastructure enjoys being pedantic.

For LLM systems, the gateway may decide whether the model recomputes thousands of tokens or reuses cached work. It may decide whether a chat response starts quickly or sits there doing the spinner dance.

That makes it architecture.

It also means teams should stop treating AI inference as a black box behind a single endpoint.

The endpoint is not the system. The model is not the system. The prompt is not the system.

The system is the model, prompt shape, runtime, cache behavior, gateway policy, hardware placement, telemetry, and cost model all arguing in one room.

Very normal. Very relaxing.

the boring win

I like this direction because it moves AI infrastructure away from vague magic and toward inspectable systems.

Prefix-cache-aware routing is not a motivational poster. It is an engineering claim you can measure. If it works, you should see better time to first token, better token throughput, lower recomputation, and a nicer bill.

That is healthy.

The best AI platforms will not be the ones with the fanciest chat screenshots. They will be the ones that understand where the expensive state lives and route work accordingly.

In the old web world, a load balancer mostly had to know where traffic could go.

In the AI world, the gateway increasingly needs to know where the useful past already is.

That is the shift.

Your AI gateway is a cache scheduler now.

Sorry. Another simple thing became a platform problem.

To test my projects, I use Railway. If you want $20 USD to get started, use this link.