VK Tech explains why similar request rates can conceal sharply different LLM latency, and when prefix-aware routing may outperform round-robin.
VK Tech has outlined how prefix-aware routing can reduce latency for self-hosted LLMs running across multiple Kubernetes replicas. The company argues that conventional round-robin balancing can discard KV-cache locality and worsen time to first token, even when requests per second remain almost unchanged.
In a cited Red Hat test using OpenShift 4.20, four NVIDIA L4 GPUs and a Qwen-family model, round-robin produced a p95 time to first token of 745 milliseconds, versus 272 milliseconds with prefix-aware routing. Throughput was close at 0.52 and 0.49 requests per second, respectively. The controlled workload comprised 11 concurrent conversations with 10 turns each and slightly more than 100 requests overall.
The difference arises because a processed prefix remains in the KV cache of the replica that computed it. If a later request containing the same system prompt, RAG document or conversation history reaches another pod, that replica must process the prefix again. A basic request counter also treats short prompts and prompts containing thousands of tokens equally, hiding duplicated GPU work.
The Kubernetes Gateway API inference extension addresses replica selection through an InferencePool, which replaces a conventional Service in this design, and a separate Endpoint Picker. The InferencePool API is stabilized as inference.networking.k8s.io/v1. According to VK Tech, selection can consider queue depth, active requests, KV-cache utilization and whether a request begins with an already processed prefix. The default scorer examines the first 16,000 characters, or roughly 4,000 English tokens; vLLM can then reuse the full matching prefix even when it extends beyond that window.
Cache affinity should not be treated as the sole routing objective. A widely shared document could direct most requests to the pod holding its fullest cached prefix while other replicas remain underused. In practice, prefix matching must be balanced against queue depth and current load. VK Tech also notes that traffic patterns affect the outcome and that there is no universal estimate for the Endpoint Picker’s latency overhead, making a workload-specific shadow test important.
Before deployment, VK Tech recommends measuring tail TTFT, token-level prefix-cache hits, queues and KV-cache occupancy, then replaying representative traffic in shadow mode. A dedicated router is most relevant when requests have long repeated beginnings and each replica handles several concurrent sessions. For short, independent prompts without a shared prefix, it adds another component without creating useful cache reuse.
Replacing a Service with an InferencePool also affects autoscaling, canary releases and dashboards. VK Tech recommends aligning HPA signals with queue depth and cache occupancy, while dividing application versions within the pool. The article lists data-plane support in Istio from version 1.27, NGINX Gateway Fabric and agentgateway, and warns that InferencePool and Endpoint Picker versions follow separate release cycles and require explicit compatibility checks.
| Metric | Round-robin | Prefix-aware routing |
|---|---|---|
| TTFT p50 | 123 ms | 92 ms |
| TTFT p95 | 745 ms | 272 ms |
| TTFT p99 | 841 ms | 674 ms |
| KV-cache hit rate | about 62% | about 90% |
| Requests per second | 0.52 | 0.49 |
Sources
Originally published by AI Feed: VK Tech Details How Prefix-Aware Routing Affects LLM KV Cache.
Top comments (0)