Context Mobility: How Cross-Model KV Cache Sharing Could Reshape Multi-Model AI Inference

Modern AI applications rarely run on a single model. A typical production pipeline might route a user query through a small model for triage, escalate it to a larger model for reasoning, then pass the result to a specialized model for formatting or verification. Each of those handoffs has a hidden cost: every model re-reads the same conversation history from scratch, recomputing its own internal representation of context it has never seen before.

Two recent papers from August–September 2026 propose a concrete fix. The idea is called cross-model KV cache sharing, and it treats the key-value (KV) states computed during one model's prefill as a transferable artifact — something that can be translated and reused by a different model rather than discarded at the model boundary.

What Is the KV Cache, and Why Does Prefill Matter?

When a transformer processes an input sequence, it computes key and value tensors at each attention layer for every input token. These tensors are stored in the KV cache and reused during autoregressive decoding, so the model doesn't have to re-attend to the full context on every generation step.

The expensive part is prefill: the initial pass where the model reads all input tokens and builds those KV tensors. For a 16K-token prompt, prefill can take hundreds of milliseconds. In a multi-model pipeline where three or four models all process the same long context, that cost multiplies — even though the underlying information is identical.

Existing prefix-caching systems already avoid redundant prefill within a single model. The gap is at model boundaries: a KV cache produced by Model A is useless to Model B because the two models have different hidden dimensions, attention configurations, and learned representations.

The Translation Approach

The paper "A Universal Context-Reuse Layer for Cross-Model KV Sharing" (Li et al., UT Dallas, August 2026) introduces a translation layer that maps the KV states from a source model into a representation the target model can consume directly. Formally:

KV_target_approx = T(KV_source)

The translation function T is learned to minimize the difference between the translated state and what the target model would have produced from native prefill. Once trained, it runs as a lightweight module that sits between the two models in the serving stack.

A complementary paper, "Cross-Model KV Cache Transfer in LLM Families" (Heo et al., August 2026), focuses on the within-family case and finds that KV states across models in the same family have substantial linear structure. On Qwen3 14B→32B, a single source layer explains 56% of variance in the target's keys and 32% in values; using multiple source layers pushes that to 79% and 65%. This means a simple closed-form ridge regression — fit on just 500 calibration sequences — can serve as the mapper, running 2.7–25× faster than re-prefilling.

What the Numbers Look Like

The UT Dallas paper evaluates three transfer scenarios:

Within-family (Qwen2.5-7B → Qwen2.5-1.5B): Translating the 7B model's KV cache into the 1.5B model's space improves the smaller model's LongBench2 accuracy from 27.59% to 34.48% — a 6.89 percentage-point gain over native 1.5B inference. The handoff itself takes 34.5 ms for an 8K–16K context, compared to 158.7 ms for native 1.5B prefill.

Cross-family (Qwen2.5-1.5B → Gemma-2-2B): At 4K context length, the KV handoff reduces Gemma's prefill cost by 67% while keeping decoding perplexity close to native Gemma baselines.

Large-to-small (Llama3.1-70B → Qwen2.5-7B): This is the most striking result. Native Qwen2.5-7B inference takes 899 ms for prefill. With a translated Llama-70B cache, the handoff takes 138 ms — an 85% reduction — while achieving 44.0% accuracy versus 45.7% for native Qwen inference. The accuracy gap is small; the latency gap is large.

Where the Approach Breaks Down

The linear mapper works well for within-family transfers and some cross-family pairs, but the Heo et al. paper reports that two of six tested pairs "degrade sharply" with a linear mapper alone. A nonlinear MLP recovers up to 37 percentage points of HellaSwag retention on those failures, at the cost of additional compute and complexity.

The UT Dallas paper is also careful to frame these as "initial evidence" rather than a production-ready system. The translation layer needs to be trained per model pair, which means maintaining a matrix of translators as the number of models in a deployment grows. For a fleet of N models, that's potentially O(N²) translators — a real engineering challenge.

There's also the question of what happens when source and target models have different tokenizers. The papers focus on cases where the same token sequence is processed by both models; cross-tokenizer transfer is left as future work.

The Broader Idea: Context Mobility

The UT Dallas paper coins the term context mobility to describe the systems abstraction being proposed: treating previously computed context as a portable resource rather than a model-local artifact. The analogy to CPU caches is apt — just as a cache miss forces a round-trip to memory, a model handoff currently forces a full prefill. Cross-model KV sharing is an attempt to make that round-trip optional.

This matters most in three scenarios that are already common in production:

1. Model cascading and routing: A cheap model handles easy queries; hard queries escalate to a stronger model. With KV sharing, the escalation doesn't restart from scratch.
2. Multi-agent systems: Multiple specialized agents operate over a shared conversation or document. Each agent currently re-prefills the shared context independently.
3. Verification pipelines: A generator model produces output; a separate verifier checks it. Both process the same input, but today they do so redundantly.

Practical Implications

For teams building multi-model inference pipelines, the immediate takeaway is that prefill cost at model boundaries is not fixed overhead — it's a target for optimization. The techniques described here are still research-stage, but the direction is clear: serving infrastructure will increasingly need to treat KV states as first-class, transferable objects rather than ephemeral per-model artifacts.

The Heo et al. approach is particularly accessible: a ridge regression fit on 500 sequences is cheap to train and fast to run. For teams already operating within a single model family (e.g., routing between Qwen3 7B and 32B), this could be a near-term practical optimization.

The harder cross-family case — translating between architecturally distinct models like Llama and Qwen — requires more work, but the latency results (899 ms → 138 ms) suggest the payoff is worth pursuing.

Both papers are available on arXiv: 2608.30963 and 2608.03893. The field of KV cache optimization is moving quickly, and cross-model sharing looks like one of the more tractable near-term advances in multi-model serving efficiency.