ReContext: How Recursive Evidence Replay Helps LLMs Actually Use Long Contexts

Large language models can now accept context windows of 128K, 1M, or even 10M tokens. But accepting a long input and reasoning well over it are two different things. A growing body of research shows that models frequently fail to retrieve or correctly use information buried in the middle of a long context — a problem known as the "lost in the middle" effect. A new paper from July 2026, ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning, proposes a training-free inference method that addresses this gap without modifying the model or adding external memory.

The Problem: Context Access ≠ Context Utilization

When you feed a 128K-token document to an LLM and ask a question, the model technically has access to every token. In practice, attention tends to concentrate on the beginning and end of the sequence, leaving the middle largely ignored. This happens for structural reasons rooted in the Transformer architecture:

• Causal attention masking gives early tokens more exposure across the sequence, creating a primacy bias.
• Rotary position embeddings (RoPE) introduce distance-based decay, so tokens far from both ends fall into an "attention dead zone."
• Training data patterns reinforce this: most documents place key information at boundaries (headings, conclusions), so models learn to weight those positions more heavily.

The result is that even when a model can process a million tokens, its effective retrieval accuracy often degrades sharply past 200K–400K tokens. Benchmarks like RULER and multi-needle NIAH tests consistently show 30–60 point accuracy drops as context length increases for most frontier models.

What ReContext Does

ReContext treats the long-context reasoning problem as an associative memory problem. The authors draw an explicit analogy:

• The context is the memory store.
• The question is the retrieval cue.
• The model's attention is the cue-trace association mechanism.
• The replay is trace reactivation — surfacing the relevant memory before generating an answer.

The method works in three stages at inference time:

1. Identify Heavy-Hitter Tokens

ReContext uses the model's own internal attention signals to find "heavy-hitter" tokens — spans that receive disproportionately high attention relative to the query. Research on attention sinks and KV-cache optimization has shown that these high-attention tokens often account for 50–80% of query-relevant information in a document.

Rather than relying on an external retriever or a separate embedding model, ReContext reads these signals directly from the model's forward pass. No additional components are needed.

2. Build a Query-Conditioned Evidence Pool

The identified spans are assembled into an ordered evidence pool. This pool is constructed recursively: each round conditions on the original context, the question, and the evidence accumulated so far. The recursion typically runs for 2–4 rounds with an evidence token budget of 8–32 tokens per round.

This recursive selection is what distinguishes ReContext from simpler approaches like context truncation or sliding-window retrieval. Instead of discarding context, it adds a focused evidence summary on top of the full input.

3. Replay Evidence Before Generation

The final evidence pool is prepended to the prompt before the model generates its answer. This replay step re-binds the model's attention to the relevant spans, counteracting the positional bias that would otherwise cause it to overlook mid-context information.

Crucially, the full original context is still present. ReContext does not prune or compress the input — it supplements it with a distilled evidence trace.

Performance Results

The authors evaluated ReContext on eight long-context benchmarks at 128K token lengths, using Qwen3-4B, Qwen3-8B, and Llama3-8B as base models. The benchmarks include Natural Questions, TriviaQA, HotpotQA, and several multi-hop reasoning tasks.

Key findings:

• ReContext achieves the best average rank across all eight benchmarks compared to baseline prompting and other training-free long-context methods.
• Mean accuracy improves by approximately 24.6% relative to vanilla prompting across the tested models.
• The gains are consistent across model sizes, suggesting the method is not dependent on a specific architecture or scale.

The improvement is particularly pronounced on multi-hop tasks, where the model needs to synthesize information from multiple locations in the context — exactly the scenario where positional bias causes the most damage.

Why Training-Free Matters

Most approaches to improving long-context reasoning require either fine-tuning the model (expensive, model-specific) or adding external retrieval infrastructure (RAG pipelines, vector databases, re-rankers). ReContext requires neither.

It runs entirely at inference time, using only the model's existing attention mechanism. This means it can be applied to any transformer-based LLM without modification, and it adds no new parameters or training data requirements. The computational overhead is modest: a few additional forward passes for the recursive evidence selection rounds.

This makes it practical for teams that want better long-context performance without committing to a full RAG architecture or a fine-tuning pipeline.

Limitations and Open Questions

ReContext is not a complete solution to the long-context problem. A few caveats worth noting:

• The method has been tested on models up to 8B parameters. Whether the same gains hold for larger models (70B+) with different attention patterns is an open question.
• The evidence token budget (8–32 tokens per round) is a hyperparameter that may need tuning for different task types. Very long or complex answers may require larger budgets.
• The recursive selection process adds latency proportional to the number of rounds. For latency-sensitive applications, this tradeoff needs to be weighed against the accuracy gains.
• The paper evaluates on English-language benchmarks. Performance on multilingual or code-heavy contexts is not yet characterized.

The Broader Context

ReContext fits into a growing set of inference-time techniques that try to extract more value from existing models without retraining. Methods like StreamingLLM address infinite-length streaming by managing attention sinks. KV-cache compression techniques like H2O and SnapKV reduce memory overhead. ReContext occupies a different niche: it improves reasoning quality over a fixed long input, rather than managing memory or extending the effective window.

As context windows continue to grow and agentic workflows push models to process longer and longer histories, the gap between nominal context length and effective reasoning depth will remain a practical concern. Training-free methods like ReContext offer one path toward closing that gap incrementally, without waiting for the next generation of model architectures.

The paper is available at arXiv:2607.02509.