Lost in the middle: why a 100k-token context window is not 100k tokens of attention

An LLM does not read a long context evenly. Bury a single fact — a needle — inside a long stack of documents and ask the model to retrieve it, and how reliably it succeeds depends heavily on where the fact sits. Facts at the very start (primacy) or the very end (recency) are recalled reliably; a fact in the middle is far more likely to be missed. Plot accuracy against needle position and you get a U-shaped curve — high at both edges, sagging in the middle. That's the finding of Liu et al., 2023, "Lost in the Middle: How Language Models Use Long Contexts."

And it gets worse as the context grows. The more documents you stuff in, the deeper the middle sags. So a 100k-token window does not mean 100k tokens of usable attention. Long context ≠ used context.

Modelling the U: primacy, recency, and a decaying floor

Map a needle in document i of N to a relative position r = (i-1)/(N-1) in [0,1]. The edges are lifted by two Gaussian bumps — primacy peaks at the start, recency peaks at the end — and in the middle both have decayed to nearly zero:

import math

def primacy(r, strength=0.97, width=0.28):
    return strength * math.exp(-(r / width) ** 2)      # strong at r=0

def recency(r, strength=1.00, width=0.22):
    return strength * math.exp(-((1 - r) / width) ** 2) # strong at r=1

def edge_advantage(r):
    return max(primacy(r), recency(r))   # ~1 at the edges, ~0 in the middle

A middle document has no edge advantage — its accuracy is just the background floor, and that floor decays as the context grows, because more documents compete for finite attention:

def middle_floor(n, floor0=0.85, k=0.055):
    return floor0 * math.exp(-k * (n - 1))

middle_floor(5)   # ~0.68  short context -> middle is still findable
middle_floor(24)  # ~0.30  the middle is sagging
middle_floor(60)  # ~0.03  the middle has effectively vanished

Combine them and you get a clean, always-U-shaped position-to-accuracy curve:

def accuracy(i, n, floor0=0.85, k=0.055):
    r = rel_pos(i, n)
    floor = middle_floor(n, floor0, k)
    acc = floor + (1 - floor) * edge_advantage(r)
    return max(0.02, min(0.995, acc))       # clamp: never 0, never 1
# n=24: start -> ~0.99, middle -> ~0.30, end -> ~1.00   (a clean U)

At the edges edge ≈ 1 so acc ≈ 1; in the middle edge ≈ 0 so acc ≈ floor. A bigger or long-context-tuned model raises floor0 and shrinks k — a flatter, higher U with a wider usable middle.

The fix is positional, not magical

If accuracy depends on position, then control the positions. Rerank retrieved chunks by relevance with a cross-encoder, keep only the few best, then interleave them onto the edges of the prompt — best and second-best at the start and end, weakest buried in the middle where a miss costs least.

def edge_first_order(ranked):
    "Put the most relevant chunks at the START and END, weakest in the middle."
    head, tail = [], []
    for idx, chunk in enumerate(ranked):
        (head if idx % 2 == 0 else tail).append(chunk)
    return head + list(reversed(tail))   # best -> edges, worst -> middle

ordered = edge_first_order(rerank(chunks, query, cross_encoder))

The practical toolkit is short: retrieve fewer, better chunks; rerank with a cross-encoder (Cohere Rerank, bge-reranker, a fine-tuned scorer); place the strongest evidence at the top and bottom of the prompt; keep the question near an edge; and reach for long-context-tuned models, which flatten the U, when you truly need length.

Above all, the takeaway: "it's in the context" is not the same as "the model used it." Verify with a needle-in-a-haystack eval before you ship — don't assume.

Place a needle at any depth, grow the context length, and watch the U-curve's middle collapse, live at: https://dev48v.infy.uk/ai/days/day54-lost-in-the-middle.html