Context Windows: RoPE, Long Context, and Why Models Forget
The window numbers keep growing. GPT-4o lists 128,000 tokens. Gemini 2.5 Pro lists 1,048,576. Llama 4 Scout claims 10 million. The numbers read like disk space, and vendors want you to think of them that way. We ran the measurements, and the window behaves nothing like storage. A model that can hold 128,000 tokens cannot use all 128,000 tokens equally. Some positions are easy to reach, some are nearly invisible, and the first few tokens get an attention share no later token ever gets. This part explains why, with numbers from our own runs on GPT-2 plus the papers that measured the same effects on frontier models.
This is Deep Part 5 of the AI Engineer Deep series. Deep Part 1 built attention by hand, Deep Part 2 built a tokenizer from scratch, Deep Part 3 trained tiny transformers and read the loss curves, and Deep Part 4 measured the KV cache and sampling. This part takes the model that came out of Part 3, runs it at the edge of its window, and asks what actually happens to memory when the context gets long. If you only read one section, read the needle test and the attention sink table. They settle more arguments about context windows than any amount of marketing.
Positions are the problem, not memory
Attention has no idea where a token sits. The dot product between a query and a key does not know whether the two tokens are neighbours or a thousand tokens apart. Somebody has to tell it, and the choice of how to tell it decides how far the model can reach. GPT-2, the model we run in this series, uses learned absolute position embeddings: one vector per position, 0 to 1,023, added to the token embedding. Position 500 and position 505 are two unrelated vectors the model memorised in training. Feed the model a token at position 1,024 and there is no vector for it at all. That is a hard wall, not a soft limit, and it is why GPT-2 stops at 1,024 tokens no matter how much memory you give it.
The field moved away from absolute positions for exactly this reason. ALiBi, from Press and colleagues in 2021, drops position embeddings entirely and adds a fixed linear penalty to the attention scores, proportional to how far apart the query and key are. A 1.3 billion parameter model trained on 1,024 token sequences extrapolated to 2,048 tokens with the same perplexity as a sinusoidal model trained on 2,048, while training 11 percent faster and using 11 percent less memory. On WikiText-103, a smaller model trained on 512 tokens reached a dev perplexity of 18.40 at an evaluation length of 3,072, beating a sinusoidal model trained on 3,072, which scored 18.67. The bias toward recent tokens is a feature, not a bug: it is the model admitting that distance matters.
RoPE, rotary position embedding, from Su and colleagues in 2021, is the design that won. Almost every open weight model since LLaMA uses it. Instead of adding a position vector, RoPE rotates the query and key vectors by an angle proportional to their position, using a base of 10,000 across the dimensions. The rotation cancels out in the dot product except for the difference between the two positions, so the attention score depends only on the relative distance. No extra parameters, no change to the attention formula, and a built in long term decay: tokens further apart score lower on average. That last property is the one vendors lean on when they stretch a window, and it is the one we measured first.
RoPE, measured by hand
We implemented RoPE in NumPy, head dimension 64, base 10,000, and checked the core claim directly: for a fixed relative distance, the score must be identical at every absolute position. It is, to six decimal places.
Query positionKey positionRelative distanceAttention score
1055-9.991476
100955-9.991476
5004955-9.991476
1,0009955-9.991476
That is the whole point of RoPE in one table. Absolute embeddings cannot do this. With RoPE, position 500 and position 995 are not memorised vectors. Only the gap between them matters, which is why a RoPE model can be stretched to lengths it never trained on by rescaling the base, while an absolute embedding model hits a wall.
Then we measured the decay. Take a query vector and the same vector as the key, the case of a query attending to an identical token further back, and average over 2,000 random vectors at each distance:
Relative distanceMean attention score
063.94
161.81
546.67
1042.16
5031.10
10035.65
50013.04
1,00017.86
4,0000.31
The score falls from 63.94 at distance zero to 0.31 at distance 4,000, a factor of about 200. But look at the middle of the table before concluding the decay is smooth. At distance 100 the score is 35.65, higher than the 31.10 at distance 50. At 1,000 it is 17.86, higher than the 13.04 at 500. The decay is real in the long run and lumpy in the middle, because the rotation is a sum of cosines at different frequencies that interfere with each other. RoPE gives you relative distance and a soft long range penalty, not a clean forgetting curve. Anybody who tells you attention decays smoothly with distance has not measured it. We also checked the EleutherAI comparison at 1.4 billion parameters on The Pile: RoPE scored 8.78 perplexity against 9.39 for learned absolute embeddings, so the design wins on quality as well as on length.
What attention actually does in a long context
The usual story about long context is dilution: more tokens means the softmax spreads thinner, so each token gets less attention. We tested that on GPT-2 by feeding contexts of 128, 256, 512, and 1,024 tokens and reading the attention distribution of the last token, averaged over all 12 heads, in layer 6 and layer 12. The entropy did not rise the way the dilution story predicts.
Context lengthLayerEntropy (nats)Top weightMass on first tokenMass on first 8 tokens
12861.620.5810.3170.552
25661.880.6160.4350.622
51261.770.5860.3680.542
1,02461.860.6190.3260.512
128122.450.4940.2430.480
256122.740.5170.2770.525
512122.600.5660.3040.562
1,024122.420.6050.3140.601
Entropy in layer 6 goes 1.62, 1.88, 1.77, 1.86 as the context grows eightfold. In layer 12 it goes 2.45, 2.74, 2.60, 2.42. It is flat, not rising. Attention does not spread thinner. It concentrates, and it concentrates on the same place every time: the first few tokens. At every length we tested, the last token gives between 48 and 62 percent of its attention to the first 8 tokens, and between 24 and 44 percent to the very first token alone. The top single weight never drops below 0.49. One token, out of 1,024, holds most of the distribution.
This is the attention sink, and the StreamingLLM paper from Xiao and colleagues in 2023 explains why it happens. The softmax must sum to one, and the first tokens are visible to every later token, so later tokens dump their leftover attention mass onto them regardless of meaning. The paper measured how load bearing those first tokens are: deleting the first 4 tokens from Llama-2-7B's 4,096 token KV cache sends perplexity from 9.59 to 3,359.95, a jump of about 350 times. Keeping just 4 sink tokens plus a sliding window restores perplexity to 5.40, against 5.43 for full recomputation. Four tokens carry the whole trick. With the sink kept, the same paper held generation stable to 4 million tokens with no fine tuning, 22.2 times faster than recomputing the window.
Our take is that the sink is both the explanation and the warning. It explains why a long system prompt still works: it sits at the start, where attention already pools. And it warns that the middle of a long context is fighting for the attention that is left after the sink and the recent tokens take theirs. That is exactly what the next section measures.
Lost in the middle, on paper and on our CPU
The cleanest measurement of this problem is Lost in the Middle, from Liu and colleagues in 2023. They gave GPT-3.5-Turbo and Claude-1.3 a question plus 20 documents, only one of which held the answer, and moved the answer document through the context. With the answer first, GPT-3.5-Turbo scored 75.8 percent. With the answer in the middle, position 10, it scored 53.8 percent. With the answer last, 63.2 percent. The curve is U-shaped, and the middle is the worst place to put the fact that matters. The closed book score, with no documents at all, was 56.1 percent, which means the middle position scored below knowing nothing: having the answer in the window in the wrong place was worse than not having it. The oracle score, with only the answer document, was 88.3 percent. The best to worst drop was 22 points, and the same U-shape appeared at 10 and 30 documents. Adding more documents barely helped: going from 20 to 50 retrieved documents improved GPT-3.5-Turbo by only about 1.5 percent. The honest conclusion of that paper is that retrieval quality beats retrieval quantity, and position beats both.
We ran a small version of the same test on GPT-2. The setup: about 880 tokens of neutral filler sentences, one needle sentence, "The secret passcode is 84712", placed at the start, the middle, or the end, then a question in the format the paper uses: "Question: What is the secret passcode? Answer: The secret passcode is". We measured the probability the model assigns to the correct 5 digit code, for three different codes and three different filler shuffles.
CodeStartMiddleEndMiddle, with distractor
847120.5230.2200.1910.116
391080.6230.1220.0090.165
562900.6390.3320.0760.144
Mean0.5950.2250.0920.141
The pattern is clear and it is not the U-shape from the paper. The start wins by a wide margin, the middle is less than half as good, and the end is the worst of all: mean probability 0.595 at the start, 0.225 in the middle, 0.092 at the end. A distractor sentence, "The old passcode was 11111", placed elsewhere in the context, cut the middle score further, from 0.225 to 0.141, about a 37 percent relative drop. At the end position the model often did not even try to answer: its top prediction was " a", with probability 0.33, as in "The secret passcode is a...", a sentence continuation instead of a retrieval.
We will not pretend this small model refutes the paper. GPT-2 is not instruction tuned, it has 124 million parameters, and it treats a question as text to continue, not as a task to perform. The big models in the paper show a U-shape because they are trained to answer, and recency helps them. Our model shows primacy only, because its attention sink and its absolute positions favour the start. The shared finding is the one that matters for builders: position moves retrieval accuracy by a factor of six in our test and by 22 points in theirs, and the middle is bad in both. If the fact that decides your answer sits in the middle of a long prompt, you have a position problem, and no amount of prompt wording fixes a position problem. Put the important fact first, or retrieve it into a short context instead.
One more honest note from this experiment. We first ran it with the question repeating the needle sentence verbatim, "The secret passcode is", instead of a real question. The numbers flipped: the start and middle scored 0.94 and 0.87, and the end collapsed to 0.05, because the model treated the repeated phrase as a sentence to continue rather than a cue to retrieve. Prompt wording moved the result more than position did. That is itself a finding, and it is why we report both runs: in a small model, format beats position, and a needle test that does not control its question format is measuring the format.
Forgetting by distance
Retrieval is one kind of forgetting. The other is decay: a fact introduced early, then buried under filler. We introduced a rare word once, "Zanzibar", in the sentence "The research station is located in Zanzibar", then added 0, 200, 500, 650, or 850 tokens of filler, then asked for the word again in "Supplies are shipped to Zanzibar every month". We measured the probability of the word at each gap, and the same for "Timbuktu" as a second word, plus a baseline with no lead sentence at all.
Gap (tokens)Zanzibar log probabilityZanzibar probabilityBoost over no lead baseline
No lead (baseline)-16.160.000000101x
0-11.590.0000092496x
200-14.380.000000575.9x
500-18.200.000000010.13x
650-14.550.000000485.0x
850-13.560.0000012913x
The lead sentence multiplies the word's probability by 96 times when it is adjacent. At 200 tokens the boost is 5.9 times. At 500 tokens it falls below the baseline entirely, to 0.13 times, which means the model is less likely to produce the word than if the lead had never been mentioned. Then it recovers: 5.0 times at 650 tokens, 13 times at 850. The Timbuktu run shows the same shape: log probability -12.85 at gap 0, -14.71 at 200, -20.38 at 500, -13.55 at 850, against a no lead baseline of -15.88. The recovery at 850 tokens held across three different filler shuffles, so it is a property of this model, not noise in the filler.
Our reading is that forgetting in a small model is not a smooth decay curve. It is lumpy and position dependent, and past about 500 tokens the local sentence matters more than anything the model read earlier. The 500 token dip is the honest surprise of this part: at that distance the model does worse than its own base rate, as if the distant mention actively interferes. We do not have a clean mechanism for the recovery at 850, and we will not invent one. What we can say is that GPT-2's absolute position embeddings make some positions easier to attend to than others, and that a forgetting curve measured at one gap tells you nothing about the next gap. If your system depends on a fact surviving 500 tokens of context, measure it at 500 tokens, not at 50.
Stretching the window after training
The good news is that windows are no longer fixed at training time. The first trick was NTK aware scaling, proposed by Reddit user bloc97 in 2023 and documented by EleutherAI: instead of compressing position indices, scale the RoPE base so the high frequency dimensions barely change and the low frequency dimensions take the compression. It extended LLaMA well past its trained length with zero fine tuning, where plain linear interpolation failed.
YaRN, from Peng and colleagues in 2023, turned that idea into a method: NTK by parts interpolation of the RoPE frequencies plus a temperature term on the softmax. It took Llama 2 7B and 13B from 4,000 tokens to 128,000, the first method to reach 128k effective context on Llama 2. The cost was small: 400 fine tuning steps for the 16x model and 200 more steps for the 32x model, under 0.1 percent of the original pretraining data, and training data capped at 64k length. Perplexity kept falling all the way to 128k despite never training on 128k text: 3.56 at 8k, 3.04 at 16k, 2.70 at 32k, 2.45 at 64k, 2.37 at 128k. Passkey retrieval at 128k scored above 99 percent for both model sizes.
LongRoPE, from Ding and colleagues in 2024, pushed further: 2,048,000 tokens, the first method past 2 million, using non uniform positional interpolation found by evolutionary search plus progressive extension, fine tuned at 256k and then interpolated again. The whole extension took at most about 1,000 fine tuning steps at training lengths within 256k, and without any fine tuning it still achieved an 8x extension, from 4k to 32k, at low perplexity.
Extension is cheap now. That is not the same as extension working. Stretching the window gives the model room to hold tokens. It does not give the model the ability to use them, and the next section is the gap between the two.
Claimed windows versus effective windows
Here is what vendors claim, and here is what the RULER benchmark measured. RULER, from Hsieh and colleagues in 2024, defines effective length as the longest tested length where a model still beats a fixed bar, 85.6, the score Llama 2 7B gets at 4k, across 13 synthetic tasks covering retrieval, multi hop tracing, aggregation, and QA. Seventeen models were tested, all claiming 32k or more. Only about half held satisfactory performance at 32k, despite near perfect scores on the vanilla needle in a haystack test.
ModelClaimed windowEffective window (RULER)Score at 4kScore at 128k
Gemini 1.5 Pro1,000,000Over 128k96.794.4
GPT-4 (1106 preview)128,00064,00096.681.2
Llama 3.1 70B128,00064,00096.566.6
Mistral 7B32,00016,00093.613.8
Yi 34B200,00032,00093.6Not reported
Read the Mistral row twice. It claims 32,000 tokens and scores 93.6 at 4k, then 13.8 at 128k, and its effective window is 16,000, half the claim. Llama 3.1 70B claims 128k, holds to 64k, and falls to 66.6 at 128k. GPT-4 holds to 64k and falls to 81.2. Only Gemini 1.5 Pro in this set holds above 128k, at 94.4. The pattern is consistent: the effective window is roughly half the claimed window, and the claimed window is a storage number, not a capability number.
The needle in a haystack test tells the same story from the other side. Greg Kamradt's test hides one fact at varying depths inside long essays and asks the model to retrieve it. GPT-4 Turbo's recall is near perfect at short lengths, begins to decline past about 64k tokens, and falls sharply at 100k and above, with failures concentrated when the needle sits near the beginning of the context. Claude 2.1 scored only 27 percent overall in the original run, then 98 percent after Anthropic changed the needle to match the haystack topic and added one prompt line telling the model to return the most relevant sentence. A 71 point swing from a prompt change, on the same model, on the same test. That is how much of long context performance is prompt craft and test design rather than raw window size.
What we actually do with long context
Five rules, each earned by a number above. First, put the important fact at the start. Our needle test retrieved it at 0.595 from the start and 0.092 from the end, and the Lost in the Middle paper puts the middle at the bottom. The start is the only position that is good in every study we read. Second, repeat the key fact. A second mention is cheap and it gives the model a second position to retrieve from, which matters when one position is bad. Third, retrieve instead of dumping. If the answer lives in one paragraph of a 100 page document, send the paragraph, not the document. Our earlier RAG chunking measurements and hybrid search guide are the practical version of this rule, and our RAG evals guide shows how to check whether the retrieval actually worked. Fourth, test your own needle. Take a real document from your own system, hide a fact at five depths, and measure retrieval at each one. Our numbers came from GPT-2 and your model will differ, but the shape, a bad middle and a position dependent curve, will almost certainly be there. Fifth, treat the claimed window as half. RULER says the effective window is roughly half the claim for most models, and the needle test says performance falls off well before the claim. Plan capacity on the effective number, not the marketing number.
The opinion underneath all of it: vendors sell context like disk space, and it behaves like working memory. A bigger window is a bigger desk, not a better memory. The model can hold more paper, but it still reads the top sheet first, forgets the middle sheet, and trips over the wording of the question. The window is not the product. Retrieval inside the window is, and that is a property of position, prompt, and measurement, not of token count.
What is next in this series
This is Deep Part 5 of the AI Engineer Deep series, a line by line revisit of the original 8 part series with real research and real measurements. Deep Part 6 takes on RAG chunking: which chunking strategies actually survive at scale, measured on real documents, building on our first chunking test and Part 4 of the original series, RAG from zero. The original series still stands as the map: Part 3, How LLMs Actually Work is where tokens and temperature first appeared, Deep Part 4 on inference is where the KV cache behind long context lives, and Part 8 on LLMOps shows what the long context bill turns into once the model ships. If you want the cost side of this story now, our measured guide to cutting the LLM API bill puts numbers on five techniques.
Previous: LLM Inference: KV Cache, Sampling, and Temperature Measured
Next: Deep Part 6: RAG Deep: Chunking Strategies Tested at Scale (coming next in this series)
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)