A model trained at 2k tokens does not "get confused" past 2k. It breaks, and the reason is arithmetic you can compute in a few lines.
RoPE encodes position as a rotation by m · theta_i. Attention then depends only on the relative angle (m-n) · theta_i — absolute position cancels. I verified that identity on 5,000 random pairs before trusting anything built on it.
Each dimension has a wavelength, the distance over which it completes a full rotation:
lambda_i = 2*pi / theta_i = 2*pi * base^(2i/d)
At d=64 and base 10000, the slowest wavelength is 47,117 tokens. Inside a 2k training window that dimension swept a tiny arc.
Measure it yourself: https://dev48.infy.uk/ai/days/day63-context-extension.html
The diagnosis is two lines
function outOfRange(d, base, L){
return ropeFreqs(d, base).filter(t => wavelength(t) > L).length;
}
11 of 32 dimension pairs have a wavelength longer than a 2048-token window. Ask them for position 8000 and they produce angles that never appeared in training. Not slightly-worse angles — unseen ones. That is why naive extrapolation gives confident nonsense rather than graceful degradation.
Three fixes, and what each actually costs
Position Interpolation — divide every position by s = L'/L. Position 8000 becomes 2000, inside the trained range. Nothing is ever asked for an unseen angle. The cost lands entirely on the high frequencies: at 4× extension, adjacent tokens are four times harder to tell apart in exactly the dimensions that encode word order. Local resolution kept: 25%.
NTK-aware — raise the base instead: base' = base * s^(d/(d-2)), which is 10000 → 41,829 at 4×. Barely touches the fast dimensions, stretches the slow ones. Local resolution kept: 100%. Often needs no finetuning, which is why it spread as a config tweak before it had a paper.
YaRN — decide per dimension by wavelength. Many cycles inside training, leave alone; never cycled, interpolate fully; in between, ramp. At these settings that splits 9 untouched, 12 ramped, 11 fully interpolated.
Two things my own page got wrong
The decay curve was plotting noise. I averaged rotated dot products over independent random q and k — whose expected value is exactly 0 at every distance. There was no decay to see. RoPE's long-term decay is about a vector losing alignment with itself: sum_i |q_i|^2 * cos(d * theta_i). Fixed, the curve peaks at distance 0 and falls away, as it should.
The out-of-range metric compared rescaled frequencies against themselves, which always reports zero and would have made every method look safe.
What none of this fixes
All three make positions representable. None teaches the model to use information at 30k tokens — that is a data and training question, and it is why a model advertising a huge window can still fail a needle test in the middle of it. Extending the context and using the context are different problems.
Top comments (0)