Rotary position embeddings do something that sounds impossible. They inject each token's absolute position into the query and key — and the attention score between any two tokens depends only on the distance between them. Move a whole sentence ten tokens later and every score inside it is unchanged.
👉 Live, runs in your browser: https://dev48.in/dl/day81-rotary-embeddings.html
The identity, in one line
Write a 2-D pair of the query at position m as a complex number q, and the key at n as k. RoPE multiplies them by e^(imθ) and e^(inθ):
Re[ (q·e^(imθ)) · conj(k·e^(inθ)) ] = Re[ q·conj(k)·e^(i(m−n)θ) ]
The absolute positions appear only as their difference. Verified over 42 combinations of offset and shift — including a shift of 4000 — worst deviation 1e−15. Additive sinusoidal embeddings, by contrast, give a different score for the same offset at different absolute positions.
function rope(vec, pos, base = 10000) {
const th = thetas(vec.length, base);
const out = new Array(vec.length);
for (let i = 0; i < vec.length / 2; i++) {
const a = vec[2*i], b = vec[2*i+1], ang = pos * th[i];
out[2*i] = a * Math.cos(ang) - b * Math.sin(ang);
out[2*i+1] = a * Math.sin(ang) + b * Math.cos(ang);
}
return out;
}
🔴 Now the part people get wrong
"Attention under RoPE decays with distance." Plot it carelessly and you get a flat line of random scatter, because two independent random vectors have an expected dot product of zero at every distance. The curve you drew was sampling noise.
The usual next move is to average the magnitude instead. That does not rescue it either:
| offset | q against itself | mean raw score, 600 pairs | mean |score|, same pairs |
|---|---|---|---|
| 0 | 10.60 | −0.07 | 3.17 |
| 16 | 8.05 | −0.37 | 3.12 |
| 256 | 4.95 | −0.28 | 3.04 |
| 1024 | 0.69 | −0.38 | 3.02 |
Three columns, same vectors, same offsets. The mean raw score never leaves ±0.38 against a typical magnitude of 3.2. The mean |score| moves 4% across the whole range. Only the self column collapses — and that is the decay people are describing: the pairs start perfectly in phase and dephase at different rates until their cosines cancel.
It is dephasing, not attenuation, and it is a statement about a fixed vector.
🔴 The bug that produced the first, wrong measurement
Each "random" vector got its own Park-Miller generator seeded by a nearby integer. Nearby LCG seeds give correlated streams. The supposedly independent mean wandered to ±2.5 — an order of magnitude past its standard error — and looked like structure.
One generator for the whole study, and it drops to ±0.38. If your "independent samples" come from rng(seed + i), they are not independent.
🔴 NTK scaling cannot touch the fastest pair
Train to 512, ask about 4096. It is tempting to blame the fast pairs, but they were already wrapping constantly inside the training window — another twenty turns is nothing new.
The pair that leaves familiar ground is the slowest one. Its period is longer than the whole context, so across training it swept only 2.6% of a turn; push the position eightfold and it swings into angles that never once appeared.
| strategy | slowest pair's angle | |
|---|---|---|
| trained, at 512 | 0.162 rad | what the model saw |
| untouched, at 4096 | 1.295 rad | 8× past anything in training |
| position interpolation | 0.162 rad | back exactly where it trained |
| NTK-aware base | 0.162 rad | back exactly where it trained |
Both repairs land it in the same place. They differ at the fast end: θ₀ = base⁰ = 1 whatever you do to the base, so NTK cannot slow the fastest pair at all, while interpolation slows every pair equally and costs resolution between adjacent tokens.
How it is checked
The verifier multiplies complex numbers rather than using the real 2×2 form, extracts the rotation as a matrix and tests RᵀR = I with det = +1, and re-measures the decay with a Marsaglia polar sampler so a bug in the page's generator cannot reproduce itself in the check.
It caught a sign error in my closed form — + where the derivation gives − on the sin term.
226 verifier asserts, 28 in-page checks, 0 failures.
Top comments (0)