RoPE Explained: Encoding Token Position by Rotating Query and Key Vectors

Self-attention is order-blind. The raw QKᵀ scores are pure content dot-products, so a Transformer has to be told where each token sits. The 2017 sinusoidal recipe adds a positional vector to every token; ALiBi adds a distance bias to the scores. RoPE (Su et al., 2021) does neither — it encodes position by rotating the query and key. It's the scheme behind LLaMA, GPT-NeoX, PaLM, and Mistral, and once you see the trick it's beautifully simple.

Rotate the pairs

Split each d-dimensional query/key vector into d/2 two-dimensional pairs. For a token at position p, spin pair i by an angle θ = p · ω_i. Rotation is a plain 2×2 matrix — and because it's orthogonal, it preserves the norm, which is why RoPE never rescales Q or K:

def rot2(x, y, a):                 # rotate the pair (x, y) by angle a
    c, s = np.cos(a), np.sin(a)
    return x*c - y*s,  x*s + y*c   # R(a) @ [x, y]

The frequency schedule

Each pair rotates at its own fixed rate — a geometric ladder from fast to slow:

def rope_freqs(d, base=10000.0):
    i = np.arange(d // 2)                 # one frequency per PAIR
    return base ** (-2.0 * i / d)         # omega_i, geometric ladder

rope_freqs(8)
# [1.0, 0.1, 0.01, 0.001]                # fast -> slow
# wavelengths 2*pi/omega : 6.28 .. 6283  # local -> long-range

High-frequency pairs (pair 0 spins at 1 radian/step) encode fine, local position; low-frequency pairs barely move and encode coarse, long-range position. It's a built-in multi-scale code. base = 10000 is the original LLaMA value.

The relative-offset property — the whole point

Here's the magic. A dot product of two rotated vectors depends only on the difference of their angles, because of the rotation identity R(mθ)ᵀ R(nθ) = R((n−m)θ). So the attention score rope(q, m) · rope(k, n) becomes a function of the relative offset m − n alone — even though each vector was rotated by its absolute position:

def score(q, k, m, n, base=10000.0):
    return rope(q, m, base) @ rope(k, n, base)

q = np.random.randn(8); k = np.random.randn(8)
assert np.allclose(score(q, k, 5, 3), score(q, k, 7, 5))   # same offset 2
assert np.allclose(score(q, k, 9, 9), q @ k)               # offset 0 -> raw q.k

Positions (5,3) and (7,5) score identically — both have offset 2. Laid out over an (m, n) grid, the score is constant along every diagonal: a Toeplitz matrix.

RoPE attention

In the attention layer you rotate every query and key by its position before the dot products. V is untouched, nothing is added to the inputs, and the resulting score matrix is automatically relative:

def rope_attention(Q, K, V, base=10000.0):
    N, d = Q.shape
    Qr = np.stack([rope(Q[p], p, base) for p in range(N)])   # rotate queries
    Kr = np.stack([rope(K[p], p, base) for p in range(N)])   # rotate keys
    scores = (Qr @ Kr.T) / np.sqrt(d)                        # relative by construction
    scores[np.triu_indices(N, 1)] = -np.inf                  # causal mask
    A = np.exp(scores - scores.max(1, keepdims=True))
    A /= A.sum(1, keepdims=True)
    return A @ V, A

Why it stretches to long context

Push a model past its trained length and the fast pairs alias. Two fixes reuse the exact same code: position interpolation squeezes positions (p → p/s) so old angles cover new lengths; NTK-aware scaling raises the base so slow pairs stretch while fast ones stay sharp. Both only touch the frequency schedule — the rotation is unchanged, and the relative-offset property survives.

That's the trade-off table in one line: sinusoidal is absolute and added to inputs; ALiBi is relative and added to scores; RoPE is relative and applied by rotating Q and K, preserving norm and scaling cleanly to long context.

Slide the query and key positions, watch the 2D sub-planes rotate and the score stay glued to constant diagonals, and read the full six-step build here: https://dev48v.infy.uk/dl/day55-rope.html