Rotary Position Embeddings: How Rotating Queries and Keys Encodes Relative Position

Transformers have no recurrence and no convolution, so on their own they cannot tell whether a token came first, fifth, or five-hundredth. Early transformers solved this by adding a position vector to each token embedding. That works, but it mixes "what the token means" with "where the token sits" in a single sum, and it forces the model to learn how position should interact with content from data alone.

Rotary Position Embeddings, introduced as RoFormer in 2021, take a different route. Instead of adding position, RoPE rotates the query and key vectors by an angle that depends on their absolute position. The beautiful consequence is that the attention score between two tokens then depends only on their relative distance. Absolute position goes in, relative position comes out.

The core idea in one picture

Imagine each query and key as a vector in a plane. Before attention is computed, rotate the query for position m by angle m times theta, and rotate the key for position n by angle n times theta. When you take their dot product, the two rotations partly cancel, and what survives depends only on the difference m minus n.

That is the whole trick. Nothing is added to the token embedding itself. The value vectors are left untouched, so position changes how tokens attend to each other, not what information they carry once attended to.

The maths: rotation matrices and relative attention

Start with the standard attention score between a query at position m and a key at position n, before any position encoding:

$$\text{score}(m, n) = q_m^{T} k_n$$

RoPE replaces the raw query and key with rotated versions. Let R(m) be a rotation by angle m times theta. Then:

$$q_m' = R(m) q_m, \quad k_n' = R(n) k_n$$

The new attention score is:

$$q_m'^{T} k_n' = q_m^{T} R(m)^{T} R(n) k_n = q_m^{T} R(n - m) k_n$$

The last step uses the defining property of rotations: rotating by n and then un-rotating by m is the same as rotating by n minus m. The score is therefore a function of the query content, the key content, and the relative offset n minus m, but not of m or n separately. This is exactly the property a relative position encoding should have, and RoPE gets it by construction rather than by learning it.

In two dimensions, the rotation matrix is the familiar one:

$$R(m) = \begin{pmatrix} \cos(m\theta) & -\sin(m\theta) \ \sin(m\theta) & \cos(m\theta) \end{pmatrix}$$

From two dimensions to the full model

Real query and key vectors have hundreds of dimensions, not two. RoPE handles this by splitting the vector into pairs of coordinates and rotating each pair by a different frequency. For a vector of dimension d, pair i (counting from 0) is rotated by angle m times theta_i, where:

$$\theta_i = 10000^{-2i/d}$$

The lowest pairs rotate fast with position, so they are sensitive to small changes in distance. The highest pairs rotate slowly, so they change gradually over long distances. Together, the pairs give the model a multi-scale sense of distance, much like the sine and cosine waves of the original absolute position encoding, but applied multiplicatively through rotation instead of additively.

There is an equivalent complex-number view that makes implementation cleaner. Treat each pair (x_0, x_1) as a complex number x_0 plus i times x_1. Rotating by m times theta_i is then just multiplication by the complex exponential:

$$f(x, m) = x \cdot e^{i m \theta_i}$$

This is why RoPE implementations in libraries often work with complex tensors or with interleaved sine and cosine multiplications. The two views are the same operation written in different notation.

Why rotation beats addition

Absolute sinusoidal encodings add a position vector p_m to the token embedding x_m. Expand the attention score and you get four terms: content-content, content-position, position-content, and position-position. The model must learn to disentangle them, and the interaction between content and position is unconstrained.

RoPE keeps content and position separate by design. Position acts only through the rotation applied to queries and keys, and the resulting score is provably a function of relative distance. Three practical benefits follow:

1. No position vector to learn or to extrapolate badly. The rotation formula is defined for any integer position, so in principle there is no hard maximum length baked into a lookup table.
2. Attention decay with distance emerges naturally. As the relative distance grows, the rotations for different frequency pairs drift out of phase, and the expected attention score between unrelated tokens tends to shrink. This gives the model a soft bias toward nearby tokens without a hard attention window.
3. Values stay position-free. Because values are not rotated, a token contributes the same content wherever it sits. Position governs routing (who attends to whom), not payload (what is passed along).

A minimal implementation

The code below shows the idea for one pair of coordinates, using the complex-number view. Real implementations vectorise this across all pairs and all heads, but the logic is identical.

import torch

def apply_rope_pair(x_pair, positions, theta):
    # x_pair: (..., 2) real tensor holding (x0, x1) for one pair
    # positions: (...) integer positions
    # theta: scalar frequency for this pair
    angles = positions * theta  # shape (...,)
    cos = torch.cos(angles).unsqueeze(-1)
    sin = torch.sin(angles).unsqueeze(-1)
    x0, x1 = x_pair[..., 0:1], x_pair[..., 1:2]
    rotated_0 = x0 * cos - x1 * sin
    rotated_1 = x0 * sin + x1 * cos
    return torch.cat([rotated_0, rotated_1], dim=-1)

# Example: rotate a query at position 5 and a key at position 8
# by the same frequency, then check that only the gap (3) matters.
q = torch.tensor([[1.0, 0.0]])
k = torch.tensor([[1.0, 0.0]])
theta = 0.5
q_rot = apply_rope_pair(q, torch.tensor([5]), theta)
k_rot = apply_rope_pair(k, torch.tensor([8]), theta)
score = (q_rot * k_rot).sum()
# score equals cos((8 - 5) * theta) = cos(1.5), as the theory predicts
print(score.item(), torch.cos(torch.tensor(1.5)).item())

Two implementation details matter in practice. First, RoPE is applied to queries and keys only, after their linear projections and before the attention dot product. Applying it to values is a common bug that changes what tokens transmit rather than how they are routed. Second, the pairing convention matters: some implementations pair adjacent coordinates (0 with 1, 2 with 3), while others pair coordinates half a dimension apart (the "GPT-NeoX" style). Both are valid rotations, but mixing them up between training and inference silently changes the encoding, so the convention must match the pretrained weights.

Length extrapolation: where the theory meets reality

RoPE is often described as giving free length extrapolation, because the rotation formula works for any position. In practice, extrapolation is much messier. A model trained on sequences of length 2,048 has only ever seen relative distances up to 2,048, and the slow-rotating pairs have barely completed a fraction of a turn within that range. Feed it a sequence of length 32,768 and those pairs enter angle ranges never seen in training, while the attention softmax faces many more competing tokens than it was calibrated for.

This is why a family of RoPE scaling methods exists. They all adjust the frequencies theta_i, or equivalently the base 10000, to stretch the rotation schedule over longer contexts:

• Linear position interpolation rescales positions by the ratio of new length to old length, so a 32,768-token context is squeezed back into the familiar 2,048 range. It works surprisingly well with a little fine-tuning but blurs fine-grained distance distinctions.
• NTK-aware scaling changes the base instead of the positions, stretching low frequencies more than high ones, on the reasoning that high-frequency pairs already complete many turns within training length and need less adjustment.
• YaRN combines frequency rescaling with a temperature adjustment to the attention softmax, compensating for the entropy growth that comes from attending over many more tokens.

The lesson is not that RoPE fails at long context, but that extrapolation is a property of the whole trained system - frequencies, attention calibration, and training distribution - not of the rotation formula alone. If you extend context, expect to validate on genuinely long inputs, not just to trust the formula.

Limitations and common pitfalls

RoPE is elegant, but it is not a complete theory of position. Keep these limits in mind:

• Relative is not always enough. Tasks that need absolute position, such as counting from the start of a sequence or locating the tenth item, get no direct absolute signal from RoPE alone. Models recover some of it indirectly, but it is not guaranteed.
• The decay is a tendency, not a guarantee. The argument that attention decays with distance holds in expectation over random content vectors. For specific learned queries and keys, long-distance attention can still be strong, which is sometimes exactly what you want.
• Head dimension must be even. Pairs need two coordinates each. An odd head dimension forces padding or a different scheme, and silently dropping a coordinate changes the model.
• RoPE interacts with attention variants. With grouped-query attention or multi-query attention, the same rotation schedule applies to shared keys, which is fine, but the reduced number of key heads changes how much positional diversity the model can express.
• Do not confuse RoPE with ALiBi. ALiBi adds a fixed linear penalty proportional to distance directly to the attention score. RoPE rotates vectors so distance emerges from geometry. Both encode relative position, but they are different mechanisms with different extrapolation behaviour, and they are sometimes combined rather than substituted.

When to reach for RoPE

RoPE is now the default position encoding in most large language models for good reason: it is simple, parameter-free, provably relative, and cheap to compute. Reach for it when you want position to shape attention routing without polluting token content, and when you need a principled starting point for long-context scaling. Reach for something else, or an addition to RoPE, when your task genuinely needs absolute position, hard distance cutoffs, or a position signal that survives into the values themselves.

The deeper lesson of RoPE is a design principle worth stealing: instead of asking a model to learn a structural property from data, build that property into the operation itself. Additive position encodings ask the model to discover that attention should depend on relative distance. RoPE guarantees it, and leaves the model's capacity for everything else.

---

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial