Here's a fact that surprised me the first time it clicked: self-attention has no idea what order your tokens are in.
Feed it "the cat sat" and feed it "sat cat the", and it produces the same set of representations, just shuffled to match. That's because attention builds each token's output as a weighted sum over every other token, and the weights come purely from query-key dot products of the contents. Nothing in that math mentions position. Permute the inputs, and the outputs permute right along with them.
RNNs never had this problem. They walk the sequence one step at a time, so order is baked into the computation for free. Transformers threw that away to get parallelism, and the bill came due: if you want the model to know that "dog bites man" differs from "man bites dog", you have to tell it where each token sits.
Add the position in
The original Transformer fix is almost embarrassingly simple. Give every position its own vector, and add it to the token embedding before the first attention layer:
x = TokenEmbedding(token) + PositionalEncoding(pos)
Addition, not concatenation, so the model width stays the same and the network learns to blend content and position however it likes. Now two copies of the same word at different positions are different input vectors, and attention can react to that.
The only real question is: what should PositionalEncoding(pos) be?
The sinusoidal recipe
Vaswani et al. picked sines and cosines:
PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Each consecutive pair of dimensions is a sine/cosine at one frequency. The trick is that the frequency changes across dimensions: omega_i = 10000^(-2i/d). Low dimensions oscillate fast, high dimensions crawl. It's a geometric ladder of wavelengths, from about 2*pi up to roughly 10000*2*pi.
If you plot the whole thing as a position-by-dimension heatmap, you get the signature look: rapid vertical stripes on the left that fade into slow, broad bands on the right. That picture is the frequency ladder.
Why sinusoids specifically? Four reasons that all matter:
- Bounded. Everything sits in [-1, 1], so adding position never blows up the embedding, no matter how long the sequence.
- Unique. Across the full ladder, every position gets its own fingerprint, a bit like fast and slow bits together naming a number on a binary clock.
- Extends. sin and cos are defined for any real number, so the encoding exists for positions you never trained on. In principle you can run longer sequences.
- Relative. This is the good one.
The relative-position trick
Shift a position by k, and every sin/cos pair rotates by a fixed angle k * omega_i. That rotation doesn't depend on the absolute position at all. So PE(pos + k) is just a fixed linear function of PE(pos).
The payoff shows up in the dot product:
PE(a) . PE(b) = sum_i cos((a - b) * omega_i)
Read that carefully: it depends only on the offset a - b, never on where a and b actually are. The similarity between positions 3 and 5 is identical to the similarity between 300 and 302. Plot similarity against offset and you get a clean curve that peaks at 0 and decays with distance. That is exactly why attention can learn a pattern like "look two tokens back" as one constant relationship instead of memorizing it separately for every location.
Learned, and then RoPE
You don't have to use a formula. BERT and GPT-2 just make position a trainable lookup table, nn.Embedding(max_len, d), and let gradient descent fill it in. It's flexible and often a touch better in-distribution, but it has a hard ceiling at max_len (position 5000 has no row if you trained to 512) and it can't extrapolate.
The modern answer is RoPE, rotary position embedding, used by LLaMA, PaLM, and most current LLMs. RoPE adds nothing to the embeddings. Instead it rotates each query and key by an angle proportional to its position, right before the attention dot product. Because a dot product of two rotated vectors depends only on the difference of their angles, the score becomes an explicit function of the relative offset. Zero extra parameters, and it extrapolates to long contexts far better than absolute schemes.
That's the arc of the whole topic: attention needs a position signal, and the field has drifted from adding absolute vectors up front toward injecting relative offsets right where the query meets the key.
I built an interactive version of all of this: click a position, watch its encoding vector, and see the similarity-vs-offset curve update live. Play with it here:
Top comments (0)