Self-attention treats every token in the sequence the same way: it computes attention scores based purely on content, comparing queries against keys regardless of position. Left alone, self-attention has no notion of order — "the cat sat on the mat" and "mat the on sat cat the" would look identical to it. Positional embeddings are how we teach the model where each token sits in the sequence.
Self-Attention Doesn't See Order
The attention mechanism from Phase 3 computes a weighted sum over value vectors, where the weights come from a similarity between queries and keys. Nothing in that computation depends on the index of a token — it's a set operation, not a sequence operation. If you shuffled the input tokens and shuffled the output in the same way, the result would be unchanged (this property is called permutation equivariance).
That's a problem for language, where order carries meaning. The fix is not architectural — we don't change attention itself — it's additive: we inject positional information directly into the input embeddings before they ever reach the attention layers. Each token's embedding gets a positional vector added to it, so that by the time attention runs, position is already baked into the representation it's comparing.
Two Ways to Encode Position
There are two broad families of positional encoding: fixed (sinusoidal) and learned.
The original Transformer paper used fixed sinusoidal functions — sine and cosine waves of different frequencies, one pair per pair of embedding dimensions. This has an elegant property: the encoding for any position can be expressed as a linear function of the encoding for any other position, which in theory lets the model generalize to sequence lengths it never saw during training.
Learned positional embeddings, by contrast, are just another embedding table — exactly like the token embedding table from Phase 2, but indexed by position instead of by token id. Position 0 gets a trainable vector, position 1 gets a different trainable vector, and so on, up to some maximum length. GPT-2 and MiniGPT both use this approach: simpler to implement, and in practice it performs comparably to sinusoidal encoding for the sequence lengths these models actually train on.
MiniGPT uses learned positional embeddings for that reason — it keeps the implementation symmetric with the token embedding table, and there's no need for the extrapolation properties sinusoidal encoding offers when the context window is fixed and known in advance.
contextWindow, Not maxSeqLen
One naming decision worth calling out: MiniGPT's positional embedding table is sized by a field called contextWindow, not maxSeqLen. This isn't just a style preference. "Max sequence length" suggests an incidental limit — the longest input you happen to support. "Context window" names what the number actually is: the boundary of what the model can attend to at all. Every position beyond it simply has no embedding to look up.
That distinction matters for readability. A future maintainer reading contextWindow immediately understands why the value matters (it defines the model's attention horizon), rather than treating it as an arbitrary array-bounds constant to work around.
Implementation Guided by Testing
The positional embedding table in MiniGPT is a straightforward parallel to the token embedding table: a matrix of shape [contextWindow, dModel], where row i holds the learned vector for position i.
public class PositionalEmbedding {
private final float[][] weights; // [contextWindow][dModel]
private final int contextWindow;
private final int dModel;
public PositionalEmbedding(int contextWindow, int dModel) {
this.contextWindow = contextWindow;
this.dModel = dModel;
this.weights = new float[contextWindow][dModel];
initializeWeights();
}
private void initializeWeights() {
Random random = new Random();
float bound = 1.0f / (float) Math.sqrt(dModel);
for (int pos = 0; pos < contextWindow; pos++) {
for (int d = 0; d < dModel; d++) {
weights[pos][d] = (random.nextFloat() * 2 - 1) * bound;
}
}
}
public float[] embed(int position) {
if (position >= contextWindow) {
throw new IllegalArgumentException(
"Position " + position + " exceeds contextWindow " + contextWindow);
}
return weights[position];
}
public float[][] embedSequence(int sequenceLength) {
if (sequenceLength > contextWindow) {
throw new IllegalArgumentException(
"Sequence length " + sequenceLength + " exceeds contextWindow " + contextWindow);
}
float[][] result = new float[sequenceLength][dModel];
for (int pos = 0; pos < sequenceLength; pos++) {
result[pos] = embed(pos);
}
return result;
}
}
The bounds check in embed is not defensive boilerplate — it's the direct enforcement of the contextWindow concept from the previous section. A unit test that asserts this exception is thrown for position == contextWindow is what actually pins down the invariant; without it, an off-by-one in a caller could silently read garbage or throw an unrelated array-index exception with a far less useful message.
Combining token and positional embeddings is simple element-wise addition: the final input to the transformer blocks is tokenEmbedding[i] + positionalEmbedding[i] for each position i. Both live in the same dModel-dimensional space, which is precisely why dModel had to be decided once, up front, in Phase 3 — every component downstream depends on that shared dimensionality.
What This Means at Production Scale
The contextWindow choice has real consequences at scale. MiniGPT's table is tiny by design. GPT-2 XL's context window is 1024 tokens; Llama 3.1's is 128,000; some production systems now advertise context windows in the millions. Since the positional embedding table (for the learned variant) is [contextWindow, dModel], a naive learned-embedding approach doesn't scale gracefully to those lengths — this is exactly why modern large-scale models have largely moved to relative or rotary positional encodings (RoPE), which encode position as a function of the relative distance between tokens rather than as a fixed per-position lookup table. That's a natural next question, but outside the scope of what MiniGPT needs to demonstrate the core mechanism.
For MiniGPT's purposes, the fixed-size learned table is the right tradeoff: it's the simplest correct implementation of "the model needs to know where each token is," and it makes the connection between contextWindow and the model's attention horizon completely explicit in the code.
Top comments (0)