Every Transformer-based language model, from BERT to GPT, relies on a mechanism called attention to determine how different tokens in a sequence relate to one another. At the heart of this mechanism are three representations: Query (Q), Key (K), and Value (V).
Let's understand what they are, why we need them, and how they work together to help Transformers process language.
1. What Are Query, Key, and Value (QKV)?
To understand QKV, imagine searching for information in a large collection of documents. You have a question you want answered, each document has information that helps you decide whether it is relevant, and each document contains the actual content you might retrieve.
The attention mechanism follows a conceptually similar process: it determines what information a token should gather from other tokens in the sequence.
Query (Q) — What information am I looking for?
A Query represents what a token is looking for in other tokens. It helps determine which tokens contain information relevant to the current token.
Key (K) — What information do I match on?
A Key represents the features used to match a token against a Query. It helps determine how relevant that token is to another token's Query.
Value (V) — What information do I contribute?
A Value represents the information a token contributes when other tokens attend to it. The attention mechanism combines Values according to the attention weights.
In simple terms:
- Query determines what a token is looking for.
- Key helps determine how relevant another token is.
- Value supplies the information that gets incorporated.
These are not three different tokens. They are three different representations of tokens, each serving a distinct purpose in the attention mechanism.
2. Why Do We Need Q, K, and V?
Words get their meaning from context. Consider the word bank in these two sentences:
- "I deposited money in the bank."
- "We sat on the bank of the river."
The word is the same, but its meaning changes depending on the surrounding words. A Transformer needs a way to understand these relationships.
This is where self-attention comes in. It allows each token to gather relevant information from other tokens in the sequence.
But how does a token decide which other tokens are relevant and what information to gather from them?
That's why we need Query, Key, and Value.
For example, in "The animal was tired because it had been running," the token it can attend to animal and other relevant words to build a context-aware representation.
Without this mechanism, the model would have a much harder time connecting information across a sequence. QKV gives self-attention a flexible, learnable way to determine which tokens should influence one another and how their information should be combined.
In short: Queries and Keys help determine where attention goes, while Values determine what information is gathered.
3. How Are Q, K, and V Generated?
Now that we understand why Q, K, and V are needed, let's look at how a Transformer actually generates them.
It starts with the input representations of the tokens. These are arranged into a matrix called (X), where each row represents a token.
The Transformer then uses three learnable weight matrices to create Query, Key, and Value:
Q = X × W_Q
K = X × W_K
V = X × W_V
Here, (W_Q), (W_K), and (W_V) are learnable weight matrices. Each transforms the input representations into a different representation for its specific role.
Why three matrices? Because the model needs to learn what information is useful for calculating attention scores and what information should be passed along through the Values.
These matrices are learned during training through backpropagation and gradient-based optimization.
The important point is that Q, K, and V are not manually defined. The model learns how to generate them from the input representations.
4. How Do Queries and Keys Calculate Attention?
Once the Transformer generates Q, K, and V, it needs to determine how relevant each token is to the others. This is where Queries and Keys come into play.
Step 1: Calculate Attention Scores
The model multiplies the Query matrix (Q) by the transpose of the Key matrix (K):
S = Q × Kᵀ
Each score represents how strongly a token's Query matches another token's Key. Higher scores generally lead to higher attention weights, although these scores are learned representations rather than direct measures of semantic similarity.
Step 2: Scale the Scores
Before applying softmax, the model divides the scores by the square root of the Key dimension, (d_k):
S_scaled = (Q × Kᵀ) / √dₖ
This scaling helps prevent the dot products from becoming excessively large, which can cause softmax to produce very peaked distributions and make optimization less effective.
Step 3: Apply Softmax
Finally, softmax converts the scaled scores into normalized attention weights:
A = softmax((Q × Kᵀ) / √dₖ)
Each row of (A) contains the attention weights for one token, indicating how its attention is distributed across the available tokens. In causal language models, an attention mask prevents tokens from attending to future positions.
In short: Queries and Keys calculate relevance scores, scaling helps stabilize them, and softmax converts them into attention weights.
5. How Do Values Produce Contextual Information?
After calculating the attention weights, the Transformer uses them to determine how much information to gather from each token. This is where Values (V) come into play.
The attention weights are multiplied by the Value matrix to produce the attention output:
O = A × V
Substituting the attention formula gives us the complete scaled dot-product attention equation:
Attention(Q, K, V) = softmax((Q × Kᵀ) / √dₖ) × V
Here, each token's output is a weighted combination of the Value vectors from the tokens it is allowed to attend to.
For example, when processing "The animal was tired because it had been running," the representation of it can incorporate information from animal and other relevant tokens, helping the model build a context-aware representation.
The model does not simply select the token with the highest attention score. Instead, it generally combines information from multiple tokens according to their attention weights.
In short: Queries and Keys determine how attention is distributed, while Values carry the information that gets combined to produce the attention output.
6. QKV in Multi-Head Attention
So far, we've explored how a single attention mechanism uses Query, Key, and Value to determine relationships between tokens and combine information. But what if a Transformer needs to capture different types of relationships at the same time?
This is where Multi-Head Attention comes in.
Instead of using a single attention mechanism, the Transformer uses multiple attention heads, each with its own learned Query, Key, and Value projections. Different heads can learn to focus on different patterns, such as grammatical relationships or connections between entities.
Each head calculates attention independently:
headᵢ = Attention(X × WᵢQ, X × WᵢK, X × WᵢV)
The outputs from all heads are then concatenated and passed through a final learnable projection matrix:
MultiHead(Q, K, V) = Concat(head₁, ..., headₕ) × Wᴼ
Here, (h) is the number of attention heads, and (W^O) is the output projection matrix.
By allowing multiple attention heads to operate in parallel, Transformers can learn different patterns of relationships within the same sequence and combine them into a single representation.
In short: Each attention head learns its own way of gathering information, and the output projection combines the results into a unified representation.
Conclusion
Query, Key, and Value form the foundation of the attention mechanism in Transformers.
- Query determines what information a token is looking for.
- Key helps determine which tokens are relevant.
- Value carries the information that gets incorporated into the output.
Together, they enable self-attention to calculate relationships between tokens and build context-aware representations. Multi-Head Attention extends this idea by allowing multiple attention mechanisms to learn different patterns in parallel.
Understanding QKV is an important step toward understanding how Transformers process language and how modern large language models work under the hood.
Top comments (1)
Clear walkthrough, and the retrieval analogy fits well. A natural follow-up is the KV cache: at inference only the new token needs a fresh Query, while the Keys and Values of earlier tokens are stored and reused. That is why generation stays fast but memory grows with context length, and it shows concretely why Q is separate from K and V. Writing the shapes out in a tiny NumPy example is also a good way to check the formulas.