An Intuitive Explanation of Q, K, and V in Transformer Self-Attention

Hi, I’m Matsuken, a data scientist working on AI and machine learning at a Japanese technology company.

In this article, I’ll explain Self-Attention—one of the key mechanisms inside Transformers—with a particular focus on building an intuitive understanding of Q, K, and V.

Transformer-based generative models such as GPT compute Self-Attention as follows:¹

$$O = \mathrm{softmax}\left(\frac{QK^{\mathrm{T}}}{\sqrt{d_k}} + M\right)\cdot V$$

This operation lets each token build a context-aware representation using the tokens available up to its position, which in turn helps the model work with long-range context. Our goal here is to make the meaning of this equation easy to grasp intuitively.

This is an English version of my original Japanese article, which I originally wrote and published on Qiita.

Overview

Simplifying assumptions used in this article

• For clarity, we will treat each token as a whole word. In actual GPT-family models, text is split into tokens that do not necessarily correspond one-to-one with words. This simplification makes the intuition easier to follow.
• Q, K, and V are not the original word embeddings. They are sequences of vectors obtained by applying separate linear transformations to each token representation. For intuition, however, each row in the diagrams will be treated as the vector associated with the token at that position.

Creating the Query, Key, and Value from the Input

In Transformer Self-Attention, each token representation in X is transformed into three sequences of vectors: Query (Q), Key (K), and Value (V). An intuitive way to think about their roles is:

• Query: what information this position is looking for.
• Key: what kind of query this position responds to.
• Value: the actual information passed on to the output.

In a generative model such as GPT, a causal mask prevents a token from looking at future tokens. Each token therefore gathers the information it needs from itself and earlier tokens through a weighted combination.

As shown below, Q, K, and V are created by multiplying the input sequence X by separate learned weight matrices:

$$Q = XW_Q, \qquad K = XW_K, \qquad V = XW_V$$

Although these are transformed token representations rather than the words themselves, from here on we will refer to their rows by the corresponding words to keep the diagrams intuitive.

Representing Relationships Between Tokens with Q and K

The numerator in $\frac{QK^{\mathrm{T}}}{\sqrt{d_k}}$ computes a dot product between every Query vector and every Key vector, placing the results in a matrix. For this article, we can loosely interpret each dot product as the strength of the relationship between two token positions.

When both vectors are unit vectors, their dot product can be viewed as a measure of how closely their directions align. The vectors here are not unit vectors, but that geometric picture is still a useful starting point.

In Self-Attention, a dot product is best understood as a score expressing how strongly a token on the Query side wants to look at each token on the Key side. The score may sometimes behave like semantic similarity, but more generally it is a learned matching score that can also capture contextual, grammatical, and referential relationships.[^dk]

For simplicity, this article treats $d_k$ as the embedding dimension², and $\sqrt{d_k}$ in the denominator is a scaling factor³.

Focusing on “playing” as the Query, we compute its dot product with the Key vector at each token position and place the resulting scores in the row of $QK^T$ corresponding to “playing.” This row represents the relationships between “playing” and each token in the sentence.

Next comes:

$$\frac{QK^{\mathrm{T}}}{\sqrt{d_k}} + M$$

Here, M is the mask matrix. During next-token prediction, this causal mask prevents information from future positions from leaking into the current position. It adds $-\infty$ to entries above the main diagonal, so those entries become zero after softmax and are ignored.

We can now compute the attention matrix:

$$A = \mathrm{softmax}\left(\frac{QK^{\mathrm{T}}}{\sqrt{d_k}} + M\right)$$

Softmax is applied across each row so that its entries sum to 1. This converts the scores into weights for a weighted average. Positions masked with $-\infty$ receive an Attention Weight of zero. Applying this operation to every Query row produces the complete attention matrix.

Computing the Attention Output

Multiplying the attention matrix A by the Value matrix V gives the Self-Attention output:

$$O = AV$$

In other words, the Self-Attention output at a position is the sum of the Value vectors, each multiplied by its Attention Weight.

Consider the Query position for “green.” First, the model uses its Query vector and every available Key vector to compute Attention Weights. It then uses those weights to average the corresponding Value vectors. The result can be viewed as the vector for the “green” position enriched with the context that precedes it in the sentence.

Summary

This article developed an intuitive picture of the roles played by Q, K, and V in Transformer Self-Attention.

Starting from a token’s Query vector, Self-Attention compares it with the Key vectors at every visible position to obtain Attention Weights. It then uses those weights to compute a weighted average of the corresponding Value vectors the corresponding Value vectors, producing an output representation that incorporates the context relevant to that position.

In short: the Query asks, the Keys determine where to look, and the Values provide the information that is gathered.

---

1. For simplicity, this equation shows a single-head view and omits the final step used in Multi-Head Attention. ↩

2. In this article, the embedding dimension is treated as the dimension of the Query and Key vectors because Multi-Head Attention is omitted. With Multi-Head Attention, the model dimension is typically divided across multiple heads, so each head uses a smaller Query and Key dimension. ↩

3. The variance of the dot product grows roughly in proportion to $d_k$ . Without scaling, the values passed to softmax can become very large, making the distribution excessively sharp and training less stable. Dividing by $\sqrt{d_k}$ keeps the scale of these values under control. ↩