DEV Community: matsuken92

Understanding Transformer Decoding with a KV Cache

matsuken92 — Tue, 18 Aug 2026 11:17:30 +0000

Hi, I'm Matsuken, a data scientist at a Japanese technology company.

In this article, I'll explain how decoding works in a Transformer that uses a key-value (KV) cache.

This is an English version of my original Japanese article, which I originally wrote and published on Qiita.

Transformer inference consists of two stages:

Prefill: Process all input tokens—the prompt—in parallel. During this stage, the model computes the keys ( $K$ ) and values ( $V$ ) and stores them in a cache.
Decode: Generate text one token at a time. At each step, the model reads and reuses the previously computed keys and values, then appends one new row—the key and value for the latest token—to the cache.

For an explanation of the first stage, prefill, see my previous article. The stored $K$ and $V$ tensors produced by this process are collectively called the KV cache.

Given an input $X$ , an attention layer computes the queries, keys, and values as follows:

Q K V = X W_{Q} = X W_{K} = X W_{V}

It then produces the output $Y$ :

Y = softmax (\frac{Q K ^{T}}{d} + M) V

The process is illustrated below.¹

Here, $n$ is the number of tokens and $d$ is the token representation dimension.

In a decoder-only Transformer, this computation takes place in every Multi-Head Attention block: each block receives $X$ and produces $Y$ .²

Prefill

During the initial pass, the input $X$ normally contains multiple tokens, so it is an $n \times d$ matrix. This is the prefill stage covered in my previous article.

The next stage—decode—is where the KV cache comes into play.

Decode

Prefill processes the prompt and determines the first generated token. In each subsequent decode step, the model receives the token generated in the previous step and autoregressively predicts the next token.

Conceptually, we could append every generated token to the original input sequence and process the entire sequence again. In practice, this would repeatedly perform the same computations. Instead, the model computes only the new query, key, and value vectors for the latest token. It reuses the keys and values for all previous tokens from the KV cache. This reuse is the purpose of the KV cache.

Suppose the prompt is:

[BOS] two kids are playing in a swimming pool with a green colored crocodile.

and the first generated token is “Then.”

Because $n$ tokens have already been processed, the model uses the representation of token $n + 1$ to compute three new vectors: $q_{n + 1}$ , $k_{n + 1}$ , and $v_{n + 1}$ .

Next, the model updates $K$ and $V$ . It reuses the entries through position $n$ from the cache and appends $k_{n + 1}$ and $v_{n + 1}$ , respectively.

The new query $q_{n + 1}$ is multiplied by the transpose of the updated key matrix, which consists of the cached keys followed by $k_{n + 1}$ . This produces the attention-score vector $s_{n + 1}$ .

The model applies softmax to $s_{n + 1}$ and multiplies the resulting attention weights by the updated value matrix—the cached values followed by $v_{n + 1}$ . The result is $y_{n + 1}$ .

We have now obtained the $y_{n + 1}$ required by this layer.

Although the explanation above follows the operations step by step, we can also view the computation as part of a conceptual full attention matrix. It has three regions: entries that were computed in previous steps, future positions excluded by the causal mask, and the newly computed row for the current step.

The complete decode-step computation can be summarized as follows:

You might wonder whether $y_{n + 1}$ alone is enough, or whether the model also needs $y_{0}, \dots, y_{n}$ . Recall that this step required only $x_{n + 1}$ because the earlier keys and values were already cached. Therefore, computing only $y_{n + 1}$ is sufficient.

As long as $K$ and $V$ are cached, each subsequent step needs only the representation of the newest token to produce the next-token output. That is what makes decoding with a KV cache so efficient.

The model maintains a separate KV cache for every Transformer layer. If the model has $N$ layers, it therefore has $N$ cache pairs:

(K_{cache (1)}, V_{cache (1)}), \dots, (K_{cache (N)}, V_{cache (N)})

This efficiency comes with a trade-off: as the number of tokens—the context-window length—increases, the memory required to store the key and value caches also grows. Several techniques address this issue, but I'll save those for another article.

To keep the diagrams simple, they omit division by the scale factor $d$ . ↩
To focus on the KV-cache mechanism, the rest of this article represents Multi-Head Attention as a single attention head. ↩

An Intuitive Explanation of Q, K, and V in Transformer Self-Attention

matsuken92 — Mon, 17 Aug 2026 10:10:26 +0000

Hi, I’m Matsuken, a data scientist working on AI and machine learning at a Japanese technology company.

In this article, I’ll explain Self-Attention—one of the key mechanisms inside Transformers—with a particular focus on building an intuitive understanding of Q, K, and V.

Transformer-based generative models such as GPT compute Self-Attention as follows:¹

O = softmax (\frac{Q K ^{T}}{d _{k}} + M) \cdot V

This operation lets each token build a context-aware representation using the tokens available up to its position, which in turn helps the model work with long-range context. Our goal here is to make the meaning of this equation easy to grasp intuitively.

This is an English version of my original Japanese article, which I originally wrote and published on Qiita.

Overview

Simplifying assumptions used in this article

For clarity, we will treat each token as a whole word. In actual GPT-family models, text is split into tokens that do not necessarily correspond one-to-one with words. This simplification makes the intuition easier to follow.
Q, K, and V are not the original word embeddings. They are sequences of vectors obtained by applying separate linear transformations to each token representation. For intuition, however, each row in the diagrams will be treated as the vector associated with the token at that position.

Creating the Query, Key, and Value from the Input

In Transformer Self-Attention, each token representation in X is transformed into three sequences of vectors: Query (Q), Key (K), and Value (V). An intuitive way to think about their roles is:

Query: what information this position is looking for.
Key: what kind of query this position responds to.
Value: the actual information passed on to the output.

In a generative model such as GPT, a causal mask prevents a token from looking at future tokens. Each token therefore gathers the information it needs from itself and earlier tokens through a weighted combination.

As shown below, Q, K, and V are created by multiplying the input sequence X by separate learned weight matrices:

Q = X W_{Q}, K = X W_{K}, V = X W_{V}

Although these are transformed token representations rather than the words themselves, from here on we will refer to their rows by the corresponding words to keep the diagrams intuitive.

Representing Relationships Between Tokens with Q and K

The numerator in $\frac{Q K ^{T}}{d _{k}}$ computes a dot product between every Query vector and every Key vector, placing the results in a matrix. For this article, we can loosely interpret each dot product as the strength of the relationship between two token positions.

When both vectors are unit vectors, their dot product can be viewed as a measure of how closely their directions align. The vectors here are not unit vectors, but that geometric picture is still a useful starting point.

In Self-Attention, a dot product is best understood as a score expressing how strongly a token on the Query side wants to look at each token on the Key side. The score may sometimes behave like semantic similarity, but more generally it is a learned matching score that can also capture contextual, grammatical, and referential relationships.[^dk]

For simplicity, this article treats $d_{k}$ as the embedding dimension², and $d_{k}$ in the denominator is a scaling factor³.

Focusing on “playing” as the Query, we compute its dot product with the Key vector at each token position and place the resulting scores in the row of $Q K^{T}$ corresponding to “playing.” This row represents the relationships between “playing” and each token in the sentence.

Next comes:

\frac{Q K ^{T}}{d _{k}} + M

Here, M is the mask matrix. During next-token prediction, this causal mask prevents information from future positions from leaking into the current position. It adds $- \infty$ to entries above the main diagonal, so those entries become zero after softmax and are ignored.

We can now compute the attention matrix:

A = softmax (\frac{Q K ^{T}}{d _{k}} + M)

Softmax is applied across each row so that its entries sum to 1. This converts the scores into weights for a weighted average. Positions masked with $- \infty$ receive an Attention Weight of zero. Applying this operation to every Query row produces the complete attention matrix.

Computing the Attention Output

Multiplying the attention matrix A by the Value matrix V gives the Self-Attention output:

O = A V

In other words, the Self-Attention output at a position is the sum of the Value vectors, each multiplied by its Attention Weight.

Consider the Query position for “green.” First, the model uses its Query vector and every available Key vector to compute Attention Weights. It then uses those weights to average the corresponding Value vectors. The result can be viewed as the vector for the “green” position enriched with the context that precedes it in the sentence.

Summary

This article developed an intuitive picture of the roles played by Q, K, and V in Transformer Self-Attention.

Starting from a token’s Query vector, Self-Attention compares it with the Key vectors at every visible position to obtain Attention Weights. It then uses those weights to compute a weighted average of the corresponding Value vectors the corresponding Value vectors, producing an output representation that incorporates the context relevant to that position.

In short: the Query asks, the Keys determine where to look, and the Values provide the information that is gathered.

For simplicity, this equation shows a single-head view and omits the final step used in Multi-Head Attention. ↩
In this article, the embedding dimension is treated as the dimension of the Query and Key vectors because Multi-Head Attention is omitted. With Multi-Head Attention, the model dimension is typically divided across multiple heads, so each head uses a smaller Query and Key dimension. ↩
The variance of the dot product grows roughly in proportion to $d_{k}$ . Without scaling, the values passed to softmax can become very large, making the distribution excessively sharp and training less stable. Dividing by $d_{k}$ keeps the scale of these values under control. ↩