From GPT-2 to Kimi K3: How Language Models Learned to Manage Memory

Six architectures, one idea: from the KV cache to Kimi K3's hybrid of recurrent memory, global attention, and sparse experts, each a smarter way to manage a limited memory.

TL;DR: In 2019, GPT-2 (small) had 124 million parameters. In 2026, Kimi K3 has 2.8 trillion, about 22,580× more. Yet the leap was never only about size. A large part of the story is quieter: how models learned to remember, update, and retrieve information more efficiently. This post traces that one thread across six architectures, starting with the model that makes the idea easiest to see, GPT-2.

This piece is inspired by Ali Taha's excellent deep-dive "22,580: GPT-2 to Kimi K3, explained" on the Baseten blog (the six-stage structure and the 22,580 comparison follow it closely), and by the Kimi K3 paper, "Kimi K3: Open Frontier Intelligence".

---

1. GPT-2: the standard Transformer

GPT-2 is a decoder-only Transformer, the architecture from 2017's Attention Is All You Need. "Decoder-only" means it does exactly one job: predict the next token from the tokens before it. There's no separate "encoder" stage reading a source sentence the way a translation model would; GPT-2 simply continues a sequence. When the model processes any position, it is only allowed to look at that position and the ones before it, never the future.

GPT-2 works autoregressively: it predicts one token, appends it to the sequence, and then uses the whole updated sequence to predict the next one, and repeats. Every prediction rests on everything that came before it: the original prompt plus each token generated so far.

Walk through the prompt "The cat sat on the". To choose the next word, the model processes those five tokens and, from the representation at the last one (the), predicts a likely continuation, say mat. That token is appended, the model runs again (now able to attend to all six tokens), and so on. Two things are worth noticing. First, GPT-2 doesn't squeeze the past into a single running summary the way older recurrent networks (RNNs) did; it keeps a separate, directly addressable representation for every token in the window. Second, it handles the prompt in one parallel pass ("prefill"), then generates one token at a time ("decoding").

The mechanism that lets a token "read" the others is called attention. At each position, the model scores how relevant each earlier token is (and the current position itself), then blends their information in proportion to those scores, so relevant tokens contribute more and irrelevant ones less. It's the representation at the current position that carries this blended look-back forward to predict the next token. This weighted look-back is the single most important operation in a Transformer, and it is the one the later architectures work hardest to make cheaper.

Figure 1. Predicting the word after "The cat sat on the." Attention runs from the **last input token* (the), which may look at itself and all earlier tokens; the representation it produces is what predicts the next token, mat. Note that mat has no representation of its own until it is generated and fed back in.*

The hidden inefficiency

Attention at a given position needs a small bundle of information from itself and every earlier token. (Exactly what that bundle contains comes up in a moment.) If the model rebuilt that bundle for every past token from scratch at each step, it would redo almost the same work again and again, and the redundant work would pile up faster the longer the text became.

Figure 2. Each colored cell is one token's key/value being computed in a given step. Only one is genuinely new per step (green); the rest are **recomputations* of key/value projections already done (red). Rebuilding them from scratch costs about 1 + 2 + 3 + 4 = 10 projection computations just to emit 4 tokens, and grows like n². (This counts key/value projection work, not the attention comparison itself, which comes up below.)*

The fix: the KV cache

The fix starts with naming that "bundle of information." Inside every attention layer, each token's hidden representation is passed through three small learned transformations that produce three vectors:

• Q, the query: "what am I looking for?"
• K, the key: "how should I be found by others?"
• V, the value: "what information do I pass on if I'm selected?"

Attention compares the current token's Q against the K of every earlier token (and its own) to decide how much to draw from each, then mixes their Vs accordingly.

K and V are called intermediate representations because they aren't the input (the words) and they aren't the final output (the prediction). They're vectors computed inside the attention layer, on the way from one to the other. And this is what the whole fix hinges on: a past token's K and V never change when a new token arrives. "The cat sat" produces the same K and V for cat whether the next word turns out to be on, quietly, or down. Why? Attention is causal: a token's internal representation is built only from itself and the tokens before it (the mask blocks any peeking ahead), and K and V are computed straight from that representation. So a token that arrives later has no path to reach back and change them.

Because those vectors are stable, there's no need to recompute them. Each token's K and V are computed once and stored; that store is the KV cache. Every new step then computes a fresh Q, K, and V for just the current token, appends its K and V to the cache, and reuses everything already saved, which eliminates the red recomputation triangle in Figure 2.

While the KV cache saves the model from rebuilding past keys and values (the waste in Figure 2), but it doesn't save it from reading them. To produce each new token, the model still has to scan the entire cache: compare the new token against every key stored so far and mix in every value.

And that cache only grows. Writing the 10th token means scanning about 10 entries; the 1,000th token scans about 1,000; the millionth scans about a million. So every new token costs a little more than the last, and generation keeps slowing as the context gets longer. Add all that scanning up over a long passage and the total work grows with the square of the length: double the text and you do roughly four times the attention work (this is attention's well-known quadratic cost). Caching removes the redundant rebuilding; attention's built-in "look back over everything so far" cost stays, and it's a big reason long context is expensive.

Figure 3. Left: every token is projected into Q, K, V inside the attention layer. Right: the K and V vectors are cached and reused, so the cache gains one entry per token.

The catch that shapes everything after GPT-2

The KV cache trades compute for memory, and that trade has a cost: the cache grows with the length of the text. Every new token permanently adds its K and V, so the memory it occupies is roughly:

cache size ≈ (number of tokens) × (size of one token's K,V) × (number of layers).

For a short chat, that's negligible. But that cache typically has to sit in (or stream through) fast accelerator (GPU) memory while the model generates, and that memory is finite and expensive. Now imagine scaling GPT-2's full-attention design to a book-length document, or a million-token codebase: the KV cache alone can swell to many gigabytes, and cache capacity and memory bandwidth can become as much of a bottleneck as the attention math. (GPT-2's own context window was just 1,024 tokens, so this is a thought experiment about its design) Modern long-context models attack this head-on, and Kimi K3 mostly replaces the growing KV cache with a fixed-size recurrent state in the bulk of its layers, keeping compressed global attention in a periodic minority of them.

That tension, direct access to every past token but a cache that grows linearly with context length, is the thread the next architectures pull on. The first attempt, linear attention, gives up some of that direct addressability in exchange for a memory that doesn't grow at all.

2. Linear attention: compress the past into a fixed-size state

Linear attention starts from a blunt question: instead of storing a key and value for every token, what if the model kept a single fixed-size memory and folded each new token into it?

That's the whole idea. Where full attention keeps a KV cache that grows by one entry per token, linear attention maintains one fixed-size state: a memory matrix S that summarizes everything seen so far. Reading and writing never get more expensive, no matter how long the text is.

Figure 4. Full attention stores a key/value entry per token, so memory grows with length. Linear attention folds every token into one fixed-size state, so memory stays constant.

How it works, without heavy math

Recall standard attention: for a new token, you compare its query against every stored key, turn those comparisons into weights, and take the weighted average of the values. That per-query loop over all the keys is the expensive part, and the reason you must keep every key and value around if you want exact attention.

So why can't we just pre-add the past into one running total and reuse it for each new query? It comes down to how the score is computed. In standard attention, the score for a query-key pair is exp(query · key): the query and key are multiplied together and then run through an exponential. That fuses them into a single number, and you can't compute any useful piece of it until you know both the query and the key. So exact softmax attention can't be boiled down to one fixed-size summary that works for every possible future query; each new query has to be re-scored against every stored key.

Linear attention changes the scoring rule to remove that fusion. Instead of the exponential, it applies the same fixed transformation separately to the query and the key, then compares the two transformed vectors with a plain dot product. The point of that transformation is to make the score separable: it splits into a query-only part and a key-only part, instead of one exponential that welds them together.

That separation is the entire trick. Because the query-only part is the same for every past token in a given read, the model doesn't have to pair it with each key one at a time. It can fold all the transformed keys and values into a single running summary as they stream in, and then apply the transformed query to that summary once. In other words: each token adds its transformed key and value to the summary a single time, and a later query simply reads that summary, while a second small running total keeps the result normalized as a weighted average.

Two things fall out of this:

• Constant memory (per head and layer, with respect to sequence length). S and z have fixed sizes, set by the model's dimensions, not by how many tokens you've seen. A million-token document uses the same-sized summary as a ten-token one.
• It's a recurrence. Each step updates the summary from only the new token and the previous summary, exactly how a recurrent neural network (RNN) carries a hidden state, the very thing Transformers moved away from in Block 1. Linear attention quietly brings that fixed-size memory back. (This equivalence is the point of the aptly named paper Transformers are RNNs.) One nuance: during training or prompt prefill the whole sequence is known, so the summary can still be built in parallel, and linear attention doesn't give up the Transformer's parallel training. It's step-by-step generation that runs as a recurrence.

That fixed state also brings back an RNN-like capacity limit: however long the sequence gets, all of its history has to share the same finite memory. That sharing is where interference creeps in.

For the curious: the exact formulas. Write φ(x) for that fixed transformation (often called a feature map), and let each sum run over the tokens seen up to the current position. Linear attention is just a weighted average that uses the separable score φ(q) · φ(k), and regrouping the sum is what turns it into a reusable summary:

direct form:
  output = Σᵢ [φ(q) · φ(kᵢ)] vᵢ  /  Σᵢ [φ(q) · φ(kᵢ)]

same calculation, regrouped:
  S = Σᵢ φ(kᵢ) vᵢᵀ
  z = Σᵢ φ(kᵢ)
  output = [φ(q) · S]  /  [φ(q) · z]

The point of regrouping: S and z don't depend on the query, so they're built once and reused. This is a different similarity rule, and it doesn't reproduce softmax exactly. The paper's feature map is φ(x) = ELU(x) + 1, whose positive outputs keep every weight ≥ 0 and (after dividing by φ(q) · z) summing to 1, the same two properties softmax gave us.

Figure 5. Linear attention carries a fixed-size state (S, z) from step to step. Each token updates S (green), and each output is read from the current S (blue), the same update-a-hidden-state pattern as an RNN.

The catch: one small table, many memories

Constant memory isn't free. Full attention keeps a separate row for each token, so a query can pick out exactly the ones it needs. Linear attention adds every key-value pair into the same fixed-size table S. If two keys write into overlapping parts of that table, their values mix, so a later query can pull back a blend instead of one clean value. That's interference. And because the basic update only ever adds, with no way to overwrite or forget, the crowding only grows as more pairs share the same fixed space. (This is exactly the capacity problem analyzed in Linear Transformers Are Secretly Fast Weight Programmers, which motivates the next step.)

So the trade from Block 1 flips: full attention gave direct addressability at ever-growing cost; linear attention gives constant cost but crowded, interfering memory. The next three steps attack that crowding. The first asks a sharper question: when we fold a new token into S, are we writing it in the right way? That is DeltaNet.

3. DeltaNet: write the correction, not just the sum

Linear attention's flaw was that it only ever adds. Every token writes its key→value association into S on top of whatever is already there, so associations pile up and interfere. DeltaNet fixes the write: before storing a new association, it checks what the memory already holds for that key and writes only the difference.

That difference is the "delta," and the idea is a classic: the delta rule (Widrow-Hoff, 1960), the same error-correction step at the heart of early machine learning. Applied to our fixed-size memory, it turns a blind add into a deliberate read-modify-write.

Figure 6. Write the same key twice. Plain linear attention keeps both associations, so a later read returns a mixture (interference). DeltaNet instead reads what's stored, computes the correction, and writes only that, moving the entry toward the new value.

One useful picture before the mechanics. Think of S as a compressed, fuzzy dictionary: a key is a learned numeric address, a value is the information written at that address, and a query is a lookup request. Unlike a real dictionary, the addresses aren't isolated slots: nearby keys overlap, so a lookup can return a blend. Two kinds of reads happen against it: the layer's actual output is looked up with a query q, while the delta update below does its own read with the key k, to check what's already stored where it's about to write.

The delta rule in three steps

DeltaNet keeps the same fixed-size matrix S, but changes how a new key-value association is written. First, one setup detail: the modern DeltaNet formulation divides each key by its own geometric length, so every key ends up the same unit size: same direction, standardized magnitude. That common size is what lets the write strength β behave as a clean 0-to-1 dial. (This standardizing step is called L2 normalization, dividing a vector by its length, and is unrelated to L2 regularization.) When token t arrives with key k and value v:

1. Read the memory at this key: v̂ = k · S. Here k is a vector and S is a matrix, so the result is another vector: the value currently stored at that address.
2. Compare it to the new target: Δ = v − v̂, the gap between the new target value and what this key currently retrieves.
3. Write only the correction: S ← S + βₜ · (k ⊗ Δ). The symbol ⊗ is an outer product: it combines the key and correction vectors into a small matrix with the same shape as S, so it can be added straight into the memory.

The write strength βₜ (between 0 and 1) isn't a fixed knob; the model learns it from the current token (βₜ = sigmoid(Wβ · xₜ)), so it can barely touch memory, partially update, or strongly replace. After the write, a read at the same key returns (1−βₜ)·v̂ + βₜ·v: it moves toward the new value, landing exactly on it when βₜ = 1.

What it fixes, and what it doesn't

By correcting instead of piling on, DeltaNet is a much better writer: writing a key again moves what it retrieves toward the new value rather than stacking a second association on top.

But DeltaNet still lacks a separate forgetting control. It can correct what the current key retrieves, but it has no independent way to fade the rest of the state when information becomes stale. Old associations can therefore persist and keep competing for the fixed-size memory. Gated DeltaNet adds a learned decay gate that deliberately fades prior state before the next write.

4. Gated DeltaNet: give the model a forget button

Gated DeltaNet supplies the control DeltaNet was missing: a learned forget gate. To see what it fades (and to follow the upgrade in the next section), it helps to picture the memory concretely.

A key is a vector with dₖ components, called key channels. The memory S is a matrix with one row per key channel (dₖ rows in all). In the read k · S, component kᵢ weights row i, and the weighted rows combine into the retrieved value.

Gated DeltaNet's gate is a single retention factor αₜ between 0 and 1, produced fresh for each token and each head. Before the corrective delta write, it multiplies that head's entire state by αₜ (every row by the same number), so a value near 1 keeps almost all of the memory and a smaller one fades more of it. Then it runs the same read→compare→write step from Block 3.

Let S̄ be the state after fading. Gated DeltaNet reads and corrects this faded state:

S̄ = αₜ · S                     # fade the whole state
v̂ = k · S̄                      # read the faded state
S  = S̄ + βₜ · (k ⊗ (v − v̂))    # corrective delta write

The order matters (fade, then read, then correct), so the comparison is made against the faded state. Because αₜ is learned per token, the gate can output a factor near 1 to preserve the state, or a smaller one (for instance on inputs that mark a context shift) to clear space. It gives DeltaNet the dial it was missing: "how much of the past should I keep?"

Figure 7. Gated DeltaNet inserts one step before the delta write: scale the entire memory by a learned forget gate αₜ. αₜ ≈ 1 applies no global decay; a smaller αₜ forgets aggressively.

Gated DeltaNet combines DeltaNet's corrective write with the same kind of input-dependent scalar state decay used in Mamba-2. Yang, Kautz, and Hatamizadeh formalized the combination in Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025): a memory that can both correct what a key retrieves and fade the rest on purpose.

The catch: one dial for everything

Gated DeltaNet is a real step up: the model can now edit and forget. But its gate is a single scalar per head (a head is one of the parallel attention sub-units): it fades that head's entire state by one number, so every channel, every row of S, decays at the same rate. That is the limit: one scalar can't keep some rows while quickly fading others within the same head.

The fix is to give each channel, each row of S, its own retention factor. That is the KDA idea at the heart of Kimi Linear.

5. Kimi Linear: a separate retention factor for every channel

Gated DeltaNet fades a head's whole state with one number, every row of S by the same factor. Kimi Linear makes that control per-channel: it uses a separate retention factor for each row of S, one per channel. So each channel can fade at its own speed.

On each step, it multiplies row i of the previous state by that channel's retention factor αₜ,ᵢ, then applies the same corrective delta write using the current key and value.

That small change is powerful. Each αₜ,ᵢ is the fraction of channel i kept at this step: near 1 means a slow fade (a long effective memory), small means a fast fade (quick-cycling scratch space), all learned, all per token. Instead of one retention knob per head, the model has a whole bank of them, and can hold some learned dimensions of the state far longer than others, something a single scalar can't express.

Figure 8. Gated DeltaNet applies one decay to a head's whole state (left). Kimi Linear promotes that decay to a per-key-channel vector (right), so each channel fades at its own learned rate (α is the fraction of a channel kept per step).

This per-channel gated delta update is the KDA module (Kimi Delta Attention), introduced in the Kimi Linear report. It keeps a fixed-size recurrent state that writes correctively and forgets channel by channel, and never grows with sequence length.

Kimi Linear is the larger architecture around it: mostly KDA layers, interleaved with occasional global-attention layers, so cheap recurrent memory is punctuated by full attention that can reach across the whole context. Kimi K3 inherits this hybrid and modifies KDA's implementation: it bounds the decay parameterization so low-precision (BF16) computation stays stable, and swaps Kimi Linear's low-rank output gate for an input-dependent full-rank one.

Where this leaves us

Trace the path from Block 2:

• Linear attention gave a fixed-size memory, but it could only add, so associations interfered.
• DeltaNet made writes corrective: move what the current key retrieves toward the target instead of piling on (though similar keys can still interfere).
• Gated DeltaNet added forgetting: one retention factor per head.
• KDA (from Kimi Linear) made forgetting fine-grained: a separate retention factor per key channel.

That is a genuinely capable fixed-size memory. But it is still a summary: everything lives compressed inside S. For most tokens that's exactly right: cheap, constant in size, and recency-aware. Yet some tasks need direct, token-level access to something far back in the context, which is what a full-attention layer provides. Kimi Linear already combines the two: most layers use KDA to maintain cheap, fixed-size running memory, while periodic global-attention layers can attend directly to earlier tokens across the context. Kimi K3 inherits that hybrid and scales it into a 2.8-trillion-parameter frontier model, adding sparse experts and cross-layer retrieval along the way.

6. Kimi K3: put each mechanism where it matters

We finally have the pieces. Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 104 billion parameters active per token and a context window of up to one million tokens. But the interesting part isn't the headline number; it's how K3 spends that capacity. K3 doesn't pick one memory mechanism. It combines four architectural mechanisms, each shaping information flow along a different axis of the network.

A hybrid backbone: cheap memory, occasional full attention

Most of K3's layers are KDA, the fixed-size recurrent memory from Block 5, a recurrence whose per-token decode cost doesn't grow with context length. But every fourth layer uses Gated Multi-head Latent Attention (Gated MLA): global attention that can attend directly to any earlier token in the causal context. The pattern is three KDA layers to one Gated MLA layer, repeated through the backbone (mostly cheap recurrent memory, punctuated by periodic full attention), with one extra Gated MLA layer closing the stack, so the final layer is always global. Stable LatentMoE and Attention Residuals (AttnRes) add the width-side and depth-side mechanisms.

Figure 9. A Kimi K3 block interleaves three KDA layers with one Gated MLA layer (a 3:1 ratio), each paired with a Stable LatentMoE feed-forward network. The block repeats through the backbone, with one extra Gated MLA layer closing it. Attention Residuals (AttnRes) let each layer select from the token embedding and compact summaries formed from earlier layers.

Why mix them? KDA is cheap and recency-aware but compresses everything into a fixed state; MLA keeps a genuine, addressable view of the whole sequence but its cache grows with length. Interleaving them 3:1 buys most of the efficiency of recurrence with periodic doses of true global reach. Two details make the MLA layers earn their keep:

• Compressed KV (latent attention). Full attention's KV cache is the villain from Block 1. MLA, introduced in DeepSeek-V2, caches one small latent vector per past token (and reconstructs the content keys and values when attention runs), so its cache is far smaller than storing full per-head K/V, though it still grows linearly with context length.
• No positional encoding (NoPE). K3 applies no explicit positional encoding to MLA's queries or keys. Instead it relies on the interleaved KDA layers for position- and recency-aware mixing, while MLA handles unrestricted global content interaction.
• An output gate. A learned per-token gate independently modulates the channels read from global attention; that gate is the "Gated" in Gated MLA.

Sparse experts: 2.8T parameters, 104B active per token

Every K3 layer performs two different jobs. First, KDA or Gated MLA lets tokens gather information from the sequence. Stable LatentMoE then transforms each token's updated representation without moving information between token positions. It decides which specialized feed-forward networks, called experts, should process that token.

Each Stable LatentMoE layer has its own router and its own pool of experts. When a token reaches the layer, the router examines its current representation and selects 16 of 896 routed experts. Two shared experts also process every token. The selected experts' results and the shared experts' results are combined into one updated token representation, which continues to the next layer. That next layer has another router and makes a new decision, so the same token may use different experts at different depths. The model is therefore not splitting the sequence into permanent streams; it is choosing a temporary team of experts at each layer.

This is only a high-level view of Stable LatentMoE; if readers are interested, I can unpack its routing, compressed expert representation, and stability mechanisms in a dedicated follow-up.

Retrieval across depth: Attention Residuals

A normal residual connection carries earlier information forward by adding every layer's update to one running representation. Nothing is deliberatelydiscarded, but information from different depths becomes blended together, so a later layer cannot directly choose how much to reuse from each earlier stage.

Attention Residuals (AttnRes) preserve several summaries from different depths. In Kimi K3, a layer can draw from the token's original embedding, summaries formed by earlier groups of layers, and the running summary from earlier layers in its current group. It assigns each available source a weight and combines them into its next input. This is attention across depth: instead of reading different token positions, the model chooses among different stages of processing for the same token.

Figure 10. Kimi K3's four mechanisms and the axis each one scales: memory along the sequence (KDA), global retrieval along the sequence (Gated MLA), retrieval along depth (AttnRes), and routing along width (Stable LatentMoE).

The point

Line the four up and the design philosophy is clear. Kimi K3 adds structure along every axis of the network (sequence for KDA memory and MLA retrieval, depth for AttnRes, and width for MoE routing) rather than simply making one big dense stack larger. It routes information and computation differently along sequence, depth, and width, placing capacity where the model can use it to remember, retrieve, route, and forget.

---

Conclusion: architecture as memory management

Step back over the whole path (GPT-2, linear attention, DeltaNet, Gated DeltaNet, Kimi Linear, Kimi K3) and one theme runs through all of it. The thread traced here was never only about making models bigger. It was also about giving them a smarter way to manage a limited memory:

• GPT-2 kept every token in its context window, and paid with a cache that grows with length.
• Linear attention fixed the size, but blurred the memory together.
• DeltaNet learned to edit it; Gated DeltaNet learned to forget; Kimi Linear learned to forget selectively.
• Kimi K3 combined that fast recurrent memory with periodic direct token-level retrieval, sparse expert routing, and retrieval across depth.

Any memory with fixed capacity eventually has to decide what to keep and what to throw away. Gating, decay, routing, and attention are simply the tools a model uses to make that decision. Seen that way, the jump from 124 million parameters to 2.8 trillion isn't really 22,580× more of the same. It's 22,580× the rounded total parameter count, spent far more wisely.

---

References

1. Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762
2. Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI
3. Katharopoulos et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. arXiv:2006.16236
4. Schlag, Irie, Schmidhuber (2021). Linear Transformers Are Secretly Fast Weight Programmers. arXiv:2102.11174
5. Yang et al. (2024). Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484
6. Yang, Kautz, Hatamizadeh (ICLR 2025). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464
7. Dao, Gu (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2). arXiv:2405.21060
8. Widrow, Hoff (1960). Adaptive Switching Circuits (the delta rule). WESCON Convention Record.
9. DeepSeek-AI (2024). DeepSeek-V2 (Multi-head Latent Attention). arXiv:2405.04434
10. Kimi Team (2025). Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692
11. Kimi Team (2026). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653
12. Ali Taha. 22,580: GPT-2 to Kimi K3, explained. Baseten blog