amananandrai

Posted on Nov 10

From Attention to 3-Bit Caches: How LLMs Learned to Remember Less (MHA, GQA, MLA, MoE, Quantisation Explained)

#ai #machinelearning #llm #deeplearning

In 2017, eight researchers published a paper with the cockiest title in computer science: Attention Is All You Need. They were right, with one small footnote that took the rest of the industry nine years to work through:

Attention is all you need. Also 16 GB of memory per conversation, a warehouse of GPUs, and a power plant.

This post is the story of that footnote. We'll start from zero (what attention even is), then follow the trail of clever hacks that turned a research toy into the chatbots you use every day: KV caches, grouped and latent attention, FlashAttention, mixture of experts, ZeRO, distillation and quantisation.

No PhD needed. If you can follow a restaurant analogy, you can follow this. There is a little maths, clearly marked, and you can skip every formula without losing the plot.

What's inside

Attention in five minutes
The KV cache: a waiter's notepad that ate the restaurant
The evolution of attention: MHA → MQA → GQA → MLA → LRKV
Taming the KV cache: three families of tricks
Mixture of experts: stop asking the dentist about your taxes
Scaling training: parallelism, ZeRO and FSDP
Distillation: a small model that copied the topper's notes
Quantisation: FP8, FP4 and TurboQuant
So how did all this make GPT-style models better?
The timeline

1. Attention in five minutes

The cocktail party

You're at a noisy party. Twenty conversations are happening at once, and yet when someone across the room says your name, your brain snaps to it. You didn't listen to everyone equally. You gave most of your attention to the few voices that mattered.

That is the whole idea. When a language model reads a sentence, every word looks at every other word and decides how much each one matters to it.

Take the sentence: "The cat sat on the mat because it was tired."

What does "it" mean? You know it's the cat (mats rarely get tired). A model works this out by letting "it" pay most of its attention to "cat".

Query, Key, Value: attention as a search engine

How does a word decide who matters? Each word carries three little vectors (lists of numbers):

Query: "here's what I'm looking for"
Key: "here's what I am"
Value: "here's what I'll tell you if you pick me"

Think of a library. Your query is the question in your head. Every book has a key (the label on the spine) and a value (the contents). You compare your question with every spine, then read mostly from the books that match best.

Remember keys and values. They are the main characters of the next three sections, and also the reason your GPU is sweating.

Multi-head attention: a panel of specialists

One round of attention only catches one kind of relationship. So transformers run many in parallel, called heads. One head might track grammar, another tracks who "it" refers to, another notices that we're inside a quotation. Each head has its own queries, keys and values. That's multi-head attention (MHA).

It's a panel interview: one interviewer checks your coding, one checks your communication, one just stares at you to see if you blink.

The transformer in one picture

Stack attention with a small "thinking" network (the feed-forward layer), repeat the pair dozens of times, and you have a transformer.

GPT-style models do one job: guess the next token. Then they add that token to the text and guess again. And again. "The" → "The cat" → "The cat sat". Everything a chatbot writes is produced one token at a time by this loop.

That loop is where the trouble starts.

2. The KV cache: a waiter's notepad that ate the restaurant

Imagine a waiter with no notepad. Every time your table adds a dish, he walks back and asks everyone to repeat the entire order from the beginning. By dessert he has heard "one butter chicken" eleven times.

A transformer without a cache behaves exactly like that. To generate token 1,000, it would recompute the keys and values for the previous 999 tokens, even though they haven't changed.

The fix is obvious: write it down. The KV cache stores the keys and values of every token already processed, in every layer. A new token only computes its own K and V, then looks up everyone else's.

Wonderful. Except the notepad keeps growing:

\text{KV bytes} = 2 \times \text{layers} \times \text{KV heads} \times d_{head} \times \text{tokens} \times \text{batch} \times \text{bytes per value}

Plug in a medium-sized example: 32 layers, 32 heads, 128 numbers per head, 16-bit values and a 32,000-token chat. You get 16 GB for a single conversation. Now serve a thousand users.

This is the plot twist of modern LLMs: for long contexts, the limit is not how fast the GPU can multiply (FLOPs). It's how much the GPU can remember. Almost every trick in the rest of this post is a way to shrink, share or sidestep that notepad.

3. The evolution of attention: MHA → MQA → GQA → MLA → LRKV

Here is the whole family in one animation. Watch the memory meter on the right.

Let's meet them using one running analogy: a class of 32 students (heads) taking notes (keys and values) in a lecture.

MHA: everyone writes their own notes

Every student writes complete notes. Great diversity of thought, 32 notebooks to store. This is the original 2017 design, used by GPT-2 and GPT-3.

MQA (2019): one notebook, photocopied

Noam Shazeer (one of the original eight) asked: what if all heads shared a single set of keys and values, and only the queries stayed separate? The cache shrinks by 32×. The catch: with one notebook, the students start thinking alike, and quality dips.

GQA (2023): study groups

The compromise that won. Split the heads into a few groups; each group shares one notebook. With 8 groups instead of 32 notebooks you save 4× memory and lose almost nothing. Llama 2 (70B), Llama 3 and Mistral all use it. If you've run an open model on your laptop, you've used GQA.

MLA (2024): flat-pack furniture

DeepSeek-V2 took a different road. Instead of sharing notebooks, compress them. Store one small "latent" vector per token, and rebuild the full keys and values only when needed. It's IKEA logic: ship the wardrobe flat, assemble at home.

In the DeepSeek-V2 setup, that's 576 numbers per token per layer. The paper notes this is as small as GQA with only about 2.25 groups, while keeping quality on par with full MHA.

The RoPE headache (and the fix)

There's a neat trick hiding in MLA. At inference time you don't even have to rebuild the keys: the "expand" matrix can be folded into the query side once, like pre-multiplying a recipe. Keys stay compressed forever.

Then RoPE walks in. RoPE (rotary position embedding) is how most modern models know word order: it rotates each key by an angle that depends on the token's position. Token 5 gets one rotation, token 500 another.

And a position-dependent rotation can't be folded into one fixed matrix. It's like trying to pre-print a single shipping label for parcels going to different cities.

DeepSeek's fix is decoupled RoPE: split the key into two parts. A big content part with no position in it (still foldable), and a small separate part that carries the rotation and is cached next to the latent.

For the formula fans:

c^{KV}t=W^{DKV}h_t,\quad k^{C}{t}=W^{UK}c^{KV}t,\quad v^{C}{t}=W^{UV}c^{KV}t,\quad k^{R}_t=\mathrm{RoPE}(W^{KR}h_t),\quad k{t,i}=\big[k^{C}_{t,i};\,k^{R}_t\big]

In words: squeeze, cache, expand; and keep position in its own small pocket.

LRKV (2026): one textbook, personal sticky notes

The newest idea, Low-Rank Key-Value attention, comes from a team at Intercom. Their observation: heads are mostly redundant, but not entirely. So give the whole class one shared full-quality textbook, and let each student add a thin stack of personal sticky notes (a low-rank residual).

Turn the sticky-note thickness (the rank, r) down to zero and you get MQA. Turn it all the way up and you're back at MHA. The authors report that a 2.5-billion-parameter model beat standard attention while using roughly half the KV cache.

The scoreboard

Method	Cached per token, per layer	Trade-off
MHA	2 × H × d_head	Full head diversity, largest cache
MQA	2 × d_head	Smallest simple cache, heads lose diversity
GQA	2 × G × d_head (G groups)	The common middle ground
MLA	d_latent + d_rope	Very small cache, extra up-projection maths
LRKV	about 2 × (d_head + H × r)	Shared base plus per-head low-rank detail

And with real numbers for our example model:

Notice the meter doesn't only go down. MQA is the smallest, yet nobody stopped there, because the goal was never "smallest cache". It was "smallest cache that still thinks well". Everything after MQA is about buying quality back cheaply.

Meanwhile, three side quests

FlashAttention (2022): eat the pizza slice by slice. Attention compares every token with every token, giving an N × N grid of scores. For 32,000 tokens, that's a billion numbers the GPU has to shuffle in and out of its slow main memory. FlashAttention computes the grid in small tiles that fit in the GPU's tiny, fast on-chip memory, keeps a running total (the "online softmax") and throws each tile away. During training it recomputes tiles when needed instead of storing them. Same exact answer, far less memory traffic, much faster.

Sliding-window attention and attention sinks. Do you remember every word of this article so far? No, you remember the last few paragraphs and the gist. Sliding-window attention lets each token look only at the last few thousand tokens (early Mistral 7B did this). Researchers then found a quirk: models dump a surprising amount of attention on the very first tokens, using them as a sort of "parking spot". Evict those and the model falls apart. Keep a handful of these attention sinks forever and you can stream text endlessly.

Context extension (RoPE scaling, YaRN). A model trained on 4k tokens panics at token 4,001, a bit like a ruler that only goes to 30 cm. Because RoPE encodes position as rotation, you can re-scale the rotation speeds, effectively stretching the ruler. YaRN does this carefully, frequency by frequency, and lets a model handle much longer text with only a little fine-tuning.

4. Taming the KV cache: three families of tricks

Shrinking the notepad through architecture is one option. There are two more, and real systems stack all three.

Spell 1: architectural magic (make less cache)

Everything from section 3: GQA, MLA, LRKV. You choose this when the model is designed; you can't bolt it on later.

Spell 2: shrinking spells (fewer bits per number)

Each cached number is normally stored in 16 bits. Do we need all 16?

FP8 and FP4 caches: store each value in 8 or 4 bits. Half or a quarter of the memory, for a small accuracy cost.
KIVI (2024): noticed that keys and values have different personalities. Keys have a few loud channels, so quantise keys per channel and values per token. Works down to about 2 bits with no retraining.
TurboQuant (Google Research, ICLR 2026): about 3 bits per value, no retraining, no calibration data. It first rotates vectors so they become predictable (PolarQuant), then adds a tiny 1-bit correction for whatever error is left (QJL). Think of it as packing a suitcase really well, then sitting on it.

More on how quantisation works in section 8.

Spell 3: system tricks (waste less)

PagedAttention (vLLM, 2023). Early servers reserved one huge continuous slab of memory per chat, sized for the longest possible reply. Most chats are short, so most of that memory sat empty, like booking a 12-seat table for two. PagedAttention borrows an idea from 1960s operating systems: cut the cache into small fixed-size blocks, park them anywhere, and keep a lookup table.

Prefix caching. Every user of a chatbot gets the same long system prompt. Why compute and store its keys and values a million times? Compute once, share the blocks. This is what's behind the "cached input tokens" discount on API pricing pages.

Continuous batching. Old approach: a bus that waits until it's full, then leaves, and nobody new boards until the last passenger gets off. Continuous batching is a shared auto-rickshaw: the moment one passenger hops out, another hops in. GPUs stay full.

Chunked prefill. One user pastes a 100-page PDF, and everybody else's chat freezes while the model reads it. Chunked prefill breaks the reading into pieces and interleaves other users' tokens between them. It's the "I'll just let you go ahead, you only have two items" of GPU scheduling.

Speculative decoding. A small, fast model drafts the next few tokens; the big model checks them all in one pass and keeps the ones it agrees with. The intern drafts the email, the boss skims and approves. The output is exactly what the big model would have written anyway, just faster, as long as the intern is usually right.

If the big model accepts each guessed token with probability α, guesses γ tokens ahead, and the draft model costs a fraction c of the big one:

\text{speed-up} = \frac{1-\alpha^{\gamma+1}}{(1-\alpha)(\gamma c+1)}

Cache offloading and eviction. When a user wanders off to make chai, move their cache to cheaper memory (CPU RAM or disk), or delete it and rebuild later. Hot conversations stay on the GPU; cold ones go to the attic.

5. Mixture of experts: stop asking the dentist about your taxes

So far we've optimised attention. The other half of every layer is the feed-forward network, and it's where most of the parameters live.

In a normal ("dense") model, every token passes through all of those parameters. That's a hospital where every patient, including the one with a paper cut, must see every specialist in the building.

A mixture of experts (MoE) adds a receptionist. A tiny router looks at each token and sends it to just a few specialists, say the top 2 out of 8.

The maths is short. The router scores every expert, keeps the top k, and blends their answers:

y=\sum_{i\in\mathrm{TopK}(g(x),\,k)} g_i(x)\,E_i(x),\qquad g(x)=\mathrm{softmax}(W_r\,x)

Total vs active parameters

This gives MoE models two sizes. Mixtral 8x7B has about 47 billion parameters in total but uses only about 13 billion for any given token. DeepSeek-V3 has 671 billion in total and roughly 37 billion active. You get the knowledge of a giant model at the per-token cost of a mid-sized one.

The lazy-router problem

Left alone, routers play favourites. One expert gets picked a bit more, so it learns a bit more, so it gets picked even more. Soon you have one exhausted expert and seven on a tea break (the second half of the animation above).

Fixes:

Load-balancing loss: a small penalty, α · N · Σ fᵢ · Pᵢ, that is lowest when work is spread evenly (fᵢ is the share of tokens expert i received, Pᵢ the share of router probability).
Auxiliary-loss-free balancing (DeepSeek-V3): instead of a penalty, quietly nudge each expert's score up or down depending on how busy it has been. Like a restaurant host steering guests to the empty section.
Capacity factor: each expert has a fixed number of seats per batch. Overflow tokens get dropped: they skip the expert layer entirely.
Shared and fine-grained experts: a couple of always-on generalists handle common knowledge, plus many small specialists instead of a few big ones.

The fine print

Experts usually live on different GPUs (expert parallelism), so tokens must be shipped to their expert and back. Every GPU talks to every other GPU, a traffic pattern called all-to-all.

And the big caveat: MoE saves compute, not memory. All the experts must sit loaded in memory, because you can't know which ones the next token will need. All the doctors are on salary even if you only see two.

6. Scaling training: parallelism, ZeRO and FSDP

Running a model is hard. Training one is harder, because training hoards memory.

Where the memory goes

For every parameter, training keeps roughly:

2 bytes for the weight itself (16-bit)
2 bytes for its gradient (which way to nudge it)
12 bytes for the Adam optimiser's bookkeeping (a full-precision copy, plus momentum, plus variance)

That's about 16 bytes per parameter before counting activations. A 7.5-billion-parameter model needs 120 GB. No single GPU has that. So we split the work.

Three ways to slice a model

It's a group project, and there are three ways to divide it.

Data parallelism: everyone gets a full copy of the assignment and a different chapter to work on, then you average the results. Simple, but every GPU needs the whole model.
Tensor parallelism: split each big matrix across GPUs. Everyone works on the same sentence at the same time, so it needs very fast connections.
Pipeline parallelism: GPU 1 holds layers 1 to 8, GPU 2 holds 9 to 16, and so on. A relay race.

GPUs coordinate with a small vocabulary of group chats, called collective operations: all-reduce (everyone sums up and gets the total), all-gather (everyone gets everyone's piece), reduce-scatter (sum, then each keeps a slice), and all-to-all (the MoE one from earlier).

ZeRO: stop making everyone carry everything

Plain data parallelism is wasteful. All 64 GPUs hold identical copies of the optimiser states, like 64 flatmates each buying their own washing machine.

ZeRO (Zero Redundancy Optimizer) shards things so each GPU keeps only its slice:

Stage 1: shard the optimiser states
Stage 2: also shard the gradients
Stage 3: also shard the weights themselves, fetching them when needed

FSDP is PyTorch's implementation of the stage 3 idea. ZeRO-Offload goes further and pushes the optimiser's work to CPU memory.

With Ψ parameters and N GPUs:

M_{\text{ZeRO-1}}=4\Psi+\frac{12\Psi}{N_d},\quad M_{\text{ZeRO-2}}=2\Psi+\frac{14\Psi}{N_d},\quad M_{\text{ZeRO-3}}=\frac{16\Psi}{N_d}\ \text{bytes}

From 120 GB to under 2 GB per GPU. The price is more chatter between GPUs, which is why the fast interconnect matters as much as the GPUs.

The pipeline bubble

The relay race has a flaw. While runner 1 is running, runners 2, 3 and 4 stand around. That idle time is the pipeline bubble.

The fix is to cut each batch into many micro-batches, so GPU 1 can hand off the first one and start the next immediately:

\text{bubble}=\frac{p-1}{m+p-1}

With 4 stages and 4 micro-batches, GPUs idle 43% of the time. With 32 micro-batches, under 9%. A smarter ordering called 1F1B (one forward, one backward) keeps the same bubble but stops activations piling up in memory.

The rest of the toolbox, quickly

Sequence / context parallelism: split one very long sequence across GPUs; ring attention passes keys and values around in a circle.
Expert parallelism: different MoE experts on different GPUs.
3-D / 4-D parallelism: real training runs combine data + tensor + pipeline (+ sequence or expert) at once.
Activation checkpointing: don't store every intermediate result for the backward pass; save a few and recompute the rest. Trades time for memory.
FP8 training: do much of the arithmetic in 8-bit floats (DeepSeek-V3 trained this way).
FlashAttention helps here too: its recomputation trick is what keeps training memory linear in sequence length.
Overlapping communication with compute: send data while the GPU is busy calculating, so nobody waits on the network.
MFU (model FLOPs utilisation): the fraction of the hardware's theoretical peak that actually goes into your model's maths. It's the fuel-efficiency number of training; around half is considered good.

7. Distillation: a small model that copied the topper's notes

You've trained a brilliant, enormous model. It's also slow and expensive. Can a small model learn to behave like it?

Dark knowledge

Show a great teacher model a photo of a husky. A plain label says: "husky". End of lesson.

But the teacher's full answer is richer: "88% husky, 7% wolf, 4% malamute, basically 0% pizza." That tells the student something a textbook label never could: huskies look like wolves and not like pizza. Geoffrey Hinton called this dark knowledge. It lives in the wrong answers.

At normal settings those small percentages are too faint to learn from. So we raise the temperature, which softens the teacher's confidence and makes the "nearly right" answers visible:

The student is trained on a blend of the true labels and the teacher's softened opinion:

\mathcal{L}=\alpha\,\mathrm{CE}\big(y,\,p_s\big)+(1-\alpha)\,T^{2}\,\mathrm{KL}\big(p_t^{(T)}\,|\,p_s^{(T)}\big),\qquad p^{(T)}=\mathrm{softmax}(z/T)

It's the difference between an answer key and a teacher who explains why option B was tempting.

Flavours of distillation

Logit distillation: copy the teacher's output probabilities (above).
Hidden-state (feature) distillation: also copy what the teacher is "thinking" in its middle layers.
Sequence-level distillation: just train the student on text the teacher wrote. Simple, and it works even when you can only see the teacher's outputs.
On-policy distillation: the student writes, the teacher marks. This matters more than it sounds.

Mean-seeking vs mode-seeking

Suppose a question has two good answers, and the student is too small to hold both. It has two ways to fail:

Classic distillation minimises forward KL: the student tries to cover everything the teacher might say, and ends up blurring between the two answers, sometimes producing a mushy mix the teacher would never say. Ask two friends for directions, average "turn left" and "turn right", and drive into a wall.

On-policy distillation minimises reverse KL: the student is only judged on what it actually says, so it learns to pick one good answer and commit. For generating text, committing is usually what you want.

Where it shows up

DistilBERT (2019): 40% smaller than BERT, nearly as good. The proof that this works for language.
Reasoning-trace distillation (2025): DeepSeek-R1's step-by-step reasoning was distilled into small open models, giving laptop-sized models that "think out loud".
Distillation + pruning: cut parts out of the big model, then use distillation to help what's left recover.

8. Quantisation: FP8, FP4 and TurboQuant

Last stop. Distillation makes the model smaller by having fewer numbers. Quantisation keeps the same count of numbers and makes each one cheaper.

Think of prices. ₹499.99 and ₹500 are the same to your wallet. A model's weights are stored with far more decimal places than they need.

Where the bits go

FP32 / FP16 / BF16: the standard floating-point formats. BF16 keeps FP32's huge range but with fewer digits, which training likes.
FP8: comes in two flavours. E4M3 (more precision) and E5M2 (more range).
INT8 / INT4: plain evenly spaced steps: 256 or 16 of them.
NF4: 4 bits, but the 16 levels are placed where weights actually cluster (a bell curve), not evenly.
FP4 (MXFP4, NVFP4): tiny 4-bit floats where a block of values shares one scale factor.

How rounding works

To quantise, pick a ruler: a step size s and an offset z. Then snap every value to the nearest mark.

q=\mathrm{clamp}!\Big(\mathrm{round}\big(\tfrac{x}{s}\big)+z,\;0,\;2^{b}-1\Big),\qquad \hat{x}=s\,(q-z),\qquad s=\frac{x_{\max}-x_{\min}}{2^{b}-1}

Absmax quantisation centres the ruler on zero. Zero-point (affine) quantisation slides the ruler to wherever the data sits.
The rounding error is never more than half a step (s/2).
Weight memory ≈ parameters × bits ÷ 8. A 70-billion-parameter model is 140 GB at 16 bits and 35 GB at 4 bits. That's the difference between "data centre" and "one very nice workstation".

The outlier problem

Ten people are in a bar. Bill Gates walks in. On average, everyone is now a billionaire.

The same thing wrecks naive quantisation. The step size depends on the largest value, so one giant outlier stretches the ruler until all the normal weights are crushed onto two or three marks (the last frame above: one outlier makes 4-bit error about five times worse). LLMs are full of such outliers, which is why "just round everything" doesn't work.

You can soften this with granularity: one ruler for the whole tensor (crude), one per channel (better), or one per small group of values (best, slightly more overhead).

The methods you'll see on model cards

Post-training quantisation (PTQ) vs quantisation-aware training (QAT): round after training (cheap) or teach the model to live with rounding during training (better, costly).
LLM.int8(): keep the few outlier dimensions in 16-bit, squash everything else to 8-bit.
SmoothQuant: move the "spikiness" from activations into weights, where it's easier to handle.
GPTQ: quantise one weight at a time and adjust the remaining ones to cancel out the error. Like fixing a wobbly table by trimming the other legs.
AWQ: look at real activations to find the small fraction of weights that matter most, and protect them.
GGUF k-quants: those Q4_K_M files you download for llama.cpp. Different layers get different bit-widths depending on how sensitive they are.
QLoRA: freeze a 4-bit (NF4) model and train small adapter layers on top. This made fine-tuning big models possible on a single consumer GPU.

Back to the KV cache

And so we return to our villain. Weights are quantised once, offline. The KV cache is created live, per user, per token, so its quantiser must be fast and must work without seeing data in advance.

That's the significance of TurboQuant: about 3 bits per cached value, with no retraining and no calibration. Compared with 16 bits, that's a cut of more than 5× applied to the largest moving part of long-context inference. The rotation step (PolarQuant) deals with the Bill Gates problem by spreading every vector's energy evenly, and the 1-bit correction (QJL) fixes the small bias that's left in attention scores.

9. So how did all this make GPT-style models better?

None of these ideas made models smarter directly. They made bigger, longer, cheaper possible, and that's what made models smarter. (OpenAI doesn't publish GPT-4-era internals, so the examples below come from open models and papers.)

The wall we hit	The idea that broke it	Where you can see it
Recomputing everything per token	KV cache	Every LLM in production
Cache too big for long chats	GQA, MLA, LRKV	Llama 2/3, Mistral (GQA); DeepSeek-V2/V3 (MLA)
Attention too slow and memory-hungry	FlashAttention	Built into PyTorch and most training stacks
Context windows stuck at a few thousand tokens	RoPE, YaRN, sliding windows, attention sinks	4k contexts became 100k+
GPUs sitting half-empty when serving	PagedAttention, continuous batching, prefix caching	vLLM and friends; "cached token" pricing on APIs
Replies too slow	Speculative decoding	Faster responses, same output
Bigger models cost more per token	Mixture of experts	Mixtral, DeepSeek-V3
Model doesn't fit on any GPU for training	ZeRO/FSDP, tensor and pipeline parallelism	Every 100B+ training run
Great model, too expensive to deploy	Distillation	DistilBERT, small "distilled" reasoning models
Model doesn't fit on your GPU	Quantisation	GGUF files, QLoRA fine-tunes, FP8 serving

Put together: GPT-2 in 2019 handled 1,024 tokens. Today's models read entire codebases in one go and answer in seconds, on hardware that isn't a thousand times bigger. The gap was closed by the tricks in this post.

10. The timeline

Year	What happened	Why it mattered
1991	Mixture of experts proposed	The "router + specialists" idea, decades early
2014	Attention for translation (Bahdanau et al.)	Models learn to focus
2015	Knowledge distillation (Hinton et al.)	Small models learn from big ones
2017	Transformer; sparsely-gated MoE	Attention becomes the whole architecture
2018–2020	GPT-1, GPT-2, GPT-3	Scale works; costs explode
2019	MQA, DistilBERT, ZeRO, Megatron	First serious efficiency push
2021	RoPE, Switch Transformer	Better positions; MoE meets transformers
2022	FlashAttention, GPTQ, LLM.int8(), SmoothQuant, speculative decoding	Speed and compression go mainstream
2023	GQA, PagedAttention (vLLM), QLoRA, AWQ, YaRN, attention sinks, Mixtral	Open models become practical to run
2024	MLA (DeepSeek-V2), KIVI, DeepSeek-V3	The KV cache becomes the main target
2025	Reasoning distilled into small open models	"Thinking" models on laptops
2026	LRKV, TurboQuant	Smaller caches without losing head diversity; ~3-bit caches

The 30-second version

Attention = every token decides which other tokens matter, using queries, keys and values.
KV cache = remember keys and values so you never recompute them. It's fast, and it eats memory.
MHA → MQA → GQA → MLA → LRKV = five answers to "how little can each token remember and still think well?"
Three spells for the cache: better architecture, fewer bits, smarter memory management.
MoE = use a few specialists per token. Saves compute, not memory.
ZeRO and parallelism = split training across GPUs without everyone carrying everything.
Distillation = a small model learns from a big one's full opinions, not just its final answers.
Quantisation = store each number with fewer bits; watch out for outliers.

The original paper said attention is all you need. Nine years of engineering later, the fuller version is: attention, plus a very good memory for what not to remember.

If this helped, tell me which section you'd like expanded into its own post. And if you spot a mistake, call it out in the comments; I'd rather be corrected than confidently wrong.