DEV Community

陈晨
陈晨

Posted on

What KV Cache is actually trading, and a Chinese handbook that shows the work

What KV Cache is actually trading, and a Chinese handbook that shows the work

Most explanations of large language models stop at “attention is a softmax over query-key scores.” That is enough to sound fluent, and not enough to see why a decoding loop gets expensive, or why every serving stack eagerly caches keys and values. I wrote a Chinese handbook to sit in that gap: not a prompt cookbook, and not a from-scratch training repo, but a mechanism-first walk along the whole LLM pipeline.

The book is LLM 机制学习手册 (LLM Mechanics Handbook). The prose is in Chinese; README_en.md is the English entry point. You can read it in the browser, with no clone and no build, at:

https://chenchen8486.github.io/LLM_learning/

If you prefer files on disk, clone the repo and open index.html, or jump straight to output/html/ch0_1_what_is_llm.html. Prebuilt pages are committed. Formulas and fonts load from a CDN on first open.

Who it is for

It is for readers who want the why behind the terms they already meet in papers and code: tokenization, rotary position embeddings, attention, pre-training and mid-training, SFT and LoRA, preference alignment, decoding, and deployment. The Chinese README describes the project as a handbook that covers training data, tokenization, embeddings and positional encoding, the Transformer architecture, pre-training, mid-training, post-training, inference and deployment, application patterns, and multimodal extensions. Each topic is explained with formulas, original SVG diagrams, and — where the mechanism exists in code — a line-level comparison to a real PyTorch implementation.

That implementation is minimind by jingyaogong, released under Apache 2.0. The handbook vendors a read-only copy as minimind-master/, keeps its original license, and cites file names and line numbers at each comparison. I am grateful to the author and contributors; this is a companion for reading the mechanisms, not a substitute for their project, and all rights in that code remain with them.

The handbook text is CC BY-NC-SA 4.0: you may share and adapt it with attribution, not for commercial use, and derivative work should stay under the same terms.

How the book is organized

Chapters run from 第 0 章 (an overview of the LLM pipeline) through 第 15 章 (multimodal extensions: ViT, CLIP, connectors, VLM training, unified architectures), plus a math-and-deep-learning appendix. The GitHub About line currently describes it as “15 章加数学附录.” (The Chinese README badge counts “16 章 + 附录,” treating chapter 0 as a chapter; the English README’s appendix numbering also lags the files slightly. The page sources go through appendix sections A.1–A.15.)

A typical page is built the same way: a “quick scan” card with the core question and a one-sentence claim, then the explanation, diagrams, a short recap, and sources. Chapter 4 follows a Chinese translation of Attention Is All You Need; other chapters lean on the original papers and official technical reports, listed at the bottom of each page.

A concrete mechanism: KV Cache

To show the tone of the book, here is the argument of 第 13 章第 2 节,KV Cache:用显存换速度的核心缓存 — without inventing any numbers that are not on that page.

Autoregressive decoding runs a forward pass for every new token. The input at step (t+1) is the input at step (t) plus one token. If you naively rerun the whole sequence each time, you recompute a lot of intermediate results that have not changed.

Why have they not changed? Causal masking, covered earlier in section 4.5 of the handbook, means the representation at position (i) depends only on positions (1) through (i). Once a position’s key and value vectors are computed, they stay put. Recomputing them on the next step is wasted work.

The section works a tiny example. Take a 3-token prompt (x_1, x_2, x_3) and generate two more tokens, (x_4) and (x_5):

  • Round 1 (produce (x_4)): 3 positions, 3 position-processings. All of this is first-time work.
  • Round 2 (produce (x_5)): 4 positions, 4 processings, but the keys and values of the first three positions are bit-for-bit the same as in round 1. Only (x_4) is new.

That is 7 position-processings in total, 3 of them repeats — and this is only two generated tokens. The page writes the general counts as

[
C_{\text{no cache}} = m\,n_0 + \frac{m(m-1)}{2}, \qquad C_{\text{with cache}} = n_0 + (m-1)
]

where (n_0) is the prompt length and (m) is the number of new tokens. For (n_0 = 3), (m = 2), that is 7 versus 4. As (m) grows, the uncached cost picks up an (m^2/2) term; the cached cost stays linear. The first full pass over the prompt is prefill; after that, each decode step should handle only the new position.

The cache itself is two growing buffers per layer: all past keys, and all past values. At a decode step you send in only the latest token, compute its (q), (k), and (v), append (k) and (v), and let (q) attend to the whole history:

[
\mathrm{out}t = \mathrm{softmax}!\left( \frac{q_t\, K{1:t}^{\top}}{\sqrt{d_h}} \right) V_{1:t}
]

This is the ordinary attention formula with a single query position. The output distribution does not change; only the work does. Queries are not cached: a query is a one-shot probe used to write that position’s output, and later steps never reuse it.

The same section then reads the mechanism in minimind (model/model_minimind.py, Apache 2.0). Three moves matter:

  1. Concatenate (Attention, lines 121–123): new xk / xv are appended to the cached keys and values along the sequence axis. Rotary embeddings are applied before the cache (line 119), so stored keys already carry position. GQA’s repeat_kv (line 124) runs after the concat, which is why the cache can store the smaller native KV-head layout.
  2. Slice (generate, lines 269–270): with the cache on, only the new suffix is fed in.
  3. Write back (line 286): the updated past_key_values are returned for the next step. use_cache=True is the default (line 262). Turning it off does not change the answers; it just recomputes everything.

The cost is memory that grows with context. For minimind’s measured config — 8 layers, 4 KV heads, head dimension 96, FP16 — the page’s formula gives (2 \times 4 \times 96 \times 2 = 1536) bytes per token per layer, or 12 KB per token across 8 layers. At 8192 tokens that is about 100 MB for a single session, close to the model’s own FP16 weights (about 128 MB). The same section notes that 8 concurrent 8K sessions would be about 800 MB of cache, already several times the weights. GQA with 4 KV heads, rather than 8, is what cuts that bill in half relative to full MHA (the page also quotes the MHA / MQA ends of the range: about 201 MB and 25 MB under the same assumptions).

If you only remember one sentence from the chapter, this is the one it leads with: KV Cache is an exactly equivalent optimization. You trade a context-linear slab of memory for not paying a quadratic amount of repeated compute, and the token distribution stays the same.

If you want to read further

I wrote the handbook in Chinese because that is the language I wanted this path to exist in. If you read English more comfortably, the pages will still show the diagrams and the formulas; a bilingual friend, or the English README, is enough to decide whether it is useful. Corrections and missing topics are welcome as issues on the repo.

Top comments (0)