<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: 陈晨</title>
    <description>The latest articles on DEV Community by 陈晨 (@chenchen8486).</description>
    <link>https://dev.to/chenchen8486</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4253319%2Fa723d889-2194-403d-9434-64a451c3e767.png</url>
      <title>DEV Community: 陈晨</title>
      <link>https://dev.to/chenchen8486</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/chenchen8486"/>
    <language>en</language>
    <item>
      <title>What KV Cache is actually trading, and a Chinese handbook that shows the work</title>
      <dc:creator>陈晨</dc:creator>
      <pubDate>Thu, 19 Nov 2026 08:19:41 +0000</pubDate>
      <link>https://dev.to/chenchen8486/what-kv-cache-is-actually-trading-and-a-chinese-handbook-that-shows-the-work-5ch5</link>
      <guid>https://dev.to/chenchen8486/what-kv-cache-is-actually-trading-and-a-chinese-handbook-that-shows-the-work-5ch5</guid>
      <description>&lt;h1&gt;
  
  
  What KV Cache is actually trading, and a Chinese handbook that shows the work
&lt;/h1&gt;

&lt;p&gt;Most explanations of large language models stop at “attention is a softmax over query-key scores.” That is enough to sound fluent, and not enough to see why a decoding loop gets expensive, or why every serving stack eagerly caches keys and values. I wrote a Chinese handbook to sit in that gap: not a prompt cookbook, and not a from-scratch training repo, but a mechanism-first walk along the whole LLM pipeline.&lt;/p&gt;

&lt;p&gt;The book is &lt;a href="https://github.com/chenchen8486/LLM_learning" rel="noopener noreferrer"&gt;LLM 机制学习手册&lt;/a&gt; (&lt;em&gt;LLM Mechanics Handbook&lt;/em&gt;). The prose is in Chinese; &lt;a href="https://github.com/chenchen8486/LLM_learning/blob/master/README_en.md" rel="noopener noreferrer"&gt;README_en.md&lt;/a&gt; is the English entry point. You can read it in the browser, with no clone and no build, at:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://chenchen8486.github.io/LLM_learning/" rel="noopener noreferrer"&gt;https://chenchen8486.github.io/LLM_learning/&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you prefer files on disk, clone the repo and open &lt;code&gt;index.html&lt;/code&gt;, or jump straight to &lt;code&gt;output/html/ch0_1_what_is_llm.html&lt;/code&gt;. Prebuilt pages are committed. Formulas and fonts load from a CDN on first open.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who it is for
&lt;/h2&gt;

&lt;p&gt;It is for readers who want the &lt;em&gt;why&lt;/em&gt; behind the terms they already meet in papers and code: tokenization, rotary position embeddings, attention, pre-training and mid-training, SFT and LoRA, preference alignment, decoding, and deployment. The Chinese README describes the project as a handbook that covers training data, tokenization, embeddings and positional encoding, the Transformer architecture, pre-training, mid-training, post-training, inference and deployment, application patterns, and multimodal extensions. Each topic is explained with formulas, original SVG diagrams, and — where the mechanism exists in code — a line-level comparison to a real PyTorch implementation.&lt;/p&gt;

&lt;p&gt;That implementation is &lt;a href="https://github.com/jingyaogong/minimind" rel="noopener noreferrer"&gt;minimind&lt;/a&gt; by jingyaogong, released under Apache 2.0. The handbook vendors a read-only copy as &lt;code&gt;minimind-master/&lt;/code&gt;, keeps its original license, and cites file names and line numbers at each comparison. I am grateful to the author and contributors; this is a companion for reading the mechanisms, not a substitute for their project, and all rights in that code remain with them.&lt;/p&gt;

&lt;p&gt;The handbook text is &lt;a href="https://github.com/chenchen8486/LLM_learning/blob/master/LICENSE" rel="noopener noreferrer"&gt;CC BY-NC-SA 4.0&lt;/a&gt;: you may share and adapt it with attribution, not for commercial use, and derivative work should stay under the same terms.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the book is organized
&lt;/h2&gt;

&lt;p&gt;Chapters run from &lt;strong&gt;第 0 章&lt;/strong&gt; (an overview of the LLM pipeline) through &lt;strong&gt;第 15 章&lt;/strong&gt; (multimodal extensions: ViT, CLIP, connectors, VLM training, unified architectures), plus a math-and-deep-learning appendix. The GitHub About line currently describes it as “15 章加数学附录.” (The Chinese README badge counts “16 章 + 附录,” treating chapter 0 as a chapter; the English README’s appendix numbering also lags the files slightly. The page sources go through appendix sections A.1–A.15.)&lt;/p&gt;

&lt;p&gt;A typical page is built the same way: a “quick scan” card with the core question and a one-sentence claim, then the explanation, diagrams, a short recap, and sources. Chapter 4 follows a Chinese translation of &lt;em&gt;Attention Is All You Need&lt;/em&gt;; other chapters lean on the original papers and official technical reports, listed at the bottom of each page.&lt;/p&gt;

&lt;h2&gt;
  
  
  A concrete mechanism: KV Cache
&lt;/h2&gt;

&lt;p&gt;To show the tone of the book, here is the argument of &lt;a href="https://chenchen8486.github.io/LLM_learning/output/html/ch13_2_kv_cache.html" rel="noopener noreferrer"&gt;第 13 章第 2 节，KV Cache：用显存换速度的核心缓存&lt;/a&gt; — without inventing any numbers that are not on that page.&lt;/p&gt;

&lt;p&gt;Autoregressive decoding runs a forward pass for every new token. The input at step (t+1) is the input at step (t) plus one token. If you naively rerun the whole sequence each time, you recompute a lot of intermediate results that have not changed.&lt;/p&gt;

&lt;p&gt;Why have they not changed? Causal masking, covered earlier in section 4.5 of the handbook, means the representation at position (i) depends only on positions (1) through (i). Once a position’s key and value vectors are computed, they stay put. Recomputing them on the next step is wasted work.&lt;/p&gt;

&lt;p&gt;The section works a tiny example. Take a 3-token prompt (x_1, x_2, x_3) and generate two more tokens, (x_4) and (x_5):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Round 1 (produce (x_4)): 3 positions, 3 position-processings. All of this is first-time work.&lt;/li&gt;
&lt;li&gt;Round 2 (produce (x_5)): 4 positions, 4 processings, but the keys and values of the first three positions are bit-for-bit the same as in round 1. Only (x_4) is new.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is 7 position-processings in total, 3 of them repeats — and this is only two generated tokens. The page writes the general counts as&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
C_{\text{no cache}} = m\,n_0 + \frac{m(m-1)}{2}, \qquad C_{\text{with cache}} = n_0 + (m-1)&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;where (n_0) is the prompt length and (m) is the number of new tokens. For (n_0 = 3), (m = 2), that is 7 versus 4. As (m) grows, the uncached cost picks up an (m^2/2) term; the cached cost stays linear. The first full pass over the prompt is prefill; after that, each decode step should handle only the new position.&lt;/p&gt;

&lt;p&gt;The cache itself is two growing buffers per layer: all past keys, and all past values. At a decode step you send in only the latest token, compute its (q), (k), and (v), append (k) and (v), and let (q) attend to the whole history:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
\mathrm{out}&lt;em&gt;t = \mathrm{softmax}!\left( \frac{q_t\, K&lt;/em&gt;{1:t}^{\top}}{\sqrt{d_h}} \right) V_{1:t}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;This is the ordinary attention formula with a single query position. The output distribution does not change; only the work does. Queries are not cached: a query is a one-shot probe used to write that position’s output, and later steps never reuse it.&lt;/p&gt;

&lt;p&gt;The same section then reads the mechanism in minimind (&lt;code&gt;model/model_minimind.py&lt;/code&gt;, Apache 2.0). Three moves matter:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Concatenate&lt;/strong&gt; (Attention, lines 121–123): new &lt;code&gt;xk&lt;/code&gt; / &lt;code&gt;xv&lt;/code&gt; are appended to the cached keys and values along the sequence axis. Rotary embeddings are applied &lt;em&gt;before&lt;/em&gt; the cache (line 119), so stored keys already carry position. GQA’s &lt;code&gt;repeat_kv&lt;/code&gt; (line 124) runs &lt;em&gt;after&lt;/em&gt; the concat, which is why the cache can store the smaller native KV-head layout.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Slice&lt;/strong&gt; (&lt;code&gt;generate&lt;/code&gt;, lines 269–270): with the cache on, only the new suffix is fed in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write back&lt;/strong&gt; (line 286): the updated &lt;code&gt;past_key_values&lt;/code&gt; are returned for the next step. &lt;code&gt;use_cache=True&lt;/code&gt; is the default (line 262). Turning it off does not change the answers; it just recomputes everything.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The cost is memory that grows with context. For minimind’s measured config — 8 layers, 4 KV heads, head dimension 96, FP16 — the page’s formula gives (2 \times 4 \times 96 \times 2 = 1536) bytes per token per layer, or &lt;strong&gt;12 KB per token&lt;/strong&gt; across 8 layers. At 8192 tokens that is about &lt;strong&gt;100 MB&lt;/strong&gt; for a single session, close to the model’s own FP16 weights (about 128 MB). The same section notes that 8 concurrent 8K sessions would be about 800 MB of cache, already several times the weights. GQA with 4 KV heads, rather than 8, is what cuts that bill in half relative to full MHA (the page also quotes the MHA / MQA ends of the range: about 201 MB and 25 MB under the same assumptions).&lt;/p&gt;

&lt;p&gt;If you only remember one sentence from the chapter, this is the one it leads with: KV Cache is an &lt;em&gt;exactly equivalent&lt;/em&gt; optimization. You trade a context-linear slab of memory for not paying a quadratic amount of repeated compute, and the token distribution stays the same.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you want to read further
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Handbook online: &lt;a href="https://chenchen8486.github.io/LLM_learning/" rel="noopener noreferrer"&gt;https://chenchen8486.github.io/LLM_learning/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The KV Cache page above, and the rest of chapter 13 (inference cost, serving, quantization, distillation, speculative decoding)&lt;/li&gt;
&lt;li&gt;Chapter 4 for attention itself, and section 5.2 if you want the MHA / GQA / MQA trade-off that makes the cache cheaper&lt;/li&gt;
&lt;li&gt;Repository, including the English README: &lt;a href="https://github.com/chenchen8486/LLM_learning" rel="noopener noreferrer"&gt;https://github.com/chenchen8486/LLM_learning&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The code being compared: &lt;a href="https://github.com/jingyaogong/minimind" rel="noopener noreferrer"&gt;https://github.com/jingyaogong/minimind&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I wrote the handbook in Chinese because that is the language I wanted this path to exist in. If you read English more comfortably, the pages will still show the diagrams and the formulas; a bilingual friend, or the English README, is enough to decide whether it is useful. Corrections and missing topics are welcome as issues on the repo.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>tutorial</category>
      <category>python</category>
    </item>
  </channel>
</rss>
