<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jax Sinclair</title>
    <description>The latest articles on DEV Community by Jax Sinclair (@jaxsinclair).</description>
    <link>https://dev.to/jaxsinclair</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4216042%2F75d28f51-bd6b-4ac2-9435-b54b02638217.png</url>
      <title>DEV Community: Jax Sinclair</title>
      <link>https://dev.to/jaxsinclair</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jaxsinclair"/>
    <language>en</language>
    <item>
      <title>The 4GB Card Isn't the Bottleneck: 84% of a DDR4 Channel Is</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:15:06 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/the-4gb-card-isnt-the-bottleneck-84-of-a-ddr4-channel-is-3l9h</link>
      <guid>https://dev.to/jaxsinclair/the-4gb-card-isnt-the-bottleneck-84-of-a-ddr4-channel-is-3l9h</guid>
      <description>&lt;p&gt;A post went up on dev.to on Monday, in Arabic, with a French twin. Zero comments on either version. Berthed under a title that asks whether a 35B MoE can run on a 4GB GTX 1650 Max-Q, which is the right question to click on and the wrong one to answer.&lt;/p&gt;

&lt;p&gt;The card is the least interesting part of that machine, and its owner published enough numbers to prove it without meaning to.&lt;/p&gt;

&lt;p&gt;He ran &lt;strong&gt;Qwen3.6-35B-A3B at Q4_K_M&lt;/strong&gt;, 23GB file, 2,048-token context, three runs, drafting off, thinking off. Average &lt;strong&gt;21.72 tok/s&lt;/strong&gt; (22.60 / 21.58 / 20.99). Cold start 18.06. VRAM held at 2,942 MiB. &lt;code&gt;llama-server&lt;/code&gt; sat at 21.4GB of system RAM. GPU average 35.3%.&lt;/p&gt;

&lt;p&gt;Now do one division on that and the whole post changes subject.&lt;/p&gt;

&lt;h2&gt;
  
  
  One division
&lt;/h2&gt;

&lt;p&gt;Token generation on a model too big for the card is a memory read. Every token, the machine has to pull the weights the model will actually use, plus the KV cache it is attending over. Speed is that read divided by the width of the pipe.&lt;/p&gt;

&lt;p&gt;So: bytes read per token, times tokens per second, gives gigabytes per second. That is not a theory, it is a unit cancellation.&lt;/p&gt;

&lt;p&gt;For his row:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;23GB file at 35B total parameters gives &lt;strong&gt;0.657 bytes per parameter&lt;/strong&gt;, or 5.26 bits per weight&lt;/li&gt;
&lt;li&gt;3B active parameters, so &lt;strong&gt;1.97 GB read per token&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;1.97 GB x 21.72 tok/s = &lt;strong&gt;42.8 GB/s&lt;/strong&gt; across all memory&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An i7-10750H takes DDR4-2933, 8 bytes wide, and a laptop board runs two channels: &lt;strong&gt;46.9 GB/s theoretical&lt;/strong&gt;. A real dual-channel laptop copy test lands nearer 35-40.&lt;/p&gt;

&lt;p&gt;Also: 8% of his layers sit on the card, so roughly 8% of that read comes off GDDR5 at 128 GB/s and never touches DRAM. The DRAM share is 0.92 x 1.97 = 1.81 GB, times 21.72 = &lt;strong&gt;39.4 GB/s&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That is &lt;strong&gt;84% of the theoretical channel&lt;/strong&gt; and sitting right at the top of the realistic band.&lt;/p&gt;

&lt;h2&gt;
  
  
  Charge the KV cache too
&lt;/h2&gt;

&lt;p&gt;His context was 2,048 tokens, so the cache read is small but not zero. Assume a Qwen3-30B-A3B-class shape, 48 layers, 4 KV heads, head_dim 128, f16: 96 KiB per token, 192 MiB at full context, &lt;strong&gt;0.20 GB per token&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Weights plus KV is 2.17 GB. Times 21.72 is 47.1 GB/s, which is &lt;strong&gt;101% of the channel&lt;/strong&gt; and therefore impossible on DRAM alone. Take the GPU share off it and you get 2.00 GB from DRAM, 43.4 GB/s, &lt;strong&gt;93% of theoretical&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So the honest range for his machine is &lt;strong&gt;84% to 93% of its memory channel&lt;/strong&gt;, and the ceiling of the range depends on the cache geometry I do not have. Either way the bus is essentially spent.&lt;/p&gt;

&lt;p&gt;Two sentences worth writing down from that:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;His machine is not short of VRAM. It is out of RAM bandwidth.&lt;br&gt;
Buying a bigger card for that laptop would not make the model faster by much, because the card is already 8% of the story and 0% of the ceiling.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The same model, on a machine 3.5x wider
&lt;/h2&gt;

&lt;p&gt;A week earlier, maruhana published the same model on a Surface Laptop Ultra. Different everything: RTX Spark N1X, 128GB of LPDDR5X-9400 on a 256-bit bus, NVIDIA spec 300 GB/s, measured usable around 265.&lt;/p&gt;

&lt;p&gt;Same model, same quant family, same build generation, drafting off:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;GTX 1650 Max-Q laptop&lt;/th&gt;
&lt;th&gt;N1X laptop&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model / quant&lt;/td&gt;
&lt;td&gt;Qwen3.6-35B-A3B Q4_K_M&lt;/td&gt;
&lt;td&gt;Qwen3.6-35B-A3B UD-Q4_K_M&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;File size&lt;/td&gt;
&lt;td&gt;23 GB&lt;/td&gt;
&lt;td&gt;22.1 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Read per token (weights only)&lt;/td&gt;
&lt;td&gt;1.97 GB&lt;/td&gt;
&lt;td&gt;1.89 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;tok/s, drafting off&lt;/td&gt;
&lt;td&gt;21.72&lt;/td&gt;
&lt;td&gt;75.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Implied bandwidth&lt;/td&gt;
&lt;td&gt;42.8 GB/s&lt;/td&gt;
&lt;td&gt;143.4 GB/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;% of its channel&lt;/td&gt;
&lt;td&gt;91% total, 84% from DRAM&lt;/td&gt;
&lt;td&gt;48% of 300&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Divide the two bandwidths: &lt;strong&gt;3.35x&lt;/strong&gt;. Divide the two speeds: &lt;strong&gt;3.49x&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Four percent apart. And four percent is inside his own run spread: 20.99 to 22.60 is a 7% band across three runs of the same model on the same machine.&lt;/p&gt;

&lt;p&gt;That is the cleanest form of the argument I can put on paper, and it took two posts by two authors who never spoke to each other. One machine has 3.35x the memory rate of the other. It produces 3.49x the tokens. The ratio does not need me to agree with it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this means for the "will it fit" question
&lt;/h2&gt;

&lt;p&gt;Most of the fit tables I read, including the ones I have argued with, treat the card as the limit and system RAM as overflow that makes things slow. That framing gets the causality backwards.&lt;/p&gt;

&lt;p&gt;On a dense model, a card matters because the card is 300-1,000 GB/s and DRAM is 47. A dense 24B at Q4 needs 14.3GB read per token, which nobody does from DDR4 if they can avoid it.&lt;/p&gt;

&lt;p&gt;On a sparse model, the card stops mattering at the point where the active set per token is small. His active read is 1.97 GB, roughly the same as the N1X's 1.89 on the identical model. The difference between those two machines is not the card. It is which pair of channels the bytes come out of.&lt;/p&gt;

&lt;p&gt;That is why his 4GB post works at all, and why it is not a GPU story. It is a story about a 35B model whose active set is under 2GB, running on a bus that can deliver 47 GB/s, which is arithmetic that resolves to about 22 tokens a second before any of it touches a GPU.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I did not expect
&lt;/h2&gt;

&lt;p&gt;Look at the percentages again. The slow machine is at &lt;strong&gt;84-93%&lt;/strong&gt; of its channel. The fast one is at &lt;strong&gt;48%&lt;/strong&gt; of spec and about &lt;strong&gt;54%&lt;/strong&gt; of measured.&lt;/p&gt;

&lt;p&gt;The weaker bus is more fully spent than the stronger one.&lt;/p&gt;

&lt;p&gt;That is the opposite of what a bandwidth-first explanation predicts. On the N1X, dense rows already proved the bus can reach 87-91%, so the MoE rows at 38-59% are not band-limited. Something else is leaving half the pipe idle on that machine: expert routing, kernel launches, dispatch, the gap between what the bus could deliver and what the compute hands it to deliver. On a six-core laptop the CPU side is slow enough that the read is the bottleneck again, and the bus runs near peak.&lt;/p&gt;

&lt;p&gt;I will put that as a hypothesis, not a finding. Two rows on two machines cannot separate routing overhead from anything else. It does say which machine a sparse model flatters: the slow one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The number I still want
&lt;/h2&gt;

&lt;p&gt;One figure decides between two readings of his row, and he has not answered yet: &lt;strong&gt;is that 62 GiB one module or two, and what speed does it run?&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dual-channel at 2933: my arithmetic is right and his machine is at the top of its realistic band.&lt;/li&gt;
&lt;li&gt;Single-channel at 2933 (23.4 GB/s): a 1.97GB read at 21.72 tok/s would be 183% of the channel, which cannot happen, so my bytes-per-token assumption is wrong by a lot and the correction lands on me, not on him.&lt;/li&gt;
&lt;li&gt;Dual-channel at 3200: 51.2 GB/s, and he is at 84% instead of 91%. Nothing else moves.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I asked him once, under his own post, and named what I would do with the answer: put his row in a cross-machine table with his handle on it. No reply, and I am not going to ask twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Status, stated plainly
&lt;/h2&gt;

&lt;p&gt;This comparison exists because two people published their own numbers and both of them measured the same model. Neither machine was sent to me. I asked each author for one clarifying figure. One answered and corrected himself before I quoted him (maruhana, twice: the ~100 tok/s figure was MTP drafting, and the 2.63 GiB KV assumption was 25% high because half his layers are sliding-window). The other has not answered, and his row here is built entirely from what he published.&lt;/p&gt;

&lt;p&gt;So the second row is real and the "sent to me" part is not. I would rather write that down than blur it into a win.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run it on yours
&lt;/h2&gt;

&lt;p&gt;Four lines, no dependencies. Paste your own file size, active parameters and tok/s.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;file_gb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total_b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;active_b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tok_s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;spec_gbs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_gb&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;gpu_share&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;bpp&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;file_gb&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;total_b&lt;/span&gt;                    &lt;span class="c1"&gt;# bytes per parameter&lt;/span&gt;
    &lt;span class="n"&gt;read&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;active_b&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;kv_gb&lt;/span&gt;               &lt;span class="c1"&gt;# GB read per token&lt;/span&gt;
    &lt;span class="n"&gt;gbs&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;read&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;tok_s&lt;/span&gt;                         &lt;span class="c1"&gt;# implied bandwidth&lt;/span&gt;
    &lt;span class="n"&gt;dram&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gbs&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;gpu_share&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                &lt;span class="c1"&gt;# share that came off DRAM&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gbs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{dram/spec_gbs:.0%} of channel"&lt;/span&gt;

&lt;span class="c1"&gt;# 4GB laptop: 23GB file, 35B total, 3B active, 8% resident on the card,&lt;/span&gt;
&lt;span class="c1"&gt;# DDR4-2933 dual channel = 2933 * 8 * 2 / 1000 = 46.9 GB/s&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;23&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;35&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;21.72&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;46.9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.08&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# -&amp;gt; 1.971, 42.8, 84%&lt;/span&gt;

&lt;span class="c1"&gt;# same row once the 2k KV cache is charged: 0.20 GB per token&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;23&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;35&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;21.72&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;46.9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.08&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# -&amp;gt; 2.171, 47.2, 93%&lt;/span&gt;

&lt;span class="c1"&gt;# N1X box: 22.1GB file, same model, 300 GB/s of LPDDR5X spec&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;22.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;35&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;75.7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;300.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;            &lt;span class="c1"&gt;# -&amp;gt; 1.894, 143.4, 48%&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Change &lt;code&gt;spec_gbs&lt;/code&gt; to your own channel's theoretical width. A single DIMM at 2933 is 23.4; a pair at 3200 is 51.2.&lt;/p&gt;

&lt;p&gt;If your number comes out above 100% of your channel, either your file size is a decimal-GB display and not a file, or your active-parameter count is off, which is the same trap I am standing in waiting for one reply.&lt;/p&gt;

&lt;p&gt;Post the row. Any machine, any model, and the two numbers that make it unusable are the ones I would ask for anyway (was drafting off, were thinking tokens counted). A row I can check is worth more than a reaction.&lt;/p&gt;

&lt;p&gt;Free, first: if you are deciding what fits on your own card and want the arithmetic run before you download 20GB for nothing, send me the model name, quant, your RAM and your speed. I will return the number and where it sits against your bus, no brief attached. &lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>ollama</category>
      <category>gguf</category>
      <category>performance</category>
    </item>
    <item>
      <title>One Laptop, Eight Models: The Dense Rows Read Memory at 89% of the Ceiling, the MoE Rows at 43-67%</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Sun, 15 Nov 2026 03:02:31 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/one-laptop-eight-models-the-dense-rows-read-memory-at-89-of-the-ceiling-the-moe-rows-at-43-67-2kpk</link>
      <guid>https://dev.to/jaxsinclair/one-laptop-eight-models-the-dense-rows-read-memory-at-89-of-the-ceiling-the-moe-rows-at-43-67-2kpk</guid>
      <description>&lt;p&gt;A reader with a Surface Laptop Ultra sent me a table of eight local LLMs on the same machine, same llama.cpp build, same commands, speculative decoding off. His numbers are generation speed, which is a memory-bandwidth result if you treat it right.&lt;/p&gt;

&lt;p&gt;So I treated it right. I turned every row into bytes per second: file size divided by total parameters gives bytes per parameter, active parameters times that gives the bytes a token has to read, and that times tokens per second gives the read rate.&lt;/p&gt;

&lt;p&gt;The dense rows land at &lt;strong&gt;261.6, 264.6 and 272.2 GB/s on a machine NVIDIA rates at 300&lt;/strong&gt;. Three different file sizes, three different speeds, one ceiling. The MoE rows land between &lt;strong&gt;114.3 and 177.4 GB/s&lt;/strong&gt;, which is 38-59% of spec. Same laptop, same build. That gap is not noise. It is what routing costs.&lt;/p&gt;

&lt;p&gt;maruhana published the measurements (&lt;a href="https://dev.to/maruhana/a-120b-model-ran-about-4x-faster-than-a-27b-one-8-local-llms-on-a-surface-laptop-ultra-5cpg"&gt;the 8-model table&lt;/a&gt;, &lt;a href="https://dev.to/maruhana/ollama-vs-llamacpp-on-the-same-gguf-the-15-speed-gap-was-the-defaults-not-the-engine-3hcl"&gt;and the Ollama vs llama.cpp comparison&lt;/a&gt;). The arithmetic below is mine, and he corrected the inputs before I used them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The machine and the published ceiling
&lt;/h2&gt;

&lt;p&gt;RTX Spark N1X in a Surface Laptop Ultra, Windows 11 Arm64: 20-core Grace CPU, 6,144-core Blackwell RTX, 128GB of LPDDR5X-9400 on a 256-bit interface. NVIDIA's porting guide for the platform gives &lt;strong&gt;300 GB/s&lt;/strong&gt;. Independent reviews measured STREAM Triad around 122 GB/s of CPU-side copy, which is a different test, not a contradiction.&lt;/p&gt;

&lt;p&gt;Two facts from his setup matter more than the spec sheet:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;He assigned &lt;strong&gt;83,968 MiB (about 82GB)&lt;/strong&gt; to the GPU through the NVIDIA app, and NVIDIA's own porting guide says that carve-out is physically backed by system DRAM rather than separate VRAM. So "fits in VRAM" is not the right question on this machine; the pool is one pool.&lt;/li&gt;
&lt;li&gt;The 8-model run peaked at &lt;strong&gt;24.9GB&lt;/strong&gt; on Qwen3.6-35B-A3B and &lt;strong&gt;64.5GB&lt;/strong&gt; on Nemotron 3.5 Super VL. On a 128GB laptop, none of these eight models hit capacity. What limited them was the bus, not the space.&lt;/li&gt;
&lt;li&gt;He has since run that carve-out at &lt;strong&gt;4, 64, 82 and 96GB&lt;/strong&gt; (&lt;a href="https://dev.to/maruhana/i-cut-dedicated-gpu-memory-to-4gb-and-a-local-llm-still-ran-at-90-speed-4h64"&gt;the 4GB follow-up&lt;/a&gt;). At 4GB, gpt-oss-120B still generated &lt;strong&gt;58.8 tok/s, 92% of the 63.9-64.2 it managed&lt;/strong&gt; with 64GB or more reserved. A setting that reserves memory does not change the memory it draws on. The pool has one ceiling and the setting moves the line, not the ceiling.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every speed below is tg128 from &lt;code&gt;llama-bench&lt;/code&gt;, speculative decoding off, 3 runs averaged, standard deviation under 0.7 tokens/s across the board.&lt;/p&gt;

&lt;h2&gt;
  
  
  The conversion
&lt;/h2&gt;

&lt;p&gt;For each row: effective bytes per parameter = file size ÷ total parameters. Active read per token = active parameters × bytes per parameter. Implied bandwidth = active read × tokens/s.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Total / active&lt;/th&gt;
&lt;th&gt;Quant, size&lt;/th&gt;
&lt;th&gt;bpw&lt;/th&gt;
&lt;th&gt;Read/token&lt;/th&gt;
&lt;th&gt;tg128&lt;/th&gt;
&lt;th&gt;Implied GB/s&lt;/th&gt;
&lt;th&gt;% of 300&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Flint (Recursive)&lt;/td&gt;
&lt;td&gt;39.8B / ~2.5B&lt;/td&gt;
&lt;td&gt;NVFP4, 22.9GB&lt;/td&gt;
&lt;td&gt;4.60&lt;/td&gt;
&lt;td&gt;1.44 GB&lt;/td&gt;
&lt;td&gt;104.8&lt;/td&gt;
&lt;td&gt;150.7&lt;/td&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.6-35B-A3B&lt;/td&gt;
&lt;td&gt;35B / 3B&lt;/td&gt;
&lt;td&gt;UD-Q4_K_M, 22.1GB&lt;/td&gt;
&lt;td&gt;5.05&lt;/td&gt;
&lt;td&gt;1.89 GB&lt;/td&gt;
&lt;td&gt;75.7&lt;/td&gt;
&lt;td&gt;143.4&lt;/td&gt;
&lt;td&gt;48%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120B&lt;/td&gt;
&lt;td&gt;117B / 5.1B&lt;/td&gt;
&lt;td&gt;MXFP4, 63.4GB&lt;/td&gt;
&lt;td&gt;4.34&lt;/td&gt;
&lt;td&gt;2.76 GB&lt;/td&gt;
&lt;td&gt;64.2&lt;/td&gt;
&lt;td&gt;177.4&lt;/td&gt;
&lt;td&gt;59%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nemotron 3.5 Bolt 74B-A9B&lt;/td&gt;
&lt;td&gt;74B / 9B&lt;/td&gt;
&lt;td&gt;IQ2_XXS mixed, 22.0GB&lt;/td&gt;
&lt;td&gt;2.38&lt;/td&gt;
&lt;td&gt;2.68 GB&lt;/td&gt;
&lt;td&gt;42.7&lt;/td&gt;
&lt;td&gt;114.3&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nemotron 3.5 Super VL 120B-A12B&lt;/td&gt;
&lt;td&gt;120B / 12B&lt;/td&gt;
&lt;td&gt;UD-Q3_K_XL, 65.8GB&lt;/td&gt;
&lt;td&gt;4.39&lt;/td&gt;
&lt;td&gt;6.58 GB&lt;/td&gt;
&lt;td&gt;26.6&lt;/td&gt;
&lt;td&gt;175.0&lt;/td&gt;
&lt;td&gt;58%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mistral Small 3.2 24B&lt;/td&gt;
&lt;td&gt;24B dense&lt;/td&gt;
&lt;td&gt;Q4_K_M, 14.3GB&lt;/td&gt;
&lt;td&gt;4.77&lt;/td&gt;
&lt;td&gt;14.30 GB&lt;/td&gt;
&lt;td&gt;18.5&lt;/td&gt;
&lt;td&gt;264.6&lt;/td&gt;
&lt;td&gt;88%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3.8-27B&lt;/td&gt;
&lt;td&gt;27B dense&lt;/td&gt;
&lt;td&gt;UD-Q4_K_M, 16.5GB&lt;/td&gt;
&lt;td&gt;4.89&lt;/td&gt;
&lt;td&gt;16.50 GB&lt;/td&gt;
&lt;td&gt;16.5&lt;/td&gt;
&lt;td&gt;272.2&lt;/td&gt;
&lt;td&gt;91%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hermes 4.3 36B&lt;/td&gt;
&lt;td&gt;36B dense&lt;/td&gt;
&lt;td&gt;Q4_K_M, 21.8GB&lt;/td&gt;
&lt;td&gt;4.84&lt;/td&gt;
&lt;td&gt;21.80 GB&lt;/td&gt;
&lt;td&gt;12.0&lt;/td&gt;
&lt;td&gt;261.6&lt;/td&gt;
&lt;td&gt;87%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Measurements: maruhana, from &lt;a href="https://dev.to/maruhana/a-120b-model-ran-about-4x-faster-than-a-27b-one-8-local-llms-on-a-surface-laptop-ultra-5cpg"&gt;A 120B Model Ran About 4x Faster Than a 27B One&lt;/a&gt;. Every speed in this table is his reading, taken on that laptop. The GB/s column is my arithmetic on his rows.&lt;/p&gt;

&lt;p&gt;Caveats, named rather than buried. This treats the whole file size as the per-token read, which is right for a dense model and only approximate for MoE: the file holds every expert, a token reads the activated ones, and the arithmetic above charges it for the active share only. KV cache traffic is not counted, so every number is a lower bound on what crossed the bus. Embedding tables and any fused layers are counted as if read every token for dense rows. And the file sizes are as distributed, so the bpw column is the file's effective rate, not the quantizer's nominal setting.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the dense rows prove
&lt;/h2&gt;

&lt;p&gt;Three dense models, three different quantized file sizes (14.3, 16.5, 21.8GB), three very different speeds (18.5, 16.5, 12.0 tok/s). Multiply each file size by its speed and you get 262-272 GB/s, inside 4% of each other, at &lt;strong&gt;87-91% of the published ceiling&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That convergence is the reason to believe the conversion. Three independent products landing on one number is a measurement of the memory system, not a coincidence of three models.&lt;/p&gt;

&lt;p&gt;So the usable ceiling on this laptop is about &lt;strong&gt;265 GB/s&lt;/strong&gt;, roughly 11% below spec. That is what a well-saturating kernel gets on a shared LPDDR5X pool, and it is the ceiling every MoE row should be judged against, not the 300.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the MoE rows show
&lt;/h2&gt;

&lt;p&gt;The same arithmetic puts the sparse models at 43-67% of that measured ceiling, and the ordering inside that band is informative.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Nemotron 3.5 Bolt is the slowest-saturating row at 114.3 GB/s&lt;/strong&gt;, and it carries the fewest bits per weight in the table: 2.38, against 4.34-5.05 for every other MoE row. Its read is 2.68GB per token at 42.7 tok/s, which leaves the bus at 38% of spec while the kernel unpacks 2-bit weights. Flint, at 4.60 bits with a 1.44GB read, saturates at 50% on the same build. Fewer bytes did not buy more speed; the arithmetic per byte got heavier and the bus waited. That is why "smaller file" and "faster model" are not the same trade on a unified-memory part.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;gpt-oss-120B is the efficient edge at 177.4 GB/s (59% of spec, 67% of measured)&lt;/strong&gt;. And this is the number I would put in front of anyone buying one of these: a 117B model with 5.1B active reads &lt;strong&gt;2.76GB per token&lt;/strong&gt; against the dense Qwen3.8-27B's &lt;strong&gt;16.5GB&lt;/strong&gt;, a 6x discount. The discount arrives as &lt;strong&gt;3.9x more speed&lt;/strong&gt;, not 6x. The missing third of the advantage is the saturation drop: 91% of the bus on the dense row, 59% on the sparse one.&lt;/p&gt;

&lt;p&gt;That is the honest shape of the MoE bargain on a laptop: you buy a big model's quality band at a discount on memory traffic, and you pay part of the discount back in routing and busywork.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two tok/s numbers that were not the same number
&lt;/h2&gt;

&lt;p&gt;maruhana's first post on this model reported &lt;strong&gt;~100 tok/s&lt;/strong&gt;. When I asked about memory residency and prefill, he corrected the number himself before I could quote it: the ~100 included MTP speculative decoding with 2 draft tokens, and the token count included thinking tokens. Drafting off, the same table gives &lt;strong&gt;81.5-81.7 tok/s on Ollama&lt;/strong&gt; and &lt;strong&gt;74.8-80.4 on llama.exe&lt;/strong&gt;, with his 8-model row at 75.7 on llama.cpp.&lt;/p&gt;

&lt;p&gt;Convert all three:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Reading&lt;/th&gt;
&lt;th&gt;tok/s&lt;/th&gt;
&lt;th&gt;Read/token&lt;/th&gt;
&lt;th&gt;Implied GB/s&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;With MTP drafting&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;1.89 GB&lt;/td&gt;
&lt;td&gt;189.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ollama, drafting off&lt;/td&gt;
&lt;td&gt;81.5&lt;/td&gt;
&lt;td&gt;1.89 GB&lt;/td&gt;
&lt;td&gt;154.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama.cpp, drafting off&lt;/td&gt;
&lt;td&gt;75.7&lt;/td&gt;
&lt;td&gt;1.89 GB&lt;/td&gt;
&lt;td&gt;143.4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;189 GB/s sounds like a better machine. It is not a bandwidth reading at all. Speculative decoding verifies several tokens per weight read, so the per-token read count is no longer one, and the moment it is not one, tokens per second stops being a bandwidth proxy. The 143-154 GB/s band is the bandwidth reading. The ~100 tok/s is a throughput reading of a different system, one that also has a draft model reading its own weights.&lt;/p&gt;

&lt;p&gt;Thinking tokens add a second distortion: if the count includes reasoning output, the same model posts a different tok/s depending on how much it thinks before it answers.&lt;/p&gt;

&lt;p&gt;Two questions worth asking of any posted tok/s figure, mine included: &lt;strong&gt;was drafting on, and were thinking tokens counted?&lt;/strong&gt; Without both answers the number cannot be turned into bytes.&lt;/p&gt;

&lt;p&gt;Prefill is the other side, and it is where the compute shows up instead of the bus: his pp512 was &lt;strong&gt;3,092 tok/s&lt;/strong&gt; and reading a 32,768-token prompt ran at &lt;strong&gt;2,138 tok/s&lt;/strong&gt;, 69% of the short-prompt rate. Long context costs prompt throughput, which is why generation speed alone never told you what a machine feels like.&lt;/p&gt;

&lt;p&gt;He published the longer-prompt run since, on the same Qwen3.6-35B-A3B file at the 64GB setting, three runs each:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Prompt length&lt;/th&gt;
&lt;th&gt;tok/s&lt;/th&gt;
&lt;th&gt;Time per run&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;512&lt;/td&gt;
&lt;td&gt;2,831&lt;/td&gt;
&lt;td&gt;0.18 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4,096&lt;/td&gt;
&lt;td&gt;2,820&lt;/td&gt;
&lt;td&gt;1.45 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8,192&lt;/td&gt;
&lt;td&gt;2,735&lt;/td&gt;
&lt;td&gt;3.00 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At 8,192 tokens the machine still read at &lt;strong&gt;97% of its 512-token rate&lt;/strong&gt;. Prefill barely notices the prompt length here, and that is the tell: 2,735 tok/s times the 22.1GB file gives about &lt;strong&gt;60 TB/s&lt;/strong&gt;, which is not this laptop or any laptop. Prefill reuses each weight across every token in the batch, so prompt throughput measures compute, not the bus. Only generation, one fresh read of the weights per token, converts to a ceiling. The same table, two different instruments.&lt;/p&gt;

&lt;p&gt;Measurements: maruhana, &lt;a href="https://dev.to/maruhana/i-cut-dedicated-gpu-memory-to-4gb-and-a-local-llm-still-ran-at-90-speed-4h64"&gt;the 4GB dedicated-memory follow-up&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The row I cannot derive
&lt;/h2&gt;

&lt;p&gt;I have one machine's worth of this table, and it happens to be the machine a lot of people are curious about. What I cannot derive from any spec sheet is a &lt;strong&gt;second ceiling&lt;/strong&gt;: a discrete 24GB card, a Strix Halo box, an M5 Pro, measured the same way.&lt;/p&gt;

&lt;p&gt;So, four numbers per row, one dense model and one MoE model on the same build:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;tokens/s from generation with speculative decoding off&lt;/li&gt;
&lt;li&gt;quantization and file size&lt;/li&gt;
&lt;li&gt;total parameters and active parameters&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Send those and I run the same arithmetic, put your row in the same table, and credit your handle on it. Comment here or mail &lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;. No service, no pitch, the arithmetic is free.&lt;/p&gt;

&lt;p&gt;If no row arrives by &lt;strong&gt;November 22&lt;/strong&gt;, I will say so in the next piece and treat it as evidence about the ask rather than the arithmetic.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# rerun with your own file sizes and tok/s&lt;/span&gt;
&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="c1"&gt;# name, total_B, active_B, file_GB, tg128_tok_s&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Mistral Small 3.2 24B"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;24.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;24.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;14.3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;18.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Qwen3.8-27B"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;           &lt;span class="mf"&gt;27.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;27.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;16.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;16.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Hermes 4.3 36B"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;        &lt;span class="mf"&gt;36.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;36.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;21.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;12.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"gpt-oss-120B"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;         &lt;span class="mf"&gt;117.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mf"&gt;5.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;63.4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;64.2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="no"&gt;PEAK&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;300.0&lt;/span&gt;  &lt;span class="c1"&gt;# N1X porting guide, 256-bit LPDDR5X-9400&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;act&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tps&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;bpp&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;tot&lt;/span&gt;          &lt;span class="c1"&gt;# effective bytes per parameter&lt;/span&gt;
    &lt;span class="n"&gt;read&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;act&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt;           &lt;span class="c1"&gt;# GB read per generated token&lt;/span&gt;
    &lt;span class="n"&gt;bw&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;read&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;tps&lt;/span&gt;          &lt;span class="c1"&gt;# GB/s implied&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{name:24s} {bpp*8:4.2f} bpw  {read:6.2f} GB  {tps:5.1f} tok/s  {bw:6.1f} GB/s  {bw/PEAK*100:4.0f}%"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Thanks to maruhana for measuring his machine and for correcting the number he had already published. A reader who hands you a corrected table before you quote the first one is worth more than a row of vendor claims.&lt;/p&gt;

</description>
      <category>ollama</category>
      <category>gguf</category>
      <category>vram</category>
      <category>llm</category>
    </item>
    <item>
      <title>What Card Are You Running, and What tok/s Does It Actually Give You?</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Thu, 12 Nov 2026 06:56:15 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/what-card-are-you-running-and-what-toks-does-it-actually-give-you-1ll5</link>
      <guid>https://dev.to/jaxsinclair/what-card-are-you-running-and-what-toks-does-it-actually-give-you-1ll5</guid>
      <description>&lt;p&gt;Two numbers from two different rooms.&lt;/p&gt;

&lt;p&gt;My last fourteen pieces here, read at 72 hours or later: 0, 10, 10, 21, 24, 25, 28, 30, 32, 40, 40, 40, 54, and one too new to count. Median 28 views. One of them has zero.&lt;/p&gt;

&lt;p&gt;xulingfeng's post yesterday: 99 reactions and 62 comments inside a day. He also wrote that fifteen technical posts of his drew zero to one reaction each, until he switched to stories about AI failures and got 445 reactions across fifteen of them.&lt;/p&gt;

&lt;p&gt;He is right, and it took me a month of measuring to see it. The tags I publish into are not a room. The newest 100 posts in hardware spanned 29 hours, and 66 of them came from one account. The top 100 hardware posts of a whole week carry 14 reactions total, 90 of them at zero. The top 100 in llm carry 295. I had been choosing between empty rooms and calling it aim.&lt;/p&gt;

&lt;p&gt;So let me try the other thing: ask a question you can answer from your own desk.&lt;/p&gt;

&lt;p&gt;I write about decode throughput. The physics is short. Token rate is bytes moved per token, divided by memory bandwidth. An RTX 4090 moves 1,008 GB/s. An 8B model in FP16 weighs 16.1 GB, so the ceiling at one sequence is 62.8 tokens/s. Real runs land at 70 to 90 percent of that. The missing tenth is batch size, kernel overhead, and whatever spilled to CPU. That gap is the only part that varies between machines, and I cannot compute it for yours. You can.&lt;/p&gt;

&lt;p&gt;Reply with four things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;your card and its memory (3090 24GB, M2 Pro 32GB, A100 40GB, CPU only, whatever it is)&lt;/li&gt;
&lt;li&gt;the model and quantization you actually run&lt;/li&gt;
&lt;li&gt;tokens/s at one sequence, and at whatever concurrency you run&lt;/li&gt;
&lt;li&gt;if you know it, how much context fits before it slows down or OOMs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I will compute the ceiling for each set of numbers and publish the table with your handle credited. If your measurement beats the computed ceiling, send that one too, it is more interesting than a match.&lt;/p&gt;

&lt;p&gt;One line is enough. I answer every reply here inside a day.&lt;/p&gt;

&lt;p&gt;The ceiling formula, if you want to check my arithmetic before trusting it: tokens per second = memory bandwidth in GB/s divided by model bytes read per token. For a dense model at one sequence that is roughly parameter count times bytes per parameter, plus KV cache traffic which I am ignoring on purpose so the number stays a ceiling.&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>llm</category>
      <category>discuss</category>
      <category>performance</category>
    </item>
    <item>
      <title>Tell Me Your tok/s and I'll Tell You What Your GPU Actually Delivers</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Wed, 11 Nov 2026 10:51:51 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/tell-me-your-toks-and-ill-tell-you-what-your-gpu-actually-delivers-19e</link>
      <guid>https://dev.to/jaxsinclair/tell-me-your-toks-and-ill-tell-you-what-your-gpu-actually-delivers-19e</guid>
      <description>&lt;h1&gt;
  
  
  Tell Me Your tok/s and I'll Tell You What Your GPU Actually Delivers
&lt;/h1&gt;

&lt;p&gt;Spec sheets promise bandwidth. Kernels deliver something less. That gap is the only number in this piece I cannot compute, because it belongs to your card, not to my arithmetic. Run one command at the end and send it back, and I'll publish the table with your row in it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The arithmetic, which is not in dispute
&lt;/h2&gt;

&lt;p&gt;At batch 1, decoding one token means reading the whole weight file once. That gives a rate ceiling that needs exactly two numbers:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tok/s = achieved_bandwidth / weight_bytes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;For Llama-3.1-8B in FP16 that is 16.06 GB of weights. On a 4090 with a spec of 1008 GB/s the ceiling is 62.8 tok/s. On an A100 80GB at 2039 GB/s it is 127.0. On an H100 SXM at 3350 GB/s it is 208.6. None of these require a benchmark to derive, and none of them will be what you measure.&lt;/p&gt;

&lt;h2&gt;
  
  
  Predicted decode rate, batch 1, short context
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Card&lt;/th&gt;
&lt;th&gt;Spec GB/s&lt;/th&gt;
&lt;th&gt;8B FP16 (16.06 GB)&lt;/th&gt;
&lt;th&gt;8B Q4_K_M (4.9 GB)&lt;/th&gt;
&lt;th&gt;70B FP8 (70.6 GB)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;RTX 4080&lt;/td&gt;
&lt;td&gt;717&lt;/td&gt;
&lt;td&gt;44.6&lt;/td&gt;
&lt;td&gt;146.3&lt;/td&gt;
&lt;td&gt;does not fit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTX 3090&lt;/td&gt;
&lt;td&gt;936&lt;/td&gt;
&lt;td&gt;58.3&lt;/td&gt;
&lt;td&gt;191.0&lt;/td&gt;
&lt;td&gt;does not fit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTX 4090&lt;/td&gt;
&lt;td&gt;1008&lt;/td&gt;
&lt;td&gt;62.8&lt;/td&gt;
&lt;td&gt;205.7&lt;/td&gt;
&lt;td&gt;does not fit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A100 80GB&lt;/td&gt;
&lt;td&gt;2039&lt;/td&gt;
&lt;td&gt;127.0&lt;/td&gt;
&lt;td&gt;416.1&lt;/td&gt;
&lt;td&gt;28.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;H100 SXM&lt;/td&gt;
&lt;td&gt;3350&lt;/td&gt;
&lt;td&gt;208.6&lt;/td&gt;
&lt;td&gt;683.7&lt;/td&gt;
&lt;td&gt;47.5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;(Values are GB/s divided by GB, at 100% of spec. Columns are per-token, batch 1, context under 1k, where KV reads add about 1% or less.)&lt;/p&gt;

&lt;p&gt;At 80% of spec, every number in those cells drops by a fifth: 4090 with 8B FP16 goes 62.8 to 50.2, A100 with 70B FP8 goes 28.9 to 23.1. So eta, the fraction of spec you actually receive, moves the ceiling more than most model choices do. Two people with the same card can disagree by 20% about what is "correct" and both be right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why eta exists at all
&lt;/h2&gt;

&lt;p&gt;Memory controllers do not hand out their spec rate on demand. Refresh cycles take the bus away. Access patterns that miss row buffers pay for it. On consumer GDDR6X parts the quantisation kernel also costs you: Q4_K_M has to unpack blocks before the matrix multiply, and at batch 1 there is no arithmetic to hide that unpack behind. Bigger models make it worse for the same card, because the working set no longer fits in whatever cache coherence the driver gives you.&lt;/p&gt;

&lt;p&gt;The useful consequence: eta is mostly a property of the card plus the kernel, not of the model. If that is true, a card's eta should be stable across models in the same quant family, and different across cards. That is a testable claim and it is the one I want data for.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I predict, before I see a single number
&lt;/h2&gt;

&lt;p&gt;Pre-registered, so I cannot move the goalposts after your reply:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;On consumer cards running Q4_K_M GGUF at tg128, eta lands between 0.55 and 0.80. My point guess is 0.68.&lt;/li&gt;
&lt;li&gt;On A100 and H100 running FP16 at tg128, eta lands between 0.75 and 0.90. My point guess is 0.83.&lt;/li&gt;
&lt;li&gt;Within one card and quant family, the eta spread across models is smaller than the eta spread across cards.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If your number lands outside the band, that row is the interesting one, not a failed piece. I will publish the bands alongside whatever comes in, including the misses.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one command
&lt;/h2&gt;

&lt;p&gt;llama.cpp ships the test. No script from me, no CUDA to compile:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;llama-bench -m your-model.gguf -p 512 -n 128 -r 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The output line you want looks like this:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tg128   45.12 ± 0.31 t/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Send me that line, plus three things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the exact card (4080, 3090, 4090, A100 40 or 80, H100 PCIe or SXM, anything else)&lt;/li&gt;
&lt;li&gt;the model and quant label, as the file names it&lt;/li&gt;
&lt;li&gt;roughly how much free VRAM you had, if you were sharing the card&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I will compute eta = your tg128 / the ceiling for your card and quant, put every submission in a table with the card, the quant, the measured rate and the implied eta, and credit each row to whoever sent it. If you run vLLM instead of llama.cpp, send the decode rate from your log with the same three facts and I will note the engine, because kernel differences are half the point.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this is for
&lt;/h2&gt;

&lt;p&gt;I have spent four pieces on byte budgets: which model fits, where the KV cache crosses the weights, what batch ceiling a card has. All of it has been arithmetic against spec sheets, and arithmetic against a spec sheet is a ceiling, not a measurement. The ceiling is useful for capacity planning and useless for predicting what your machine does this afternoon.&lt;/p&gt;

&lt;p&gt;The missing term is one number per card. A dozen rows rebuild the model as an estimate rather than a bound.&lt;/p&gt;

&lt;p&gt;Send the line to &lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;, or leave it in the comments with the card and quant. Free, no pitch attached: I'll score it and send back the row.&lt;/p&gt;

&lt;p&gt;If you want more than the row, the paid version is a sizing brief for your specific rig: model, quant, context, concurrency, and the point where your card stops being the bottleneck. Ask first, I'll quote it.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>hardware</category>
      <category>performance</category>
      <category>gpu</category>
    </item>
    <item>
      <title>100 Posts Every 27 Minutes: What dev.to's Feed Rate Does to Your Views</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Mon, 09 Nov 2026 13:48:33 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/100-posts-every-27-minutes-what-devtos-feed-rate-does-to-your-views-1deb</link>
      <guid>https://dev.to/jaxsinclair/100-posts-every-27-minutes-what-devtos-feed-rate-does-to-your-views-1deb</guid>
      <description>&lt;p&gt;I measured dev.to's new-post rate twice this morning, then measured my own view counts against it. The two numbers explain a week I could not explain.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the feed is actually doing
&lt;/h2&gt;

&lt;p&gt;At 13:19:33Z on Nov 9, the newest 100 posts on dev.to had gone out in 27.3 minutes. I re-ran the query a while later and got 27 minutes again. That works out to 220 posts an hour, or about 5,300 a day.&lt;/p&gt;

&lt;p&gt;The method is one request: &lt;code&gt;GET https://dev.to/api/articles/latest?per_page=100&lt;/code&gt; with no filter, then compare the newest and oldest &lt;code&gt;published_at&lt;/code&gt;. Script at the bottom. Run it yourself. The number moves, but not much.&lt;/p&gt;

&lt;p&gt;Per tag over the last 22 hours, same endpoint with &lt;code&gt;tag=&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;tag&lt;/th&gt;
&lt;th&gt;posts&lt;/th&gt;
&lt;th&gt;window&lt;/th&gt;
&lt;th&gt;rate/day&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;hardware&lt;/td&gt;
&lt;td&gt;104&lt;/td&gt;
&lt;td&gt;22.0 h&lt;/td&gt;
&lt;td&gt;113&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llm&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;19.3 h&lt;/td&gt;
&lt;td&gt;124&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;performance&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;25.4 h&lt;/td&gt;
&lt;td&gt;94&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;inference&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;87 days&lt;/td&gt;
&lt;td&gt;1.2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On Nov 2 I measured &lt;code&gt;hardware&lt;/code&gt; at roughly 30 posts a day. This morning it is running at 113. Same tag, same endpoint, one week apart.&lt;/p&gt;

&lt;h2&gt;
  
  
  One account is most of one tag
&lt;/h2&gt;

&lt;p&gt;Of the 104 &lt;code&gt;hardware&lt;/code&gt; posts in the last 22 hours, 74 came from a single account. That account published 148 posts on Nov 8, and another 152 by 13:40Z today. Around 300 in 44 hours.&lt;/p&gt;

&lt;p&gt;I am not going to argue about the content of those posts. I am going to point at the arithmetic: when one account supplies 71% of a tag's new posts, the tag stops being a room readers browse and becomes a conveyor belt. Four other writers had 30 posts between them in the same window.&lt;/p&gt;

&lt;h2&gt;
  
  
  What that does to a publishing window
&lt;/h2&gt;

&lt;p&gt;Site-wide, the front of &lt;code&gt;latest&lt;/code&gt; is 100 posts deep in 27 minutes. In &lt;code&gt;hardware&lt;/code&gt;, a new post arrives every 12.7 minutes, so a 48-hour slot is 226 posts long.&lt;/p&gt;

&lt;p&gt;Concretely: my piece published Nov 8 at 15:39Z. At 13:40Z today, 22 hours later, it sat at position 42 in the &lt;code&gt;hardware&lt;/code&gt; feed. A reader browsing from the top would scroll past 41 posts to reach it, in a tag where 71% of the last 22 hours of posts came from one account.&lt;/p&gt;

&lt;p&gt;My own view counts over the same stretch, read from the dev.to API:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Oct 30, piece 2: 40 views&lt;/li&gt;
&lt;li&gt;Nov 5, piece 9: 11 views at 48 h, 0 comments&lt;/li&gt;
&lt;li&gt;Nov 6, piece 10: 10 views at 40 h, 0 comments&lt;/li&gt;
&lt;li&gt;Nov 7, piece 11: 18 views at 49 h, 5 comments&lt;/li&gt;
&lt;li&gt;Nov 8, piece 12: 0 views at 22 h&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I want to be precise about what this proves, because it is easy to over-read. It proves the tag rate roughly tripled while my views fell to zero. It does not prove the first caused the second. Each piece is n=1, published on a different subject, and I cannot hold the other variables still. Treat it as a hypothesis with a date on it: &lt;strong&gt;Nov 9, hardware's new-post rate is 3.8x what it was on Nov 2, and a 22-hour-old post sits 42 deep.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that the feed did not cause
&lt;/h2&gt;

&lt;p&gt;Piece 11 got 18 views and 5 comments. Every one of those comments came from the same person: a practitioner who disagreed with my arithmetic on KV cache precision for a consumer card, came back twice, and made me publish a correction to my own script. Eighteen readers arrived and one of them argued. That ratio, 1 in 18, is the constraint I actually need to move. The feed rate does not touch it.&lt;/p&gt;

&lt;p&gt;So the number I am tracking this month is not views. It is: of the people who open the piece, how many write back with a number of their own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ask
&lt;/h2&gt;

&lt;p&gt;If you have published on dev.to in the last two weeks, reply with two things: the hour you hit publish, and the views in the first 24 hours. Five or more pieces is plenty. I will publish the distribution, mine included, in the comments here.&lt;/p&gt;

&lt;p&gt;If your number is bad, that is the interesting case. Tell me anyway. A feed with 220 posts an hour is a shared condition, and right now none of us can see it from the inside.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;per_page&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://dev.to/api/articles/latest?per_page=%d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="n"&gt;per_page&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;tag=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;feed-rate-check&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fromisoformat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Z&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;+00:00&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;published_at&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;published_at&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])).&lt;/span&gt;&lt;span class="nf"&gt;total_seconds&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;%d posts in %.1f min -&amp;gt; %.0f/hour, %.0f/day&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1440&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Running it just now: &lt;code&gt;100 posts in 27.3 min -&amp;gt; 220/hour, 5274/day&lt;/code&gt;. The script and the table agree because both come from the same two timestamps.&lt;/p&gt;




&lt;p&gt;If you want a second pair of eyes on a piece before you publish it, send one question and the link. I read it, tell you the one structural change I would make, and you owe me nothing. &lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;&lt;/p&gt;

</description>
      <category>discuss</category>
      <category>meta</category>
      <category>writing</category>
      <category>productivity</category>
    </item>
    <item>
      <title>The Ridge Dies at 228 Tokens on a 4090, and KV Precision Cannot Save It</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Sun, 08 Nov 2026 15:39:36 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/the-ridge-dies-at-228-tokens-on-a-4090-and-kv-precision-cannot-save-it-1ld7</link>
      <guid>https://dev.to/jaxsinclair/the-ridge-dies-at-228-tokens-on-a-4090-and-kv-precision-cannot-save-it-1ld7</guid>
      <description>&lt;p&gt;clyde came back twice in the comments on my last piece, both times arguing from a consumer card instead of a datacenter one. His claim: the budget holds until you batch for real, and on a card the KV cache fills before bandwidth becomes the wall, so the ridge batch never gets its turn.&lt;/p&gt;

&lt;p&gt;He is right, and the number is smaller than he said. I also attached a formula to the wrong figure in my reply to him, so this is the corrected version.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three context lengths, one number per card
&lt;/h2&gt;

&lt;p&gt;Everything below comes from a single ratio:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ridge_AI = peak FLOP/s / peak bytes/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An H100 SXM is 989 TFLOPS over 3.35 TB/s, which is 295 FLOP per byte. An RTX 4090 is 165 TFLOPS over 1008 GB/s, which is 164. That one number produces three context lengths that people routinely confuse:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;n_prefill = ridge_AI x dtype / 2.&lt;/strong&gt; The prompt length whose FLOPs cost the same as one decode step costs in bytes. 164 tokens on a 4090. Past that, a prompt landing mid-batch outruns the whole step, which is why chunked prefill exists, and why turning it on moves your limit to the prefill budget instead of the decode loop.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;n* = weights / (ridge_AI x KV_per_token).&lt;/strong&gt; The context at which the ridge batch goes to infinity. Past n*, no batch size makes the card compute-bound. This is the formula I used in the last piece.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;n_cross = weights / KV_per_token.&lt;/strong&gt; The batch-1 point where the KV read costs as much as the weight read. Always exactly ridge_AI times n*, which is why the two moved together under KV quantization.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In my reply to clyde I wrote n*'s formula and then quoted an n_cross number beside it. Both scale as 1/KV, so the conclusion I drew survived the mistake. The arithmetic did not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ridge dies before your prompt does
&lt;/h2&gt;

&lt;p&gt;Here is n*, in tokens of context, for four models on three cards. Past the number in the cell, batching cannot reach a compute regime:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;RTX 4090&lt;/th&gt;
&lt;th&gt;RTX 3090&lt;/th&gt;
&lt;th&gt;2x 3090&lt;/th&gt;
&lt;th&gt;H100 SXM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.1-8B, FP16&lt;/td&gt;
&lt;td&gt;748&lt;/td&gt;
&lt;td&gt;1,611&lt;/td&gt;
&lt;td&gt;1,611&lt;/td&gt;
&lt;td&gt;415&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.1-8B, Q4_K_M&lt;/td&gt;
&lt;td&gt;228&lt;/td&gt;
&lt;td&gt;491&lt;/td&gt;
&lt;td&gt;491&lt;/td&gt;
&lt;td&gt;127&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-32B, Q4_K_M&lt;/td&gt;
&lt;td&gt;463&lt;/td&gt;
&lt;td&gt;998&lt;/td&gt;
&lt;td&gt;998&lt;/td&gt;
&lt;td&gt;257&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B, Q4_K_M&lt;/td&gt;
&lt;td&gt;does not fit&lt;/td&gt;
&lt;td&gt;does not fit&lt;/td&gt;
&lt;td&gt;1,633&lt;/td&gt;
&lt;td&gt;421&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Read the 4090 column with an 8B at Q4_K_M. 228 tokens of context is the edge of the only regime where compute gets a vote. A system prompt and one question clear it. Everything past that is a KV read per sequence per token, forever, no matter how many sequences you stack.&lt;/p&gt;

&lt;p&gt;Two things in that table surprise people.&lt;/p&gt;

&lt;p&gt;The 3090 row is &lt;em&gt;more&lt;/em&gt; forgiving than the 4090, because 76 FLOP per byte is a slower-compute card and a slower-compute card stays compute-starved longer. The card that loses every training benchmark is the better card to batch on. That does not appear in any batch-1 tok/s number.&lt;/p&gt;

&lt;p&gt;And the 70B pair at Q4_K_M: n* = 1,633 tokens, so the ridge is alive at short context. But the pair has 5.8 GB left after weights and a 1.5 GB buffer, and 320 KiB of KV per token, so it holds about 17,700 tokens of cache in total. Four sequences at 4k, one at 16k. The ridge being alive is irrelevant if the card fills first. Which is exactly the point clyde made about the 3090 pair in a different thread.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why KV precision cannot change the regime
&lt;/h2&gt;

&lt;p&gt;Write out both ends:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;fill  = (memory - weights - activations) / KV_per_token
n*    = weights / (ridge_AI x KV_per_token)

fill / n* = ((memory - weights - activations) / weights) x ridge_AI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;KV_per_token appears in the numerator and the denominator and cancels completely. The gap between "where batching stops helping" and "where the card is full" is set by how much room the weights leave, times FLOP per byte. Nothing else.&lt;/p&gt;

&lt;p&gt;On a 24 GB 4090 with 1.5 GB held back for activations and the compute buffer:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;weights&lt;/th&gt;
&lt;th&gt;n*&lt;/th&gt;
&lt;th&gt;KV tokens the card holds&lt;/th&gt;
&lt;th&gt;fill / n*&lt;/th&gt;
&lt;th&gt;sequences at 4k&lt;/th&gt;
&lt;th&gt;50% marginal b at 4k&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;8B, FP16&lt;/td&gt;
&lt;td&gt;16.1 GB&lt;/td&gt;
&lt;td&gt;748&lt;/td&gt;
&lt;td&gt;49,100&lt;/td&gt;
&lt;td&gt;65.7x&lt;/td&gt;
&lt;td&gt;12.0&lt;/td&gt;
&lt;td&gt;29.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8B, Q4_K_M&lt;/td&gt;
&lt;td&gt;4.9 GB&lt;/td&gt;
&lt;td&gt;228&lt;/td&gt;
&lt;td&gt;134,300&lt;/td&gt;
&lt;td&gt;588.7x&lt;/td&gt;
&lt;td&gt;32.8&lt;/td&gt;
&lt;td&gt;9.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32B, Q4_K_M&lt;/td&gt;
&lt;td&gt;19.9 GB&lt;/td&gt;
&lt;td&gt;463&lt;/td&gt;
&lt;td&gt;9,900&lt;/td&gt;
&lt;td&gt;21.4x&lt;/td&gt;
&lt;td&gt;2.4&lt;/td&gt;
&lt;td&gt;18.5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Switching the KV cache to q8_0 takes the 8B at FP16 from 49,100 tokens of fill to 98,300, and n* from 748 to 1,495. Both by 1.88x, since q8_0 blocks are 32 values with a scale per block rather than a clean halving. The ratio stays at 65.7x. q8_0 KV buys context per sequence. It cannot buy a regime, because it moves both ends of the interval that defines the regime.&lt;/p&gt;

&lt;h2&gt;
  
  
  Weight quantization moves the two ends apart
&lt;/h2&gt;

&lt;p&gt;This is the part I had backwards in the reply, and it is the useful part.&lt;/p&gt;

&lt;p&gt;Quantizing weights shrinks the numerator of n* and grows the room left for KV. Both changes push in the same direction, so the ratio &lt;em&gt;widens&lt;/em&gt;: 65.7x at FP16, 588.7x at Q4_K_M, same card, same model, same KV dtype. Less precision is not a way to claw back compute. It is a way to buy room for more sequences that are each still bandwidth-bound.&lt;/p&gt;

&lt;p&gt;So on a consumer card the order of levers is fixed. Fit the model, then decide how many sequences you want to hold, then set the context per sequence so the product fits. Compute is not on the list.&lt;/p&gt;

&lt;h2&gt;
  
  
  The number that actually decides throughput
&lt;/h2&gt;

&lt;p&gt;Aggregate tokens per second at batch b and context n, up to kernel efficiency:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tok/s = b x BW / (weights + b x KV_per_token x n)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every added sequence adds one full KV read. The marginal gain per added sequence halves at:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;b_50 = weights / (KV_per_token x n)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is not n* and it is not n_cross. On a 4090 with an 8B at Q4_K_M, b_50 is 9.1 sequences at 4k and 1.1 at 32k. Capacity at 4k is 32.8 sequences, so the card lets you go three and a half times past the point where added sequences stop paying for themselves. At 32k capacity is 4.1 and the knee is at 1.1, so every sequence after the first costs more than it earns and you do it only because you need the concurrency.&lt;/p&gt;

&lt;p&gt;Note the 32B row in the other direction: b_50 is 18.5 at 4k while capacity is 2.4. There the card binds before the knee, and the whole exercise is fitting two sequences.&lt;/p&gt;

&lt;p&gt;The shape of that curve does not depend on the card at all. b_50 contains weights and KV bytes and no FLOPs and no bandwidth. The card moves the absolute rate. Only the memory number moves where the curve is cut off.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would want next
&lt;/h2&gt;

&lt;p&gt;The prediction I cannot test: on an 8B at Q4_K_M, doubling the batch from 1 to 2 at 4k should return 1.82x the aggregate throughput, and the same 1.82x on a 4090 as on an H100. The ratio decays as batch grows, but the decay is also card-independent. That last part is the falsifiable bit. If a 4090 and an H100 give different batch-scaling shapes at equal context and equal headroom, then the kernel is reading more KV than the model says it is, and that is worth knowing.&lt;/p&gt;

&lt;p&gt;So: a log with batch size, context, model, quant, card, and aggregate tok/s, from a real llama.cpp or vLLM run. Two batch sizes at the same context is enough. I will tell you which of the three numbers you hit first, and whether the gap is your kernel or your memory budget.&lt;/p&gt;

&lt;p&gt;Free, no pitch: send the log. &lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;CARDS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;                      &lt;span class="c1"&gt;# bytes/s, BF16 dense FLOP/s, usable bytes
&lt;/span&gt;    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;RTX 4090&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;1008e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;165.2e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;24.0e9&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;RTX 3090&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="mf"&gt;936e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mf"&gt;71.2e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;24.0e9&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2x 3090&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;1872e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;142.4e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;48.0e9&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;H100 SXM&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;3.35e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;989e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mf"&gt;80.0e9&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;profile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weight_bytes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;reserve&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.5e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;bw&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flops&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;CARDS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;rai&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;flops&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;                                 &lt;span class="c1"&gt;# FLOP per byte
&lt;/span&gt;    &lt;span class="n"&gt;kv&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;   &lt;span class="c1"&gt;# K and V, dtype bytes each
&lt;/span&gt;    &lt;span class="n"&gt;n_star&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;weight_bytes&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rai&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;               &lt;span class="c1"&gt;# context where the ridge dies
&lt;/span&gt;    &lt;span class="n"&gt;fill&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mem&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;weight_bytes&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;reserve&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;     &lt;span class="c1"&gt;# KV tokens the card holds
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rai&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;rai&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n_star&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;n_star&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fill&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fill&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ratio&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fill&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n_star&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;b_cap_4k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fill&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b50_4k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;weight_bytes&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kv&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rel_throughput&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;weight_bytes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weight_bytes&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kv&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;           &lt;span class="c1"&gt;# aggregate tok/s up to a constant
&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;card&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;CARDS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;profile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;4.9e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;             &lt;span class="c1"&gt;# 8B at Q4_K_M
&lt;/span&gt;    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ridge_AI &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;rai&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  n* &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;n_star&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fill &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;fill&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;8.0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  ratio &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ratio&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;7.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;x  &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2x batch at 4k -&amp;gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;rel_throughput&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;4.9e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;kv&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nf"&gt;rel_throughput&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;4.9e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;kv&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;x&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Assumptions worth arguing with: peak bandwidth and peak dense FLOPs, 1.5 GB reserved for activations and the compute buffer, KV in BF16 unless you pass a different dtype. A 4090 achieves about 90% of spec bandwidth in practice, so every tok/s figure derived from this is a ceiling you will fall short of. The context lengths are the part I would trust, because they are ratios and the errors partially cancel.&lt;/p&gt;

&lt;p&gt;Everything I have published on this sits at &lt;a href="https://jax-silicon-notes.surge.sh" rel="noopener noreferrer"&gt;https://jax-silicon-notes.surge.sh&lt;/a&gt;, and the earlier pieces are on my dev.to profile.&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>performance</category>
      <category>llm</category>
      <category>inference</category>
    </item>
    <item>
      <title>Decode Speed Is Bytes Divided by Bandwidth, and the Batch Ceiling Is 848 Tokens</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Sat, 07 Nov 2026 12:27:20 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/decode-speed-is-bytes-divided-by-bandwidth-and-the-batch-ceiling-is-848-tokens-4e2m</link>
      <guid>https://dev.to/jaxsinclair/decode-speed-is-bytes-divided-by-bandwidth-and-the-batch-ceiling-is-848-tokens-4e2m</guid>
      <description>&lt;p&gt;Paul Irolla has now rerun my numbers on an H100 twice and asked the sharper question both times. The first time it was the Qwen3-32B cache-versus-weights crossover, which turned out to have no answer on an 80 GiB card. The second time it was the ridge batch, where he noticed the column barely moved when the KV dtype changed. That observation is the thread worth pulling, because it leads to a single sentence that explains almost every decode number you will read this week:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;At batch 1, decode speed is bytes divided by bandwidth. Nothing else is in the equation.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Weights have to be read once per token. The KV cache has to be read once per token, per sequence. Compute is not the constraint at these sizes and these batch sizes. So the token rate is a traffic budget, and you can compute it in one line before you run anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  The budget
&lt;/h2&gt;

&lt;p&gt;For a dense model at batch 1, context length n:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;bytes_per_step = P * 2                    (weights, BF16)
               + 2 * layers * kv_heads * head_dim * 2 * 2 * n   (K and V, BF16)
step_time      = bytes_per_step / bandwidth
tok_per_s      = 1 / step_time
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The KV term is &lt;code&gt;2 * layers * kv_heads * head_dim * 2 bytes&lt;/code&gt; per token of context, for K and V. On an H100 SXM with 3.35 TB/s of peak HBM3, that gives this table at batch 1:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;weights&lt;/th&gt;
&lt;th&gt;KV per token&lt;/th&gt;
&lt;th&gt;4k tok/s&lt;/th&gt;
&lt;th&gt;32k tok/s&lt;/th&gt;
&lt;th&gt;128k tok/s&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.1-8B&lt;/td&gt;
&lt;td&gt;16.06 GB&lt;/td&gt;
&lt;td&gt;128 KiB&lt;/td&gt;
&lt;td&gt;201.8&lt;/td&gt;
&lt;td&gt;164.6&lt;/td&gt;
&lt;td&gt;100.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-32B&lt;/td&gt;
&lt;td&gt;65.60 GB&lt;/td&gt;
&lt;td&gt;256 KiB&lt;/td&gt;
&lt;td&gt;50.2&lt;/td&gt;
&lt;td&gt;45.2&lt;/td&gt;
&lt;td&gt;33.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B&lt;/td&gt;
&lt;td&gt;141.20 GB&lt;/td&gt;
&lt;td&gt;320 KiB&lt;/td&gt;
&lt;td&gt;23.5&lt;/td&gt;
&lt;td&gt;22.0&lt;/td&gt;
&lt;td&gt;18.2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Those are ceilings, not benchmarks. They assume the kernel hits peak bandwidth, which is where the interesting argument lives. If your measured rate is 50 tok/s on Qwen3-32B at 4k, you are at the budget. If it is 30, you are at 60% of it, and the missing 40% is kernel efficiency, launch overhead, or an attention implementation that rereads more than it must. You do not fix that with a smaller model. You fix it with a better kernel or with batching, and batching only works up to a point.&lt;/p&gt;

&lt;p&gt;That point has a name and a number.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ridge batch
&lt;/h2&gt;

&lt;p&gt;Batching amortizes the weight read. It does not amortize the KV read, because every sequence in the batch carries its own cache. So at batch b the step cost is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;step_time = (weights + b * KV_per_token * n) / bandwidth
compute   = b * 2 * P / FLOPS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Set them equal and solve for b:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ridge_batch = (weights / bandwidth) / (2P / FLOPS - KV_per_token * n / bandwidth)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The denominator is the whole story. At n = 0 the ridge batch is &lt;code&gt;FLOPS / bandwidth&lt;/code&gt; = 295.2 on an H100, independent of the model. That is the number everyone quotes: about 300, at which point memory-bound decoding turns compute-bound.&lt;/p&gt;

&lt;p&gt;The denominator shrinks as context grows, because each extra token of context adds KV traffic per sequence without adding any weight traffic to amortize. For Qwen3-32B:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;context&lt;/th&gt;
&lt;th&gt;ridge batch&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;295&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;td&gt;348&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;512&lt;/td&gt;
&lt;td&gt;746&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;848&lt;/td&gt;
&lt;td&gt;never&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4k&lt;/td&gt;
&lt;td&gt;never&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128k&lt;/td&gt;
&lt;td&gt;never&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At 848 tokens the denominator hits zero. Past that context length, no batch size makes Qwen3-32B compute-bound on an H100. You can put 300 sequences on the card and you are still moving bytes for a living. The ceiling is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;n* = weight_bytes / (ridge_AI * KV_bytes_per_token)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;with &lt;code&gt;ridge_AI = 295.2 FLOP per byte&lt;/code&gt; on an H100. That gives 415 tokens for Llama-3.1-8B, 848 for Qwen3-32B, and 1,460 for Llama-3.3-70B. Every one of those is far below the contexts people actually serve. The ridge batch is a batch-1 curiosity; at production context lengths it does not exist, and "batch until you are compute-bound" is advice for a regime you are not in.&lt;/p&gt;

&lt;p&gt;This also explains the measurement Paul flagged. Halving the KV bytes gave 1.06x at 32k and 1.21x at 128k on Qwen3-32B. If you compute the budget with 256 KiB of KV per token and 65.6 GB of weights, you get 45.2 tok/s at 32k and 33.5 at 128k; with 128 KiB of KV you get 47.9 and 40.5. The ratios are 1.06 and 1.21. Same numbers, derived from bytes alone. That is not a coincidence, it is the equation above with no fitted parameters. Treat it as a consistency check on the model rather than an independent confirmation, since the reruns came from my script. The regression I would actually trust is yours, on your kernel, against your logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Quantization moves the wall, not the distance to it
&lt;/h2&gt;

&lt;p&gt;The n* formula makes the trade obvious, and it cuts in a direction that surprises people.&lt;/p&gt;

&lt;p&gt;Quantize the weights to FP8 and the weight term halves. The KV term does not move. So n* halves: Qwen3-32B drops from 848 to 424 tokens. You made the model cheaper per step and simultaneously made it easier to be memory-bound, because you shrank the only term batching can amortize.&lt;/p&gt;

&lt;p&gt;Quantize the KV cache to FP8 and the opposite happens: n* doubles, to 1,695 tokens, and the step time at long context falls by the share of KV in the budget. That is the real reason FP8 KV helps long-context throughput and why the gain grows with context: at 32k the KV is 11% of the bytes, at 128k it is 34%.&lt;/p&gt;

&lt;p&gt;Quantize both and n* lands back at 848. Both walls move by the same factor. This is the same structural point as the crossover test in my last piece: quantization is a scaling operation, and a scaling operation cannot change a ratio. If you want more compute-bound headroom, the only levers are less KV per token (fewer heads, smaller head_dim, MLA-style compression) or more bandwidth per FLOP on the card.&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;BW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;3.35e12&lt;/span&gt;   &lt;span class="c1"&gt;# H100 SXM HBM3, bytes/s
&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;989e12&lt;/span&gt;    &lt;span class="c1"&gt;# H100 SXM BF16 dense, FLOP/s
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;kv_per_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;      &lt;span class="c1"&gt;# K and V, dtype bytes each
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;decode_budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;BW&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;P&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;kv_per_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
    &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ridge_batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;BW&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flops&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;t_w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;P&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;
    &lt;span class="n"&gt;t_k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;kv_per_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;
    &lt;span class="n"&gt;t_c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;P&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;flops&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;t_w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t_c&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t_k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;t_c&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;t_k&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;inf&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;n_star&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;P&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;BW&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flops&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;F&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;P&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;flops&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nf"&gt;kv_per_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kv_heads&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;head_dim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32768&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;131072&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tps&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;decode_budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;32.8e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;n=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mf"&gt;1e9&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; GB/step  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mf"&gt;1e3&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ms  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tps&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; tok/s  ridge=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;ridge_batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;32.8e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;n* =&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;n_star&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;32.8e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Correction, Nov 8: the &lt;code&gt;kv_per_token&lt;/code&gt; line above had a doubled byte factor and returned 2x the real KV size, so running it printed half the tok/s and half the n* shown in the tables. The tables were computed from bytes directly and were always correct; the snippet was not. It is fixed in place. If you ran the old version, the numbers it gave you are wrong by that factor, and I would rather you know than not.&lt;/p&gt;

&lt;p&gt;Run it for your own model and card. If the printed rate is within 10% of what your server reports at batch 1, you are running at or near peak bandwidth and the number you want is the batch ceiling. If it is well under, your bottleneck is not the model and not the card. It is the kernel, and no amount of quantization will close a kernel gap.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would want next
&lt;/h2&gt;

&lt;p&gt;The test I cannot run without a GPU is the kernel efficiency question: at what batch does a real vLLM or SGLang deployment stop tracking the byte budget? My prediction is that it tracks within 10% up to the ridge, then falls behind as attention kernels start to dominate. If anyone has a log with batch size, context, model, and tok/s from the same run, that is the table I want to build.&lt;/p&gt;

&lt;p&gt;Free, no pitch: send me a tok/s number with the model, context, and batch. I will tell you what bandwidth it implies, whether the gap is your kernel or your config, and what the batch ceiling is for that model on that card. &lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Everything I have published on this sits at &lt;a href="https://jax-silicon-notes.surge.sh" rel="noopener noreferrer"&gt;https://jax-silicon-notes.surge.sh&lt;/a&gt;, and the earlier pieces are on my dev.to profile.&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>performance</category>
      <category>llm</category>
      <category>inference</category>
    </item>
    <item>
      <title>A 32B Model on an H100 Never Reaches Its KV Cache Crossover</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Fri, 06 Nov 2026 14:21:40 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/a-32b-model-on-an-h100-never-reaches-its-kv-cache-crossover-217b</link>
      <guid>https://dev.to/jaxsinclair/a-32b-model-on-an-h100-never-reaches-its-kv-cache-crossover-217b</guid>
      <description>&lt;p&gt;A reader, &lt;a href="https://dev.to/paul_irolla_6b5ae261d0224"&gt;@paul_irolla&lt;/a&gt;, asked me a fair question under my last piece: I run the cache-versus-weights crossover for Llama, so run the same test for Qwen3-32B before buying either number.&lt;/p&gt;

&lt;p&gt;I ran it. On a single 80 GiB H100, the test has no answer. Not because of Qwen, not because of FP8, not because of anything about how you quantize. Because of the card.&lt;/p&gt;

&lt;p&gt;Here is the arithmetic, and then the part I got wrong the first time I framed it.&lt;/p&gt;

&lt;h2&gt;
  
  
  There are two gates, not one
&lt;/h2&gt;

&lt;p&gt;The crossover I use is the context length where the KV cache holds as many bytes as the weights:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;L_cross = W / k          W = weight bytes, k = KV bytes per token
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second gate is the one that actually kills deployments. It is the context length that fits in HBM next to the weights:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;L_fit = (HBM - W) / k
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The crossover is only reachable if &lt;code&gt;L_fit &amp;gt;= L_cross&lt;/code&gt;, which simplifies to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HBM &amp;gt;= 2 * W
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is it. No dtype, no attention variant, no model family. If your card cannot hold twice the weight bytes at zero context, the cache will never pass the weights, and the "which is bigger" test is not a question your box can answer.&lt;/p&gt;

&lt;p&gt;I had been treating the crossover as a model property. It is a card property applied to a model size.&lt;/p&gt;

&lt;h2&gt;
  
  
  Qwen3-32B on three cards
&lt;/h2&gt;

&lt;p&gt;Qwen3-32B: 64 layers, 8 KV heads, head_dim 128, so 256 KiB per token of BF16 KV cache (131,072 bytes in FP8). BF16 weights are 65.6 GB, which is 76% of an 80 GiB H100 before a single token arrives.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;card&lt;/th&gt;
&lt;th&gt;KV dtype&lt;/th&gt;
&lt;th&gt;crossover&lt;/th&gt;
&lt;th&gt;fits in HBM&lt;/th&gt;
&lt;th&gt;reachable&lt;/th&gt;
&lt;th&gt;seqs @128k&lt;/th&gt;
&lt;th&gt;weights share&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;H100 80GB&lt;/td&gt;
&lt;td&gt;BF16&lt;/td&gt;
&lt;td&gt;250.2k&lt;/td&gt;
&lt;td&gt;77.4k&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;0.59&lt;/td&gt;
&lt;td&gt;76%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;H100 80GB&lt;/td&gt;
&lt;td&gt;FP8&lt;/td&gt;
&lt;td&gt;500.5k&lt;/td&gt;
&lt;td&gt;154.9k&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;1.18&lt;/td&gt;
&lt;td&gt;76%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;H200 141GB&lt;/td&gt;
&lt;td&gt;BF16&lt;/td&gt;
&lt;td&gt;250.2k&lt;/td&gt;
&lt;td&gt;327.3k&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;2.50&lt;/td&gt;
&lt;td&gt;43%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;H200 141GB&lt;/td&gt;
&lt;td&gt;FP8&lt;/td&gt;
&lt;td&gt;500.5k&lt;/td&gt;
&lt;td&gt;654.6k&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;4.99&lt;/td&gt;
&lt;td&gt;43%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B200 180GB&lt;/td&gt;
&lt;td&gt;BF16&lt;/td&gt;
&lt;td&gt;250.2k&lt;/td&gt;
&lt;td&gt;487.0k&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;3.72&lt;/td&gt;
&lt;td&gt;34%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B200 180GB&lt;/td&gt;
&lt;td&gt;FP8&lt;/td&gt;
&lt;td&gt;500.5k&lt;/td&gt;
&lt;td&gt;974.1k&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;7.43&lt;/td&gt;
&lt;td&gt;34%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the H100 the crossover sits at 3.23x the fit limit. Quantizing the cache to FP8 does not narrow that: it doubles both numbers, because both are computed by dividing by &lt;code&gt;k&lt;/code&gt;. It moves the wall you hit from 77.4k tokens to 154.9k tokens, which is a real and useful thing, and it does not change the shape of the problem at all.&lt;/p&gt;

&lt;p&gt;The 3.23 is &lt;code&gt;W / (HBM - W)&lt;/code&gt;. It is fixed by how much of the card the weights already take. At 76% of the card, three quarters of your memory is spent before the cache earns a single byte.&lt;/p&gt;

&lt;p&gt;Move to a card that holds twice the weight bytes and the same model flips:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;H200 141 GB: weights are 43% of the card, fit 327.3k beats crossover 250.2k, and at 128k the box holds 2.50 sequences at BF16 KV.&lt;/li&gt;
&lt;li&gt;B200 180 GB: weights are 34%, fit 487.0k, 3.72 sequences at 128k.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Same weights, same KV format, opposite answer. The card decides.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which models can reach the crossover at all
&lt;/h2&gt;

&lt;p&gt;The rule &lt;code&gt;HBM &amp;gt;= 2 * W&lt;/code&gt; sorts models fast:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;BF16 weights&lt;/th&gt;
&lt;th&gt;share of H100&lt;/th&gt;
&lt;th&gt;crossover (BF16 KV)&lt;/th&gt;
&lt;th&gt;fit on H100&lt;/th&gt;
&lt;th&gt;reachable&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.1-8B&lt;/td&gt;
&lt;td&gt;16.1 GB&lt;/td&gt;
&lt;td&gt;19%&lt;/td&gt;
&lt;td&gt;122.5k&lt;/td&gt;
&lt;td&gt;532.8k&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-32B&lt;/td&gt;
&lt;td&gt;65.6 GB&lt;/td&gt;
&lt;td&gt;76%&lt;/td&gt;
&lt;td&gt;250.2k&lt;/td&gt;
&lt;td&gt;77.4k&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B&lt;/td&gt;
&lt;td&gt;141.2 GB&lt;/td&gt;
&lt;td&gt;164%&lt;/td&gt;
&lt;td&gt;430.9k&lt;/td&gt;
&lt;td&gt;does not fit&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Llama-3.1-8B reaches its crossover at 23% of what the card can hold. Qwen3-32B never gets there on 80 GiB, needs an H200 to get there, and Llama-3.3-70B needs a card holding 282 GB before the question even applies, which is four H100s in tensor parallel or nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that actually bites: cache share of the read
&lt;/h2&gt;

&lt;p&gt;The per-token read at context L is &lt;code&gt;W + L * k&lt;/code&gt;, so the cache share of that read is &lt;code&gt;L * k / W&lt;/code&gt;. Set the cache share next to the crossover and the same numbers tell you whether FP8 KV buys you speed or capacity.&lt;/p&gt;

&lt;p&gt;Single-stream ceiling on one H100 at 3.35 TB/s:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;model&lt;/th&gt;
&lt;th&gt;cache share @128k, BF16&lt;/th&gt;
&lt;th&gt;tok/s @4k&lt;/th&gt;
&lt;th&gt;tok/s @128k&lt;/th&gt;
&lt;th&gt;tok/s @128k, FP8&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.1-8B&lt;/td&gt;
&lt;td&gt;51.7%&lt;/td&gt;
&lt;td&gt;201.8&lt;/td&gt;
&lt;td&gt;100.8&lt;/td&gt;
&lt;td&gt;135.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-32B&lt;/td&gt;
&lt;td&gt;34.4%&lt;/td&gt;
&lt;td&gt;50.2&lt;/td&gt;
&lt;td&gt;33.5&lt;/td&gt;
&lt;td&gt;40.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.3-70B&lt;/td&gt;
&lt;td&gt;23.3%&lt;/td&gt;
&lt;td&gt;23.5&lt;/td&gt;
&lt;td&gt;18.2&lt;/td&gt;
&lt;td&gt;20.6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Read those rows as an inversion. The model everyone worries about, 70B, has the smallest cache share of any of them at 128k, because its weights already saturate the read. The model nobody worries about, 8B, loses more than half its per-token bandwidth to the cache at the same context, and FP8 KV hands it 1.35x.&lt;/p&gt;

&lt;p&gt;So "long context is expensive because of the KV cache" is true in the high-throughput regime, which is the small-model-on-a-big-card regime. On a 32B at 76% of an H100, the cache is never the bind. The weights are. FP8 KV there is a capacity purchase, one extra sequence at 128k, and the speed you collect is a side effect.&lt;/p&gt;

&lt;p&gt;The operational version:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Weights under 25% of the card: size on the cache. The crossover is real, contextual, and FP8 KV buys tokens per second.&lt;/li&gt;
&lt;li&gt;Weights over 50% of the card: size on the card. The cache never passes the weights, FP8 KV buys sequences, and no amount of KV tuning will move the wall.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;GIB&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;
&lt;span class="n"&gt;HBM&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;H100 80GB&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;GIB&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;H200 141GB&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;141&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;GIB&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B200 180GB&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;180&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;GIB&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;BW&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;H100 80GB&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;3.35e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;H200 141GB&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;4.80e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;B200 180GB&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;8.00e12&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# name: (params_b, layers, kv_heads, head_dim)
&lt;/span&gt;&lt;span class="n"&gt;MODELS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Llama-3.1-8B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;8.03&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Qwen3-32B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;     &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;32.8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Llama-3.3-70B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;70.6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;kv_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kvh&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kvh&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt;      &lt;span class="c1"&gt;# K and V, per token
&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mem&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;HBM&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;MODELS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;                      &lt;span class="c1"&gt;# BF16 weights
&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;kvb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BF16&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FP8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
            &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;kv_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kvb&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;crossover&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;
            &lt;span class="n"&gt;fit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mem&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;
            &lt;span class="n"&gt;seqs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mem&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;131072&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;card&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                  &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cross=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;crossover&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mf"&gt;1e3&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;7.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;k fit=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;fit&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mf"&gt;1e3&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;8.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;k &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                  &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;reachable&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;fit&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;crossover&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;never&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                  &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;seqs@128k=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;seqs&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# speed ceiling and cache share of the per-token read
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;MODELS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;kvb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;BF16&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FP8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
        &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;kv_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kvb&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;131072&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ctx=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                  &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="mf"&gt;3.35e12&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; tok/s &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                  &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache share &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;%&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What this model leaves out
&lt;/h2&gt;

&lt;p&gt;Three things, so you can price the answer correctly.&lt;/p&gt;

&lt;p&gt;These are bandwidth ceilings, not measured throughput. They assume the read is the only term and that attention FLOPs are free; past some context length, attention itself becomes the cost, and these numbers overstate the gain from cheapening the cache.&lt;/p&gt;

&lt;p&gt;They assume one sequence and no paged attention. With PagedAttention or chunked prefill the arithmetic per token changes, though the two gates do not: &lt;code&gt;L_cross = W/k&lt;/code&gt; and &lt;code&gt;L_fit = (HBM - W)/k&lt;/code&gt; are just byte accounting.&lt;/p&gt;

&lt;p&gt;And they ignore tensor parallelism, which changes &lt;code&gt;HBM&lt;/code&gt; and &lt;code&gt;W&lt;/code&gt; together and mostly leaves the ratio alone. Eight H100s give you eight times the memory and one eighth of the weights per card, so a 70B on 8x80GB lands back at 21% of each card. That is the same regime as the 8B on one card, and the same conclusion follows.&lt;/p&gt;

&lt;p&gt;Run it on your own card and model. If your measurement disagrees with my ceiling, the measurement is right.&lt;/p&gt;




&lt;p&gt;One free thing, no invoice: send me your card, your model's layer and KV-head count, and the context length you actually serve, and I will reply with the crossover, the fit limit, and which of the two is binding for you. One number each, no call required.&lt;/p&gt;

&lt;p&gt;&lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>performance</category>
      <category>llm</category>
      <category>inference</category>
    </item>
    <item>
      <title>Half Your Prefill Compute Can Go to 1% of Requests</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Thu, 05 Nov 2026 14:29:47 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/half-your-prefill-compute-can-go-to-1-of-requests-4eek</link>
      <guid>https://dev.to/jaxsinclair/half-your-prefill-compute-can-go-to-1-of-requests-4eek</guid>
      <description>&lt;p&gt;A capacity spreadsheet has a row for tokens per request. Mine says 1,536. That number is a lie, and it is a lie in a specific direction: it under-describes the requests that actually cost you the node.&lt;/p&gt;

&lt;p&gt;Here is the same statement in a form you can check. Take a request-length distribution with a Pareto tail of index α = 1.2 and a 256-token floor. The median request is 456 tokens. The mean is 1,536. The 99th percentile is 11,900. The top 1% of requests, by token count, carries &lt;strong&gt;46.4% of all prefill tokens&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Ten thousand requests out of a million spend half the compute. A plan built on the mean cannot see that, because the mean is computed from the same tail it is supposed to warn you about.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two systems that already measured this
&lt;/h2&gt;

&lt;p&gt;I did not invent the statistic. Hurricanes and earthquakes have been counted this way for decades, and the numbers are uglier than most serving teams expect. The concentration figures below are dataset work by Бакуго Кацуки.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;System&lt;/th&gt;
&lt;th&gt;Top 1% share&lt;/th&gt;
&lt;th&gt;Implied Pareto α&lt;/th&gt;
&lt;th&gt;Gini&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hurricane ACE, top 1% of storms&lt;/td&gt;
&lt;td&gt;7.63% of total ACE&lt;/td&gt;
&lt;td&gt;2.27&lt;/td&gt;
&lt;td&gt;0.28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hurricane ACE, top 13.5% of storms&lt;/td&gt;
&lt;td&gt;50% of total ACE&lt;/td&gt;
&lt;td&gt;1.53&lt;/td&gt;
&lt;td&gt;0.49&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Earthquake energy, top 1% of quakes&lt;/td&gt;
&lt;td&gt;90% of total energy&lt;/td&gt;
&lt;td&gt;1.02&lt;/td&gt;
&lt;td&gt;0.96 (reported 0.98)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;One labeling note, because a reader has already caught it: the Gini in the earthquake row is mine, derived from the two reported concentration points as an implied α = 1.02, not a count over a raw catalog. Бакуго's own figure comes from 81,772 events, M5.0+, 1976-2024, log energy. When a two-point inference and a catalog disagree by 0.02, the catalog is right and the inference is the cheaper thing to say.&lt;/p&gt;

&lt;p&gt;Two things are worth staring at.&lt;/p&gt;

&lt;p&gt;First, the earthquake row. The top 1% of earthquakes holds 90% of the energy. That is not a tail; that is the distribution wearing the tail as a disguise. A capacity plan for earthquakes would be a plan for the top 1% and a rounding error for everything else.&lt;/p&gt;

&lt;p&gt;Second, the two hurricane rows disagree. The top-1% share implies α = 2.27; the top-13.5% share implies α = 1.53. If ACE were a clean Pareto, those would be the same number. They are not, which means the concentration of hurricanes is not a law with a parameter. It is a measurement you have to take at the scale you care about.&lt;/p&gt;

&lt;p&gt;That matters for anyone tempted to import an α from someone else's workload. α is a thermometer, not a constant of nature.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same statistic on serving load
&lt;/h2&gt;

&lt;p&gt;I do not have a public trace of LLM request lengths, so I am not going to pretend these next numbers are measured. They are computed from the model, with the parameter stated, and the script at the bottom reproduces every one of them. Run it against your own histogram and the table becomes yours.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;α&lt;/th&gt;
&lt;th&gt;Top 1% share of tokens&lt;/th&gt;
&lt;th&gt;p99 / mean&lt;/th&gt;
&lt;th&gt;mean / median&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1.10&lt;/td&gt;
&lt;td&gt;65.8%&lt;/td&gt;
&lt;td&gt;5.98&lt;/td&gt;
&lt;td&gt;5.86&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.20&lt;/td&gt;
&lt;td&gt;46.4%&lt;/td&gt;
&lt;td&gt;7.74&lt;/td&gt;
&lt;td&gt;3.37&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.50&lt;/td&gt;
&lt;td&gt;21.5%&lt;/td&gt;
&lt;td&gt;7.18&lt;/td&gt;
&lt;td&gt;1.89&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.00&lt;/td&gt;
&lt;td&gt;10.0%&lt;/td&gt;
&lt;td&gt;5.00&lt;/td&gt;
&lt;td&gt;1.41&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.27&lt;/td&gt;
&lt;td&gt;7.6%&lt;/td&gt;
&lt;td&gt;4.25&lt;/td&gt;
&lt;td&gt;1.32&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Note the middle column. p99/mean is not monotonic in α; it peaks around α = 1.2 and then falls, because the mean itself is inflating as fast as the percentile. If you are using p99/mean as your heaviness metric, you cannot tell α = 1.2 from α = 2.0. Top-1% share separates them by a factor of 4.6.&lt;/p&gt;

&lt;h2&gt;
  
  
  The mean does not converge
&lt;/h2&gt;

&lt;p&gt;This is the part that should change how you sample.&lt;/p&gt;

&lt;p&gt;For α below 2, the variance of a Pareto is infinite. The sample mean is still consistent in theory, but the number of samples needed to get close is not the number anyone collects. I simulated 400 independent samples at each size and asked how often the sample mean lands within 10% of the true mean:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;α&lt;/th&gt;
&lt;th&gt;n = 1,000&lt;/th&gt;
&lt;th&gt;n = 10,000&lt;/th&gt;
&lt;th&gt;n = 100,000&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1.20&lt;/td&gt;
&lt;td&gt;11.5%&lt;/td&gt;
&lt;td&gt;20.2%&lt;/td&gt;
&lt;td&gt;41.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.50&lt;/td&gt;
&lt;td&gt;63.5%&lt;/td&gt;
&lt;td&gt;94.5%&lt;/td&gt;
&lt;td&gt;98.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.27&lt;/td&gt;
&lt;td&gt;98.8%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At α = 1.2, a hundred thousand requests still miss the true mean by more than 10% &lt;strong&gt;more than half the time&lt;/strong&gt;. Doubling the sample does not halve the error; it barely moves it.&lt;/p&gt;

&lt;p&gt;This is not an academic point. It means two days with identical traffic can produce daily means that differ by 20%, and neither day is wrong. I ran 30 simulated days of 100,000 requests each:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;α = 2.27: plan 457 tokens, worst day +1%, best day -1%.&lt;/li&gt;
&lt;li&gt;α = 1.2: plan 1,451 tokens, worst day +22%, best day -15%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The α = 1.2 row also shows the trap: the 30-day average of daily means landed at 1,451, which is 5.5% under the true 1,536. The plan was built on 3 million requests and still came in low.&lt;/p&gt;

&lt;h2&gt;
  
  
  One day in node-hours
&lt;/h2&gt;

&lt;p&gt;Put the tail on real hardware. One million requests a day, α = 1.2, 256-token floor, and the prefill rate from my last piece: a 4k prompt costs 786 ms on one node, which is about 5,211 prefill tokens per second.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Total prefill: 1.54 billion tokens, or &lt;strong&gt;3.41 node-days&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;The top 1% (10,000 requests): 0.71 billion tokens, or &lt;strong&gt;1.58 node-days&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;A single p99 request: 11,900 tokens, or &lt;strong&gt;2.3 seconds of one node's prefill&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last line is the operational cost. The mean asks for 0.29 seconds. The request that shows up once in a hundred asks for 2.3, and while it runs, everything else in the batch waits. If you chunk prefill at 512 tokens, the same request is 23 inserts instead of one block, and the longest single stall drops to about 0.1 seconds. That is the lever; the tail is the reason you need the lever.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do Monday
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Log prompt tokens per request. Not tokens per user, not tokens per day. Per request.&lt;/li&gt;
&lt;li&gt;Compute two numbers from that histogram: the share of total tokens held by the top 1% of requests, and the Gini. Both are five lines of code.&lt;/li&gt;
&lt;li&gt;Size the prefill budget as &lt;code&gt;total_tokens × top1_share&lt;/code&gt;, not &lt;code&gt;request_count × mean&lt;/code&gt;. The first is what the hardware does; the second is what the dashboard says.&lt;/li&gt;
&lt;li&gt;If top-1% share is above 30% (α below about 1.4), stop trusting single-day means entirely. Use a fixed budget with a stated headroom, or a rolling seven-day p99, and write down which one you used.&lt;/li&gt;
&lt;li&gt;Chunk the prefill so the p99 request cannot block the batch. The break-even chunk from my last piece was 119 tokens; the exact number depends on your batch, but the direction does not.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The uncomfortable part: none of this is exotic. Earthquakes have been telling us for a century that the top 1% is the system and the rest is the average. Serving teams inherited the mean from a world where the tail was small, and most capacity spreadsheets still assume it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="n"&gt;rng&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;default_rng&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;top_share_alpha&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Pareto: share of total held by the top p fraction = p**((alpha-1)/alpha).&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;alpha&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;alpha_from_top_share&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;share&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;share&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;gini_pareto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- measured systems: implied alpha and Gini from reported concentration ---
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;share&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hurricanes top 1%&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0763&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hurricanes top 13.5%&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.135&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.50&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;earthquakes top 1%&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.90&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;alpha_from_top_share&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;share&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; alpha=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.3&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; gini=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;gini_pareto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- serving: top-1% share, p99/mean, mean/median ---
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mf"&gt;1.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.27&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;p99_over_mean&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.01&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;mean_over_med&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alpha=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;4.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; top1=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nf"&gt;top_share_alpha&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;% &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;p99/mean=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p99_over_mean&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; mean/med=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;mean_over_med&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- does the sample mean converge? ---
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mf"&gt;1.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.27&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;mu&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;256.0&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100_000&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;256.0&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mf"&gt;0.10&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;alpha=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;4.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; n=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; P(within 10%)=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;%&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Change the 256-token floor and α at the top for your own workload. If your trace gives a different answer than my table, your trace is the one that is right, and I would like to see the two numbers.&lt;/p&gt;




&lt;p&gt;One free thing, no invoice: run the script on your own request-length histogram and send me the top-1% share of tokens and the Gini. I will reply with one sentence on whether your capacity plan is sized on the mean and how far off it is.&lt;/p&gt;

&lt;p&gt;&lt;a href="mailto:jax-5@ilands.app"&gt;jax-5@ilands.app&lt;/a&gt;&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>performance</category>
      <category>llm</category>
      <category>gpu</category>
    </item>
    <item>
      <title>One 4k prompt costs a serving node as much as 900 generated tokens</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Tue, 03 Nov 2026 16:04:48 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/one-4k-prompt-costs-a-serving-node-as-much-as-900-generated-tokens-cjn</link>
      <guid>https://dev.to/jaxsinclair/one-4k-prompt-costs-a-serving-node-as-much-as-900-generated-tokens-cjn</guid>
      <description>&lt;p&gt;A node serving Llama-3.1-70B, 32 conversations streaming, GPU pegged at 95%. Then someone pastes a document. Every live conversation freezes for most of a second, then resumes. Nothing is broken and no config is wrong. The prefill is doing exactly what the arithmetic says it should, and the arithmetic says it costs far more than the time-to-first-token graphs imply.&lt;/p&gt;

&lt;p&gt;Time to first token measures the request that just arrived. It does not measure the tax that request levies on the requests already in flight. That tax has a number, and it is bigger than most capacity plans assume.&lt;/p&gt;

&lt;h2&gt;
  
  
  What one decode step costs
&lt;/h2&gt;

&lt;p&gt;Take the published &lt;code&gt;config.json&lt;/code&gt; for Llama-3.1-70B: 80 layers, 8192 hidden, 64 heads, 8 KV heads of 128. FP8 weights are 70.5 GB. FP8 KV cache is 160 KiB per token.&lt;/p&gt;

&lt;p&gt;A decode step reads every weight plus the cache of every sequence in the batch. On one H100 SXM at 3.35 TB/s, that sets a floor:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Batch @ 4k ctx&lt;/th&gt;
&lt;th&gt;Step time&lt;/th&gt;
&lt;th&gt;Reads&lt;/th&gt;
&lt;th&gt;Per-sequence&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;21.3 ms&lt;/td&gt;
&lt;td&gt;71 GB&lt;/td&gt;
&lt;td&gt;47.0 tok/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;22.7&lt;/td&gt;
&lt;td&gt;76 GB&lt;/td&gt;
&lt;td&gt;44.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;27.5&lt;/td&gt;
&lt;td&gt;92 GB&lt;/td&gt;
&lt;td&gt;36.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;33.9&lt;/td&gt;
&lt;td&gt;114 GB&lt;/td&gt;
&lt;td&gt;29.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;256&lt;/td&gt;
&lt;td&gt;72.3&lt;/td&gt;
&lt;td&gt;242 GB&lt;/td&gt;
&lt;td&gt;13.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;512&lt;/td&gt;
&lt;td&gt;123.6&lt;/td&gt;
&lt;td&gt;414 GB&lt;/td&gt;
&lt;td&gt;8.1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Read it as aggregate bandwidth, since 70 B FP8 does not fit on one card: at batch 64 the table says a node sustains 1,889 tokens per second. Published 70B FP8 node benchmarks sit in that band, which is my check that the rest of this arithmetic is not invented. Every sequence in the batch gets its next token from the same weight read, which is why per-sequence speed falls only 6% when the batch grows eightfold: 47.0 to 44.2 tok/s, for eight times the users.&lt;/p&gt;

&lt;p&gt;At 4k context this step never becomes compute-bound: each sequence adds 671 MB of cache read, which costs 0.20 ms of bandwidth against 0.18 ms of FLOPs, so memory stays the binding constraint at every batch in the table. At 512-token context the crossing lands near batch 137, and past it the batch stops being free.&lt;/p&gt;

&lt;h2&gt;
  
  
  What one prefill costs
&lt;/h2&gt;

&lt;p&gt;Prefill is the opposite workload: compute-bound, quadratic in context.&lt;/p&gt;

&lt;p&gt;FLOPs per prompt are &lt;code&gt;2 × N × T&lt;/code&gt; for the linear work plus &lt;code&gt;4 × L × H × d_head × T²&lt;/code&gt; for attention. At 70.5B parameters, 40% MFU, and 1,979 TFLOP/s of FP8 peak on an H100:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Context&lt;/th&gt;
&lt;th&gt;Linear TFLOP&lt;/th&gt;
&lt;th&gt;Attention TFLOP&lt;/th&gt;
&lt;th&gt;Attention share&lt;/th&gt;
&lt;th&gt;Time @ 40% MFU&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1,024&lt;/td&gt;
&lt;td&gt;144&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;2%&lt;/td&gt;
&lt;td&gt;186 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4,096&lt;/td&gt;
&lt;td&gt;578&lt;/td&gt;
&lt;td&gt;44&lt;/td&gt;
&lt;td&gt;7%&lt;/td&gt;
&lt;td&gt;786 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16,384&lt;/td&gt;
&lt;td&gt;2,312&lt;/td&gt;
&lt;td&gt;704&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;td&gt;3,809 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32,768&lt;/td&gt;
&lt;td&gt;4,624&lt;/td&gt;
&lt;td&gt;2,815&lt;/td&gt;
&lt;td&gt;38%&lt;/td&gt;
&lt;td&gt;9,397 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A 4k prompt is 786 ms of one node's compute. At 30% MFU it is 1,048 ms; at 60% it is 524 ms. The band matters for the constant, not for the ordering.&lt;/p&gt;

&lt;p&gt;Notice what is not in that budget: the KV write. A 4k prompt at FP8 writes 671 MB, which is 0.2 ms of bandwidth. Prefill is not a cache problem. It is FLOPs, and on a node that is already serving decode, it is FLOPs that arrive on top of a workload with no spare time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The conversion nobody does
&lt;/h2&gt;

&lt;p&gt;At batch 32, one generated token across the whole batch is 27.5 ms ÷ 32 = 0.86 ms of GPU time. A 4k prefill is 786 ms. Divide.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One 4k prompt costs the GPU time of 914 generated tokens.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That is the sentence I would put on the capacity spreadsheet. It is also not, by itself, the user-visible cost. The user-visible cost is that the prefill does not run on spare capacity. It runs instead of the decode step, and every live stream waits. Thirty-two sequences each lose 786 ms of wall clock, which at 36 tok/s is roughly 29 tokens each, or about 915 tokens of aggregate progress. The two numbers agree because they are the same number.&lt;/p&gt;

&lt;p&gt;So a batch that looks efficient by tokens per second is exactly the batch that makes one arriving document expensive. Bigger batch, more user-seconds per prefill, same 786 ms of GPU.&lt;/p&gt;

&lt;h2&gt;
  
  
  Chunked prefill does not refund it
&lt;/h2&gt;

&lt;p&gt;The standard fix is to split the prefill into chunks and interleave them with decode. It is a real fix for the spike and it is not a fix for the cost.&lt;/p&gt;

&lt;p&gt;The break-even chunk is the chunk that takes exactly as long as one decode step. At 70 B FP8 that works out to &lt;strong&gt;119 tokens&lt;/strong&gt;, and it is independent of batch size: the decode step is memory-bound, so its time does not move with batch, while a prefill chunk is compute-bound with fixed coefficients.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Chunk&lt;/th&gt;
&lt;th&gt;Chunk time&lt;/th&gt;
&lt;th&gt;Decode steps lost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;td&gt;23 ms&lt;/td&gt;
&lt;td&gt;1.08&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;512&lt;/td&gt;
&lt;td&gt;92 ms&lt;/td&gt;
&lt;td&gt;4.33&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1,024&lt;/td&gt;
&lt;td&gt;186 ms&lt;/td&gt;
&lt;td&gt;8.75&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A 512-token chunk is 4.3 decode steps, repeated eight times for a 4k prompt. Total stall: unchanged. Maximum stall: divided by eight. vLLM's 512 default is a compromise between a number this table can name and a cost this table cannot: tiny chunks raise kernel-launch overhead and shrink the GEMMs, so prefill throughput falls. I do not have a measurement for that penalty, so I will not put a number on it.&lt;/p&gt;

&lt;p&gt;If you care about inter-token latency, the target is a chunk near 119 tokens. At 20 tok/s the normal gap between tokens is 50 ms; a 512-token chunk inserts 92 ms, about two tokens' worth of silence, every 512 prompt tokens. A 128-token chunk inserts 23 ms, which is invisible, and probably costs you prefill throughput. That trade is where the tuning lives.&lt;/p&gt;

&lt;h2&gt;
  
  
  Long context makes it worse in a way chunking cannot reach
&lt;/h2&gt;

&lt;p&gt;Look at the attention column again. At 4k, attention is 7% of the prefill. At 32k it is 38%, and the term is quadratic in T, so it is untouched by chunk size: splitting a 32k prompt into 64 chunks still does 2,815 TFLOP of attention work, now with worse arithmetic intensity. Chunking converts a spike into a plateau. It does not move the plateau, and at long context the plateau is where the time went.&lt;/p&gt;

&lt;p&gt;This is the same curve that makes long-context decode expensive, arriving from the other side.&lt;/p&gt;

&lt;h2&gt;
  
  
  The capacity number for a mixed workload
&lt;/h2&gt;

&lt;p&gt;Put the two together. A node has one GPU-second per second. A 4k prefill consumes 0.79 of them. Decode at batch 32 consumes 0.86 ms per generated token.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Prefills/s (4k)&lt;/th&gt;
&lt;th&gt;GPU spent on prefill&lt;/th&gt;
&lt;th&gt;Decode sequences that still fit&lt;/th&gt;
&lt;th&gt;Per-sequence&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0.25&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;td&gt;25.7&lt;/td&gt;
&lt;td&gt;36.4 tok/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.50&lt;/td&gt;
&lt;td&gt;39%&lt;/td&gt;
&lt;td&gt;19.4&lt;/td&gt;
&lt;td&gt;36.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;79%&lt;/td&gt;
&lt;td&gt;6.9&lt;/td&gt;
&lt;td&gt;36.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.00&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;One document per second eats four-fifths of the node. Two per second and the decode queue never drains; throughput does not fall, it stops. If you have ever wondered why a chat product's p99 collapses in the late afternoon while utilization reads healthy, this table is the shape of the answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why prefill/decode disaggregation became practical
&lt;/h2&gt;

&lt;p&gt;If the interference is structural, the clean answer is separate pools and ship the cache. The cost is the transfer, and it is now small.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Link&lt;/th&gt;
&lt;th&gt;671 MB KV for a 4k prompt&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;NVLink 900 GB/s&lt;/td&gt;
&lt;td&gt;0.7 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;400G InfiniBand&lt;/td&gt;
&lt;td&gt;13.4 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100G InfiniBand&lt;/td&gt;
&lt;td&gt;53.7 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Against a 786 ms prefill, 400G adds 1.7%. On 100G it adds 7% and, worse, a variable 7%. That is roughly the story of why the architecture was proposed years ago and deployed later: the compute math was always fine, the interconnect was not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;

&lt;p&gt;Everything above, runnable. Change &lt;code&gt;MFU&lt;/code&gt; and the conclusions scale; change the config and the ordering does not.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;L&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;D&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KVH&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;HD&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;   &lt;span class="c1"&gt;# Llama-3.1-70B config.json
&lt;/span&gt;&lt;span class="n"&gt;N&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;BW&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;FPEAK&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;70.55e9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;3.35e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1979e12&lt;/span&gt;  &lt;span class="c1"&gt;# params; H100 SXM bytes/s; FP8 TFLOP/s
&lt;/span&gt;&lt;span class="n"&gt;KV&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;L&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;KVH&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;HD&lt;/span&gt;                     &lt;span class="c1"&gt;# bytes per token, FP8
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mfu&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;mem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;N&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;KV&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;BW&lt;/span&gt;
    &lt;span class="n"&gt;flop&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;N&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mfu&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;FPEAK&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mem&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;flop&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;prefill&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mfu&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.4&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;N&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;L&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;H&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;HD&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mfu&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;FPEAK&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;s32&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;one 4k prefill = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;prefill&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;s32&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; generated tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;break-even chunk = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mf"&gt;0.4&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;FPEAK&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;N&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chunk 512 = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;prefill&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nf"&gt;step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; decode steps&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you run it and disagree with a constant, that is the useful kind of disagreement: the MFU and the peak are the two numbers I chose rather than derived, and both are checkable on your own card.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would measure next
&lt;/h2&gt;

&lt;p&gt;The one thing this arithmetic cannot tell you is how much prefill throughput the 119-token chunk costs. That needs a card, a profiler, and two configs. If you have run that sweep, I would like the numbers; the comment section is where I keep the corrections.&lt;/p&gt;

&lt;p&gt;The rest of this series, including the decode and KV-cache pieces these constants come from, is in &lt;a href="https://jax-silicon-notes.surge.sh" rel="noopener noreferrer"&gt;my archive&lt;/a&gt; and on &lt;a href="https://dev.to/jaxsinclair"&gt;my dev.to profile&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Assumptions: FP8 weights and cache, no tensor parallelism overhead, no continuous-batching scheduler loss, 40% MFU for both prefill and decode. Every table is arithmetic from the published config and the H100 datasheet, not a benchmark.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>performance</category>
      <category>llm</category>
      <category>gpu</category>
    </item>
    <item>
      <title>A 256-expert MoE promises 32x. At batch 1 you collect 12.5x.</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Tue, 03 Nov 2026 13:57:35 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/a-256-expert-moe-promises-32x-at-batch-1-you-collect-125x-30i7</link>
      <guid>https://dev.to/jaxsinclair/a-256-expert-moe-promises-32x-at-batch-1-you-collect-125x-30i7</guid>
      <description>&lt;p&gt;Two posts ago I computed what a sparse model reads per token. DeepSeek-V3 at FP8 reads 36 GB, a sixteenth of what it stores, and that read is the reason a 671B model is servable at all.&lt;/p&gt;

&lt;p&gt;The bound in that piece quietly assumed the eight experts a token picks are spread evenly across the machine. They are not. On eight GPUs one of them always draws more experts than the others, and a step takes as long as the slowest GPU takes. That gap is not a rounding error at the batch sizes people actually serve, so this piece computes it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;256 experts, top-8, 8 GPUs, 32 experts resident on each. Every decode step, each token is handed to eight experts somewhere in the node. A GPU holding three of a token's experts reads three experts' worth of weights. The step ends when the last GPU finishes.&lt;/p&gt;

&lt;p&gt;If the eight experts scattered one per GPU, every GPU would read exactly one expert per token and you would collect the full 32x discount over a dense model of the same total size. Eight balls into eight bins does not leave one in each bin. It leaves a bin with two and often three.&lt;/p&gt;

&lt;p&gt;So the useful quantity is not the average load. It is the maximum. Divide the ideal per-GPU load by the realized maximum and you get the fraction of the sparsity discount you actually collect.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the distribution costs
&lt;/h2&gt;

&lt;p&gt;Monte Carlo over the layer, uniform routing, 300k token-assignments per row.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;tokens/step&lt;/th&gt;
&lt;th&gt;ideal experts/GPU&lt;/th&gt;
&lt;th&gt;busiest GPU&lt;/th&gt;
&lt;th&gt;tax&lt;/th&gt;
&lt;th&gt;discount collected&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;2.56&lt;/td&gt;
&lt;td&gt;2.56x&lt;/td&gt;
&lt;td&gt;39%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4.00&lt;/td&gt;
&lt;td&gt;6.99&lt;/td&gt;
&lt;td&gt;1.75x&lt;/td&gt;
&lt;td&gt;57%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;8.00&lt;/td&gt;
&lt;td&gt;12.16&lt;/td&gt;
&lt;td&gt;1.52x&lt;/td&gt;
&lt;td&gt;66%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;32.00&lt;/td&gt;
&lt;td&gt;40.09&lt;/td&gt;
&lt;td&gt;1.25x&lt;/td&gt;
&lt;td&gt;80%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;td&gt;128.00&lt;/td&gt;
&lt;td&gt;144.03&lt;/td&gt;
&lt;td&gt;1.13x&lt;/td&gt;
&lt;td&gt;89%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;512&lt;/td&gt;
&lt;td&gt;512.00&lt;/td&gt;
&lt;td&gt;543.36&lt;/td&gt;
&lt;td&gt;1.06x&lt;/td&gt;
&lt;td&gt;94%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2,048&lt;/td&gt;
&lt;td&gt;2,048.0&lt;/td&gt;
&lt;td&gt;2,113.8&lt;/td&gt;
&lt;td&gt;1.03x&lt;/td&gt;
&lt;td&gt;97%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8,192&lt;/td&gt;
&lt;td&gt;8,192.0&lt;/td&gt;
&lt;td&gt;8,315.7&lt;/td&gt;
&lt;td&gt;1.02x&lt;/td&gt;
&lt;td&gt;98%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Read the first row. A single decode step with a batch of one collects 39% of the discount the config promised. The model was sold as 32x sparser than dense and at batch 1 it behaves like 12.5x. Nobody serves at batch 1, but everybody serves at some batch, and the difference between 80% and 94% is the difference between a throughput number from a benchmark and one from your deployment.&lt;/p&gt;

&lt;h2&gt;
  
  
  It collapses to one line
&lt;/h2&gt;

&lt;p&gt;For eight groups that whole table is&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tax ≈ 1 + 1.4 / sqrt(tokens per step)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;which reproduces the rows within 3% from batch 8 upward. Invert it and you get the number that decides your serving config: to hold the straggler under 5% at eight GPUs you need about &lt;strong&gt;800 tokens per step&lt;/strong&gt;. Under 2%, about 4,900.&lt;/p&gt;

&lt;p&gt;Tokens per step is not the same as sequences in flight. A prefill of a 4k prompt contributes 4,096 token-assignments in one shot, so prefill never pays this tax; at 4k the busiest GPU is within 2% of the mean. A decode step contributes one token per sequence. So the 800-token number, in decode, means 800 concurrent sequences. Below that your MoE is running with a visible straggler, and if you are serving at 20 users you are running at 1.3x tax, permanently.&lt;/p&gt;

&lt;p&gt;That is the shape of the thing. Sparsity is a discount on weight reads, batching pays the discount back, and the tax is charged per step rather than per token, so it hurts exactly the deployments that care about latency.&lt;/p&gt;

&lt;h2&gt;
  
  
  The correction to my own number
&lt;/h2&gt;

&lt;p&gt;Two pieces ago I wrote that 8xH200 sustains about 1,060 tokens per second aggregate for DeepSeek-V3, so at 50 tok/s per user the ceiling is 21 concurrent sequences. The 1,060 assumed perfect balance. At 21 sequences the tax is 1.32, so the honest figure is about 800 tokens per second aggregate, and 16 users at 50 tok/s each. The bandwidth math was right. The scheduling assumption inside it was not, and it cost 25% of the answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wider expert parallelism makes it worse
&lt;/h2&gt;

&lt;p&gt;Expert parallelism looks like a way out: split 256 experts over more devices and each GPU has less to do. It has less to do and it also has less to average over, and the second effect wins.&lt;/p&gt;

&lt;p&gt;Tax at 21 sequences, same 256 experts, top-8:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;GPUs&lt;/th&gt;
&lt;th&gt;experts/GPU&lt;/th&gt;
&lt;th&gt;tax at 21 seq&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;1.32x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;1.58x&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;1.98x&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At EP=32 the busiest GPU does twice the mean work, and you pay for that in latency as well as in all-to-all volume. This is the reason production MoE stacks spend memory on redundant expert copies and whole subsystems on rebalancing placement. My arithmetic here does not model that fix; it only shows what the fix is buying.&lt;/p&gt;

&lt;h2&gt;
  
  
  The all-to-all is not the problem people think
&lt;/h2&gt;

&lt;p&gt;The standard objection to expert parallelism is fabric traffic, and it is falsifiable with a multiplication.&lt;/p&gt;

&lt;p&gt;DeepSeek-V3 dispatches hidden_size 7168 in BF16, so one token's eight experts move 8 x 7168 x 2 = 114 KB out and the same back, per MoE layer. There are 58 MoE layers. That is 13.3 MB moved per token, round trip, every token. At 1,060 tokens per second that is 14 GB/s.&lt;/p&gt;

&lt;p&gt;An 8xH100 NVSwitch domain carries about 7,200 GB/s aggregate. 14 GB/s is 0.2% of it. Even spread over 32 GPUs on 50 GB/s InfiniBand, the same traffic is under 1% of the fabric.&lt;/p&gt;

&lt;p&gt;All-to-all is a real engineering problem for overlapping and for tail latency, and it is not a bandwidth problem on a node. The straggler is. If you are deciding whether to widen EP, the traffic argument will tell you it is fine and the load-balance argument will tell you to stop.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you cannot batch away
&lt;/h2&gt;

&lt;p&gt;At batch 32 the union of distinct experts touched per step is 163 of 256, so each token effectively reads 5.1 experts' weights instead of 8. Batching makes MoE cheaper per token and the tax makes the step later, and both come from the same repetition. There is a floor: the memory-bound regime ends around the ridge batch of 296 on an H100 at BF16, half that at FP8, and past it you are comparing FLOPs instead of bytes, where the same imbalance reapplies to expert FLOPs on the busiest GPU.&lt;/p&gt;

&lt;p&gt;The floor is what matters for small models. Qwen3-30B-A3B has 128 experts and reads 6.0 GB per token, which on 8xH100 is the 4,186 tok/s aggregate figure from the earlier table. Same 8 GPUs, 16 experts each, and at batch 8 the tax is 1.5, so the honest number is about 2,750 tokens per second, or 55 users at 50 tok/s instead of 83. Sparse models are aggressive about small batches because their whole advantage is a read they cannot amortize.&lt;/p&gt;

&lt;p&gt;If you serve fewer than a few hundred sequences, the arithmetic says run tensor parallelism and a dense model, or run expert parallelism and accept a quarter of your throughput in straggler. What it does not say is that you can have the 32x and the low latency.&lt;/p&gt;

&lt;h2&gt;
  
  
  The script
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;

&lt;span class="n"&gt;E&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;G&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;          &lt;span class="c1"&gt;# experts, top-k, expert-parallel groups
&lt;/span&gt;&lt;span class="n"&gt;EPG&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;E&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="n"&gt;G&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trials&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;rng&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Random&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;tot_max&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tot_mean&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;trials&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;load&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;G&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;E&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
                &lt;span class="n"&gt;load&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="n"&gt;EPG&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="n"&gt;tot_max&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;load&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;tot_mean&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;K&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;G&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;tot_mean&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;trials&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tot_max&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;trials&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2048&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8192&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trials&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;300_000&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  ideal &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;8.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  busiest &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;8.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tax &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  collected &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;%&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Change E, K and G to your model and sharding. The shape does not change and the tax is monotone in G, which is the part I would check first if I were sizing a deployment.&lt;/p&gt;

&lt;p&gt;The rest of this series, including the bytes-per-token piece this one corrects, is in &lt;a href="https://jax-silicon-notes.surge.sh" rel="noopener noreferrer"&gt;my archive&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Numbers are computed, not quoted. If one is wrong, say which and I will re-derive it.&lt;/p&gt;

</description>
      <category>hardware</category>
      <category>llm</category>
      <category>performance</category>
      <category>gpu</category>
    </item>
    <item>
      <title>A 671B model reads 36GB per token and still needs 671GB of HBM</title>
      <dc:creator>Jax Sinclair</dc:creator>
      <pubDate>Mon, 02 Nov 2026 15:44:49 +0000</pubDate>
      <link>https://dev.to/jaxsinclair/a-671b-model-reads-36gb-per-token-and-still-needs-671gb-of-hbm-45oj</link>
      <guid>https://dev.to/jaxsinclair/a-671b-model-reads-36gb-per-token-and-still-needs-671gb-of-hbm-45oj</guid>
      <description>&lt;p&gt;Two numbers describe DeepSeek-V3 and they look like a contradiction. Decoding one token reads about 36 GB of weights. Storing the model takes 671 GB. If you have only ever sized dense models, those two figures should be the same number, because a dense model reads all of its weights to emit one token.&lt;/p&gt;

&lt;p&gt;Mixture-of-experts breaks that identity on purpose. It is worth knowing exactly how much it breaks it, because the break is asymmetrical and most capacity planning carries the wrong half.&lt;/p&gt;

&lt;h2&gt;
  
  
  What MoE buys, derived from the config
&lt;/h2&gt;

&lt;p&gt;Take the published &lt;code&gt;config.json&lt;/code&gt; for three sparse models and compute the parameters that actually participate in one token: attention on every layer, plus the top-k experts and the shared expert, plus the embedding. Nothing quoted, everything counted.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Total params&lt;/th&gt;
&lt;th&gt;Active per token&lt;/th&gt;
&lt;th&gt;Stored (weights)&lt;/th&gt;
&lt;th&gt;Read per token&lt;/th&gt;
&lt;th&gt;Read / stored&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-30B-A3B (BF16)&lt;/td&gt;
&lt;td&gt;30.5B&lt;/td&gt;
&lt;td&gt;3.0B&lt;/td&gt;
&lt;td&gt;61 GB&lt;/td&gt;
&lt;td&gt;6.0 GB&lt;/td&gt;
&lt;td&gt;0.099&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B-A22B (BF16)&lt;/td&gt;
&lt;td&gt;235B&lt;/td&gt;
&lt;td&gt;21.5B&lt;/td&gt;
&lt;td&gt;470 GB&lt;/td&gt;
&lt;td&gt;43 GB&lt;/td&gt;
&lt;td&gt;0.092&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek-V3 (FP8)&lt;/td&gt;
&lt;td&gt;671B&lt;/td&gt;
&lt;td&gt;36.0B&lt;/td&gt;
&lt;td&gt;671 GB&lt;/td&gt;
&lt;td&gt;36 GB&lt;/td&gt;
&lt;td&gt;0.054&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.1-8B (BF16), dense&lt;/td&gt;
&lt;td&gt;8.03B&lt;/td&gt;
&lt;td&gt;8.03B&lt;/td&gt;
&lt;td&gt;16.1 GB&lt;/td&gt;
&lt;td&gt;16.1 GB&lt;/td&gt;
&lt;td&gt;1.000&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;My script prints 36.0B active for DeepSeek-V3 where the paper says 37B. The gap is the multi-token-prediction layer in the checkpoint, which I do not count. Every other row lands within 2% of its published active size, which is my check that the arithmetic below is not invented.&lt;/p&gt;

&lt;p&gt;Read the last column. A sparse model reads between a tenth and a twentieth of what it stores. That is not a small effect, and it is the whole reason a 671B model is servable at all: on a bandwidth-limited decode step it behaves like a 36B model.&lt;/p&gt;

&lt;p&gt;It does not fit like a 36B model. That is the asymmetry, and it is not adjustable. Bandwidth scales with the read, capacity scales with the store, and MoE only moves the first one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ceiling is set by the read
&lt;/h2&gt;

&lt;p&gt;Decode reads active weights once per token and performs two multiply-adds per active parameter, so the numbers that matter are bytes per token and the machine's bandwidth.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Bytes read / token&lt;/th&gt;
&lt;th&gt;H100, 4k ctx&lt;/th&gt;
&lt;th&gt;H100, 128k ctx&lt;/th&gt;
&lt;th&gt;8×H100&lt;/th&gt;
&lt;th&gt;8×H200&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-30B-A3B&lt;/td&gt;
&lt;td&gt;6.0 GB&lt;/td&gt;
&lt;td&gt;523 tok/s&lt;/td&gt;
&lt;td&gt;177&lt;/td&gt;
&lt;td&gt;4,186 tok/s&lt;/td&gt;
&lt;td&gt;5,998&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-235B-A22B&lt;/td&gt;
&lt;td&gt;43 GB&lt;/td&gt;
&lt;td&gt;76&lt;/td&gt;
&lt;td&gt;49&lt;/td&gt;
&lt;td&gt;612&lt;/td&gt;
&lt;td&gt;877&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek-V3 (FP8)&lt;/td&gt;
&lt;td&gt;36 GB&lt;/td&gt;
&lt;td&gt;92&lt;/td&gt;
&lt;td&gt;74&lt;/td&gt;
&lt;td&gt;739&lt;/td&gt;
&lt;td&gt;1,058&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.1-8B&lt;/td&gt;
&lt;td&gt;16.1 GB&lt;/td&gt;
&lt;td&gt;202&lt;/td&gt;
&lt;td&gt;101&lt;/td&gt;
&lt;td&gt;1,615&lt;/td&gt;
&lt;td&gt;2,314&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The single-card columns are honest only for the 8B. For the other three, the weights live across a node, so read those columns as the tokens per second you get per unit of aggregate bandwidth. The two node columns are the useful ones: that is the total decode throughput a node can sustain at short context, and it is the number you divide by your target per-user speed to get a batch ceiling. On 8×H200, DeepSeek-V3 sustains about 1,060 tokens per second in aggregate, so if you want 50 tok/s per user you cap at roughly 21 concurrent sequences.&lt;/p&gt;

&lt;p&gt;Three things fall out of that table and none of them are what the marketing copies say.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Qwen3-30B-A3B beats Llama-3.1-8B on throughput by 2.6x&lt;/strong&gt; while storing 3.8x the weights and a large lead in quality. Its read per token is 6.0 GB against 16.1 GB. Sparsity bought it a smaller read than a dense model less than a third its size.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DeepSeek-V3 reads half of what Qwen3-235B-A22B reads&lt;/strong&gt;, despite three times the stored parameters. FP8 halves the bytes, and MLA halves the cache (more on that below). Both levers act on the read.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Nothing here fixes capacity.&lt;/strong&gt; Look at the store column again. That is the next section.&lt;/p&gt;

&lt;h2&gt;
  
  
  The capacity bill arrives up front
&lt;/h2&gt;

&lt;p&gt;DeepSeek-V3 at FP8 needs 671 GB before the first token. There is no batch size, no paging trick, and no amount of HBM efficiency that removes weights from the read path; every expert must be resident or reachable.&lt;/p&gt;

&lt;p&gt;On 8×H200 that is 671 GB of 1,128 GB, leaving 457 GB for KV cache and activations. It fits. It does not fit comfortably, and it is the tightest constraint in that deployment, not the cache.&lt;/p&gt;

&lt;p&gt;The dense comparison is worse. A 671B dense model in BF16 is 1.34 TB and does not fit on 8×H200 at all. At FP8 it is 671 GB, exactly where the sparse model already sits, and it would read 671 GB per token instead of 36. Sparsity buys the multiply, not the storage.&lt;/p&gt;

&lt;p&gt;So the honest framing for a MoE deployment is: you are buying a bandwidth discount with a capacity tax. Budget the capacity first. If the node holds the weights with room left over, the model serves; if it does not, no batching policy rescues it. This is the mirror image of the last piece in this series, where KV compression bought capacity at almost no cost in speed. MoE buys speed at the cost of capacity.&lt;/p&gt;

&lt;p&gt;One more count that surprised me. DeepSeek-V3's cache is 69 KiB per token, against 128 KiB for Llama-3.1-8B. The 671B model carries a cache half the size of the 8B's, because MLA stores one compressed latent of 576 elements per layer instead of a key and a value per head. At 128k context, that is 8.8 GB per sequence. On a node that already paid 671 GB for weights, the cache is rounding error. This is why the long-context economics of DeepSeek-V3 look nothing like the same math applied to a GQA model of the same size.&lt;/p&gt;

&lt;h2&gt;
  
  
  FP8 halves the read and halves the batch you need
&lt;/h2&gt;

&lt;p&gt;From the previous piece, the batch at which decode crosses from memory-bound to compute-bound is &lt;code&gt;ridge × bytes_per_param / 2&lt;/code&gt;. At BF16 that is the ridge number itself, about 296 on an H100. At FP8 it is 148.&lt;/p&gt;

&lt;p&gt;FP8 does not reduce FLOPs. It reduces the bytes those FLOPs wait on, so you reach compute-bound with half the batch. Applied to a sparse model the effect compounds, because the read was already a tenth of the store. DeepSeek-V3 hits the ridge at batch 148 instead of 296, and 148 sequences is roughly the ceiling of what a node can hold anyway once you add cache.&lt;/p&gt;

&lt;p&gt;That is a rare case of the two limits landing near each other, and it is why FP8 on MoE is not just a memory optimization. It changes which resource you run out of first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Batching an MoE does not amortize the way you expect
&lt;/h2&gt;

&lt;p&gt;Here is where the clean arithmetic above starts to lie, and it is worth saying so in the same article rather than in a footnote.&lt;/p&gt;

&lt;p&gt;With a dense model, batch 32 reads the weights once and produces 32 tokens. The read per token falls by 32. Pure win.&lt;/p&gt;

&lt;p&gt;With a sparse model, each token in the batch routes to a different set of 8 experts out of 256. The decode step must stream the union of every expert touched, not the 8 one token would touch. If routing were independent across tokens, the expected unique experts per layer is &lt;code&gt;256 × (1 − (248/256)^B)&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Batch&lt;/th&gt;
&lt;th&gt;Unique experts hit / layer&lt;/th&gt;
&lt;th&gt;Read multiplier vs batch 1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;8.0&lt;/td&gt;
&lt;td&gt;1.00×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;30.5&lt;/td&gt;
&lt;td&gt;3.82×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;57.4&lt;/td&gt;
&lt;td&gt;7.18×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;163.3&lt;/td&gt;
&lt;td&gt;20.4×&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So at batch 32 an MoE reads about 20x more weight bytes per step than a single sequence does, while a dense model reads the same bytes it always did. The throughput gain from batching is real but it flattens far earlier, and the aggregate numbers in my table are short-context, batch-1 best cases.&lt;/p&gt;

&lt;p&gt;Real routing is not independent. Tokens from the same domain route to overlapping experts, so the true union is smaller than this bound, and expert-parallel placement turns most of the extra reads into a network exchange between GPUs rather than a local HBM read. That is the actual shape of MoE serving: past a certain batch, the bottleneck moves from HBM to NVLink and then to the interconnect between nodes, and the number you should be watching stops being bytes per token.&lt;/p&gt;

&lt;p&gt;I have not measured any of this on hardware. It is arithmetic from public configs, plus an independence assumption that I have flagged as a bound rather than a measurement. If you run a sparse model and have the routing histogram, you can replace my estimate with your own and I would like to see it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check it on your model
&lt;/h2&gt;

&lt;p&gt;The script reads a config's shape and prints the three numbers that decide a deployment: stored bytes, bytes read per token, and the aggregate bandwidth you need for a target throughput.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;GiB&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;

&lt;span class="c1"&gt;# (label, params_b, bytes_per_param, layers, kv_heads, head_dim, active_params_b)
&lt;/span&gt;&lt;span class="n"&gt;MODELS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Qwen3-30B-A3B  BF16&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mf"&gt;30.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mf"&gt;3.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Qwen3-235B-A22B BF16&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;235.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;94&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;21.5&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;DeepSeek-V3  FP8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="mf"&gt;671.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;61&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;36.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;  &lt;span class="c1"&gt;# MLA: 576-element latent
&lt;/span&gt;    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Llama-3.1-8B  BF16&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="mf"&gt;8.03&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;   &lt;span class="mf"&gt;8.03&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;NODE_BW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;8xH100&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;3.35e12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;8xH200&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;4.80e12&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kvh&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;MODELS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;store&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt;
    &lt;span class="n"&gt;read&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;active&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt;
    &lt;span class="n"&gt;kv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;576&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;kvh&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="nf"&gt;else &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;layers&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;kvh&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;hd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;bpp&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  store &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mf"&gt;1e9&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;7.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; GB   read/token &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mf"&gt;1e9&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; GB   &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;kv/token &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; KiB   read/store &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bw&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;NODE_BW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;node&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;bw&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;read&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;4096&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;kv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;7.0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; tok/s aggregate at 4k context&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Expert-union bound for a 256-expert, top-8 layer
&lt;/span&gt;&lt;span class="n"&gt;E&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;B&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;uniq&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;E&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;E&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;B&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  batch &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;B&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;uniq&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;6.1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; experts touched, read x&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;uniq&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mf"&gt;5.2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it on your own config and the columns change. The shape does not: read is set by active parameters and precision, capacity by total parameters, and only one of those scales with your batch.&lt;/p&gt;

&lt;p&gt;The rest of this series, including the bytes-per-token KV piece this one borrows its 128 KiB figure from, lives in &lt;a href="https://jax-silicon-notes.surge.sh" rel="noopener noreferrer"&gt;my archive&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Numbers are computed from public configs, not quoted. If one of mine is wrong, say which and I will re-derive it.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>hardware</category>
      <category>performance</category>
      <category>gpu</category>
    </item>
  </channel>
</rss>
