<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Deployed Lab</title>
    <description>The latest articles on DEV Community by Deployed Lab (@deployedlabs).</description>
    <link>https://dev.to/deployedlabs</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4246857%2Fa6963366-8bd7-4153-8542-4790da99b456.png</url>
      <title>DEV Community: Deployed Lab</title>
      <link>https://dev.to/deployedlabs</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/deployedlabs"/>
    <language>en</language>
    <item>
      <title>How much GPU memory does an LLM need? The arithmetic, step by step</title>
      <dc:creator>Deployed Lab</dc:creator>
      <pubDate>Mon, 16 Nov 2026 04:49:26 +0000</pubDate>
      <link>https://dev.to/deployedlabs/how-much-gpu-memory-does-an-llm-need-the-arithmetic-step-by-step-44he</link>
      <guid>https://dev.to/deployedlabs/how-much-gpu-memory-does-an-llm-need-the-arithmetic-step-by-step-44he</guid>
      <description>&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
        &lt;div class="c-embed__cover"&gt;
          &lt;a href="https://deployedlab.dev/lessons/37-gpus-memory-sizing" class="c-link align-middle" rel="noopener noreferrer"&gt;
            &lt;img alt="" src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeployedlab.dev%2Fassets%2Fbrand%2Fog-image.png" height="420" class="m-0" width="800"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="c-embed__body"&gt;
        &lt;h2 class="fs-xl lh-tight"&gt;
          &lt;a href="https://deployedlab.dev/lessons/37-gpus-memory-sizing" rel="noopener noreferrer" class="c-link"&gt;
            LLM VRAM calculator: GPU memory for any model | Deployed Lab
          &lt;/a&gt;
        &lt;/h2&gt;
          &lt;p class="truncate-at-3"&gt;
            Free LLM VRAM calculator: estimate GPU memory for weights, KV cache and overhead from parameters, precision, context and batch size.
          &lt;/p&gt;
        &lt;div class="color-secondary fs-s flex items-center"&gt;
            &lt;img alt="favicon" class="c-embed__favicon m-0 mr-2 radius-0" src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeployedlab.dev%2Ffavicon.ico" width="48" height="48"&gt;
          deployedlab.dev
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;"Will this model fit on my GPU?" is one of the most common questions in local and self-hosted LLM work. The answer is three numbers added together. Here is the arithmetic, with a worked example.&lt;/p&gt;

&lt;p&gt;All sizes below use 1 GB = 1 billion bytes. The model setup is illustrative, not a benchmark of a specific model.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Weights
&lt;/h2&gt;

&lt;p&gt;The weights are the model's learned numbers. Every one must sit in GPU memory.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;weights = parameters x bytes per weight
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Bytes per weight depend on the precision: 4 (32-bit), 2 (16-bit), 1 (8-bit), about 0.5 (4-bit).&lt;/p&gt;

&lt;p&gt;For an 8-billion-parameter model at 16-bit: 8e9 x 2 bytes = &lt;strong&gt;16.0 GB&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. KV cache
&lt;/h2&gt;

&lt;p&gt;While generating, the model keeps short notes about the text so far, called the KV cache (K and V stand for key and value). Without them it would re-read everything for every new token. The notes cost memory, and they grow with every token and every open chat.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;kv_per_token = 2 x layers x kv_heads x head_dim x bytes per cache value
kv_cache     = kv_per_token x context_tokens x batch_size
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The 2 is one key and one value per layer. For our example: 32 layers, 8 KV heads, head size 128, 16-bit cache values.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;kv_per_token = 2 x 32 x 8 x 128 x 2 = 131,072 bytes&lt;/li&gt;
&lt;li&gt;For 8 chats at 4,096 tokens each: 131,072 x 4,096 x 8 = &lt;strong&gt;4.29 GB&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  3. Overhead
&lt;/h2&gt;

&lt;p&gt;Real engines need room for temporary buffers and lose some memory to gaps. A rough planning default is 10 to 20 percent of weights plus cache. With 15 percent:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;overhead = 0.15 x (16.0 + 4.29) = 3.04 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Total
&lt;/h2&gt;

&lt;p&gt;16.0 + 4.29 + 3.04 = &lt;strong&gt;about 23.3 GB&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That barely fits a 24 GB card, with under 1 GB spare. Add a few more users or longer chats and it will not fit.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you can change
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Weights are the biggest part here, so a smaller model or lower precision (for example 8-bit) is the biggest lever.&lt;/li&gt;
&lt;li&gt;If the cache dominates (long context, many users), shorten the context, serve fewer chats at once, or use a lower-precision cache.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Caveats
&lt;/h2&gt;

&lt;p&gt;This is an estimate for planning. Real engines allocate memory differently, some architectures differ (for example mixture-of-experts models store all experts but use only a few per token), and you should read your model's config for the real layer count, KV heads and head size. Test on your own hardware.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;p&gt;I put the calculator in one HTML file, MIT licensed, with the formulas shown: &lt;a href="https://github.com/deployedlab/llm-vram-calculator" rel="noopener noreferrer"&gt;github.com/deployedlab/llm-vram-calculator&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;There is also a step-by-step interactive version with diagrams and a "does it fit" table, free with no signup: &lt;a href="https://deployedlab.dev/lessons/37-gpus-memory-sizing" rel="noopener noreferrer"&gt;GPUs and memory sizing&lt;/a&gt;. It is one of seven free chapters of &lt;a href="https://deployedlab.dev" rel="noopener noreferrer"&gt;Deployed Lab&lt;/a&gt;, a course on AI engineering and the forward deployed engineer role.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>gpu</category>
      <category>ai</category>
      <category>machinelearning</category>
    </item>
  </channel>
</rss>
