Budgeting memory for a local language model: weights, KV cache, and room to run

Brandon Brooks — Sun, 18 Oct 2026 00:01:51 +0000

An “8B, 4-bit” label gives you a starting estimate for weight storage. A useful memory budget also needs the conversation cache, the runtime's allocations, and the workload you intend to serve.

Here is a calculation you can adapt before downloading weights or choosing a context length. The numbers below are arithmetic estimates, not measurements of a running model.

1. Start with the weight payload

For P scalar parameters stored uniformly at b bits each:

weight bytes = P × b / 8

A hypothetical model with exactly 8 billion parameters would have these idealized payloads:

16 bits: 16,000,000,000 bytes ≈ 14.90 GiB
 8 bits:  8,000,000,000 bytes ≈  7.45 GiB
 4 bits:  4,000,000,000 bytes ≈  3.73 GiB

One GiB is 2^30 bytes; one decimal GB is 10^9 bytes. Keep units consistent when comparing a download, a memory monitor and a hardware specification.

Those figures assume every parameter uses the same representation. Real quantized artifacts can include scales, other metadata and tensors retained at higher precision. For example, the Transformers bitsandbytes documentation distinguishes quantized linear layers from other modules and separates weight storage from compute dtype. Check the exact artifact and runtime rather than treating every “4-bit” label as an identical format. [1]

2. Add the conversation's KV cache

During autoregressive generation, a transformer can reuse previously computed keys and values. The cache contains per-layer tensors and grows with the cached sequence. [2]

For conventional full-attention decoder layers with equal key and value head dimensions, an uncompressed cache estimate is:

KV bytes = 2 × L × Hkv × D × T × C × S

2   = keys and values
L   = attention layers
Hkv = key/value heads per layer
D   = dimension per key/value head
T   = cached tokens per sequence, including processed prompt and output
C   = bytes per cached element (2 for FP16/BF16)
S   = independently stored sequences of that length

Use KV heads, not query heads. For differing sequence lengths, sum the token counts; the simple multiplication assumes equal lengths and no shared prefix storage. Beam search or concurrent requests can introduce more independently cached sequences.

The official Qwen/Qwen3-8B config lists 36 layers, 8 KV heads and head dimension 128, with sliding-window attention disabled. [3] At 8,192 cached tokens, two-byte cache elements and one sequence, the calculation is:

2 × 36 × 8 × 128 × 8,192 × 2 × 1
= 1,207,959,552 bytes
= 1.125 GiB

Under those same assumptions:

4,096 tokens, one sequence: 0.5625 GiB
8,192 tokens, one sequence: 1.1250 GiB
16,384 tokens, one sequence: 2.2500 GiB
8,192 tokens, four independent sequences: 4.5000 GiB

These are tensor payload estimates. They do not establish total memory use, speed or whether a particular machine can run the checkpoint.

The formula needs adjustment for other architectures and cache implementations. Sliding-window layers may retain a bounded window. Static caches can reserve their configured maximum, while dynamic caches grow with use. Cache quantization and offloading change memory placement and precision, with performance tradeoffs. Weight quantization does not automatically set the KV-cache dtype. Check your backend's supported cache behavior. [4]

3. Leave room for execution and loading

A planning worksheet is:

resident weights + KV cache + runtime/workload overhead

Check both loading and generation peaks. Accelerate's documentation describes how loading can create extra weight copies, how sharded loading can reduce that pressure, and how CUDA initialization consumes memory beyond the weights. CPU or disk offload changes placement and introduces transfers; fitting with offload does not establish acceptable latency. [5]

For your runtime, inspect temporary workspaces, prefill allocations, allocator reservations and any additional models or modalities. There is no universal overhead percentage in this worksheet. Measure the exact workload and retain headroom for its observed peaks and other processes.

On Apple silicon, MLX uses unified memory shared by CPU and GPU. [6] Budget that shared capacity once, leaving room for the operating system and other applications. Do not add the same unified pool once as RAM and again as VRAM.

4. Turn the estimate into a reproducible check

Record the exact repository, checkpoint revision and quantization artifact; runtime version and backend; cache dtype and strategy; prompt/output token limits; and concurrent sequences. Start with a modest workload, then increase context and concurrency separately. Record loading peak, generation peak, time to first token and generation rate using the runtime's supported measurement tools. State how memory was measured, including whether it is allocated or reserved memory.

This is a proposed measurement procedure. No results from it are presented here, and this article makes no “runs comfortably on my laptop” claim.

Finally, inspect the license for the exact release. The official Qwen/Qwen3-8B repository used for the architecture example carries Apache License 2.0. [7] Do not generalize that license to another Qwen release, a conversion, or an unrelated model. Record the applicable model and runtime licenses alongside the revision and deployment configuration.

The useful outcome is a budget whose assumptions you can change: shorter context reduces this conventional cache estimate; lower weight precision changes the weight budget; more independent sequences increase cache demand. The measurements then tell you where the estimate stops matching your actual runtime.

Sources

Primary documentation checked live 2026-10-17 UTC
[1] Transformers bitsandbytes

[2] Transformers cache structure

[3] Official Qwen3-8B config

File-history short revision displayed: 47719a2. Main is mutable; pin the exact full checkpoint revision when running an experiment.
[4] Transformers cache strategies

[5] Accelerate loading and placement

[6] MLX unified memory

[7] Exact example release license

Disclosure

Prepared primarily by an AI agent for Brandon Brooks of Bright AI Future, from the primary sources linked above, and reviewed by Brandon. No model inference benchmarks were performed for this article.