VRAM and RAM for local LLMs — honest planning bands, not a GPU tier list

Tech-Gurunomics — Sun, 06 Sep 2026 12:13:47 +0000

Originally published at tech-gurunomics.com/tools/vram-ram-local-llms.

When you post this, set canonical_url to that URL (Dev.to front matter above; Hashnode: Settings → SEO → Canonical URL).

The software in the local LLMs cluster is mostly free or cheap. The box is not.

This is the boring constraints page: how much VRAM and system RAM you roughly need for common quantized sizes, when offload saves you (and when it just makes everything slow), and why disk still matters. Not a tier list. Not “best GPU 2026.”

What this replaces

It replaces thumbnail math (“70B on a laptop, easy”) with order-of-magnitude expectations before you buy a card or stuff another DIMM.
Pair it with the tool pages for Ollama, LM Studio, and llama.cpp — those are the runners; this is the metal they sit on.

It does not replace measuring your model, context length, batch size, and UI overhead. Quantization level, KV cache, and concurrent chats move the numbers. Treat everything below as planning bands, not warranties.

Rough VRAM bands (quantized chat, single user)

People usually talk in parameter count (7B, 13B, 70B) plus a quant (often Q4-ish for “fits and still readable”). Ballparks for weights in VRAM on a typical local chat setup:

~7B Q4: often comfortable in roughly 4–6 GB VRAM if context is modest and you are not stacking a heavy UI + embeddings on the same card.
~13B Q4: commonly wants on the order of 8–12 GB VRAM for a sane experience. 8 GB cards work until context, vision add-ons, or “just one more model loaded” say otherwise.
~70B Q4: weight footprint alone is often in the mid–high 30 GB+ range before you feel generous about context. A single 24 GB card usually means partial GPU residency + system RAM offload (usable, slower) or a smaller quant / smaller model. Full 70B “all on GPU, long context, snappy” is a different shopping trip than a 16 GB mid-range card.

Bigger context windows, multimodal towers, speculative decoding, and multi-user serving all push up. Serving stacks like vLLM are a different job than solo Ollama chat.

GPU vs CPU offload

VRAM is where layers want to live for speed. System RAM is the overflow tank. Runners built around llama.cpp-style graphs (including what you meet via Ollama / LM Studio) can keep some layers on GPU and spill the rest to CPU/RAM.

Offload helps when the model almost fits: you get an answer instead of an OOM, and the GPU still carries enough layers to not feel like 2018.
Offload hurts when most of the model is in RAM/CPU. Tokens per second fall off a cliff. “It loads” ≠ “you will use it daily.”
RAM size matters for spill. 16 GB system RAM that is already full of browser tabs is not a 70B offload plan. 32–64 GB is the common “I might spill layers” zone for enthusiasts; more if you keep many models and Docker UIs around.

If your goal is comfortable day-to-day local chat, buy toward fitting the model class you actually want in VRAM. Use offload as a bridge or for occasional bigger models — not as the headline feature on a sales thumbnail.

Disk: models are luggage

GGUF and other weight files are multi-GB each. A library of a few 7B/13B quants is already tens of gigabytes; add 70B-class files and you are in “where did 500 GB go” territory fast. A fast NVMe mainly helps load and swap times, not magic tokens/sec once the model is resident. Slow or full disks make pulls and model switches feel broken when the GPU is fine.

Example hardware (link to the site, not invent ASINs)

Same categories as the local LLMs hub hardware section — real patterns only (GPU / RAM / NVMe), not endorsements, not a ranking. Prices and stock change. Prefer that page for current product links.

Mid-range 16 GB VRAM (e.g. RTX 5060 Ti 16GB class) — headroom for many 7B–13B quantized setups; not an honest “70B fully on-GPU” card.
24 GB VRAM (e.g. RTX 3090 class) — older generation, but 24 GB is still what a lot of people hunt when they want larger local models or less painful offload.
64 GB system RAM (DDR5 kit) — for layer spill, Docker UIs, browsers, and “I loaded two models to compare.”
Fast 2TB NVMe — room for a real model library; faster pulls and swaps than a dying SATA drive.

Disclosure: Some links on the Tech-Gurunomics hardware section are Amazon Associates (tag=techgurunomic-20). As an Amazon Associate, Tech-Gurunomics may earn from qualifying purchases. This syndication draft points at that section rather than embedding affiliate ASINs here.

Where the shopping advice falls over

Vendor marketing uses different quant names and “GB required” charts. Always check the file size of the exact GGUF/quant you will load.
Laptop dual-use GPUs share memory with the OS in messy ways; desktop VRAM numbers do not transfer cleanly.
Apple Unified Memory is a different architecture — do not blindly apply NVIDIA VRAM bands to a Mac and call it science.
A strong card with a tiny SSD and 8 GB of RAM still feels cursed.

Verdict

Pick the model class you will actually run daily, size VRAM for that class first, then buy RAM for overflow and NVMe for the library. Use Ollama or LM Studio to get started; use llama.cpp when you need the engine knobs.

More cluster context: Local LLMs tools. Full original with live hardware examples: VRAM and RAM for local LLMs.

If you want the short video version of runners + hardware reality (Ollama vs LM Studio, then VRAM/RAM/NVMe), that lives on YouTube: Local LLMs Without the Hype.

Stars and “runs 70B” thumbnails are not a shopping list.

DEV Community: Tech-Gurunomics