DEV Community

Cover image for What a 32GB GPU actually lets you run locally
Spicy
Spicy

Posted on

What a 32GB GPU actually lets you run locally

#ai #tutorial #llm #machinelearning

The jump to 32GB of VRAM on consumer cards (hello, 5090) is a bigger deal for local AI than the gaming benchmarks suggest. The 24GB ceiling on the 3090/4090 was the thing quietly capping what you could run without brutal quantization. So the real question isn't "is it fast," it's "what can I actually load now, and how do I do it." Here's the practical version.

The fastest path: Ollama

If you just want models running in five minutes, Ollama is the easiest on-ramp.

# Linux / macOS
curl -fsSL https://ollama.com/install.sh | sh

# then pull and run (these are 4-bit quantized by default)
ollama run llama3.1:8b      # ~5 GB, runs on almost anything
ollama run qwen2.5:32b      # ~20 GB, comfortable on a 32 GB card
ollama run llama3.3:70b     # ~40 GB, tight; will spill to system RAM
Enter fullscreen mode Exit fullscreen mode

That's genuinely it for a working local chat model. The interesting part is knowing which line your card can actually handle.

The VRAM math (so you stop guessing)

The rough rule for how much memory a model needs:

VRAM ≈ (parameters in billions) × (bytes per parameter) + KV cache
Enter fullscreen mode Exit fullscreen mode

Bytes per parameter depends on quantization:

  • FP16 = 2 bytes → an 8B model ≈ 16 GB
  • Q8 (8-bit) ≈ 1 byte → 8B ≈ 8 GB
  • Q4 (4-bit) ≈ 0.5 byte → 70B ≈ ~35 GB, 32B ≈ ~16 GB

So on a 32 GB card, the sweet spot is a 32B model at 4-bit with plenty of headroom, or a 70B at 4-bit if you keep the context modest. That 70B-at-home case is exactly what 24 GB couldn't do cleanly.

Two things people forget:

  • Context eats VRAM too. The KV cache grows with your context length. A big context window on a big model can add several GB on top of the weights, which is what pushes a "should fit" model into offloading.
  • Offloading isn't failure, it's just slower. Tools will put the overflow layers in system RAM. It runs, it's just not full-speed.

When you want more control: llama.cpp

Ollama is great until you want to tune quantization, context, or GPU layer offload by hand. Then go to llama.cpp with a GGUF model:

# build with CUDA, then:
./llama-cli -m model-Q4_K_M.gguf \
  -ngl 99 \        # offload all layers to GPU
  -c 8192 \        # context length
  -p "Explain KV cache like I'm five"
Enter fullscreen mode Exit fullscreen mode

-ngl (number of GPU layers) is the dial that matters: push it as high as your VRAM allows, drop it if you run out. The Q4_K_M quant is usually the best quality-to-size tradeoff for local use; go smaller (Q3) only if you're desperate for space, since logic degrades fast below 4-bit.

The honest takeaway

32 GB doesn't make you a data center, but it moves the line from "8B toys and heavily-crippled 70Bs" to "comfortable 32B models and usable 70Bs." For building agents or fine-tuning locally, that's the difference between prototyping on your own machine and renting cloud GPUs by the hour.

I wrote up why the architecture (not just the VRAM) makes this shift real, including Blackwell's FP4 support, here: RTX 5090 Blackwell: Why Local AI Developers Can't Ignore It

What's the biggest model you're running locally, and on what card? Curious where people are drawing the cloud-vs-local line now.

Top comments (0)