How Much VRAM Do You Really Need to Run a 70B LLM?

Peter Gedeon — Wed, 23 Sep 2026 06:20:26 +0000

How Much VRAM Do You Really Need for a 70B LLM?

Running a large language model locally sounds simple until you start looking at GPU memory.

A model has 70 billion parameters. Your GPU has 24 GB, 32 GB, 48 GB, or maybe 80 GB of VRAM.

So will it fit?

Unfortunately, parameter count alone does not answer that question.

To estimate how much VRAM an LLM actually needs, you need to consider at least four things:

Model weights
Quantization
KV cache
Runtime overhead

And if the model does not fit entirely in VRAM, you also need to think about CPU offloading and multi-GPU inference.

Let's break it down.

The Simple VRAM Calculation

At the most basic level, the memory required for model weights can be estimated with:

Model memory ≈ parameters × bits per parameter ÷ 8

For a 70-billion-parameter model:

Precision	Approx. raw weight memory
FP16 / BF16	140 GB
INT8	70 GB
6-bit	52.5 GB
5-bit	43.75 GB
4-bit	35 GB
3-bit	26.25 GB

These numbers are only a starting point.

A "4-bit" model does not necessarily occupy exactly 35 GB in VRAM.

Quantization formats often include scales, metadata, higher-precision tensors, and other overhead. Different quantization methods can therefore produce noticeably different memory requirements even when both are described as "4-bit."

This is why looking only at the advertised quantization level can be misleading.

Why Quantization Changes Everything

Without quantization, a 70B model is far beyond the memory capacity of normal consumer GPUs.

At FP16:

70 billion × 2 bytes ≈ 140 GB

That means even an 80 GB accelerator cannot hold the raw weights entirely in memory.

At INT8:

70 billion × 1 byte ≈ 70 GB

Now the model becomes feasible on very large accelerator cards, although there still needs to be room for runtime overhead and KV cache.

At approximately 4-bit:

70 billion × 0.5 bytes ≈ 35 GB

Suddenly the model becomes practical on configurations with around 48 GB of GPU memory.

This is why quantization has been so important for local LLM inference.

Instead of needing several enterprise accelerators, heavily quantized models can sometimes run on workstation GPUs or multiple consumer cards.

The tradeoff is that increasingly aggressive quantization can affect model quality.

So the goal should not simply be:

Use the smallest model possible.

A better goal is:

Use the highest-quality quantization that fits comfortably within the hardware you have.

A 24 GB GPU Is Not Enough for a Typical 70B 4-Bit Model

Cards with 24 GB of VRAM are extremely useful for local AI.

But 24 GB is still substantially below the roughly 35 GB theoretical weight requirement of a 70B model at exactly 4 bits per parameter.

That means something has to change.

You can:

use a more aggressive quantization
move part of the model into system RAM
split the model across multiple GPUs
use a smaller model

CPU offloading is particularly interesting because it allows models much larger than GPU memory to run.

But capacity and performance are two different problems.

CPU Offloading: More Capacity, Less Speed

Suppose you have:

GPU VRAM: 24 GB
System RAM: 64 GB

You potentially have enough total memory to store a quantized 70B model.

The runtime can keep some layers in GPU memory while storing the remaining layers in normal system RAM.

That works.

But system RAM bandwidth is dramatically lower than modern GPU VRAM bandwidth.

For autoregressive LLM inference, weights may need to be accessed repeatedly as each token is generated.

If part of those weights must travel between CPU memory and the GPU, token generation can slow considerably.

So when evaluating hardware for local AI, I separate two questions:

Can it run?

and

Can it run fast enough to be useful?

Those are not the same thing.

Why 32 GB of VRAM Is an Interesting Middle Ground

A 32 GB GPU gets much closer.

The theoretical size of a 4-bit 70B model is still around:

35 GB

So a straightforward 4-bit model will generally still exceed 32 GB before accounting for additional memory requirements.

However, more aggressive quantizations can bring some 70B-class models within range.

That makes 32 GB cards interesting for users willing to trade some model fidelity for the ability to stay mostly—or entirely—on the GPU.

But fitting the weights is only part of the problem.

There is another large consumer of GPU memory:

context.

The Forgotten Part of LLM Memory: KV Cache

When an LLM processes a conversation, it stores information associated with previous tokens in a structure called the KV cache.

The longer the conversation becomes, the larger that cache becomes.

So a model that fits comfortably at:

4,096 tokens

may consume considerably more memory at:

32,768 tokens

or:

131,072 tokens

This creates one of the most common mistakes when estimating GPU requirements.

Someone downloads a model that appears to require 22 GB of memory and assumes it will fit comfortably on a 24 GB GPU.

Then the runtime loads:

model weights
KV cache
CUDA buffers
temporary tensors
kernels
graph allocations

and suddenly there isn't enough memory.

Context Length Can Change the Hardware Requirement

Consider two people running the exact same model.

User A

Runs:

4K context
1 concurrent request

User B

Runs:

128K context
4 concurrent requests

They may have dramatically different memory requirements even though they're using the same model.

This is especially important for:

coding agents
document analysis
RAG systems
long conversations
autonomous agents
multi-user inference servers

If you're buying hardware for AI, model size should never be considered separately from intended context length.

Runtime Overhead Matters Too

Even after accounting for weights and KV cache, you should avoid planning a system that uses exactly 100% of available VRAM.

Inference frameworks need working memory.

Depending on the runtime, that can include:

temporary tensors
CUDA kernels
attention workspace
graph capture
memory fragmentation
quantization buffers

A machine that technically fits a model with 200 MB of VRAM remaining may be much less useful than one with several gigabytes of headroom.

For that reason, I generally think of GPU memory as a budget, not a hard model-size limit.

What About 48 GB?

Around 48 GB of VRAM is where 70B-class local inference becomes significantly easier.

A 4-bit 70B model with a theoretical weight size around 35 GB leaves substantially more room for:

quantization overhead
KV cache
runtime allocations
longer context

That does not mean every 70B model and every context configuration will fit.

But compared with 24 GB or 32 GB, you have far more flexibility.

This is one reason older professional GPUs with large VRAM capacities can remain interesting for AI even when newer gaming GPUs have considerably more raw compute.

For LLM inference, sometimes:

the slower GPU that fits the whole model is more useful than the faster GPU that doesn't.

Two 24 GB GPUs vs One 48 GB GPU

This is another common question.

At first glance:

24 GB + 24 GB = 48 GB

So two 24 GB GPUs should behave exactly like one 48 GB GPU.

Not quite.

A runtime can distribute model layers or tensors across both GPUs, allowing the combined memory capacity to hold a larger model.

But the GPUs still have physically separate memory pools.

Communication must occur over:

PCI Express
NVLink on hardware that supports it
another interconnect

The topology and inference framework therefore matter.

Two GPUs can dramatically expand the models you are able to run, but they do not magically become a single GPU.

Still, for local AI enthusiasts, used high-VRAM consumer GPUs can sometimes create very interesting price-to-memory configurations.

VRAM Isn't the Only GPU Specification That Matters

Once the model fits, another specification becomes increasingly important:

memory bandwidth.

LLM token generation frequently involves moving large amounts of model data through memory.

That means two GPUs with similar compute capability can behave very differently depending on:

memory bandwidth
memory architecture
quantization kernels
software support

Other important factors include:

supported data types
Tensor Core capabilities
PCIe generation
multi-GPU topology
CUDA or ROCm support
inference framework compatibility

This is why comparing AI GPUs purely by TFLOPS is often misleading.

A Better Way to Choose a GPU for Local LLMs

Instead of asking:

What is the fastest GPU?

I recommend asking these questions in order.

1. What model do I want to run?

8B?

32B?

70B?

Mixture-of-Experts model?

2. What quantization am I willing to use?

FP16?

FP8?

INT8?

Q6?

Q5?

Q4?

3. What context length do I actually need?

4K?

32K?

128K?

More?

4. Can the model fit entirely in VRAM?

If not:

Can it be split across GPUs?
Can some weights be offloaded to RAM?

5. What performance do I need?

There is a huge difference between:

2 tokens/sec

and:

50 tokens/sec

Both configurations technically "run" the model.

Only one may be pleasant to use interactively.

Quick Rule of Thumb for a 70B Model

Here is a deliberately simplified way to think about it.

24 GB VRAM

A normal 4-bit 70B model will not fit entirely in VRAM.

Expect aggressive quantization, CPU offloading, or multiple GPUs.

32 GB VRAM

Closer, but still below the theoretical size of a standard 4-bit 70B model.

Aggressive quantization may make some configurations possible.

48 GB VRAM

A much more comfortable target for 4-bit 70B-class inference.

Context length and runtime overhead still matter.

80 GB VRAM

Enough for approximately 8-bit weights in theory, although KV cache and runtime overhead must still be accounted for.

FP16 70B remains far above the capacity of a single 80 GB GPU.

Don't Buy a GPU Based on Parameter Count Alone

The most important lesson is that:

parameters ≠ VRAM requirement

The real calculation is closer to:

Model weights
+ KV cache
+ runtime overhead
+ safety margin
= required GPU memory

And even after answering that question, you still need to consider memory bandwidth and software support to estimate actual performance.

I built CompareAIHardware around exactly this problem: comparing GPUs, accelerators, VRAM capacity, memory bandwidth, and model requirements from the perspective of people actually trying to run AI workloads locally.

The next time you see someone ask:

"Can I run a 70B model on my GPU?"

the correct answer probably isn't simply yes or no.

The better answer is:

Which 70B model, which quantization, which context length, and how much of it needs to stay in VRAM?

If you're building a local AI machine, those four questions can save you a very expensive GPU purchase.

DEV Community: Peter Gedeon