How much VRAM do you need to run or fine-tune a 70B model?

Serving a 70B model takes about 141 GB of GPU memory for the weights at 16-bit, 73 GB at FP8 and 40 GB at 4-bit, plus about 0.33 GB of KV cache for every 1,000 tokens in flight. Fine-tuning takes about 50 GB with QLoRA, 150 GB with LoRA and 1.1 TB for full fine-tuning with AdamW. In practice, one 80 GB GPU covers 4-bit serving and QLoRA, one H200 or two H100s cover FP8 serving, and full fine-tuning with AdamW needs a whole 8 x B200 machine.

We worked these numbers out from Llama 3.3 70B's config file and the real checkpoint sizes on Hugging Face, and Qwen2.5 72B comes out almost the same. This post shows where each gigabyte goes, which GPU setups work and what they cost per hour, and ends with a short script that runs the same math for any model. We use decimal gigabytes throughout and count an 80 GB GPU as 80 GB.

Where does the memory go when you serve a 70B model?

Three things share the GPU: the weights, the KV cache and a working reserve for the CUDA context, activations and the serving engine itself.

The weights are the easy part, because the checkpoint size on disk is what they take in memory. Here are three public versions of the same model:

Precision	Checkpoint on Hugging Face	Weights in memory
16-bit (BF16)	meta-llama/Llama-3.3-70B-Instruct	141.1 GB
8-bit (FP8)	RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic	72.7 GB
4-bit (AWQ INT4)	casperhansen/llama-3.3-70b-instruct-awq	39.8 GB

We summed each repository's safetensors files on October 13, 2026. The quantized versions come out a few GB above "parameters x bits" because the embedding and output layers stay at 16-bit. Those two layers hold 2.1 billion parameters, or 4.2 GB, and 4-bit formats also store scaling factors.

For the working reserve, we plan with 90% of each GPU's memory. That is close to what vLLM claims by default through its --gpu-memory-utilization setting. Whatever is left after the weights becomes KV cache, minus a few GB vLLM keeps for activations, so treat the KV figures below as upper bounds.

How big is the KV cache for a 70B model?

About 0.33 MB per token, and it grows with every request you serve at once. The formula is 2 (keys and values) x layers x key-value heads x head size x 2 bytes. Llama 3.3 70B has 80 layers, 8 key-value heads and a head size of 128, so each token costs 2 x 80 x 8 x 128 x 2 = 327,680 bytes. Qwen2.5 72B has the same shape and the same cost.

What that means for real traffic:

• One 8,192-token conversation holds 2.7 GB of cache.
• 32 of those at once hold 85.9 GB, more than twice the 4-bit weights.
• One request that uses the full 131,072-token context window needs 43 GB by itself.

The 8 key-value heads matter a lot. Llama 3 uses grouped-query attention, where 64 query heads share 8 key-value heads. With one key-value head per query head, the cache would be 8 times larger, about 2.6 MB per token.

A common rule of thumb adds 20% to the weights to cover the cache and everything else (Modal's VRAM guide uses it, for example). For 35 GB of 4-bit weights, that allowance is about 7 GB, which holds roughly 21,000 tokens. That is enough for a handful of chats but not a busy endpoint. Size the cache from your traffic instead: peak concurrent requests x (prompt + output tokens) x 0.33 MB.

You can also store the cache in FP8 with vLLM's --kv-cache-dtype fp8, which halves it to about 0.16 MB per token. Check quality on your own evaluations before you rely on it.

Which GPUs can serve a 70B model?

Any setup where the weights fit inside 90% of total memory with room left for your cache. These are the configurations we would actually consider:

Setup	Weights	Left for KV cache	Tokens in flight, at most	Listed price per hour
1 x A100 80 GB, 4-bit	39.8 GB	32.2 GB	about 98,000	$1.09
1 x RTX PRO 6000 96 GB, 4-bit	39.8 GB	46.6 GB	about 142,000	$1.35
1 x H200 141 GB, FP8	72.7 GB	54.2 GB	about 165,000	$3.59
2 x H100 80 GB, FP8	72.7 GB	71.3 GB	about 218,000	$5.20
1 x B200 180 GB, FP8	72.7 GB	89.3 GB	about 272,000	$5.50
1 x B300 288 GB, 16-bit	141.1 GB	118.1 GB	about 360,000	$6.94
4 x H100 80 GB, 16-bit	141.1 GB	146.9 GB	about 448,000	$10.40

"Left for KV cache" is 90% of total GPU memory minus the weights, and tokens are that figure divided by 327,680 bytes. Prices checked October 13, 2026: hourly rates listed on the Nodus pricing page, multiplied by the GPU count.

Two setups look fine on paper and fail in practice. Two 80 GB GPUs at 16-bit leave 2.9 GB after the weights, about 9,000 tokens across every request combined. Two 24 GB cards at 4-bit leave 3.4 GB, about 10,000 tokens. Both load the model and then have almost no room to serve it.

Precision also limits your hardware. FP8 runs natively on Hopper and Blackwell GPUs such as the H100, H200, B200 and B300, but the A100 has no FP8 support in hardware, so use a 4-bit checkpoint there.

Our rule: choose the precision from your own quality evaluations first. Then multiply your peak concurrent requests by their total tokens and 0.33 MB, and take the cheapest row with that much room. If even one request needs the full 128K context, you need at least 43 GB of cache on top of the weights.

How much VRAM does fine-tuning a 70B model take?

Far more than serving, because training keeps gradients and optimizer state for every parameter it updates. The ZeRO paper puts mixed-precision training with Adam at 16 bytes per parameter. That is 2 bytes for 16-bit weights, 2 for gradients and 12 for an FP32 master copy plus Adam's two moment estimates.

• Full fine-tuning: 70.55 billion x 16 bytes = 1,129 GB. An 8-bit optimizer from bitsandbytes stores each moment in 1 byte instead of 4, which brings it to 10 bytes per parameter, or 706 GB.
• LoRA: the 16-bit base stays frozen at 141.1 GB. Rank-16 adapters on every linear layer add about 207 million trainable parameters, 0.3% of the model. In FP32 with Adam that is 16 bytes each, or 3.3 GB.
• QLoRA: the base is stored in 4-bit NF4. The linear layers take 34.2 GB, the double-quantized scaling constants about 1.1 GB (0.127 bits per parameter, from the QLoRA paper) and the 16-bit embedding and output layers 4.2 GB. That is 39.5 GB, plus the same 3.3 GB of adapters.

Activations come on top and depend on sequence length. With gradient checkpointing, each layer keeps only its input: 4,096 tokens x 8,192 hidden size x 2 bytes x 80 layers = 5.4 GB per 4,096-token sequence. The FP32 logits over a 128,256-token vocabulary add about 2.1 GB. Call it 7.5 GB per sequence on each GPU, as an estimate.

Method	Before activations	With one 4,096-token sequence	Smallest setups that fit	Listed price per hour
QLoRA	43 GB	about 50 GB	1 x A100 80 GB, 1 x H100, 1 x RTX PRO 6000	from $1.09
LoRA	144 GB	about 152 GB	1 x B200, 1 x B300, 2 x H200, 4 x H100	from $5.50
Full, 8-bit Adam	706 GB	about 766 GB on 8 GPUs	8 x H200 (1,128 GB)	$28.74
Full, AdamW	1,129 GB	about 1,189 GB on 8 GPUs	8 x B200 (1,440 GB) or 8 x B300	from $44.00

Same prices as the serving table. Anything spread over several GPUs needs sharding, such as PyTorch FSDP or DeepSpeed ZeRO-3, so that each GPU holds only its share of the weights, gradients and optimizer state. Two 8 x H100 machines hold 1,280 GB, which leaves about 9 GB per GPU before activations for full AdamW training. That is too tight for most setups.

Can you fine-tune a 70B model on a single GPU?

Yes, with QLoRA on any GPU with 80 GB or more. The QLoRA paper fine-tuned a 65B model on one 48 GB GPU. A 70B Llama 3 needs about 43 GB before activations, so on a 48 GB card such as the L40S or RTX A6000 you have about 5 GB left. That is enough for short sequences only.

LoRA on a 16-bit base fits on one GPU only if it is a B200 or a B300. Full fine-tuning of a 70B model has no single-GPU option.

How do you check the numbers for your own model?

Run the same math against its config. This script reads config.json and the parameter count from the Hugging Face API, then prints the estimates above. It uses only the Python standard library.

"""Rough GPU memory estimate for serving and fine-tuning a Hugging Face model."""
import json, os, sys, urllib.request

def get(url):
    req = urllib.request.Request(url)
    if os.environ.get("HF_TOKEN"):  # needed for gated repos such as meta-llama
        req.add_header("Authorization", f"Bearer {os.environ['HF_TOKEN']}")
    with urllib.request.urlopen(req) as r:
        return json.load(r)

repo = sys.argv[1]
context = int(sys.argv[2]) if len(sys.argv) > 2 else 8192   # tokens per sequence
seqs = int(sys.argv[3]) if len(sys.argv) > 3 else 1          # sequences in flight
cfg = get(f"https://huggingface.co/{repo}/resolve/main/config.json")
params = get(f"https://huggingface.co/api/models/{repo}")["safetensors"]["total"]

head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
kv_per_token = 2 * cfg["num_hidden_layers"] * kv_heads * head_dim * 2   # K and V, 2 bytes each
GB = 1e9

print(f"{repo}: {params / 1e9:.1f}B parameters")
for name, bytes_per_param in [("16-bit", 2), ("8-bit", 1), ("4-bit", 0.5)]:
    print(f"  weights at {name}: {params * bytes_per_param / GB:6.1f} GB")
print(f"  KV cache: {kv_per_token / 1e6:.3f} MB per token, "
      f"{kv_per_token * context * seqs / GB:.1f} GB for {seqs} x {context:,} tokens")
print(f"  full fine-tune, AdamW mixed precision (16 B/param): {params * 16 / GB:,.0f} GB + activations")
print(f"  LoRA on a 16-bit base: {params * 2 / GB:,.0f} GB + adapters + activations")
print(f"  QLoRA on a 4-bit base: {params * 0.5 / GB:,.0f} GB + a few GB + adapters + activations")

We ran it on October 13, 2026 against an ungated copy of Llama 3.3 70B, for 32 conversations of 8,192 tokens:

$ python vram.py unsloth/Llama-3.3-70B-Instruct 8192 32
unsloth/Llama-3.3-70B-Instruct: 70.6B parameters
  weights at 16-bit:  141.1 GB
  weights at 8-bit:   70.6 GB
  weights at 4-bit:   35.3 GB
  KV cache: 0.328 MB per token, 85.9 GB for 32 x 8,192 tokens
  full fine-tune, AdamW mixed precision (16 B/param): 1,129 GB + activations
  LoRA on a 16-bit base: 141 GB + adapters + activations
  QLoRA on a 4-bit base: 35 GB + a few GB + adapters + activations

The 8-bit and 4-bit lines are pure parameter math, so add the few GB that real checkpoints carry, as in the first table. The KV formula assumes standard grouped-query attention. Models with a different design, such as multi-head latent attention, need their own formula.

How do you get a GPU with enough memory on Nodus?

Ask for memory instead of a GPU name. With minMemory and no type, Nodus accepts any GPU with at least that much memory and picks the one that is cheapest to finish your run (GPU guide):

spec:
  resources:
    gpu: {minMemory: 80Gi, count: 1}

Or name a setup from the tables and check the estimate before anything starts. After pip install nodus-compute and nodus login, the dry run prints the expected cost and start time without launching:

nodus get gpus --gpu H200
nodus run --dry-run --gpu H100:2 -- python serve.py

For fine-tuning, a TrainingJob on the nodus/sft runtime (Beta) picks GPUs from presets measured for each model, method (Full, LoRA or QLoRA) and quantization. nodus get trainingruntime/sft -n nodus -o yaml lists them, and nodus apply -f sft.yaml --dry-run -o yaml shows what your run would get and cost (training guide). While a job runs, nodus top jobs shows its GPU memory, so you can compare the estimate with reality.

A 70B fine-tune usually runs for hours, so checkpoint it. How to make GPU training survive spot preemption covers the setup, and Nodus vs Modal prices a few common training jobs on both platforms.

FAQ

Why does vLLM run out of memory when the weights fit?

vLLM checks at startup that at least one request of the model's maximum length fits in the KV cache. Llama 3.3 70B defaults to 131,072 tokens, which needs 43 GB, so a tight setup refuses to start. Lower --max-model-len to the longest request you actually serve, or make room with an FP8 KV cache.

Do Mixture-of-Experts models follow the same math?

For weights, count total parameters, not active ones. Every expert has to sit in GPU memory even if only a few run for each token, so a model with 70B total parameters needs the same weight memory as a dense 70B.

Can a 70B model run on one 24 GB or 32 GB GPU?

Not entirely in GPU memory, since even the 4-bit weights are about 40 GB. Tools such as llama.cpp can keep some layers in CPU RAM, but those layers then run at CPU memory speed.

How much disk does a 70B fine-tune need?

The 16-bit weights are 141 GB. A resumable full fine-tuning checkpoint adds the FP32 master weights and both Adam moments, about 12 bytes per parameter or another 850 GB. On Nodus, a pinned Hugging Face revision is imported once into a Volume and reused by later TrainingJobs, so you do not pay GPU time to download it again.

More guides like this on the Nodus Compute blog.