70B dense models are the point where a single consumer GPU stops being enough. Here is what the memory math says for the three setups people usually compare.
The baseline: Llama 3.3 70B at Q4_K_M
About 40.7 GB of weights. Add a KV cache and an activation buffer:
| Context | Approx. memory |
|---|---|
| 4K | ~46.1 GB |
| 8K | ~47.5 GB |
| 32K | ~55.7 GB |
| 128K | ~88.7 GB |
Option 1: two RTX 3090s (48 GB total)
- 4K: 46.1 of 48 GB, 96% of the pair. It loads, but there is no margin, so plan for a headless box.
- llama.cpp's auto-fit keeps ~1 GiB free per card, so its effective budget is ~46 GB. Pass
-ngl alland an explicit-cso a model that doesn't fit fails loudly instead of silently offloading layers to CPU. - No NVLink needed: in the default layer split only boundary activations cross cards.
- Want comfort? A generic Q3_K_M of a 70B at 8K is ~38.4 GB.
Option 2: DGX Spark 64GB
- Llama 3.3 70B Q4_K_M at 32K: ~55.7 GB, inside the comfortable line. This is "the dense-70B machine".
- GPT-OSS 120B does not fit: MXFP4 at 8K is ~66.5 GB, Q4_K_M at 8K is ~62.9 GB (98% of the pool, with the OS sharing it).
Option 3: DGX Spark 128GB
- Everything above, plus the 100B-class MoE releases: GPT-OSS 120B, GLM-4.5-Air Q4_K_M (~68.7 GB at 8K), Llama 4 Scout Q4_K_M (~70.7 GB at 8K).
- At 4K context, 116 of the 129 models in the index fit comfortably (111 on the 64GB, 103 on a 32GB RTX 5090).
Speed is a different story
Both Sparks have 273 GB/s. Generation reads the whole dense model per token, so Llama 3.3 70B Q4_K_M tops out around 7 tok/s on either. An RTX 5090 moves 1,792 GB/s, about 6.6x more, but only on models it can hold. MoE models read only active experts, which is why they feel much faster on a Spark.
Rule of thumb
- Dense 70B, short context, budget: dual 24GB cards.
- Dense 70B with long context: Spark 64GB.
- 100B+ MoE: Spark 128GB.
Full walkthroughs with commands:
- Dual 3090 + llama.cpp: https://quantized.uk/cookbook/dual-gpu-70b-llamacpp/
- DGX Spark 64GB vs 128GB: https://quantized.uk/cookbook/dgx-spark-64gb-vs-128gb/
- Try your own model/context: https://quantized.uk/tools/vram-calc/
All numbers are calculator estimates; if your measured usage differs, I'd like to hear it.
Top comments (2)
One number the table leaves implicit, and it is the one that decides the 3090 pair: for a 70B dense model at bf16 the KV cache is 320 KiB per token (80 layers, 8 KV heads, head_dim 128, K and V). That is 0.33 GB per 1k tokens of context, 2.68 GB per 8k.
Which means the pair's ceiling is not 8k. Q4_K_M weights at 40.7 GB against llama.cpp's ~46 GB budget leaves about 5.3 GB, so 16k tokens of KV if you run with no activation buffer at all and roughly 4k once you give activations and the compute buffer 4 GB. The 8k row at 47.5 GB is already past that budget, so "no margin" is understating it: at 8k the pair has negative room, not thin room.
If you want context on the pair, the lever is KV dtype, not a smaller weight quant. FP8 KV halves the per-token cost to 160 KiB and puts 8k inside the same budget. Q4 to Q3 buys margin only, because weights are a fixed per-token read and context is the variable one.
The 7 tok/s figure is weights-only, which is fair for the Spark. At 4k add 1.31 GB of KV to the 40.7 GB read: 42.0 / 273 GB/s = 154 ms, 6.5 tok/s. It only matters if you are measuring, not sizing.
双 3090 在 4K 就用到 96% 这个数字很说明问题：能装下不等于能用，KV 稍微涨一点、或者多开一个并发就得往 CPU 卸载，速度直接掉下来。还有个容易忽略的点：Spark 只有 273 GB/s 带宽，70B Q4 每 token 要读 40.7GB 权重，算下来理论上限也就 6-7 tok/s 左右，显存大不代表跑得快。