This article was originally published on runaihome.com
The Llama 3.1 405B instruction-tuned model is the most capable open-weights model Meta has released. Running it locally means having 243 GB of fast memory — not a distributed cluster, not a $50,000 server rack, just a box that fits on a desk. The only consumer hardware that meets that bar is the Mac Studio with M3 Ultra, and even that has become significantly harder to buy in 2026 than it was at launch.
Before you spend $4,000 to $6,000 on one, here is the real picture: which models actually run, what performance to expect, and the supply crisis that changed the math.
Why GPU-only setups fail above 48 GB
The RTX 4090 is the fastest consumer GPU for local AI — 1,008 GB/s of memory bandwidth, excellent CUDA ecosystem, and 24 GB VRAM. It handles every model that fits in those 24 GB extremely well.
A Llama 3.1 405B in Q4 quantization weighs 243 GB. An RTX 4090 holds about 10% of that. The other 90% lives in system RAM and crosses the PCIe bus on every single token generated. PCIe 5.0 x16 delivers roughly 30–35 GB/s in real-world throughput — around 30× slower than VRAM bandwidth for the weights that got offloaded.
A dual-RTX-4090 setup gives you 48 GB of VRAM at roughly the same bandwidth. That still offloads 195 GB of a 405B Q4 model, pushing token generation below 2 tokens per second. You can watch the cursor blink between words.
The Mac Studio M3 Ultra avoids this entirely. Its unified memory — 96 GB base, or more in earlier configurations — sits on the same memory controller at 819 GB/s, accessible to both CPU and GPU compute at full speed. There is no bus to cross, no offload hierarchy, no performance cliff. This architectural difference is the entire reason Apple Silicon dominates the "massive model on a single box" benchmark category. At 100B+, the dual-GPU setup barely functions while the Mac Studio runs conversationally.
The 2026 DRAM shortage: what happened to the 512 GB option
When Apple launched the 2025 Mac Studio, the M3 Ultra could be ordered with up to 512 GB of unified memory — the only consumer purchase capable of loading DeepSeek R1's 671 billion parameters entirely in-memory. That option is now gone.
In March 2026, Apple quietly removed the 512 GB memory upgrade from the Mac Studio configuration page, citing supply constraints driven by a global DRAM shortage. The 256 GB upgrade price simultaneously jumped by $400, putting a 256 GB M3 Ultra at approximately $5,999. During an earnings call, Tim Cook acknowledged the Mac mini and Mac Studio could take "several months to reach supply demand balance."
Then on May 5, 2026, 9to5Mac reported that Apple had cut the last remaining memory upgrade options entirely.
As of May 2026: A new Mac Studio M3 Ultra ships with 96 GB and cannot be configured higher. If you need more memory, your options are pre-owned units from the 2025 launch window, authorized resellers still holding inventory, or waiting for the M5 generation (expected later in 2026 with an M5 Ultra chip that may restore high-memory configurations — no confirmed specs or pricing yet).
This matters enormously to the article's title. "What actually works" has a different answer depending on whether you already own a 512 GB unit or are buying today.
The 100B+ model landscape: not all large models are the same
The phrase "100B+ parameters" covers two fundamentally different architectures:
Dense models touch every parameter on every forward pass. The full weight matrix must reside in memory.
Mixture-of-Experts (MoE) models activate only a subset of "expert" layers per token, so compute is sparse — but every expert's weights still live in memory, because the router can't predict which experts will fire before running them. The compute savings are real. The memory savings are not.
| Model | Type | Total params | Active per token | Q4 file size | Min memory config |
|---|---|---|---|---|---|
| Command R+ | Dense | 104B | 104B | 62.8 GB | 96 GB ✓ |
| DBRX | MoE | 132B | 36B | ~74 GB | 96 GB ✓ |
| Mixtral 8x22B | MoE | 141B | 39B | ~80 GB | 96 GB (tight) |
| Llama 3.1 405B | Dense | 405B | 405B | 243 GB | 256 GB |
| DeepSeek R1 | MoE | 671B | ~37B | ~448 GB | 512 GB |
| DeepSeek V3 | MoE | 671B | ~37B | ~448 GB | 512 GB |
DBRX's 36B active parameter count is the reason it outperforms Mixtral 7B-class models on some benchmarks while fitting in the 96 GB base config. You are not getting a "free" 132B model — you are getting a model with 132B weights loaded but only 36B compute per token. The quality is roughly that of a well-trained 30–40B dense model, not a 132B dense model. Worth knowing before you run inference on it expecting Llama-405B-level quality.
What each configuration can run in practice
M3 Ultra 96 GB ($3,999 — the only config available new today):
Command R+ 104B (62.8 GB Q4_K_M) loads with approximately 30 GB of headroom for KV cache. At 8K context, KV cache overhead for a 104B model adds 2–4 GB — you have room. This is a fully usable configuration for long-context reasoning tasks.
DBRX 132B (~74 GB Q4) is similar — tight but workable at 4K to 8K context. Mixtral 8x22B (~80 GB Q4) leaves less than 16 GB for KV cache; stay under 4K context to avoid overflow.
Llama 3.1 405B does not fit. At 243 GB it exceeds the 96 GB limit at any reasonable quantization — even Q2 brings it to approximately 100 GB, still above the ceiling with no room left for inference state.
M3 Ultra 256 GB ($5,999 at launch — sold out new, used market only):
Llama 3.1 405B at Q4_K_M (243 GB) fits with about 13 GB left for KV cache — enough for 8K context but tight for 32K. If you regularly work with long documents, Q3 quantization drops the model to roughly 180 GB and frees 76 GB for context. The quality degradation from Q4 to Q3 on a 405B model is smaller than it would be on a 7B model — there are enough parameters to absorb the precision loss.
M3 Ultra 512 GB ($7,999 — no longer orderable from Apple since March 2026, used units only):
DeepSeek R1 and V3 671B in 4-bit quantization use approximately 448 GB, leaving 64 GB for KV cache. For short-to-medium context inference (up to 8K tokens), this is a complete, high-quality configuration. Past 16K context, KV cache overhead begins competing for that 64 GB buffer.
Verified benchmark numbers
The following numbers come from published community benchmarks and hardware reviews, all on M3 Ultra hardware with MLX framework unless noted.
Llama 3.1 405B on M3 Ultra 512 GB:
- Q4 quantization: ~31 tok/s at 1K context
- Q2 quantization: ~48 tok/s at 1K context (faster, but quality degradation is noticeable on multi-step reasoning)
- Prompt prefill (time to first token) at 16K context: approximately 10 minutes — this is not a typo, and it is one of the model's real practical limits for long-input use cases
For interactive chat with prompts under 2K tokens, the time to first token is under 30 seconds. The 10-minute penalty only bites when you are feeding the model long documents or large system prompts.
DeepSeek R1 671B on M3 Ultra 512 GB:
- ~17–18 tok/s generation (4-bit quantization, MLX)
- Power consumption: under 200W for the entire system
- Uses 448 GB of unified memory, leaving 64 GB for KV cache
DeepSeek V3 671B on M3 Ultra 512 GB:
- >20 tok/s generation in 4-bit quantization (MLX)
For context on why the GPU comparison breaks down: a dual-RTX-4090 running DeepSeek R1 671B with the bulk of weights offloaded over PCIe would produce under 2 tokens per second — an order of magnitude slower, while pulling 400W+ of system power. The Mac Studio achieves competitive throughput at under 200W and without the complexity of a multi-GPU build.
Top comments (0)