The VRAM Wall: Why You Can't Self-Host a Frontier Agent
A field note on the economics of AI sovereignty: the arithmetic that quietly decides you will rent your foundation, not own it.
Every argument about "owning your AI stack" runs into the same wall, and it is made of memory. Not clever memory — the boring kind, measured in gigabytes of VRAM. If you want the stability of running a frontier-class model on hardware you control, you have to pay the entry fee. Here is what that fee actually looks like, in numbers you can check.
The weights alone
The baseline rule for inference is roughly 2 GB of VRAM per billion parameters at FP16, and about 1 GB per billion at 8-bit (FP8 / INT8). Quantization trades a little quality for a lot of memory; 4-bit roughly quarters it again, with a steeper quality cost.
Now plug in a real frontier-class open model. A 685-billion-parameter model is about 685 GB of weights at 8-bit by the per-parameter rule — and real deployments land closer to 800 GB once you count the layers that do not quantize cleanly, the embeddings, and runtime overhead. The 2026 crop of frontier open weights — the 700-billion-to-1.6-trillion-parameter class — do not fit on any single accelerator made. An NVIDIA H200 carries 141 GB of HBM3e, so a 685B model at 8-bit needs five to six of them just for weights, before the model has done a single useful thing.
That is the number people skip when they say "just run it locally." Locally, for a frontier-class model, means a multi-GPU box — often multi-node — before you have served one token.
Context is not free
Weights are the fixed cost. The variable cost is the KV cache: the attention state the model keeps in VRAM for every token in the current context. It grows with the context window, and for the long contexts that make an agent useful, it grows fast.
Take a smaller model to see the shape of it clearly: a 70B model at FP16 spends on the order of 2.5 MB of VRAM per token of context. Fill a 32K-token window and that is roughly 80 GB of KV cache alone — for a model a tenth the size of our frontier example, and on top of its weights. The frontier model's own cache stacks on top of that ~800 GB, not instead of it. Push toward the 100K+ contexts an agentic workload actually wants, across concurrent sessions, and the cache can rival the model. Budget an extra 10–20% over the weight footprint as a floor, and much more if you run long prompts at any real concurrency.
So the honest inference number for a frontier-class agent is not 822 GB. It is 822 GB plus context plus headroom plus the throughput you need to not be waiting on it — comfortably into the terabyte-of-VRAM range for a serious setup.
Fine-tuning multiplies everything
Inference is the cheap half. The reason you would want to own a model rather than rent one is to shape it — to fine-tune it to your domain so it stops drifting when a vendor changes something upstream. Fine-tuning does not cost the weight footprint. It costs a multiple of it.
Full fine-tuning holds the weights, the gradients, and the optimizer state (for Adam-class optimizers, two more full-size tensors per parameter) resident at once. That is why training-class memory for a large model runs several times the inference footprint. Parameter-efficient methods (LoRA and friends) cut this dramatically and are the right call for most people — but they adapt a frozen base you still do not own the pretraining of, and they do not remove the base's multi-hundred-gigabyte resident cost. The moment you want true ownership — your weights, your training, your guarantee that the thing does not change unless you change it — the hardware bill leaves the realm of "a workstation" and enters "a cluster with a cooling plan."
What the wall actually decides
Add it up and the wall is clear. A frontier-class agent you fully own — weights, context, throughput, the ability to fine-tune — is a 1 TB+ VRAM commitment plus the power, cooling, networking, and the standing ML-ops competence to keep it fed. That is not a hobbyist number. It is not even a most-startups number. It is a "this is a line item on the infrastructure budget with its own headcount" number.
Which means, for almost everyone, the rational choice is to rent the model. And renting the model is fine — it is genuinely the right call for most workloads. But it comes with a clause worth reading out loud: when you rent the model, the landlord sets its behavior, on their schedule, for their reasons. Rented foundations move. They get "improved." They get steered from a distance. Your reproducibility is a courtesy the vendor extends, not a property you hold.
The VRAM wall is why that clause is nearly universal. Almost nobody can afford to own the foundation, so almost everybody builds on ground somebody else can repour. That is not a moral failing; it is arithmetic. But it is worth naming, because the industry mostly does not. "Own your AI" is sold as a mindset. It is actually a memory budget, and the budget is enormous.
The pragmatic middle
None of this means give up. It means be precise about which layer you can afford to own and which you cannot.
You will almost certainly rent the model. So spend your ownership budget where it is affordable and where drift hurts most: own your data, own your orchestration, own the layer that decides how the rented model is used, and own the infrastructure it runs on so at least the ground beneath the rented part is yours. Pin versions where the vendor lets you. Measure behavior so you notice when it changes. Assume the model underneath you will move, and build so that when it does, you find out fast and nothing customer-facing breaks silently.
The VRAM wall is real and it is not coming down soon — the frontier keeps getting bigger faster than memory gets cheaper. So treat sovereignty as a spectrum, not a switch. Own the parts you can hold. Rent the parts you cannot. And never mistake a rented foundation for a stable one, however good it is today.
If you're building agent systems that run in production — or you want to see what owning the infrastructure layer looks like in practice — I run support and infrastructure at Pulsed Media. Seedboxes and storage on our own hardware in our own datacenter in Finland. Open-source platform (PMSS, GPL v3), 1Gbps or 10Gbps, EU jurisdiction, 14-day money-back.
Top comments (0)