Let's be honest up front: if you want fast, cheap, high-quality generation, call an API. A CPU VPS will not beat a datacenter full of GPUs, and anyone telling you otherwise is selling something.
So why self-host inference at all? One reason, and it's a good one: the model on your server never sends your prompts anywhere.
What CPU inference actually looks like
You can run real models on CPU with enough RAM. A 7–8B model quantized to 4-bit works. A 13B works. You can even push a 30B-class model if you've got the memory. What you can't do is make it fast — output comes at a few tokens per second, not the instant stream an API gives you.
That's the honest trade. For interactive chat it's frustrating. For background work — summarizing documents overnight, classifying a queue, enriching data on a schedule — a few tokens per second is completely fine, and nobody's watching the clock.
The RAM math
The model has to sit in memory, quantized or not, plus overhead for context and the runtime:
7–8B, 4-bit — around 6–8 GB. Runs on a mid plan.
13B, 4-bit — roughly 10–16 GB.
30B-class, quantized — 24–48 GB, depending on quantization.
Bigger, or higher precision — you're into 64–80 GB fast — and past 80 GB no single Pro box fits, still CPU-slow.
This is why "run a local model" quietly becomes a high-memory question. The model is the memory footprint. Add a RAG index or agents on the same box and the numbers stack.
Ollama vs vLLM, briefly
Ollama is the easy door in — install, ollama run, done. It's the right tool for a private single-user setup where you just want the model available. vLLM is built for serving throughput: more setup, better under concurrent load, worth it when you're actually handling volume. Start with Ollama; reach for vLLM when you're serving real traffic.
Where private-first genuinely wins
The case for self-hosted inference isn't speed or cost — it's that some data can't leave. Legal documents. Medical records. Proprietary code. Anything where sending the prompt to a third-party API is off the table for policy or trust reasons. A slower model that runs entirely on your machine beats a fast one that logs everything you send it.
And if the data is that sensitive, the payment probably should be private too. Renting the box with crypto and no KYC keeps the whole chain — server, model, prompts, billing — off anyone's identity records. That's the actual pitch: not "cheaper inference," but "inference nobody else can see."
Bottom line
For speed and quality, use an API — no shame in it. Self-host when privacy is the requirement and slower is acceptable. Size for the model plus its context plus anything else sharing the box, and don't expect GPU speed from CPU.
When the model needs real memory, the Pro line runs 32 to 80 GB with a dedicated IP and nightly backups — enough to hold a serious quantized model with room for context around it.
Top comments (0)