My local AI ran on the CPU for a week and I didn't notice

My first Ollama setup felt fast. One evening, one model, one question, a clean answer streaming back at a speed that made cloud AI feel unnecessary.

A few days later, with a bigger model and a longer question, the cursor crawled. No error. No warning. Just slow, in a way that made me suspect the model, the network, my own question — everything except the actual cause, which turned out to be almost embarrassingly mundane.

Here's what was actually going on, and the two things that caused it.

The setup

Ollama runs as its own service on Proxmox, with the GPU passed through instead of running on the CPU alone. The whole point of local AI, for me, is that no request ever leaves the house — but without usable speed, that's a nice principle nobody actually uses day to day.

In front of it sits a web UI for chatting from the browser; behind it, the Ollama API for whatever I automate later. Access works exactly like the rest of the homelab: the identity provider in front, reachable only through the tunnel, locked down the same way as everything else. No port opened just because it's "only my own chat."

The real work wasn't standing the service up. It was sizing it correctly — which model actually fits the GPU I have, what context length is realistic, and making sure the GPU Proxmox reports as "available" is the one Ollama actually gets.

Mistake 1: a model that almost fit — and Ollama never said a word about it

VRAM is a hard limit, not a soft warning. A model that fits entirely on the GPU runs there and runs fast. A model that almost fits gets partially offloaded to the CPU by Ollama, silently, with no error and nothing in the response that flags it happened.

That's what makes it dangerous. It doesn't feel like a failure — it feels like "a bit sluggish today," and a bit sluggish points you everywhere except the graphics card sitting there fully idle.

It got worse when I raised num_ctx — the maximum context length, how much text the model can hold in its head at once — because I wanted longer conversations. A bigger context needs memory on top of the model itself, and that extra chunk was exactly what pushed the whole thing past the GPU's limit, even though the model alone would have fit fine.

The lesson: model size and context length share the same VRAM budget, not separate ones. After changing either one, check actual GPU utilization — not just whether the answer "feels" fine.

Mistake 2: another VM already had the GPU, and nothing told me

GPU passthrough on Proxmox isn't multi-tenant. A passed-through card belongs to exactly one VM or container, never several at once.

I'd briefly set up a second VM using the same card for something unrelated, then just shut that VM down afterward instead of removing its passthrough config. On the next host reboot, that VM grabbed the card again automatically, before the Ollama container even started — and Ollama fell back to plain CPU inference without complaint, because from its point of view there simply was no GPU.

No crash, no warning. Just a service running roughly ten times slower than it should for the better part of a week, until I noticed while cleaning up the VM list.

The lesson: GPU pinning is exclusive and renegotiated on every boot. After a host restart, it's worth a quick check of which VM actually holds the card before blaming the AI service for being slow.

The setup, condensed

Same result, without the quiet CPU week:

1. Know the VRAM ceiling before pulling a model, not after. How much graphics memory is actually available, and how much does the model need at the quantization you picked?
2. Set num_ctx deliberately, not generously. Every increase eats memory the model itself needed.
3. Check actual GPU utilization after any change — not the response speed you feel, the number the hardware reports.
4. Assign GPU passthrough to exactly one VM, and fully remove test configurations on any other VM — don't just power it off.
5. Re-check which VM holds the card after every host reboot, before assuming the AI service itself is the problem.
6. Keep access locked down the same way as everything else — identity provider in front, reachable only through the tunnel, no exception just because it's "only the chat."

What changed

The gap between a local model that feels cloud-fast and one that crawls rarely comes down to the model itself — usually it's VRAM running out quieter than expected, or a card another VM already claimed. The reason for doing this at all hasn't changed: every request stays on my own network, nothing leaves the house, and that's worth the occasional re-tuning.

---

I write about self-hosting in more depth — one homelab component per issue, always including what broke — in my German-language newsletter Souveränes Homelab: souveraenes-homelab.beehiiv.com. The English write-ups land here.

Määäx