Run AI Locally: Ollama on a Cheap VPS, No GPU Needed

My monthly AI bill crossed ₹4,000 last quarter, and I couldn't point at a single thing I had to show for it. Chat subscriptions, API credits for little scripting experiments, a writing assistant I used twice. Renting intelligence, one invoice at a time.

So I did what I always do when a subscription annoys me: I tried to self-host it. And it turns out you can run genuinely useful language models on a cheap VPS with no GPU at all — not fast, not frontier, but good enough for a surprising number of daily tasks, private, and priced at whatever you're already paying for the server.

Here's the honest version of that experiment, including where it falls flat.

What "no GPU" actually gets you

The mental model that holds everything together: a 3-billion-parameter model quantized to 4-bit fits comfortably in ~2 GB of RAM and runs fine on a modern CPU. An 8-billion model wants ~5 GB and runs acceptably — think "patient typing," not ChatGPT snappiness. Past that, without a GPU, you're waiting long enough that you'll stop using it.

So the realistic tier list for CPU-only:

• 3B models (llama3.2:3b, phi3.5, qwen2.5:3b) — fast enough for summarizing, reformatting, explaining errors, drafting. My most-used size by far.
• 7–8B models (llama3.1:8b, mistral, gemma2) — noticeably smarter, noticeably slower. Fine when the answer matters more than the wait.
• 13B+ — technically runs on a big enough box. Practically, you'll make tea.

RAM is the whole game. A $6–10 VPS with 8 GB is the sweet spot; 4 GB works if you only run 3B models and keep the rest of the box lean.

Install: one command, because of course it is

Ollama has done for local models what Docker did for containers — made the happy path one line:

curl -fsSL https://ollama.com/install.sh | sh

That gives you the ollama CLI and a systemd service listening on 127.0.0.1:11434. Pull and run a model:

ollama run llama3.2:3b

First run downloads the weights (a couple of GB), then you're chatting in the terminal. ollama ps shows what's loaded, ollama rm <model> frees the disk. The model catalog is huge — run ollama search on newer versions or browse the site.

Make it reachable from other machines on your network — and only your network:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

sudo systemctl restart ollama

Warning, and I mean this: do not expose port 11434 to the internet. There is no authentication. None. Ollama assumes a trusted network, and the internet is not one. If you want access from outside, put it behind a reverse proxy with TLS and basic auth — we have a full guide on nginx reverse proxy setup — or better, wire it into your VPN. I use my existing WireGuard setup; zero new attack surface.

The part that makes it actually usable: a UI

Chatting in a terminal is fun for exactly one evening. What makes local models part of your daily workflow is Open WebUI — the friendliest self-hosted LLM frontend going, and it speaks to Ollama natively. One container:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart unless-stopped \
  ghcr.io/open-webui/open-webui:main

Point your browser at port 3000 and you're done. Multiple models, chat history, web search, document upload, user accounts — it feels like the hosted products, except the model and your prompts never leave your server.

If your VPS is already running other containers, this slots into a docker-compose file in about ten lines. Same story as the rest of our self-hosted stack — the full media server build uses the same pattern.

Where local wins, and where it doesn't

Wins, honestly earned:

• Privacy. Drafts, logs, half-formed thoughts — none of it leaves the box. For a privacy nerd who's been deGoogling everything else, this closes a big gap.
• Flat cost. The VPS was already paid for. The marginal cost of a thousand prompts is electricity.
• No rate limits, no policy drift. The model does what it did yesterday and will do tomorrow.

Losses, just as honest:

• It will not match the frontier. Complex reasoning, long multi-step coding tasks, anything requiring current knowledge without RAG — the big hosted models win, and it's not close.
• Speed. First token takes a few seconds on CPU for 8B models. You adapt, but you notice.
• You own the uptime. When it's down at 2 AM, that's your 2 AM.

My actual usage settled into a pattern: local for drafting, summarizing, reformatting, and "explain this log line" questions. Hosted for the hard ten percent. That split cut my AI spending by roughly three quarters — the subscription I kept gets used for things that genuinely need it.

Worth it?

If you're already paying for a VPS and you like owning your tools: absolutely, the evening it takes to set up is paid back within a month. If you'd have to rent a server just for this, do the math against your actual API bill first — and remember the real prize here isn't the money.

Every year there's a new must-rent service, and every year self-hosting gets easier. When AI stops being cheap — and pricing history says it will — the people who kept a local option warm will shrug and keep working. That's the bet I'm making, and so far it's a good one.