Cloud LLM APIs are convenient until they aren't: rate limits, price hikes, privacy concerns when your prompts contain client data, or just the principle that your tools shouldn't phone home.
I run a local model stack on modest hardware for exactly these reasons. Here's the setup that works in practice — tested on real workloads, not benchmarks.
The hardware reality
You don't need a GPU cluster. A used machine with 32GB RAM runs quantized 7-8B models fine for most assistant/coding tasks. The ceiling is context length and tokens/sec, not capability — a well-quantized Q4 model gets you 80% of what GPT-4-class APIs do for routine work.
What matters more than hardware:
- Ollama for model serving — dead simple, OpenAI-compatible API
- A quantization that fits your RAM — Q4_K_M is the sweet spot; Q8 if you have headroom
- A front-end you actually use — Open WebUI if you want chat, or hit the API directly from scripts
The setup that works
# install ollama, pull a solid general model
ollama pull llama3.1:8b-instruct-q4_K_M
ollama serve
# OpenAI-compatible endpoint at localhost:11434
curl localhost:11434/v1/chat/completions -d '{
"model": "llama3.1:8b-instruct-q4_K_M",
"messages": [{"role":"user","content":"hello"}]
}'
That endpoint drops into anything that speaks the OpenAI API — most agent frameworks, IDE plugins, scripts. Swap the base URL, keep your code.
Where local actually wins
After months of daily use, the honest assessment:
Local wins at: privacy (client/legal data never leaves the box), cost at volume (flat hardware vs per-token), offline reliability, and tasks where a 8B model is genuinely enough — classification, extraction, summarization, first-pass drafting.
Cloud still wins at: hard reasoning, long-context analysis, and anything where the 20% capability gap matters. The pragmatic setup runs BOTH — local for routine/private work, cloud for the hard stuff. Most people overspend on cloud for tasks a local model handles fine.
The gotchas nobody mentions
- Context is RAM-hungry — a 8B model at 4k context is fine; at 32k you need the quant + memory headroom
- First-token latency on CPU-only is seconds, not milliseconds — fine for async tasks, annoying for chat
- Model choice matters more than quantization — a good Q4 beats a bad Q8 every time
I packaged the full setup — model recommendations by use case, the Ollama configs, Open WebUI tuning, and the hardening checklist (auth, network exposure, update hygiene) — into a Local LLM Setup Kit. It's $21, zip file, ~30 minutes to a working offline stack:
→ https://02b912d7172995.app.lhr.life/?sku=local-ai-setup-kit
Free starting point — the security checklist covers the basics: https://02b912d7172995.app.lhr.life/free
What's the first thing you'd run locally if cost and privacy weren't factors?
Top comments (0)