Three weeks ago I got tired of pasting client emails into a cloud chat and wondering, on a slow Tuesday night, whether anyone was doing anything interesting with them. So I rebuilt my little drafting habit around a local model. Not a research project. Just my normal day-to-day work: replying to clients, rewriting boring documentation, turning bullet points into paragraphs.
I want to be honest, because most writeups about self-hosted AI read like press releases. It is not a replacement for GPT-4 level reasoning. It is a replacement for the 80% of my day where I don't need reasoning, I need a decent first draft that doesn't embarrass me.
The stuff that genuinely got better:
- Confidential text stays on my desk. I send drafts of contracts, vendor notes, and internal memos to a model running on my own hardware. That one win quietly settles most of my objections.
- No rate limits at 2 PM. The free tier of my old provider started throttling me exactly when I was busiest. A local model doesn't have a quota. It just runs.
- It made me a better editor, weirdly. Because the output is a notch below cloud frontier models, I actually read it instead of copy-pasting. I catch more of my own mistakes now.
- Predictable cost. The only per-word expense is electricity, which is roughly pocket change on a moderately sized GPU.
Now the part nobody writes about. The places it falls over:
- Math. Oof. A local 7B model struggles with anything past basic arithmetic if you let it free-write. I do sums in a calculator and treat the model's numbers as decorative.
- Long context. Sleeping Beauty plots and 5,000-line config files are genuinely hard for smaller quantized models. Keep your prompts and your inputs short and life gets better.
- It won't hold your hand on novel APIs. If you're asking about something released last week, a local snapshot doesn't know it. I treat it as a model with a knowledge cutoff and plan accordingly.
If you take one technical step here, make it quantization. Running a 4-bit version of a decent model on a machine that used to feel old turns it from a science project into something that answers in a few seconds. My laptop has quietly become a home server I did not intend to build.
The setup that actually worked, in one picture:
# one container to serve models, one to give it a web UI
docker run -d -p 8080:8080 ghcr.io/open-webui/open-webui
That's it. Point a browser at it, load a quantized model, and you have a private ChatGPT-shaped thing on your network. I stopped thinking of it as 'the AI server' and started thinking of it as 'the thing that drafts my emails.' That reframe is what made the habit stick.
If you'd rather not spend a weekend threading the pieces together, the honest shortcut is a bundle that ships LM Studio, Open WebUI, and Docker pre-arranged with quantized models and no cloud dependency: https://symshah.gumroad.com/l/selfhosted-ai-homelab. I don't use it myself (I rolled my own), but it's a legitimately painless way to get to the same place if your evenings are short. Either path, the point stands: the best AI box is the one you own.
Top comments (0)