I stopped pasting my company's code into free chatbots. A local model is where it lives now.

I watched an engineer paste a chunk of our internal codebase into a free chatbot the other day. He didn't think twice. And honestly, I don't blame him, because for a long time there was no good alternative. It was either the free cloud thing or nothing.

Here's the thing nobody says out loud: every prompt you type into those services is training data or a support ticket or both. And when the prompt is a function from your company's proprietary system, that's not a harmless question. That's a leak you didn't even notice.

I'm the IT and network admin here. Part of my job is being the guy who says the uncomfortable thing about data. So a while back I set myself a rule: anything that touches company IP does not leave this building. And if I wanted the convenience of an AI assistant, that meant I needed one that runs inside my four walls.

The part that changed my mind

The shift wasn't about benchmark scores. It was about the phrase "what goes in stays in." I can look at the machine that's running my LLM and know exactly what it does with my text. No ambiguous privacy policy, no third party in the chain, no terms of service that changed overnight. That peace of mind is the whole point.

And for my day-to-day work, the local model is genuinely good enough. I use it to explain odd log lines, to draft awkward IT emails, to sanity-check a firewall rule before I click apply. Nothing in that list needs a frontier model. It needs something that understands the question and won't rat me out.

The practical bit

Getting it up and running isn't a weekend project. It's more like an afternoon. You need three pieces working together: a model runner, a friendly web interface, and Docker to glue it together. The setup that finally stuck for me looks roughly like this:

# three pieces, one docker compose file
lmstudio        # runs the quantized model locally
open-webui      # the chat frontend everyone on the team can use
watchtower      # keeps the images from going stale

Quantized models are the secret that makes this feasible on hardware you already own. A 4-bit quantized model runs fine on a machine that would choke on the full version. You trade a little fluency for a lot of privacy, and for internal use it's a trade I make every time.

The engineers can still get their answers. The difference is those answers now cost nothing, leak nothing, and nobody has to explain to a data-protection officer why our source code is sitting in someone else's logs.

That's the pitch, really. Not a faster AI. A safer one. If you've ever winced watching someone hand your company's internals to a stranger, I put together a bundle that walks you through the whole local setup, start to finish, at https://symshah.gumroad.com/l/selfhosted-ai-homelab. It's pay-what-you-want, and it's built for people who'd rather host their own brain than rent one.