Building a private AI that runs entirely on your own machine — and the one quiet setting that stood between it and working
By Ashish Nadar — Scholar in AI-Enabled Software
The document was right there.
I had spent the better part of an hour preparing it — every fact about the company, cleanly written, uploaded, connected. Then I asked my new AI assistant the simplest question I could think of.
"Where is the company headquartered?"
It paused. And then, politely and with complete confidence, it told me it had no idea what company I was talking about.
I sat with that for a second. The answer was in the file I had just handed it. It wasn't missing information. It was looking straight past information it already had.
That moment is where this story actually lives. Not in the setup — the setup was easy. In the gap between a system that looks like it works and one that actually does.
Why I Wanted an AI That Never Leaves Home
Start with the discomfort, because that's where the idea came from.
Every time I used a cloud AI tool for something even slightly sensitive, the same question surfaced: where is this data going? With most tools, the honest answer is — somewhere else. Your words leave your machine, travel to a server you'll never see, and come back.
For casual questions, fine. For anything private, that quiet trip is the whole problem.
So I set a constraint that turned into a project: what if the AI never left my machine at all? The model, the data, every conversation — all in one place. Mine.
Not a smaller version of a cloud tool. A genuinely private one, because nothing about it ever goes online.
With cloud AI, your data makes a round trip you never see. With local AI, nothing leaves the machine — ever.
The Setup Was the Easy Part
Here's what surprised me most: you need very little.
An engine to run the model on your own hardware. An open-source model — a free, downloadable brain. A short set of instructions to give it an identity. And your own documents, so it answers from your world instead of guessing.
Four pieces. Within an afternoon I had a generic AI running locally, offline, answering questions without a single byte leaving the machine.
Four components. One afternoon. A private AI running entirely on your own machine — runtime, model, identity and your documents.
That part felt like magic. The next part felt like a wall.
Where Confidence Turns Into a Blind Spot
An out-of-the-box model is a generalist. It knows a little about everything and nothing about you.
So I gave it a name, a role, a professional voice — and one rule I cared about most: never pretend to know things it doesn't. On that, it delivered. When I later asked for a price that appeared in none of my files, it didn't invent a number. It admitted the gap and pointed to who to ask.
That honesty mattered. An AI that confidently makes things up is worse than no AI at all.
But there was still the small matter of it insisting my company didn't exist.
The Bug That Nearly Beat Me
This is the part the tutorials quietly skip. So I won't.
Everything was configured. Document uploaded. Connected. Settings correct. And still —
"I don't have information about that company."
Then I noticed something maddening. When I attached the document by hand, for that one question, the answer was flawless — correct facts, a source citation, exactly right. But the moment I relied on the automatic path? Nothing. Same document. Same question. Two different worlds depending on how the file arrived.
I went in circles. Rewrote the instructions. Swapped in a bigger, smarter model. Flipped every setting I could find. The manual way worked; the automatic way refused, every time.
The cause, when I finally found it, was almost poetic.
The system was built to make the AI reach out and fetch the document itself — like handing someone a library card instead of the book. A large model knows to go use that card. The smaller one I was running just sat there holding it, never walking to the shelf. So it answered from memory — and about my company, its memory was blank.
The fix was to stop handing it a card and start handing it the book. Put the whole document in front of it, every time. No fetching required.
One setting. I changed it, asked again, and the answer came back correct — automatically, with its source attached. I celebrated more than a grown professional probably should.
Same document. Same question. Two completely different results — depending on whether the model was handed the book or just a library card.
How I Actually Built It
Since this is the part fellow builders will want, here's the concrete stack — all of it open-source and free.
The runtime: Ollama.
It downloads and runs open models locally with a single command:
ollama pull llama3.1:8b
You're running a model on your own hardware, offline. It exposes a local API that everything else talks to.
The model: an open-weight Llama.
I ran a small one first (3B), which was fast but flaky at harder tasks, then moved to an 8B for more reliable behavior. On a CPU-only laptop, expect slower responses at the larger size — a real tradeoff.
The interface: Open WebUI.
A self-hosted, ChatGPT-style web UI that connects to Ollama automatically. Run it in Docker:
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
--name open-webui \
ghcr.io/open-webui/open-webui:main
This is also where you give the model an identity (a system prompt) and where the retrieval lives.
The knowledge: built-in RAG.
Open WebUI lets you create a "Knowledge" collection, upload documents, and attach it to your model. It handles the embedding and retrieval — I used the nomic-embed-text embedding model, also served through Ollama, so nothing left the machine:
ollama pull nomic-embed-text
The bug — and the actual fix.
Retrieval worked when I attached the collection manually in chat (via #), but not when it was attached at the model level.
The cause: native function calling. In that mode, the model is expected to call a retrieval tool to fetch the document — and small local models call tools unreliably, so it never fired and answered ungrounded.
The fix was one toggle:
On the knowledge attachment, switch from Focused Retrieval to "Use Entire Document."
That injects the full document into context every message instead of relying on a tool call. For a small document it's ideal — and it sidesteps the tool-calling problem entirely.
Note: For large corpora you'd instead disable native function calling and keep focused retrieval, since injecting everything won't scale.
That single setting was the difference between "I don't know that company" and a correct, cited answer — automatically.
What the Gap Was Really Made Of
None of this was a failure of the model, or the documents, or the idea. Every piece was right.
The gap was one layer of behavior — a system expecting a small model to do something small models don't reliably do. The whole thing worked the instant I stopped asking it to reach, and simply gave it what it needed up front.
That's the pattern worth keeping. The failure wasn't in what the system knew. It was in how it was asked to reach for it.
And two things turned out to matter more than raw capability: whether the assistant is honest about what it doesn't know, and whether it's actually handed what it does. Get those two right and a modest machine running open tools becomes something you'd genuinely trust.
The Question Worth Sitting With
The next time you build something that looks finished — configured, connected, green across the board — before you call it done, ask the simplest question you can and watch what it actually does.
Because the distance between "this should work" and "this works" isn't usually a broken part.
It's one quiet assumption nobody thought to check.
Ashish Nadar is a Scholar in Secure Cloud, Operational Excellence, and AI-Enabled Software, researching how AI tooling reshapes the way systems are built, secured, and trusted.
Portfolio → ashishnadar.com
Read more on Medium → medium.com/@ashish.w.nadar
Top comments (0)