I am building a Local AI Knowledge System on Refurbished Office PCs

That is the whole story: https://cyberdeck-ai-knowledge-system.org/ and it has to be build not downloaded using Ubuntu and Python and Ollama LLMs.

---

title: "I am building a Local AI Knowledge System on Refurbished Office PCs"
published: false
description: "A Beowulf cluster running Ollama and a RAG pipeline over my Obsidian vault. No cloud, no subscriptions, about $150 of second-hand hardware — and a lot of things that didn't work."
tags: ai, opensource, homelab, python

cover_image:

I have a confession: my AI knowledge system runs on computers that were headed for a recycling bin, but made it onto eBay. DeepSeek told me yours cold tell this story:

Three Dell OptiPlex machines, bought second-hand in Sham Shui Po for roughly $150 total. No GPU. No cloud account. No monthly subscription. They sit on a shelf under my desk, connected by a cheap gigabit switch, and together they answer questions about my own notes, documents, and bookmarks.

I call it Cyberdeck, after the sci-fi concept of a machine you build and shape yourself. It's Apache 2.0 licensed. And it is not a product — it's a project you have to be willing to build. I hope we kick off something here and find people to exploit the Apache license to its fullest, by creating their own version, with their own code and dedicated use case.

This Thesis is the build log: what I did, what broke, and the one technical assumption I got badly wrong.

The problem I was actually trying to solve

I am coming from a System Analyst background, lost my job and than AI was the next big hype. Instead of being promoted for my Knowledge Management team build; I lost the corporate game and left instead of turning Director and EVP. AIs are great for reasoning and fit perfectly in what I did before. I'd accumulated years of knowledge in scattered places if I had continued my corporate tenure. I had an Obsidian vault with a few thousand notes. Bookmarks. PDFs. And increasingly, long conversations with cloud LLMs that contained genuinely useful synthesis — which I then lost, because those conversations live in someone else's database and are gone the moment I stop paying.

Three things bothered me:

1. My knowledge was queryable by someone else, but not by me. I could ask a cloud model anything, but I couldn't ask it about my own material without uploading all of it.
2. The subscription model doesn't fit how I learn. I want to own the tool, not rent access to it.
3. I wanted to understand the stack. I've been curious about distributed systems and local LLM inference for years. Reading about MPICH and RAG pipelines only gets you so far.

So I set constraints for myself: must run offline, must use hardware I could afford, must not require a GPU, must be something I could extend myself.

The hardware

Node	Spec	Cost
node01	i5-8500, 16GB RAM, 256GB SSD	~$50
node02	i5-8500, 16GB RAM, 256GB SSD	~$50
node03	i5-8500, 16GB RAM, 256GB SSD	~$50

Plus a gigabit switch I already owned and about $12 of cabling.

If you're in Hong Kong, the second-hand market around Sham Shui Po and Apliu Street is genuinely good for this. Ex-lease office PCs are abundant, cheap, and boringly reliable. That last property matters more than you'd think — I wanted to spend my time on the software, not on hardware troubleshooting.

Why a cluster and not just one machine?

This is the part where I need to be honest, because it's the assumption I got wrong.

I assumed that distributing an LLM across three nodes would make inference faster. It doesn't, not in any meaningful way for a single query.

LLM inference doesn't parallelize the way a typical HPC workload does. You can't easily shard a transformer across machines over gigabit Ethernet without the interconnect becoming the bottleneck. Splitting layers across nodes means shipping activations back and forth, and at 1Gbps that overhead eats any theoretical gain.

Where the cluster does help:

• Throughput, not latency. Three nodes can serve three concurrent queries. One node serves one well and two badly.
• Parallel embedding and indexing. When I re-index my vault, the chunking and embedding work spreads across nodes and finishes roughly 3× faster. This is real, measurable, and it's where MPICH actually earns its place.
• Memory headroom. Each node holds its own model in RAM, so I can run different models for different jobs without swapping.

So the honest framing is: this is a cluster for throughput and capacity, not for speeding up a single response. If you only ever ask one question at a time, one machine is the right answer, and you should build one machine.

I kept the cluster because I re-index often, I run several models, and — frankly — because I wanted to learn how to build one. But I'd rather you know that upfront than spend $150 discovering it.

The software stack

The whole thing is standard, boring, open-source software glued together with Python:

• MPICH for distributing indexing and embedding jobs
• Ollama for local LLM inference (llama3.1:8b for generation, nomic-embed-text for embeddings)
• Qdrant as the vector store, running in Docker on node01
• Python for the RAG pipeline and the Obsidian integration
• Obsidian itself, unchanged — I read from the vault, I don't write to it

If you want to build one

The repository is Apache 2.0 and open to contributions:

Website: cyberdeck-ai-knowledge-system.org

GitHub: github.com/iboneumann