Building a Local AI Knowledge System with Docker, Ollama, Qdrant and Open WebUI

Today I continued experimenting with MyZubster, an open-source project I'm working on while learning how local AI systems, knowledge retrieval and verifiable profiles can work together.

The goal was simple:

Can a local AI answer questions using a personal knowledge base instead of relying only on what the language model already knows?

The stack we tested included:

• Docker
• Ollama
• Qdrant
• Open WebUI
• a Python API
• Markdown knowledge files
• local embeddings

Starting the local stack

The project uses Docker Compose.

After starting Docker Desktop, the services came online:

• api
• ollama
• qdrant
• open-webui

The API reported healthy and Open WebUI became available locally.

This gave us a completely local environment where we could experiment with AI retrieval without depending on a hosted LLM.

Building the knowledge base

The project contains Markdown documents inside a knowledge directory.

A Python ingestion script:

1. reads the documents;
2. splits them into chunks;
3. creates embeddings using Ollama;
4. stores the vectors and metadata in Qdrant.

The embedding model used during the experiment was:

nomic-embed-text

After rebuilding the knowledge index, the ingestion completed with:

Loaded 40 knowledge chunks; skipped 1 empty documents

So the pipeline itself was working.

Then we found an interesting problem

We asked the AI:

What is the GitHub repository of N4K48? Give me the complete link without abbreviating it.

The model answered confidently — but returned the wrong GitHub URL.

That was useful.

It showed an important distinction:

a working RAG pipeline does not automatically mean reliable answers.

Docker was running.

Qdrant contained the knowledge.

Ollama was generating embeddings.

Open WebUI could talk to the model.

But somewhere between document ingestion, retrieval and generation, the information reaching the model was still not reliable enough.

Investigating document chunking

The original ingestion system divided documents approximately every 1,000 characters with an overlap.

That approach is simple, but arbitrary character boundaries can split meaningful information.

URLs are a good example.

A GitHub URL divided between chunks can become incomplete or lose the context that explains what the URL represents.

So we inspected and modified the knowledge ingestion logic to make document segmentation more meaningful and reduce arbitrary cuts.

Then we deleted the previous knowledge points and rebuilt the Qdrant index.

The result was 40 newly generated knowledge chunks.

The experiment isn't finished yet.

The next step is to inspect the retrieval path between:

question → embedding → Qdrant → retrieved knowledge → API → Ollama

and determine exactly where the incorrect repository information is being introduced.

A second experiment: knowledge profiles

At the same time, we tested the MyZubster Knowledge Profile Builder.

Instead of treating every statement as verified knowledge, the Builder distinguishes between:

• declared knowledge;
• external evidence;
• information still requiring verification.

We created three private knowledge drafts covering:

1. professional transport experience;
2. Docker and local AI experimentation;
3. learning and collaboration during the MyZubster project.

The drafts were saved privately and remained available after reopening them.

They are not public yet.

Before publishing them, we're checking how evidence and verification should be attached to each claim.

What I learned

The most interesting lesson wasn't that we managed to run an LLM locally.

It was that running the model is the easy part.

The difficult part is building a trustworthy path from:

human knowledge

to:

documents

to:

chunks

to:

embeddings

to:

retrieval

to:

LLM answer

Every step can change what the final model sees.

And a model can produce a perfectly fluent answer even when one of those steps supplied the wrong information.

That's why provenance and verification matter.

What's next

The next experiments will focus on:

• inspecting Qdrant retrieval results;
• improving knowledge chunking;
• testing complete URLs and source metadata;
• distinguishing declared knowledge from verified evidence;
• testing the three Knowledge Profile drafts;
• improving the reliability of answers generated from the local knowledge base.

This is still an experiment, and some parts are intentionally unfinished.

I'll document what works — and what breaks — as the project develops.

---

MyZubster / N4K48

Building, testing and learning in public.