Building a RAG Knowledge Assistant: Chunking, Embeddings, Vector Search, and Answers That Cite Their Sources

Ask a general-purpose LLM about your company's refund policy and it will answer confidently. It will also be wrong, because it has never seen your policy.

That gap is why Retrieval-Augmented Generation (RAG) exists. Instead of hoping the model "knows" your data, you find the relevant text first and hand it to the model along with the question. The model stops guessing and starts reading.

Guidely is an internal knowledge assistant I'm building at Zone01 Kisumu, with a FastAPI backend and a React/Vite frontend. You upload company docs, it indexes them, and when you ask a question it returns a short answer with the sources it used. In this post I'll break down how the pipeline works, what each stage costs you, and where RAG systems quietly fail.

The Problem With Asking an LLM Directly

A bare LLM has two weaknesses for this job. It doesn't know your private documents, and when it doesn't know something it tends to produce a plausible answer anyway. For a support tool, a confident wrong answer is worse than no answer, because people act on it.

Pasting every document into the prompt isn't the fix either. Context windows are limited, long prompts are slow and expensive, and models are worse at finding a needle in a huge haystack than in a small one. You want to send the model only the few passages that matter.

Why Not Just Use Keyword Search?

Keyword search is the obvious shortcut, and it works until it doesn't:

• Vocabulary mismatch breaks it. "How do I reset my password?" and "I forgot my login credentials" share almost no words, yet they ask the same thing.
• It returns pages, not answers. Users still dig through results to find the relevant paragraph.
• It can't explain itself in plain language. You get a list of links, not a readable answer.

Semantic search fixes the first problem, and an LLM fixes the other two. RAG combines both.

The Architecture

The design has two phases. Indexing happens offline, when documents are added or changed. Querying happens online, on every question.

Indexing:

1. Parse the uploaded file into raw text.
2. Chunk the text into small overlapping pieces.
3. Embed each chunk into a vector.
4. Store the vectors, plus the chunk text and file name, in a vector index.

Querying:

1. Embed the question with the same model.
2. Search the index for the top-k most similar chunks.
3. Generate an answer from those chunks and the question.
4. Return the answer together with the sources that were used.

The important property is that the model only sees what retrieval hands it. Everything downstream depends on that handoff.

Chunking: The Most Underrated Decision

Embedding a 40-page PDF as one vector produces mush. So documents are split into chunks, typically 500 to 1,000 tokens, with a small overlap between neighbours so a sentence cut at a boundary still has its context on the other side.

Chunk size is a trade-off. Too big, and retrieval gets vague because one chunk covers several topics. Too small, and each chunk loses the context it needs to make sense on its own. Splitting on natural boundaries like headings and paragraphs beats cutting at a fixed character count, because it keeps ideas intact.

Every chunk also keeps metadata: the file it came from and its position in that file. That metadata is what later lets the UI say exactly where an answer came from.

Embeddings and the Vector Store

An embedding turns text into a list of numbers so that similar meaning means nearby vectors. That is what lets the password question match the login question despite sharing no words.

For storage, Guidely uses FAISS: it runs locally, needs no account or external service, and is fast. With normalized vectors and inner-product search, the score is cosine similarity. An exact flat index is perfectly fine up to hundreds of thousands of chunks. Approximate indexes matter later, and hosted options like Pinecone or Weaviate make sense once you need managed infrastructure. Start with the simplest thing that works.

One rule matters here: the question and the chunks must be embedded with the same model. Vectors from different models live in different spaces and can't be compared.

Generating a Grounded Answer

The "G" in RAG is where you enforce honesty. The prompt gives the model the numbered chunks and a strict set of instructions:

• Answer using only the provided context.
• Cite sources by number.
• If the context doesn't contain the answer, say you don't know.

Each of those does real work. "Only the context" reduces hallucination. Numbered sources let the model cite, and let the UI map each citation back to a real file and snippet. The "I don't know" escape hatch stops the model from inventing an answer when retrieval fails. Setting the temperature to zero keeps the answers stable and reproducible.

The API then returns clean JSON: the answer, plus the list of sources with file names and snippets. Always showing sources turns "trust me" into "check for yourself."

Things That Bite You in Production

Re-embedding everything. Embedding costs money and time, and documents rarely change. Hash each file's content at index time and skip it if the hash is unchanged. Log cache hits against new embeddings. If you re-ask a question and the embedding count goes up, something is broken.

Exact terms and IDs. Pure vector search is weak on error codes, product names, and identifiers, which is exactly where keyword search is strong. Hybrid search runs both and merges the rankings, and it is often the single biggest quality upgrade you can make.

Coarse ranking. Vector search is fast but approximate. A common upgrade is to retrieve a wide net, say the top 20, then use a cross-encoder reranker that reads the question and each chunk together to pick the best few. You trade a little latency for better precision.

Follow-up questions. A user asks "what about for contractors?" and retrieval has nothing to search for. Rewrite follow-ups into standalone questions before embedding them, using the conversation history.

Prompt injection. Your documents become part of the prompt, so a document containing "ignore previous instructions" is an attack vector. Keep system instructions separate from retrieved content, label the context clearly as data, and never let retrieved text trigger actions without validation.

Failure states. Real systems hit empty queries, missing API keys, corrupted files, model timeouts, and searches that find nothing. Return proper 4xx and 5xx responses with clear messages, and show friendly error states in the UI. "No relevant documents found" is a good answer. A confident hallucination is the worst one.

How Do You Know It Works?

"It seems to work" isn't a metric. Guidely tracks a small set of checks, some logged automatically by the backend and some validated by hand:

• Retrieval@3 (manual): for 10 to 20 questions whose answers exist in the sample docs, how often does the correct passage appear in the top 3 chunks? Target: 80% or more.
• Answer reference coverage (manual): does every answer cite at least one valid source? Target: 90% or more.
• Source precision (manual): does the highlighted snippet actually support the answer? Target: 80% or more.
• Latency (auto-logged): time from question to final JSON. Target: median under 3 seconds and p95 under 5 seconds with a warm cache.
• Embedding cache effectiveness (auto-logged): repeated queries on unchanged docs should cause zero re-embedding.

The most useful habit is evaluating retrieval separately from generation. If the right chunk isn't in the top 3, no prompt will save you. Check retrieval first, then check the answers, and when quality drops you'll know which half of the pipeline to blame.

Lessons Learned

• Retrieval quality is the ceiling. The LLM can only be as good as the context you hand it.
• Chunking and prompts matter more than model choice. Upgrading the model rarely fixes a bad pipeline.
• Test embedding and retrieval before the UI. Wire up the interface last, once search returns the right chunks.
• Cache from day one. Hash-based skipping is easy to build early and painful to retrofit.
• Always show your sources. Citations are what make the tool usable by real people.
• Log everything cheap. Latency, cache hits, retrieved chunk IDs, and error counts make debugging far easier.

Conclusion

RAG isn't magic. It's search plus a well-instructed reader, and once you see it that way, every improvement becomes an ordinary engineering problem: better chunks, hybrid search, reranking, query rewriting. If you're building your first one, start small: five documents, FAISS, and a strict prompt. Get retrieval working and measured before you touch anything else, and the rest of the system gets much easier to reason about.