How to build a RAG system from scratch in Python (chunk embed retrieve cite)( https://ai.studybydoing.in)

Most "RAG tutorials" hand you a framework and a .from_documents() one-liner, and you never actually see what happens inside. So I built one by hand — chunking, embeddings, a tiny
vector store, hybrid retrieval, re-ranking, and cited generation — to understand each moving part. Here's the mental model and the two pieces that matter most.

## What RAG actually is

An LLM only knows what was in its training data. RAG (Retrieval-Augmented Generation) lets it answer questions about your private/current documents by retrieving relevant snippets at
query time and putting them in the prompt. The model then answers from that supplied context — facts, not guesses.

The pipeline has two timelines:

• Offline (build the index once): Documents → Chunk (+metadata) → Embed → Vector store.
• Online (per user question): Question → Retrieve (vector + keyword) → Re-rank → Build context → LLM → Answer + citations.

In one line: RAG = look things up first, then answer from what you found. The offline row is a librarian shelving books; the online row is you asking a question and getting the right
pages handed to you before you write your reply.

## The highest-leverage decision: chunking

Models retrieve chunks, not whole documents — so how you split matters more than almost anything else:

• Too big → irrelevant text dilutes the answer and wastes tokens.
• Too small → facts get split across chunks.

  import re

  def chunk_text(text, source, target_words=120, overlap=25):
      """Split on paragraphs, then pack into ~target_words chunks with overlap."""
      paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
      chunks, buf = [], []
      for p in paras:
          buf.append(p)
          if sum(len(x.split()) for x in buf) >= target_words:
              chunks.append(" ".join(buf))
              buf = buf[-1:]  # carry last para as overlap
      if buf:
          chunks.append(" ".join(buf))
      return [{"text": c, "source": source, "id": f"{source}#{i}"} for i, c in enumerate(chunks)]

The rules that survived the labs:

• Split on semantic boundaries (paragraphs, headings) — not blind fixed windows.
• Keep 10–20% overlap so a fact near a boundary survives in at least one chunk.
• Carry metadata (source, section, URL, date) — you need it for citations and filtering.
• Aim ~100–250 words for precision; add a "parent document" fallback for context.

## Embeddings + a vector store in ~15 lines

An embedding turns text into a vector where similar meanings sit close together. Store each chunk's vector; at query time, embed the question and find the nearest chunks by cosine
similarity.

  import numpy as np
  from sentence_transformers import SentenceTransformer  # swap for any provider

  _model = SentenceTransformer("all-MiniLM-L6-v2")

  def embed(texts):
      # normalize -> cosine similarity collapses into a plain dot product
      return np.asarray(_model.encode(texts, normalize_embeddings=True))

  class VectorStore:
      def add(self, chunks):
          self.chunks = chunks
          self.vecs = embed([c["text"] for c in chunks])   # (N chunks x d)

      def search(self, query, k=4):
          q = embed([query])[0]
          sims = self.vecs @ q                             # similarity to every chunk, one step
          top = np.argsort(-sims)[:k]
          return [(self.chunks[i], float(sims[i])) for i in top]

Two things clicked for me here:

1. embed() is the one place text becomes numbers — wrap it behind a single function so you can swap providers without touching anything else. Golden rule: embed questions and documents with the same model.
2. This is genuinely what Pinecone / FAISS / pgvector do under the hood — they just add persistence, scale, and speed. Building the naive version first makes the real ones far less magical.

## Then: hybrid retrieval, re-ranking, and citations

The last labs add the parts that separate a demo from something usable:

• Hybrid search — combine meaning-based (vector) with exact-word (BM25) so you don't miss literal matches like error codes or names.
• Re-ranking — reorder candidates so the best rise to the top before they hit the prompt.
• Grounded, cited generation — answer only from retrieved context, cite the source chunk, and refuse when the context doesn't contain the answer (this is what kills hallucinations in practice).

## The mental model to keep

• RAG beats a bigger context window / fine-tuning when your knowledge is private, changing, or needs citations.
• Chunking is the highest-leverage knob — get it wrong and nothing downstream saves you.
• Retrieval is hybrid + re-rank, not just "nearest vector."
• Generation must be grounded and honest — cite, or refuse.

---

If you want to run each stage yourself (the lesson has an in-browser Python terminal, no setup) the full build is here 👉 Build a RAG System From
Scratch. It's part of a free course that builds RAG, agents, eval, and production LLM systems by hand:
ai.studybydoing.in.