Patience Mpofu

Posted on Sep 6

I Built a RAG Pipeline From Scratch Without Using LangChain — Here's Every Decision I Made

#security #ai #langchain #python

Every RAG tutorial starts the same way.

from langchain import ...

I did the opposite. I built a complete retrieval-augmented generation pipeline from scratch — no LangChain, no LlamaIndex, no framework abstractions. Just Python, a vector store, an embedding model, and the Claude API.

Not because frameworks are bad. Because I wanted to understand what a RAG pipeline actually is underneath the abstractions before trusting a framework to hide it from me.

This article is about every design decision in that pipeline — what I built, why I built it that way, and what I'd do differently at production scale.

What the Pipeline Does

In one sentence: ingest documents into a local vector store using local embeddings, then answer questions with Claude grounded in retrieved context.

In practice:

# Ingest a directory of documents
python cli.py ingest data/

# Ask a question
python cli.py query "What does this document say about authentication?"

The output is Claude's answer plus the sources it used, with similarity distances so you can see how confident the retrieval was.

The Three-Component Architecture

I split the pipeline into three components with clean interfaces between them. This wasn't accidental — it was the most important design decision in the whole project.

cli.py
  ├── rag/loader.py     — reads files, splits into chunks
  ├── rag/store.py      — embeds chunks, stores in Chroma, retrieves by similarity  
  └── rag/pipeline.py   — embeds the question, retrieves chunks, calls Claude

Each component has one job. Each can be replaced independently. The loader doesn't know about the store. The store doesn't know about Claude. The pipeline doesn't know how files are loaded or how chunks are stored — it just gets chunks back from the store and sends them to Claude.

This separation is the difference between a prototype and a maintainable system. The README documents the swap points explicitly:

Replace rag/store.py with a Pinecone or pgvector client — the interface stays the same
Swap SentenceTransformerEmbeddingFunction for Voyage AI embeddings — one line change
Replace the paragraph-based chunker with a token-aware splitter — pipeline doesn't change I built for replaceability because production RAG systems almost always need to swap components. You start with Chroma locally and move to Pinecone when you need scale. You start with sentence-transformers and move to a hosted embedding API when you need better quality. The clean interfaces make those migrations surgical rather than rewrites.

Why Not LangChain

The honest answer: LangChain is a reasonable choice for most production RAG work. It has good abstractions, a large ecosystem, and handles a lot of boilerplate.

The reason I didn't use it here is the same reason I didn't use a SAST tool before building one: I wanted to understand what was happening at each step before trusting an abstraction to handle it for me.

LangChain's RetrievalQA chain, for example, handles the retrieve-then-generate loop in a few lines. But it makes choices about prompt format, retrieval strategy, and context assembly that you might not notice until they cause a problem. When something goes wrong — wrong answer, retrieved wrong chunks, context overflow — you need to understand the pipeline well enough to diagnose it.

Building without a framework means every choice is explicit and visible. The prompt template that tells Claude to use only the retrieved context is written by me, not generated by a chain. The retrieval parameters are set by me, not defaulted by a library. When the pipeline gives a wrong answer, I know exactly where to look.

The CLI Design

The entry point is a simple subcommand CLI:

import argparse
from pathlib import Path
from rag import pipeline, store
from rag.loader import load_and_chunk

def cmd_ingest(args):
    documents = load_and_chunk(Path(args.path))
    count = store.add_documents(documents)
    print(f"Ingested {count} chunks from {args.path}")

def cmd_query(args):
    result = pipeline.answer(args.question, top_k=args.top_k)
    print(result["answer"])
    print("\nSources:")
    for src in result["sources"]:
        print(f"  - {src['source']} (distance={src['distance']:.4f})")

Two commands: ingest and query. The ingest command is idempotent — running it twice on the same documents doesn't create duplicate chunks because Chroma deduplicates by document ID. The query command returns both the answer and the sources with their similarity distances, which is important for debugging retrieval quality.

The top_k parameter is exposed as a CLI flag with a default of 5. This is a tunable parameter that significantly affects answer quality — too few chunks and you miss relevant context, too many and you dilute the prompt with noise. Exposing it at the CLI level means you can experiment without touching the code.

The Technology Choices

Python — the natural choice for ML/AI tooling. The embedding libraries, vector store clients, and LLM SDKs all have first-class Python support.

sentence-transformers/all-MiniLM-L6-v2 — a local embedding model that runs on CPU without GPU, produces 384-dimensional vectors, and is fast enough for development use. The key property: it runs entirely locally, which means no API key, no latency, no cost per embedding. For a development pipeline processing hundreds of documents, this matters.

Chroma — a local vector database that persists to disk. Zero infrastructure — no Docker, no cloud account, no configuration. Run it from Python, it creates a chroma_db/ directory, done. For a local development pipeline this is exactly the right choice.

Claude — for generation. The retrieval pipeline finds the context; Claude synthesises the answer. The system prompt explicitly instructs Claude to answer using only the retrieved context and to cite sources — which is how you prevent hallucination in RAG systems.

rag/config.py — a central configuration file with CHUNK_SIZE, CHUNK_OVERLAP, and other tunable parameters. Every magic number in the pipeline lives here, not scattered across files.

What This Isn't

Being honest about scope matters.

This is a local development pipeline, not a production system. It has no authentication, no access control, no multi-tenancy, no monitoring, no rate limiting. A single user, a single Chroma collection, a single local machine.

The README explicitly documents the swap points for production migration — but the swaps aren't implemented. A production RAG system would need hosted vector storage, API-based embeddings for consistency, access control at the retrieval layer, audit logging, and prompt injection defences.

Those gaps are the subject of article 5 in this series. This article is about what was built and why.

The Design Principle That Guided Everything

Every decision in this pipeline came back to one principle: make the components independently replaceable without changing the interfaces.

Loader reads files and returns chunks. Store takes chunks and returns relevant ones. Pipeline takes a question and returns an answer with sources. Each component's interface is stable even as the implementation can change.

That principle is what makes this a useful portfolio project beyond just "I ran some code." It demonstrates architectural thinking — the ability to design a system that can evolve without requiring a rewrite every time a component needs to change.

That's the same thinking that applies to production AI systems at scale. The embedding model will need to change as better ones emerge. The vector store will need to scale. The generation model will need to be updated. Systems designed for replaceability survive those changes; systems designed around specific tools don't.

The full source code is at github.com/pgmpofu/rag-pipeline.

Next up: chunking strategy — why I split on paragraph boundaries instead of token count, and what the overlap parameter actually does.

DEV Community