DEV Community

Anshul Rajpal
Anshul Rajpal

Posted on

RAG Systems Are Eating the World: Building Retrieval-Augmented Generation in Python

#rag #ai #python

What Is RAG and Why Should You Care?

If you have been building with LLMs, you have hit the wall: hallucination. Your model confidently tells you about a library that does not exist, cites a paper that was never written, or fabricates API endpoints.

Retrieval-Augmented Generation (RAG) fixes this by connecting your LLM to your actual data. Instead of relying solely on what the model memorized during training, RAG retrieves relevant documents at query time and feeds them as context.

This pattern is now the backbone of production AI applications, from enterprise search to coding assistants.

The Core Architecture

A RAG pipeline has three stages:

1. Ingestion (Indexing)

Your documents need to be chunked, embedded, and stored in a vector database.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

splitter = RecursiveCharacterTextSplitter(
 chunk_size=500,
 chunk_overlap=50,
 separators=["

", "
", ". ", " "]
)

docs = splitter.split_documents(your_documents)

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Chroma.from_documents(docs, embeddings, persist_directory="./chroma_db")
Enter fullscreen mode Exit fullscreen mode

Key decisions:

Parameter Recommendation
Chunk size 300-800 tokens
Overlap 10-20% of chunk size
Embedding model all-MiniLM-L6-v2 for speed, bge-large for quality
Vector DB Chroma for prototyping, Pinecone or Qdrant for production

2. Retrieval

When a user asks a question, embed it and find the closest document chunks.

from langchain.chains import RetrievalQA
from langchain_community.llms import Ollama

retriever = vectorstore.as_retriever(
 search_type="mmr",
 search_kwargs={"k": 4, "fetch_k": 20}
)

llm = Ollama(model="llama3.1:8b")

qa_chain = RetrievalQA.from_chain_type(
 llm=llm,
 chain_type="stuff",
 retriever=retriever,
 return_source_documents=True,
)
Enter fullscreen mode Exit fullscreen mode

3. Generation with Context

The magic happens when retrieved chunks become part of the prompt:

def ask(question: str) -> dict:
 result = qa_chain.invoke({"query": question})
 print(f"Answer: {result['result']}")
 for i, doc in enumerate(result['source_documents']):
 print(f" Source [{i+1}]: {doc.metadata.get('source', 'Unknown')}")
 return result

ask("How does the authentication system work?")
Enter fullscreen mode Exit fullscreen mode

Production Hardening

The basic pipeline works, but production RAG needs more.

Hybrid Search

Combine vector similarity with keyword matching for better recall:

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 4

vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

ensemble = EnsembleRetriever(
 retrievers=[vector_retriever, bm25_retriever],
 weights=[0.6, 0.4]
)
Enter fullscreen mode Exit fullscreen mode

Reranking

Your retriever returns candidates; a reranker picks the best ones:

from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank

reranker = CohereRerank(model="rerank-v3.5", top_n=3)
compression_retriever = ContextualCompressionRetriever(
 base_compressor=reranker,
 base_retriever=ensemble
)
Enter fullscreen mode Exit fullscreen mode

Query Transformation

Users ask messy questions. Transform them before retrieval:

  • HyDE (Hypothetical Document Embeddings): Generate a hypothetical answer, embed it, search for similar real docs
  • Multi-query: Generate multiple search queries from one question, merge and deduplicate results

Real-World Gotchas

After building RAG systems in production, here is what trips people up:

1. Chunking destroys context. If your code examples span multiple chunks, the retriever might only find half the function. Use semantic chunking or overlap generously.

2. Embeddings lie about relevance. Cosine similarity of 0.85 does not mean "very relevant." Always evaluate with a human-labeled test set.

3. The context window is a bottleneck. Even with large context windows, stuffing too many retrieved chunks degrades performance. Rerank aggressively.

4. Freshness matters. If your docs update daily but your index updates monthly, your RAG is already stale.

The Minimal Stack for 2025

Here is what I would use to build a RAG system today:

  • Embedding: bge-large-en-v1.5 (best open-source)
  • Vector DB: Qdrant (self-hosted) or Pinecone (managed)
  • LLM: Llama 3.1 8B via Ollama for dev, Claude or GPT-4o for production
  • Framework: LangChain for prototyping, LlamaIndex for structured data extraction
  • Evaluation: RAGAS framework for automated quality metrics

Wrapping Up

RAG is not magic, it is plumbing. The quality of your retrieval pipeline directly determines the quality of your LLM outputs. Get the retrieval right, and even a smaller model will outperform GPT-4 on your domain-specific tasks.

The best RAG systems treat retrieval as a first-class engineering problem, not an afterthought.

Start simple. Measure everything. Then optimize.

Building something with RAG? Drop your architecture in the comments. I would love to compare approaches.

Top comments (0)