I've been learning about RAG (Retrieval-Augmented Generation) and wanted to understand how it actually works beyond just reading about it.
So, I decided to build my own Agentic Hybrid RAG System using Python and open-source AI tools.
The goal was simple: allow a user to ask questions and get answers based on information stored in a custom knowledge base.
But instead of using only one retrieval method, I wanted to combine semantic search and keyword search to make the retrieval more reliable.
What is RAG?
RAG stands for Retrieval-Augmented Generation.
In a normal LLM application, the model generates an answer based on its trained knowledge. With RAG, we first retrieve relevant information from an external knowledge base and then provide that information to the LLM.
The basic idea is:
User Question
↓
Retrieve Relevant Information
↓
Give Context to LLM
↓
Generate Answer
This is useful when we want an AI system to answer questions using our own documents or knowledge base.
Why Hybrid Search?
While building the project, I learned that there isn't one perfect way to search documents.
That's why I used two retrieval techniques:
FAISS for semantic search and BM25 for keyword search.
FAISS uses embeddings to find content that is similar in meaning.
For example:
"How can I authenticate my account?"
It can find a document talking about:
"User authentication and login methods"
even if the exact words aren't the same.
BM25 works differently. It focuses more on matching important keywords.
So if someone searches:
"FAISS vector search"
BM25 can give higher importance to documents containing those exact terms.
By combining both approaches, the system can benefit from meaning-based search as well as exact keyword matching.
How My System Works
The overall architecture looks like this:
User Question
↓
CodeAgent
↓
Knowledge Base Search
↓
┌────────┴────────┐
↓ ↓
FAISS BM25
Semantic Search Keyword Search
↓ ↓
└────────┬────────┘
↓
Hybrid Scoring
↓
Relevant Chunks
↓
Qwen 2.5 72B
↓
Final Answer
For semantic search, I used the all-MiniLM-L6-v2 Sentence Transformer model to convert text into embeddings.
These embeddings are stored and searched using FAISS.
For keyword retrieval, I used BM25.
I then combine the scores using:
Hybrid Score =
0.7 × Vector Score +
0.3 × Keyword Score
I chose a 70/30 split so semantic similarity has more influence while keyword matching still contributes to the final ranking.
Document Chunking
Before searching the documents, I split them into smaller chunks using LangChain's RecursiveCharacterTextSplitter.
My current configuration is:
Chunk Size: 500
Chunk Overlap: 100
Chunking is important because sending an entire large document to the model isn't always useful.
Smaller, relevant pieces of information allow the retrieval system to find more precise context.
The overlap also helps prevent important information from being lost between two chunks.
Adding an Agent
I didn't want the application to be just a simple:
Question → Search → Answer
pipeline.
I used smolagents and a CodeAgent with a custom knowledge_base_search tool.
The agent can use the search tool to retrieve information from the knowledge base before generating an answer.
For the LLM, I used:
Qwen/Qwen2.5-72B-Instruct
This gives the system an additional reasoning layer on top of the retrieval process.
The Biggest Thing I Learned
The biggest lesson from this project was that RAG isn't just about using an LLM.
There are many components involved:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Search
+
Keyword Search
↓
Ranking
↓
Relevant Context
↓
LLM
↓
Answer
If the retrieval step returns irrelevant information, even a powerful LLM can struggle to provide a good answer.
So, improving the retrieval process can be just as important as choosing the LLM.
Challenges
Getting all the components to work together wasn't always straightforward.
I faced compatibility and API issues while working with different versions of the AI libraries. Debugging those issues actually helped me understand the frameworks better.
Instead of simply following a tutorial, I had to understand what each component was doing and how information was moving through the system.
That was probably the most valuable part of the project for me.
What's Next?
The current system is a foundation, and there are several things I'd like to improve:
- Add a reranking model
- Evaluate retrieval accuracy properly
- Improve chunking strategies
- Add better hallucination handling
- Build a cleaner user interface
- Experiment with different hybrid-search weights
I'm still learning about RAG and agentic AI, so this project is definitely not finished.
But building it from the ground up gave me a much better understanding of how retrieval, embeddings, vector databases, keyword search, agents, and LLMs come together to build an AI application.
Final Thoughts
This project started as an attempt to understand RAG, but it ended up teaching me much more about building AI systems in general.
The main takeaway for me is:
A good AI application isn't just about having a powerful model. It's about building a good system around that model.
And that's what I'm continuing to explore.
Top comments (0)