Building an Agentic Hybrid RAG System with FAISS, BM25, and smolagents

Halan P Babu — Mon, 31 Aug 2026 05:47:58 +0000

I've been learning about RAG (Retrieval-Augmented Generation) and wanted to understand how it actually works beyond just reading about it.

So, I decided to build my own Agentic Hybrid RAG System using Python and open-source AI tools.

The goal was simple: allow a user to ask questions and get answers based on information stored in a custom knowledge base.

But instead of using only one retrieval method, I wanted to combine semantic search and keyword search to make the retrieval more reliable.

What is RAG?

RAG stands for Retrieval-Augmented Generation.

In a normal LLM application, the model generates an answer based on its trained knowledge. With RAG, we first retrieve relevant information from an external knowledge base and then provide that information to the LLM.

The basic idea is:

User Question
      ↓
Retrieve Relevant Information
      ↓
Give Context to LLM
      ↓
Generate Answer

This is useful when we want an AI system to answer questions using our own documents or knowledge base.

Why Hybrid Search?

While building the project, I learned that there isn't one perfect way to search documents.

That's why I used two retrieval techniques:

FAISS for semantic search and BM25 for keyword search.

FAISS uses embeddings to find content that is similar in meaning.

For example:

"How can I authenticate my account?"

It can find a document talking about:

"User authentication and login methods"

even if the exact words aren't the same.

BM25 works differently. It focuses more on matching important keywords.

So if someone searches:

"FAISS vector search"

BM25 can give higher importance to documents containing those exact terms.

By combining both approaches, the system can benefit from meaning-based search as well as exact keyword matching.

How My System Works

The overall architecture looks like this:

                    User Question
                          ↓
                       CodeAgent
                          ↓
                Knowledge Base Search
                          ↓
                 ┌────────┴────────┐
                 ↓                 ↓
               FAISS              BM25
          Semantic Search     Keyword Search
                 ↓                 ↓
                 └────────┬────────┘
                          ↓
                   Hybrid Scoring
                          ↓
                  Relevant Chunks
                          ↓
                    Qwen 2.5 72B
                          ↓
                     Final Answer

For semantic search, I used the all-MiniLM-L6-v2 Sentence Transformer model to convert text into embeddings.

These embeddings are stored and searched using FAISS.

For keyword retrieval, I used BM25.

I then combine the scores using:

Hybrid Score =
0.7 × Vector Score +
0.3 × Keyword Score

I chose a 70/30 split so semantic similarity has more influence while keyword matching still contributes to the final ranking.

Document Chunking

Before searching the documents, I split them into smaller chunks using LangChain's RecursiveCharacterTextSplitter.

My current configuration is:

Chunk Size: 500
Chunk Overlap: 100

Chunking is important because sending an entire large document to the model isn't always useful.

Smaller, relevant pieces of information allow the retrieval system to find more precise context.

The overlap also helps prevent important information from being lost between two chunks.

Adding an Agent

I didn't want the application to be just a simple:

Question → Search → Answer

pipeline.

I used smolagents and a CodeAgent with a custom knowledge_base_search tool.

The agent can use the search tool to retrieve information from the knowledge base before generating an answer.

For the LLM, I used:

Qwen/Qwen2.5-72B-Instruct

This gives the system an additional reasoning layer on top of the retrieval process.

The Biggest Thing I Learned

The biggest lesson from this project was that RAG isn't just about using an LLM.

There are many components involved:

Documents
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Search
   +
Keyword Search
   ↓
Ranking
   ↓
Relevant Context
   ↓
LLM
   ↓
Answer

If the retrieval step returns irrelevant information, even a powerful LLM can struggle to provide a good answer.

So, improving the retrieval process can be just as important as choosing the LLM.

Challenges

Getting all the components to work together wasn't always straightforward.

I faced compatibility and API issues while working with different versions of the AI libraries. Debugging those issues actually helped me understand the frameworks better.

Instead of simply following a tutorial, I had to understand what each component was doing and how information was moving through the system.

That was probably the most valuable part of the project for me.

What's Next?

The current system is a foundation, and there are several things I'd like to improve:

Add a reranking model
Evaluate retrieval accuracy properly
Improve chunking strategies
Add better hallucination handling
Build a cleaner user interface
Experiment with different hybrid-search weights

I'm still learning about RAG and agentic AI, so this project is definitely not finished.

But building it from the ground up gave me a much better understanding of how retrieval, embeddings, vector databases, keyword search, agents, and LLMs come together to build an AI application.

Final Thoughts

This project started as an attempt to understand RAG, but it ended up teaching me much more about building AI systems in general.

The main takeaway for me is:

A good AI application isn't just about having a powerful model. It's about building a good system around that model.

And that's what I'm continuing to explore.

DEV Community: Halan P Babu

Building an Agentic Hybrid RAG System with FAISS, BM25, and smolagents

How My System Works