I Built an Agentic Hybrid RAG System with FAISS and BM25

I've been learning about Retrieval-Augmented Generation (RAG) and wanted to understand how it actually works by building something myself.

Instead of just following tutorials, I decided to create a small project where I could experiment with document retrieval, embeddings, vector search, keyword search, and an AI agent.

The result is my Agentic Hybrid RAG System, built using Python, FAISS, BM25, smolagents, and a large language model.

🔗 GitHub: https://github.com/Sonuolikkara/AgenticAI

What is RAG?

RAG stands for Retrieval-Augmented Generation.

The basic idea is to give an LLM relevant information from an external knowledge base before asking it to generate an answer.

Instead of:

User Question → LLM → Answer

a RAG system works more like:

User Question
      ↓
Retrieve relevant information
      ↓
Provide the information to the LLM
      ↓
Generate an answer

I wanted to understand this process practically, so I built my own retrieval pipeline.

Why Hybrid Search?

While working on the project, I came across an interesting problem.

There are different ways to search for relevant information.

A semantic search system can understand the meaning behind a query, while a keyword search system is better when exact words or technical terms matter.

So I decided to combine both.

My project uses:

• FAISS → semantic/vector search
• BM25 → keyword-based search

For example, imagine the knowledge base contains:

"Users can authenticate using their registered credentials."

A semantic search can potentially understand the relationship between:

"How can I authenticate?"

and the information above, even though the wording is different.

On the other hand, keyword search can be useful when someone searches for a specific term such as:

FAISS vector search

Combining both approaches gives the retrieval system two different ways of finding useful information.

How My System Works

The overall flow of my project looks like this:

                    User Question
                          ↓
                       CodeAgent
                          ↓
                  Knowledge Base
                     Retrieval
                    ↙         ↘
                FAISS         BM25
             Vector Search  Keyword Search
                    ↘         ↙
                   Retrieved
                    Context
                       ↓
                      LLM
                       ↓
                  Final Answer

The agent can use a custom knowledge-base search tool to retrieve information before generating the final response.

Creating Embeddings

For semantic retrieval, I use the:

all-MiniLM-L6-v2

sentence-transformer model.

The documents are converted into embeddings, which are numerical representations of the text.

These embeddings are then indexed using FAISS.

This allows the system to search for chunks that are semantically similar to the user's question.

BM25 Keyword Search

FAISS isn't the only retrieval method in my project.

I also use BM25 for keyword-based retrieval.

BM25 works differently from vector search. Instead of comparing the meaning of sentences through embeddings, it considers the occurrence and importance of words in the documents.

This can be useful when the exact terminology in the user's question is important.

So the two retrieval methods complement each other:

FAISS → "What information has a similar meaning?"

BM25 → "What information contains important matching words?"

Document Chunking

Another important part of the project is document chunking.

Large documents aren't always useful to send directly into a retrieval pipeline.

So I split the documents into smaller pieces using a recursive character text splitter.

The current configuration I experimented with is:

Chunk Size   : 500
Chunk Overlap: 100

The overlap helps preserve context between neighboring chunks.

For example:

Document
─────────────────────────────────────
       Chunk 1
            ┌───────────────┐
            │               │
            └───────────────┘
                    Chunk 2
                    ┌───────────────┐
                    │               │
                    └───────────────┘

This makes retrieval more manageable and can help prevent useful information from being separated too aggressively.

Adding an AI Agent

I also wanted to experiment with the idea of an agent, rather than building only a fixed retrieval pipeline.

For this, I used smolagents and a CodeAgent.

The agent has access to a custom knowledge-base search tool.

So instead of the application simply doing:

Question → Search → Answer

the agent can use the search capability as a tool during the process of generating the response.

For the language model, I used:

Qwen/Qwen2.5-72B-Instruct

The Overall Pipeline

Putting everything together, the process looks roughly like this:

Documents
    ↓
Document Chunking
    ↓
Embeddings
    ↓
FAISS Index
    +
BM25 Index
    ↓
User Question
    ↓
Hybrid Retrieval
    ↓
Relevant Chunks
    ↓
Agent
    ↓
LLM
    ↓
Final Response

Each component has a different responsibility.

That's what made this project interesting for me.

What I Learned

Before building this project, I mainly thought about RAG as:

"Give documents to an LLM and ask questions about them."

After building it, I realized there is much more happening in between.

There are several important steps:

• Preparing the documents
• Splitting them into chunks
• Creating embeddings
• Building a vector index
• Performing keyword search
• Combining retrieval results
• Selecting useful context
• Giving that context to an agent
• Generating the final response

The retrieval stage is especially important.

If the system retrieves irrelevant information, the LLM doesn't magically know which information should have been retrieved.

That made me understand why retrieval quality matters so much in RAG applications.

Challenges I Faced

Getting the different AI libraries and components to work together was not always easy.

I faced compatibility and API-related issues while experimenting with the project.

There were times when something didn't work as expected, and I had to go back and understand what that particular component was actually doing.

Honestly, debugging these problems taught me more than simply running a working tutorial.

It forced me to understand how the different parts of the system communicate with each other.

What's Next?

This project is still a learning project, and there are several things I want to experiment with next.

Some of them are:

• Add a reranking model
• Evaluate retrieval quality properly
• Experiment with different chunking strategies
• Improve hallucination handling
• Compare different hybrid-search configurations
• Build a better user interface
• Improve the overall agent workflow

I especially want to compare pure vector search, pure keyword search, and hybrid search to see how much difference each approach makes.

Final Thoughts

I started this project because I wanted to understand RAG better.

Instead, I ended up learning about embeddings, vector databases, keyword search, document chunking, agents, and LLM-based applications.

The project is far from perfect, but that's actually what makes it useful for me.

I'm still learning, experimenting, and improving it.

If you're also learning RAG, Agentic AI, or Generative AI, you can check out the code and experiment with it yourself.

🔗 GitHub Repository: https://github.com/Sonuolikkara/AgenticAI

This is one of the projects I'm building while learning AI engineering, and I'll be sharing more of what I learn along the way.