Building an Agentic RAG AI Agent with FAISS, BM25 and Qwen

Sachin Sajukumar — Sun, 30 Aug 2026 04:56:46 +0000

I recently completed a project as part of my AI Engineering learning journey: an Agentic RAG AI Agent that answers user questions based on a provided knowledge base.

The main goal of this project was to build an AI assistant that does not simply rely on its general knowledge. Instead, it first searches a knowledge base, retrieves relevant information, and then generates an answer using that retrieved context.

What is the idea behind the project?

Large Language Models can sometimes answer questions using their pre-trained knowledge, even when that information is not available in the documents provided to the system. To address this, I built a Retrieval-Augmented Generation (RAG) pipeline.
The workflow looks like this:

User Query
↓
Hybrid Search
↓

FAISS || BM25
Vector || Keyword
Search || Search
↓
Relevant Knowledge Base Chunks
↓
Retrieved Context
↓
Qwen Language Model
↓
Final Answer

Technologies Used

Python – Main programming language
Sentence Transformers – Generating text embeddings
FAISS – Semantic vector similarity search
BM25 – Keyword-based search
Hybrid Search – Combining vector and keyword search
Qwen2.5-72B-Instruct – Generating the final response
Hugging Face Transformers – Loading and using the language model
Visual Studio Code – Development environment

How the system works

Knowledge Base

The system starts with a predefined knowledge base containing information about specific topics.
The goal is to ensure that the AI assistant answers questions only from this available information.

Document Chunking

Large documents are divided into smaller chunks.
This helps the retrieval system find the specific section of a document that is most relevant to the user's question.

Embedding Generation

Each text chunk is converted into a numerical representation called an embedding using a Sentence Transformer model.
These embeddings allow the system to perform semantic search based on meaning rather than only exact keywords.

FAISS Vector Search

When a user asks a question, the question is also converted into an embedding.
FAISS compares the query embedding with the document embeddings and retrieves the most semantically similar chunks.

BM25 Keyword Search

In addition to semantic search, I also implemented BM25 for keyword-based retrieval.
This is useful for finding exact terms, technical concepts, and important keywords.

Hybrid Search

The results from FAISS and BM25 are combined to improve retrieval quality.
This gives the system the advantages of both semantic understanding and exact keyword matching.
Hybrid Search = Vector Search + Keyword Search

Answer Generation with Qwen

The most relevant chunks are combined into a context.

The Qwen language model receives:

The retrieved knowledge base context
The user's question
Strict instructions to answer only from the provided context The model then generates the final answer.

Example Interaction: Question Inside the Knowledge Base

You: What is RAG?

Searching knowledge base...

Agent: RAG stands for Retrieval-Augmented Generation.
It is an AI approach that combines information retrieval
with language generation. The system retrieves relevant
information from a knowledge base and provides it as context
to a language model to generate an answer.

In this case, the system finds relevant information in the knowledge base and uses it to generate the answer.

Example Interaction: Question Outside the Knowledge Base

You: Who is the Prime Minister of India?

Searching knowledge base...

Agent: Sorry, I cannot answer this question because
the required information is not available in the
provided knowledge base.

One of the important features of this project is attempting to prevent the AI from answering questions that are outside the provided knowledge base.

The model is instructed to:

Answer only from the retrieved context
Avoid using unsupported general knowledge
Avoid guessing or creating information
Reject questions when sufficient information is not available

Challenges I Faced

During development, I initially experimented with a CodeAgent approach. However, the language model sometimes generated unsupported tool calls and did not consistently follow the code format required by the agent framework.

For example, instead of using the available knowledge base search tool, the model attempted to call a tool that did not exist.

I simplified the architecture by using a direct RAG pipeline:
User Query
↓
Retrieve Relevant Context
↓
FAISS + BM25 Hybrid Search
↓
Provide Context to Qwen
↓
Generate Final Answer

This approach was simpler and more reliable for my project. I also experimented with different Qwen model sizes. I moved to Qwen2.5-72B-Instruct to reduce response time and make the project more practical for local execution.

What I Learned

Through this project, I gained hands-on experience with:

Retrieval-Augmented Generation
Text chunking
Text embeddings
Vector databases and similarity search
FAISS
BM25
Hybrid search
Prompt engineering
Knowledge-grounded AI systems
Large Language Model integration
Building an AI project in Python and VS Code

Future Improvements

Some features I would like to add in the future include:

PDF and document upload support
A web interface using Gradio or Streamlit
Source citations with every answer
Better relevance threshold checking
Persistent vector storage
Conversation history
Support for multiple knowledge bases
RAG evaluation metrics

This project was a great hands-on learning experience for understanding how modern AI assistants can be grounded in external knowledge.

I'm continuing to learn and build more AI Engineering projects. Feedback and suggestions are welcome!

DEV Community: Sachin Sajukumar

Building an Agentic RAG AI Agent with FAISS, BM25 and Qwen