Building a Hybrid Agentic RAG System with FAISS, BM25 and Qwen

Juan Pereira — Sun, 30 Aug 2026 19:47:27 +0000

Large Language Models (LLMs) are powerful at understanding and generating text, but they do not automatically have access to information stored in private or domain-specific documents. Retrieval-Augmented Generation (RAG) addresses this problem by retrieving relevant information from an external knowledge base and providing it to the language model as context.

In this project, I built a small hybrid Agentic RAG system that combines semantic search, keyword-based search, and language generation. The system uses Sentence Transformers for embeddings, FAISS for vector search, BM25 for keyword retrieval, and Qwen as the language model.

Workflow

User Query → Hybrid Retrieval → Relevant Context → Qwen LLM → Final Answer

Architecture

Document Processing

The system starts with a small knowledge base containing documents related to Artificial Intelligence, Large Language Models, RAG, embeddings, vector search, hybrid search, and RAG evaluation.

Before retrieval, the documents are divided into smaller chunks using recursive character-based splitting. The main configuration uses a chunk size of 300 characters with 50 characters of overlap. This allows the system to retrieve focused sections of a document while retaining some surrounding context.

The original eight documents were divided into 12 chunks using the main configuration.

I also experimented with different chunk sizes. Smaller chunks produced more retrieval units, while larger chunks reduced the total number of chunks. This showed how chunk size can affect the balance between focused retrieval and contextual information.

Semantic Search with Embeddings

After chunking, each piece of text is converted into a numerical representation called an embedding.

The project uses the all-MiniLM-L6-v2 Sentence Transformer model, which produces 384-dimensional embeddings.

The main advantage of embeddings is that they allow the system to search based on meaning rather than only exact words.

For example, a user might ask a question using different wording from the knowledge base, but the system can still identify documents that discuss the same concept.

FAISS is then used to compare the query embedding with the stored document embeddings and retrieve the most semantically similar chunks.

Keyword Search with BM25

Semantic search is useful, but it is not always the best choice.

Some queries contain exact technical terms, identifiers, names, or specific phrases where keyword matching is important. To handle these cases, the project also uses BM25, a traditional information-retrieval algorithm.

This gives the system two complementary retrieval methods:

FAISS focuses on semantic similarity.
BM25 focuses on keyword and term matching.
Hybrid Retrieval

Instead of relying on only one retrieval method, the project combines both.

The FAISS and BM25 scores are normalized and then combined using weighted scoring. The default configuration gives more importance to semantic retrieval while still maintaining a contribution from keyword retrieval.

This approach is useful because semantic search and keyword search have different strengths. Combining them can provide more useful candidates for the final answer.

The system also experiments with different vector and keyword weights to understand how changing the balance affects retrieval.

Retrieval-Augmented Generation

Once the most relevant chunks have been identified, they are combined into a context that is provided to the language model.

The important distinction is that the retriever does not generate the final answer.

Its job is to find relevant information.

The LLM then receives:

User Question + Retrieved Context

and generates the final natural-language response.

This helps ground the response in the information available in the knowledge base rather than relying entirely on the model's internal knowledge.

Agentic Layer

The original implementation also introduces an agentic component using smolagents.

The retrieval system is exposed as a tool that the agent can use. In this architecture, the roles are separated:

The agent acts as the decision-making layer.
The retriever finds relevant information.
The LLM generates the final response.

This makes the system more flexible than a simple question-and-answer pipeline because the agent can interact with available tools.

During testing, the original agent integration encountered a parsing issue with the Qwen model because the generated output did not match the format expected by the CodeAgent. The standalone implementation therefore uses a more explicit retrieval-to-generation flow to make the final user interaction more predictable.

Experiments

The project explores several important RAG parameters.

Chunk Size - Different chunk sizes were tested, including 100, 200, 300, 500, and 800 characters. The experiment demonstrated that increasing chunk size reduces the number of chunks, while smaller chunks create more granular retrieval units.
Top-K Retrieval - Different values of top-k were tested to determine how many relevant chunks should be retrieved for a query. A smaller value provides less context, while a larger value provides more information but may also introduce irrelevant content.
Hybrid Weights - Different combinations of semantic and keyword weights were also tested, ranging from vector-only retrieval to keyword-only retrieval.

These experiments demonstrate that retrieval performance depends heavily on how the retrieval system is configured.

Evaluation

The project also considers several dimensions for evaluating a RAG system:

Retrieval relevance
Answer correctness
Groundedness
Completeness
Overall response quality

Other useful metrics for a larger implementation would include retrieval accuracy, Recall@K, MRR, latency, and cost.

The current project establishes the evaluation framework but does not provide enough completed benchmark results to make a formal performance claim.

Potential Applications

This architecture can be adapted to several real-world applications.

Company Knowledge Assistant - Internal company policies, HR documents, onboarding material, and technical documentation could be indexed so employees can ask questions about company procedures.
Technical Documentation Assistant - Software documentation, API references, and troubleshooting guides could be used to create an assistant capable of answering developer questions.
Educational Assistant - Lecture notes, textbooks, and course materials could be used as the knowledge base for a study assistant.
Customer Support - Product manuals, FAQs, and troubleshooting documents could be indexed to provide support responses based on official product information.

What I Learned

One of the main lessons from this project is that building a RAG system is much more than simply connecting a document to an LLM.

The quality of the final answer depends on several stages:

Document Quality → Chunking → Embeddings → Retrieval → Context → LLM Generation

If the retrieval system provides poor or irrelevant information, even a capable language model may produce a poor answer.

This project helped me understand how different retrieval techniques complement each other and how the retrieval layer plays an important role in building reliable AI applications.

Conclusion

This project demonstrates a complete hybrid Agentic RAG pipeline combining semantic embeddings, FAISS vector search, BM25 keyword retrieval, hybrid ranking, and Qwen-based language generation.

The key idea is simple:

Retrieval finds the relevant information, while the LLM uses that information to generate the answer.

By combining semantic and keyword search, the system can handle both questions based on meaning and queries where exact terminology matters.

The next step would be to replace the small manually created knowledge base with real-world documents such as PDFs or company documentation, followed by more systematic evaluation using larger datasets and quantitative retrieval and generation metrics.

DEV Community: Juan Pereira

Building a Hybrid Agentic RAG System with FAISS, BM25 and Qwen