Building a Closed-Domain Agentic AI Knowledge Assistant with Hybrid RAG

Sanjay Sajukumar — Sun, 30 Aug 2026 04:56:47 +0000

I recently built an "Agentic AI Knowledge Assistant" that combines "Retrieval-Augmented Generation (RAG), hybrid search, and an LLM agent" to answer questions strictly from a predefined knowledge base.

One of the main goals of this project was to address a common problem with LLM applications: the model should not answer questions using its general pretrained knowledge when the required information is not available in the knowledge base.

How it works

The system follows a retrieval-first approach:
User Question → Hybrid Retrieval → Relevant Context → AI Agent → Final Answer:
The knowledge base is divided into smaller chunks using "LangChain's RecursiveCharacterTextSplitter". Each chunk is converted into embeddings using:
"sentence-transformers/all-MiniLM-L6-v2"
For retrieval, I implemented two approaches:

FAISS for semantic vector search
BM25 for keyword-based search These are combined using a weighted hybrid scoring mechanism: "Hybrid Score = 0.7 × Vector Score + 0.3 × BM25 Score" This allows the system to benefit from both semantic similarity and exact keyword matching.

The workflow looks like this:
User Query
↓
Hybrid Search
↓

FAISS || BM25
Vector || Keyword
Search || Search
↓
Relevant Knowledge Base Chunks
↓
Retrieved Context
↓
Qwen Language Model
↓
Final Answer

Agentic AI Layer

The retrieval system is exposed to the agent through a custom:
knowledge_base_search() tool.
The project uses smolagents CodeAgent along with the:
"Qwen/Qwen2.5-72B-Instruct" model for response generation.
The agent retrieves relevant information from the knowledge base before generating a response.

Closed-Domain Knowledge Restriction

One of the most important features of this project is the strict knowledge-base-only approach.
The assistant is instructed not to use:

Wikipedia
Internet searches
External websites
External APIs
External documents
General pretrained knowledge
Guessing or assumptions

If the knowledge base does not contain sufficient information, the intended response is:

"The knowledge base does not contain enough content to answer this question."
This makes the system more suitable for applications where responses need to remain within a controlled information domain.

Technologies Used

Python
LangChain
Sentence Transformers
FAISS
BM25
Hugging Face Transformers
smolagents
Qwen 2.5 72B Instruct

What I Learned

Building this project helped me understand how different components of an AI application work together rather than treating an LLM as a standalone system.

In particular, I gained practical experience with:

Document chunking
Text embeddings
Vector databases/search
BM25 retrieval
Hybrid search
Retrieval-Augmented Generation
AI agents and tools
LLM integration
Closed-domain AI systems
Controlling unsupported LLM responses

Future Improvements

Some improvements I would like to implement next include:

Adding PDF and document ingestion
Persistent vector storage
A web-based interface
Better retrieval evaluation
A program-level relevance gate before sending queries to the LLM
Conversation history while maintaining the closed-domain restriction

This project was a great hands-on experience in understanding how RAG + Hybrid Search + Agentic AI + LLMs can be combined to build a more controlled AI assistant.

DEV Community: Sanjay Sajukumar

Building a Closed-Domain Agentic AI Knowledge Assistant with Hybrid RAG