Retrieval-Augmented Generation (RAG) has fundamentally changed how AI systems access and reason over external knowledge. Instead of relying purely on what a model learned during training, RAG lets the model pull in fresh, relevant documents at query time, grounding its answers in real data.
But as real-world use cases grow more complex, the single-agent RAG model starts to show cracks. What happens when your knowledge lives in three different places? Documentation, support tickets, and live web data each require different retrieval strategies. A single retriever trying to do it all will either miss important context or drown the LLM in irrelevant noise.
Multi-Agent RAG is the answer. Instead of one agent doing everything, you build a team: specialised agents that each own a knowledge source, a smart router that decides who to call, and a synthesis agent that assembles the final answer. This post walks you through exactly how to build it with LangChain.
Use Case
Imagine you are building a support chatbot for a SaaS product. Users ask questions like:
• “How do I configure OAuth in your API?”
• “Was the login bug from last month ever fixed?”
• “What are the latest changes in the v3.0 release?”
Each question demands a different knowledge source. The first needs product documentation. The second needs the support ticket history. The third needs a live web search. A single RAG agent would attempt to blend all three, producing diluted, confused answers.
Multi-Agent RAG assigns each source to a dedicated agent. The router reads the intent behind the question and dispatches only the right agents. The result is faster, more precise, and far more scalable.
Multi-Agent RAG Architecture
Before diving into code, it is important to understand the full flow. The diagram below shows how a user query travels through the system and becomes a final answer:
Figure 1: Multi-Agent RAG — end-to-end data flow
The system has five distinct stages:
Setting Up the Agents
The foundation of the system is simple: each agent gets its own vector store, its own retriever, and a tightly scoped system prompt. The tighter the scope, the higher the retrieval precision. Here is how the shared infrastructure is wired together:
`from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain. agents import AgentExecutor, create_openai_functions_agent
from langchain. tools.retriever import create_retriever_tool
llm = ChatOpenAI(model="gpt-4o", temperature=0)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
Separate vector stores per knowledge source
docs_vs = FAISS.from_documents(docs_documents, embeddings)
tickets_vs = FAISS.from_documents(ticket_documents, embeddings)
`
Why separate vector stores? Mixing all documents into one store forces the retriever to score across unrelated domains. Separate stores keep similarity scores meaningful and prevent cross-domain noise.
The agent factory function below is reusable across all knowledge sources. Notice how the description parameter is the most important part — it tells the router when to call this agent:
`def build_agent(vectorstore, name, description):
tool = create_retriever_tool(
vectorstore.as_retriever(search_kwargs={"k": 5}),
name=name, description=description
)
prompt = ChatPromptTemplate.from_messages([
("system", f"You are a retrieval agent for {name}. Be precise and concise."),
("human", "{input}"),
("placeholder", "{agent_scratchpad}")
])
agent = create_openai_functions_agent(llm, [tool], prompt)
return AgentExecutor(agent=agent, tools=[tool])
docs_agent = build_agent(docs_vs, "docs_retriever", "Product docs and API guides")
tickets_agent = build_agent(tickets_vs, "tickets_retriever", "Customer support ticket history")`
The Router Agent
The router is the brain of the system. It reads the incoming question and decides, with reasoning, which retrieval agents to activate. The key design choice is to make the output structured JSON. This makes parsing reliable and the routing logic transparent and debuggable.
Using temperature=0 for the router is critical. You want deterministic, consistent routing decisions, not creative ones. The few-shot examples in the prompt do the heavy lifting:
`ROUTER_PROMPT = """
Route the query to the right agents. Return valid JSON only:
{"agents": [...], "reasoning": "..."}
Agents available:
- docs_retriever → technical docs, API reference, how-to guides
- tickets_retriever → past support tickets, known bugs, resolutions
Example 1: "How do I reset my API key?"
{"agents": ["docs_retriever"], "reasoning": "API key management is in the docs"}
Example 2: "Was the 2FA bug from March ever fixed?"
{"agents": ["tickets_retriever", "docs_retriever"],
"reasoning": "Need ticket history for bug context plus docs for fix details"}
"""`
The reasoning field is not just cosmetic. Log it in production. It becomes your best debugging tool when the router makes unexpected decisions.
Parallel Retrieval & Context Aggregation
Once routing is decided, the selected agents execute in parallel using Python’s asyncio. This is one of the biggest wins in multi-agent RAG: you pay the latency cost of only the slowest agent, not the sum of all agents. On a typical setup, three agents running in parallel feel almost as fast as one.
After retrieval, the context aggregator removes duplicate content before passing it to synthesis. Deduplication prevents the LLM from seeing the same passage twice, which wastes context window space and can confuse the final answer:
async def retrieve_parallel(query, agent_names):
tasks = [AGENT_MAP[n].ainvoke({"input": query})
for n in agent_names if n in AGENT_MAP]
results = await asyncio.gather(*tasks)
# Remove duplicate content using hashing
seen, unique = set(), []
for r in results:
h = hash(r["output"])
if h not in seen:
seen.add(h)
unique.append(r["output"])
return unique
Synthesis & Final Generation
The synthesis agent is where it all comes together. It receives the deduplicated context from every retrieval agent and uses an LLM to produce a coherent, grounded final answer. The prompt design here is critical: you want the LLM to stay within the provided context and flag when it cannot answer, rather than hallucinating.
async def answer_query(query):
routing = route_query(query)
contexts = await retrieve_parallel(query, routing["agents"])
combined = "\n\n---\n\n".join(contexts)
prompt = f"""
Answer using ONLY the context provided below.
If the context is insufficient, say: "I don't have enough information."
Context:
{combined}
Question: {query}
"""
return (await llm.ainvoke(prompt)).content
The phrase "ONLY the context provided below" is your guard against hallucination. Without it, AI will confidently fill gaps with its training knowledge, bypassing your retrieval entirely
Prompt Engineering Strategies
Of all the components in this system, prompt quality has the highest leverage. A better router prompt improves every downstream step. Here are the techniques that made the biggest difference:
Few-Shot Prompting
Providing 2–3 examples of routing decisions in the system prompt is the single most effective technique. The model learns domain boundaries implicitly from examples, without requiring you to write exhaustive rule sets. It generalises surprisingly well to edge cases you never explicitly covered.
Structured JSON Output
Forcing the router to return JSON instead of natural language makes the integration reliable. Pair this with temperature=0 and you get near-deterministic routing that is easy to unit test. Always validate the output schema before passing it downstream.
Context Anchoring in Synthesis
In the synthesis prompt, explicitly telling the LLM to stay within the provided context, and to admit when it cannot answer significantly reduces hallucination. Adding "cite the source" instructions further improves factuality and user trust.
Evaluation & Optimization
Building the pipeline is only half the job. You need a systematic way to measure how well it works and, more importantly, which component is responsible when it fails. The three metrics below give you full observability across the system:
Run your evaluation suite after every prompt change, not just after code changes. A one-line tweak to the router prompt can shift routing accuracy by 10–15 percentage points in either direction.
Future Work
This architecture is a starting point. Here are the most impactful extensions to explore next:
Agent memory: give each retrieval agent short-term memory to avoid redundant lookups across multi-turn conversations
Self-correcting loops: let the synthesis agent evaluate context quality and trigger a re-retrieval pass if it judges the context insufficient
Dynamic agent creation: spin up new agents on-the-fly when new knowledge sources are connected, without redeploying the system
Hierarchical routing: nested router layers for very large deployments with dozens of specialise knowledge domains
Cost-aware routing: route to cheaper or cached agents first, escalating to more expensive retrievers only when lower tiers fail to find relevant context
Conclusions
Multi-Agent RAG is not about complexity for its own sake. It is about giving each part of your knowledge base the specialised attention it deserves. When you separate concerns properly, every piece of the system becomes easier to optimise, test, and debug independently.
Specialisation wins: a retrieval agent scoped to one knowledge source consistently outperforms a generalist retriever covering multiple domains
Measure before you optimise: define routing accuracy, context precision, and answer faithfulness targets before writing a single line of code
Parallelism is nearly free: running three retrieval agents concurrently costs almost the same latency as running one
The router prompt is the system’s quality ceiling: invest disproportionately in few-shot examples and output format constraints here
Start simple: two agents and a basic router are enough to validate the pattern; add complexity only when your evaluation data justifies it
Ready to build your Multi-Agent RAG system?
Start with two agents, measure your routing accuracy, and iterate.
Have questions or ideas? Drop them in the comments below.
Top comments (0)