DEV Community: Gowtham

AI Agents: How LLMs Evolve from Generating Text to Taking Action

Gowtham — Tue, 07 Apr 2026 06:59:21 +0000

For the past two years, the world has been captivated by the "Chatbot Era." We learned to prompt Large Language Models (LLMs) to write emails, summarize documents, and generate code. However, a significant friction point remained: the "Human-in-the-Loop" bottleneck. You would get the text from the AI, but then you—the human—had to manually copy that code into a terminal, send that email, or update that database. The AI provided the intelligence, but you provided the hands.

That paradigm is shifting. We are entering the era of AI Agents. Unlike standard LLMs that simply predict the next token in a sentence, AI Agents use LLMs as a central reasoning engine to navigate software, use tools, and complete multi-step goals autonomously. They don't just tell you how to solve a problem; they execute the solution.

TL;DR: The Agentic Shift

AI Agents are autonomous systems powered by LLMs that can reason, use external tools (APIs), and manage their own memory to achieve complex goals. While traditional LLMs are passive (responding to prompts), AI Agents are active (executing tasks). This evolution turns AI from a digital assistant into a digital workforce capable of handling end-to-end business processes.

What Exactly is an AI Agent?

To understand an AI Agent, think of an LLM as a "brain in a vat." It is incredibly knowledgeable but has no way to interact with the physical or digital world directly. An AI Agent gives that brain a body, tools, and a mission.

An AI Agent is defined by four core components:

The Brain (LLM): The core model (like GPT-4, Llama 3, or Claude) that handles reasoning, planning, and decision-making.
Planning: The ability to break down a complex goal (e.g., "Research this company and find the best person to contact") into smaller, actionable steps.
Memory: Short-term memory (context window) and long-term memory (vector databases) that allow the agent to learn from previous steps and retain information across sessions.
Tool Use (Action): The ability to call external APIs, browse the web, run code, or access internal databases to perform tasks.

Why AI Agents Matter: Beyond the Hype

The transition from text generation to action is not just a technical curiosity; it is a fundamental shift in economic productivity. According to recent industry benchmarks, agentic workflows can improve task success rates by up to 40% compared to zero-shot prompting because the agent can "self-correct" when it encounters an error.

Autonomy and Efficiency

Traditional automation (like RPA) is rigid. If a website layout changes by one pixel, the bot breaks. AI Agents are resilient. Because they use "reasoning," they can look at a changed interface, understand the new context, and adapt their strategy to complete the task. This reduces the maintenance burden on IT teams.

Complex Problem Solving

Most business tasks are not single-turn interactions. They involve loops. An agent can start a task, realize it's missing information, search for that information, update its plan, and then proceed. This "chain-of-thought" processing allows for the automation of high-level roles in research, legal analysis, and software engineering.

24/7 Operations at Scale

AI Agents don't sleep. Enterprises can deploy multiple agents simultaneously to handle a sudden surge in customer support tickets or data processing tasks without hiring a single additional staff member.

The Anatomy of an Agentic Workflow: How It Works

How does an agent actually "take action"? Most modern agents follow a framework known as ReAct (Reason + Act). Here is a simplified breakdown of the process:

Step 1: Goal Decomposition

The user provides a high-level objective: "Find the three cheapest flights from London to New York for next Friday and send the options to my Slack." The agent doesn't just search; it creates a plan: 1. Access calendar to confirm dates. 2. Use a flight API to fetch prices. 3. Compare prices. 4. Format the message. 5. Use the Slack API to send it.

Step 2: Tool Selection and Function Calling

The agent identifies which "tools" it needs. In this case, it might call a "FlightSearch" function. The LLM generates the exact JSON code required to talk to that API. This is the moment where text becomes a command.

Step 3: Observation and Iteration

After the tool returns data (e.g., "No flights found for that specific date"), the agent observes the result. Instead of giving up, it reasons: "Since no flights are available Friday, I will check Thursday and Saturday." It loops back to Step 1 until the goal is achieved or deemed impossible.

Real-World Use Cases for AI Agents

Organizations are already moving past the experimentation phase and deploying agents into production environments. Here are three sectors seeing immediate impact:

Customer Experience and Support

Standard chatbots can answer "What is your return policy?" An AI Agent can actually process the return. It can verify the user's identity, check the order history in the CRM, generate a shipping label via a logistics API, and update the inventory database—all while maintaining a natural conversation with the customer.

Cybersecurity and Cloud Monitoring

In IT infrastructure, speed is everything. An AI Agent integrated with cloud monitoring services can detect network anomalies, autonomously isolate the affected server, trigger a backup, and begin a preliminary forensic analysis — all before a human engineer has opened their laptop.

Software Development (DevOps)

AI Agents like Devin or OpenDevin are now capable of writing code, running it in a sandbox environment, reading the error logs, and fixing their own bugs. For businesses, this means faster sprint cycles and the ability to automate routine maintenance tasks like dependency updates or documentation generation.

Building and Deploying AI Agents: The Infrastructure Requirement

While building a simple agent is easy with frameworks like LangChain, AutoGPT, or CrewAI, deploying them at an enterprise scale is a significant challenge. AI Agents are computationally expensive. They require multiple calls to an LLM for a single task, which can lead to high latency and costs.

To run agents effectively, you need:

Low-Latency Inference: Agents need quick responses to maintain a fluid workflow.
Secure API Orchestration: You are giving an AI the keys to your software. Security must be "baked in" to ensure the agent doesn't perform unauthorized actions.
Scalable Compute: As agents take on more concurrent tasks, the underlying infrastructure must scale horizontally without manual intervention.

The Challenges: Why We Still Need Humans

Despite their potential, AI Agents are not "set and forget." There are three primary hurdles to widespread adoption:

Hallucinations in Action: If an LLM hallucinates a fact, it's annoying. If an AI Agent hallucinates a bank transfer, it's catastrophic. Implementing "guardrails" and human-in-the-loop checkpoints is essential.
Infinite Loops: Sometimes agents get stuck in a "reasoning loop," trying the same failing action repeatedly. This wastes tokens and money.
Security (Prompt Injection): If an agent has access to your email, a malicious actor could send you an email that "tricks" the agent into forwarding your passwords. Robust security protocols are non-negotiable.

Key Takeaways

Evolution: AI is moving from "Generative" (making things) to "Agentic" (doing things).
Core Components: Agents combine LLM reasoning with planning, memory, and tool use (APIs).
Business Value: Agents reduce manual work, adapt to changing environments, and scale operations without increasing headcount.
Infrastructure is Key: Reliable, secure, and scalable cloud infrastructure is required to host and manage autonomous systems at enterprise scale.

Conclusion: The Future is Agentic

The leap from generating text to taking action marks the true beginning of AI's impact on enterprise operations. AI Agents represent a shift from AI as a toy to AI as a tool — and eventually, AI as a teammate. For businesses, the goal is no longer just to implement AI, but to build a cohesive ecosystem of agents that handle the operational heavy lifting.

Frequently Asked Questions (FAQs)

What is the difference between an AI Agent and a Chatbot?

A chatbot is designed for conversation and information retrieval. It waits for a user prompt and provides a response. An AI Agent is designed for goal completion; it can use tools, browse the web, and perform multi-step tasks autonomously to achieve a specific objective.

Do I need to know how to code to use AI Agents?

While many frameworks like LangChain require coding knowledge, new no-code agent platforms are emerging. However, for enterprise-grade agents that interact with internal data, professional deployment is recommended to ensure security and reliability.

Are AI Agents safe for business use?

They can be, provided they are implemented with proper guardrails. This includes "Human-in-the-loop" approvals for sensitive actions, restricted API permissions, and hosting on secure cloud environments to prevent data leaks.

What are the best frameworks for building AI Agents?

Currently, the most popular frameworks are LangChain (for orchestration), CrewAI (for multi-agent systems), AutoGPT (for autonomous research), and Microsoft’s AutoGen. The choice depends on whether you need a single agent or a team of agents working together.

How much do AI Agents cost to run?

The cost depends on the complexity of the task and the number of "turns" or LLM calls required. Because agents iterate and self-correct, they use more tokens than a standard chatbot. Optimising your infrastructure and using efficient models can help manage these costs.

Deploying CVAT on AWS for Image and Video Annotation

Gowtham — Tue, 24 Mar 2026 06:40:16 +0000

Building a computer vision model starts with labelled data, and that labelling work is where a surprising amount of ML project time disappears. CVAT (Computer Vision Annotation Tool) is one of the strongest open-source options for the job. It handles bounding boxes, polygons, segmentation masks, keypoints, and object tracking across images and video.

The challenge most teams hit is not CVAT itself but the infrastructure around it. This post covers deploying a pre-configured CVAT environment on AWS EC2 so you can skip the Docker Compose setup and get straight to annotating.

What the pre-built AMI includes

Multi-format annotation - bounding boxes, polygons, segmentation masks, keypoints, ellipses, cuboids, and video object tracking

Export-ready datasets - YOLO (v5 through v11), COCO, Pascal VOC, TFRecord, and LabelMe formats

OpenCV-powered assists - semi-automatic annotation, keyframe interpolation on video, and label manipulation utilities

S3 storage integration - pre-wired, no manual boto3 configuration needed

Pre-configured environment - delivered ready to use; no Docker Compose debugging on first boot

Launching CVAT on EC2

Subscribe and launch from AWS Marketplace

Find the CVAT AMI in AWS Marketplace and subscribe. Choose Launch through EC2 rather than 1-Click, so you have full control over instance configuration before anything starts.

Configure the instance

The key decisions at launch:

Instance type: t3.large works for individual annotators or small teams. For concurrent sessions or heavy video workloads, move to c5.2xlarge or above.

Key pair: Select or create one. You will need SSH access shortly to retrieve admin credentials.

Network settings: Allow inbound traffic on port 8080. Restrict the source to your team's IP range rather than leaving it open to all. For remote teams, placing CVAT behind an Application Load Balancer with HTTPS is worth the extra step before any production annotation begins.

Storage: Generous EBS sizing matters if you are working with video. Plan for at least 100 GB for any non-trivial dataset.

Access the CVAT interface

Once the instance is running, copy the Public IPv4 address from the EC2 dashboard and open:

http://<EC2_PUBLIC_IP>:8080

On first load, you may see a "Cannot connect to the server" message. This is expected. The CVAT backend services take 60 to 90 seconds to fully initialise. Click OK, wait a moment, and refresh the page. The login screen will appear.

Retrieve admin credentials

SSH into the instance:

bash
ssh -i your-key.pem ubuntu@<EC2_PUBLIC_IP>

Then run:

bash
sudo cat /opt/cvat/superuser.txt

This outputs the auto-generated superuser username and password. Copy both and use them to sign in.

Annotation workflow

Once logged in, the pattern inside CVAT is consistent regardless of annotation type:

Create a project and define your label schema. Labels map directly to the classes your model will learn, for example, car, pedestrian, traffic_light, for a detection task.

Create a task inside the project. Upload raw images or a video file directly through the interface, or point to a pre-configured S3 bucket path.

Assign annotators using CVAT's built-in role system. Annotators label; reviewers validate before export. Rejected frames route back to annotators with comments, keeping quality control inside the same platform.

Annotate using the tool that fits the task:

Bounding box for object detection
Polygon, for instance, segmentation
Key points for pose estimation
Tracking mode for video — auto-interpolates object positions between labelled keyframes, cutting annotation time significantly on longer clips

Export once the review cycle is complete.

Export formats for model training

Choose the format that matches your training framework directly:

Exported archives include label files and images structured exactly as the framework expects. No post-processing or conversion step required.

Before going to production

Snapshot your configured instance. Once label schemas, user accounts, and storage integrations are set up the way you want, take an AMI snapshot. You can launch from it later if you need to scale to a larger instance type or recover quickly.

Back up annotation data. CVAT stores its database on the instance. Export completed tasks as archives before stopping or terminating the instance. A scheduled S3 sync of the CVAT data directory is good practice for ongoing projects.

Pre-annotation workflow. If you are partway through a training run, use early model checkpoints to generate draft annotations on unlabelled batches, import those predictions back into CVAT as pre-annotations, and have annotators correct rather than label from scratch. The time saved on large batches is substantial.

Wrapping up

Annotation infrastructure is easy to underestimate, but the quality and consistency of your labelling pipeline have a direct effect on how quickly a model converges and how reliable its outputs are.

Running CVAT on your own EC2 environment keeps training data inside your own VPC, avoids per-seat SaaS pricing, and gives you a reproducible setup you can snapshot and relaunch at any point. The pre-configured AMI removes the setup friction that usually slows teams down when starting with self-hosted CVAT- learn more

Retrieval-Augmented Generation: The Complete Guide

Gowtham — Sat, 21 Mar 2026 11:34:31 +0000

How RAG fixes the fundamental limitations of large language models — and becomes the foundation of every production AI system worth building.

Large language models are remarkable at generating fluent, coherent text. They have absorbed billions of documents and can discuss almost any topic with apparent fluency. But beneath the surface lies a fundamental architectural constraint: LLMs are frozen at the moment of their training. They know nothing that happened after their cutoff date. They have access to no data you have not already baked into their weights. And when they are uncertain, they do not say so — they confabulate plausibly.

This is not a bug in a specific model. It is an intrinsic property of how transformer-based language models work. The question, then, is not how to fix the model — it is how to build a system around the model that compensates for this limitation while preserving everything that makes LLMs so powerful.

That system is Retrieval-Augmented Generation.How RAG fixes the fundamental limitations of large language models — and becomes the foundation of every production AI system worth building.

That system is Retrieval-Augmented Generation.

Important: Hallucination is not a fixable bug — it is a structural property of language models. RAG does not remove hallucination from the model. It removes the conditions that cause it: the model no longer needs to invent facts it doesn't know, because you give it those facts at query time.

What is Retrieval-Augmented Generation?

RAG is an AI architecture pattern that augments a language model's context window with information retrieved from an external knowledge source at inference time. Instead of relying solely on parametric memory — the knowledge baked into model weights during training — a RAG system retrieves relevant documents, passages, or data points from a corpus and injects them into the prompt before generation occurs.

The original paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., Meta AI Research, 2020), demonstrated that this simple architectural change — add retrieval, add context — produces models that are more factual, more up-to-date, and more attributable than pure parametric models. Every major AI deployment in 2026 that requires factual accuracy is built on some variant of this pattern.

The three problems RAG solves

Before RAG, production deployments of LLMs faced three structural problems that no amount of prompt engineering could fix:

🧠 Knowledge Cutoff LLMs are frozen at their training date. No new research, no new products, no current events — unless you retrain, which costs millions.

🌀 Hallucination When models don't know, they generate the most plausible-sounding answer. At enterprise scale, this is catastrophic for trust and liability.

🔒 No Private Data Your internal documents, your CRM, your proprietary knowledge — none of it is in any LLM. RAG bridges this gap without exposing your data to model training.

Architecture Overview

A RAG system has two distinct pipelines: an offline indexing pipeline that runs once (or on a schedule) to prepare your knowledge base, and an online inference pipeline that runs at every query. Understanding both is essential for building systems that are both accurate and fast.

Offline indexing pipeline

The offline pipeline transforms raw documents — PDFs, web pages, databases, wikis, code repositories — into a searchable vector index. This pipeline runs when you first set up the system, and again whenever your source documents change.

01 — Document Loading Source documents are loaded from wherever they live: S3 buckets, SharePoint, databases, APIs, or local filesystems. Document loaders parse the raw format into clean text.

02 — Chunking Documents are split into overlapping chunks — typically 256 to 1024 tokens. Chunking strategy is one of the most important tuning decisions in a RAG system. Too small: loss of context. Too large: retrieval noise.

03 — Embedding Each chunk is converted into a dense vector representation using an embedding model. Semantically similar chunks produce similar vectors. This is what enables meaning-based search.

04 — Vector Storage Vectors and their associated text chunks are stored in a vector database. The database builds an Approximate Nearest Neighbour (ANN) index for sub-millisecond similarity search at scale.

Online inference pipeline

The online pipeline runs at every user query and is what users interact with. Latency here matters.

01 — Query Embedding The user's question is converted to a vector using the same embedding model used at index time. This ensures the query and documents exist in the same semantic space.

02 — Retrieval The vector database finds the top-K chunks most similar to the query vector. This is semantic search: "Can I get my money back?" retrieves the same chunks as "What is your refund policy?"

03 — Context Assembly Retrieved chunks are assembled into a context window and prepended to the user's query in the LLM prompt. The model now has the relevant facts it needs.

04 — Generation The LLM generates a response grounded in the retrieved context. Because the relevant facts are present in the prompt, the model has no reason to invent them.

Note: The key insight: the LLM is not asked to remember facts. It is asked to reason over facts you have already provided. This is why grounding works.

Code example

The following example builds a complete RAG system using LangChain. It works with any LLM — Ollama for local models, or any cloud-hosted model through LangChain's unified interface.

# Python · LangChain · Works with any LLM

# Step 1: Load documents
from langchain_community.document_loaders import PyPDFDirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

loader = PyPDFDirectoryLoader("./knowledge_base/")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64
)

chunks = splitter.split_documents(docs)

# Step 2: Embeddings + Vector DB
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

vectordb = Chroma.from_documents(
    documents=chunks,
    embedding=OpenAIEmbeddings(),
    persist_directory="./chroma_db"
)

retriever = vectordb.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 5, "fetch_k": 10}
)

# Step 3: LLM + RAG
from langchain_community.llms.ollama import Ollama
from langchain.chains import RetrievalQA

llm = Ollama(model="llama3.1")

rag = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True
)

# Step 4: Query
result = rag.invoke(
    {"query": "What is our Q3 revenue target?"}
)

print(result["result"])
print(result["source_documents"])

RAG vs. Fine-Tuning vs. Base LLM

A common question when adopting LLMs for enterprise use is whether to fine-tune a model on your data or use RAG. The answer depends on what problem you are solving — and the two approaches are not mutually exclusive.

Fine-tuning teaches the model to behave differently. It is best for tasks involving style, format, domain-specific reasoning patterns, or specialized vocabulary that the base model does not handle well. RAG teaches the system to access information it does not have. It is best for tasks requiring current, private, or attributable facts. The distinction is behavior versus knowledge.

When to combine both: The highest-performing production systems often combine RAG with fine-tuning. Fine-tune the model on your domain's style, terminology, and reasoning patterns. Use RAG to supply the current facts at query time. This hybrid approach gives you the best of both: domain-adapted reasoning grounded in real, up-to-date information.

Real-World Applications

RAG is not a research prototype. It is the architectural foundation of the most widely deployed AI systems in production as of 2026.

Enterprise knowledge management Companies lose an estimated 2.5 hours per employee per day to information search. RAG systems built over internal wikis, documentation, and process documents convert this cost into a productivity gain. Employees query in natural language and receive cited, accurate answers in seconds.

Implementations: Notion AI, Confluence AI, Microsoft Copilot for SharePoint. Common outcomes: 40–60% reduction in time-to-answer for internal queries, measurable reduction in support ticket volume as employees self-serve.

Software development tooling Enterprise code assistants built on RAG index a company's internal codebase, API documentation, architecture decision records, and runbook documentation. Unlike generic coding assistants, these systems understand the company's proprietary libraries, internal naming conventions, and past architectural decisions. Developers receive context-specific suggestions, not generic code completions.

Evaluating Your RAG System
A RAG system that has not been evaluated is a liability. The RAGAS framework (Retrieval Augmented Generation Assessment) provides a principled set of metrics for measuring RAG pipeline quality.
RAGAS metrics

Important: 73% of RAG deployments have no automated evaluation pipeline. They discover failures when users complain, by which point trust is already damaged. Build evaluation in from day one — not as an afterthought.

Advanced RAG Patterns

Basic RAG — embed, retrieve, generate — is the foundation. Production systems extend this pattern in several important ways.

Hybrid Search Pure dense retrieval (vector similarity) misses exact keyword matches. BM25-based sparse retrieval misses semantic equivalences. Hybrid search combines both: a weighted sum of dense and sparse retrieval scores that outperforms either approach in isolation. Independent benchmarks show 30–40% better recall compared to vector-only retrieval across most enterprise domains.

HyDE (Hypothetical Document Embedding). Instead of embedding the user's query directly, HyDE first prompts the LLM to generate a hypothetical answer document, then embeds that document for retrieval. The intuition is that a hypothetical answer document is more semantically similar to actual answer documents in the corpus than the raw query is. This consistently improves retrieval quality, particularly for short or ambiguous queries.

Reranking Initial retrieval uses fast approximate methods (ANN search) that optimise for speed over precision. A cross-encoder reranker re-scores the top-K retrieved candidates with a more expensive but more accurate model, reordering them before passing to the LLM. Cross-encoder rerankers improve top-1 precision by 15–25% at the cost of additional latency — a worthwhile tradeoff for high-stakes queries.

Agentic RAG In agentic RAG, the LLM is not a passive consumer of retrieved context — it actively decides what to retrieve, when to retrieve, and how to use what it finds. The model can issue multiple retrieval calls, critique its own retrieved context, request clarification, and iterate. This enables complex, multi-hop reasoning that is impossible with single-shot retrieval. The tradeoff is higher latency and cost per query.

Five-step implementation path
Start with a small, high-quality document corpus. Quality beats quantity in RAG. A curated 1,000-document corpus outperforms a messy 100,000-document corpus.

Choose a chunking strategy appropriate to your document types. Fixed-size for uniform documents. Semantic chunking for mixed content. Hierarchical chunking for structured documents like manuals or legal contracts.

Select an embedding model and benchmark it on your domain before committing. The best general-purpose model is not always the best for your specific use case.

Build evaluation in from the start. Instrument with RAGAS metrics before you ship. Set target thresholds: Faithfulness ≥ 0.90, Answer Relevance ≥ 0.80.

Iterate on retrieval quality before iterating on generation quality. Most RAG failures are retrieval failures, not generation failures. Fix the retrieval first.

Conclusion

Retrieval-Augmented Generation is not a feature or a plugin. It is an architectural pattern that fundamentally changes what is possible to build with language models. It transforms LLMs from static encyclopedias into dynamic reasoning systems that can access your data, stay current, cite their sources, and operate within the boundaries your organisation requires.

The foundational concepts in this post — the offline indexing pipeline, the online inference pipeline, the RAGAS evaluation framework, and the comparison with fine-tuning — are the building blocks for everything that follows. In Part 02, we go deeper into the retrieval layer: why basic vector search is insufficient for production workloads, and how Hybrid Search, HyDE, and Reranking address its limitations.

Scalable Multi-Agent Retrieval Systems Using LangChain

Gowtham — Thu, 26 Feb 2026 12:29:17 +0000

Retrieval-Augmented Generation (RAG) has fundamentally transformed how AI systems access and reason over external knowledge. Instead of relying purely on what a model learned during training, RAG allows the model to retrieve fresh, relevant documents at query time, grounding its responses in real, up-to-date data.

However, as real-world use cases grow more complex, the traditional single-agent RAG architecture begins to show limitations. What happens when your knowledge exists across multiple sources? Product documentation, historical support tickets, and live web data each require distinct retrieval strategies. A single retriever attempting to handle all of them either misses critical context or overwhelms the LLM with irrelevant noise.

Multi-Agent RAG addresses this challenge. Instead of one agent handling everything, you build a coordinated system: specialised agents that own individual knowledge sources, a routing agent that decides which agents to activate, and a synthesis agent that composes the final grounded answer. In this post, we will walk through how to build this architecture using LangChain.

Use Case

magine you are developing a support chatbot for a SaaS product. Users might ask:

“How do I configure OAuth in your API?”

“Was the login bug from last month ever resolved?”

“What are the latest changes in the v3.0 release?”

Each question requires access to a different knowledge source. The first depends on product documentation. The second relies on support ticket history. The third may require recent release notes or even live web updates.

A single RAG agent would attempt to blend all sources into one retrieval step, often producing diluted or confused answers.

Multi-Agent RAG assigns each knowledge source to a dedicated retrieval agent. A router interprets the user’s intent and activates only the relevant agents. The result is faster, more precise, and significantly more scalable.

Multi-Agent RAG Architecture

Before writing code, it is important to understand the complete system flow. The diagram below illustrates how a user query moves through the architecture to produce a final answer.

Figure 1: Multi-Agent RAG — end-to-end data flow

The system consists of five major stages:

Setting Up the Agents

The foundation of the system is straightforward: each knowledge source gets its own vector store, retriever, and tightly scoped system prompt. The narrower the scope, the higher the retrieval precision.

Here is how the shared infrastructure is initialized:

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.tools.retriever import create_retriever_tool

llm = ChatOpenAI(model="gpt-4o", temperature=0)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

Separate vector stores per knowledge source

docs_vs = FAISS.from_documents(docs_documents, embeddings)
tickets_vs = FAISS.from_documents(ticket_documents, embeddings)

Why separate vector stores?

Combining all documents into one store forces the retriever to score similarity across unrelated domains. Isolating stores ensures cleaner similarity matching and reduces cross-domain noise.

The agent factory function below can be reused for each knowledge source. Notice that the description parameter plays a crucial role — it informs the router when this agent should be invoked.

def build_agent(vectorstore, name, description):
tool = create_retriever_tool(
vectorstore.as_retriever(search_kwargs={"k": 5}),
name=name, description=description
)
prompt = ChatPromptTemplate.from_messages([
("system", f"You are a retrieval agent for {name}. Be precise and concise."),
("human", "{input}"),
("placeholder", "{agent_scratchpad}")
])
agent = create_openai_functions_agent(llm, [tool], prompt)
return AgentExecutor(agent=agent, tools=[tool])

docs_agent = build_agent(docs_vs, "docs_retriever", "Product documentation and API guides")
tickets_agent = build_agent(tickets_vs, "tickets_retriever", "Customer support ticket history")

The Router Agent

The router is the decision-making core of the system. It analyzes the incoming query and determines which retrieval agents to activate.

The key design decision here is structured JSON output. This ensures routing decisions are transparent, deterministic, and easy to debug.

Setting temperature=0 for the router is essential. Routing requires consistency, not creativity.

ROUTER_PROMPT = """
Route the query to the correct agents. Return valid JSON only:
{"agents": [...], "reasoning": "..."}

Agents available:

docs_retriever → technical documentation, API references, how-to guides
tickets_retriever → support tickets, bug reports, issue resolutions

Example 1: "How do I reset my API key?"
{"agents": ["docs_retriever"], "reasoning": "API key management is covered in documentation"}

Example 2: "Was the 2FA bug from March resolved?"
{"agents": ["tickets_retriever", "docs_retriever"],
"reasoning": "Ticket history provides context; documentation confirms the fix"}
"""
The reasoning field is not decorative — log it in production. It becomes invaluable when debugging routing decisions.

Parallel Retrieval and Context Aggregation

After routing, selected agents execute in parallel using asyncio. This is one of the most significant advantages of multi-agent RAG: latency is determined by the slowest agent, not the sum of all agents.

Once retrieval completes, context aggregation removes duplicate content. Duplicate passages waste context window space and may distort synthesis.

async def retrieve_parallel(query, agent_names):
tasks = [AGENT_MAP[n].ainvoke({"input": query})
for n in agent_names if n in AGENT_MAP]
results = await asyncio.gather(*tasks)

seen, unique = set(), []
for r in results:
    h = hash(r["output"])
    if h not in seen:
        seen.add(h)
        unique.append(r["output"])
return unique

Synthesis and Final Answer Generation

The synthesis agent receives the deduplicated context and produces the final grounded response.

Prompt discipline is critical here. The model must remain strictly anchored to retrieved context.

async def answer_query(query):
routing = route_query(query)
contexts = await retrieve_parallel(query, routing["agents"])
combined = "\n\n---\n\n".join(contexts)

prompt = f"""
Answer using ONLY the context provided below.
If the context is insufficient, say: "I don't have enough information."

Context:
{combined}

Question: {query}
"""
return (await llm.ainvoke(prompt)).content

The phrase “ONLY the context provided below” significantly reduces hallucination by preventing the model from relying on its internal training knowledge.

Prompt Engineering Strategies

Prompt quality has the highest leverage across the system.

Few-Shot Prompting

Providing 2–3 routing examples dramatically improves classification accuracy.

Structured Output

Enforcing JSON ensures integration reliability and supports automated validation.

Context Anchoring

Explicitly instructing the model to rely only on retrieved context improves factual consistency.

Evaluation and Optimization

Deployment without evaluation is risky. You should measure:

Routing accuracy

Context precision

Answer faithfulness

Re-run evaluations after prompt changes, not just code updates. Small prompt tweaks can shift routing accuracy significantly.

Future Improvements

Agent memory for multi-turn continuity
Self-correcting retrieval loops
Dynamic agent creation for new knowledge sources
Hierarchical routing layers
Cost-aware routing strategies

Conclusion

Multi-Agent RAG is not about unnecessary complexity. It is about giving each knowledge source the specialisation it deserves.

Specialisation improves retrieval precision
Measure before optimising
Parallelism minimises latency overhead
Router prompt quality defines system reliability
Start simple and scale intentionally

Begin with two agents. Measure routing performance. Iterate deliberately.