Syeed Talha

Posted on Jul 21

Agentic RAG in LangChain

#ai #rag #agents #langchain

In the previous post, we built a 2-step RAG system where retrieval always happens — no matter what the question is. You ask something, it searches, it answers. Simple and predictable.

But think about this: what if someone just says "Hello! How are you?" — do you really need to search a vector database for that? Of course not. That's the problem Agentic RAG solves.

What is Agentic RAG?

In 2-step RAG, you control when retrieval happens. The flow is fixed:

Query → retrieve → answer

In Agentic RAG, the LLM decides when retrieval happens:

Query → agent thinks → "do I need to search?" → maybe searches → answer

The agent treats the search as a tool it can pick up or put down depending on what the question actually needs. Sometimes it searches once, sometimes multiple times, sometimes not at all.

2-Step RAG vs Agentic RAG

	2-Step RAG	Agentic RAG
Who controls retrieval?	You (the developer)	The LLM (the agent)
Always retrieves?	Yes, every time	Only when needed
Can search multiple times?	No	Yes
Good for?	Simple, predictable Q&A	Complex, dynamic questions

What We're Building

Same knowledge base as before — 5 documents about LangChain concepts. But this time, instead of always retrieving, we give the LLM a search tool and let it decide when to use it.

We'll run 3 queries:

A factual question that needs retrieval
A casual greeting that doesn't need retrieval
Another factual question that needs retrieval

Let's walk through the code.

Step 1: Same Knowledge Base, Same Vector Store

from langchain_core.documents import Document
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS

docs = [
    Document(page_content="LangChain is a framework for building LLM applications. It provides tools for chaining, agents, and retrieval."),
    Document(page_content="Agents in LangChain use an LLM to decide which tools to call. They can reason step-by-step."),
    Document(page_content="RAG stands for Retrieval-Augmented Generation. It combines retrieval with LLM generation for grounded answers."),
    Document(page_content="Vector stores like FAISS and Chroma store embeddings and enable semantic search over documents."),
    Document(page_content="LangGraph is a library for building stateful, multi-actor applications with LLMs."),
]

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(docs, embeddings)

Nothing new here. We embed the documents and store them in FAISS — same as before.

Step 2: Wrapping Search as a Tool

This is where Agentic RAG differs from 2-step RAG. Instead of calling retrieval directly in your code, you wrap it in a @tool decorator and hand it to the agent.

from langchain.tools import tool

@tool
def search_docs(query: str) -> str:
    """Search the knowledge base for relevant information. Use this when you need to look up facts about LangChain, RAG, agents, or related topics."""
    results = vectorstore.similarity_search(query, k=2)
    return "\n\n".join(d.page_content for d in results)

A few things to notice here:

The docstring matters a lot. The agent reads it to understand what the tool does and when to use it. Write it clearly — treat it like an instruction to the LLM, not just a comment for developers.

similarity_search(query, k=2) does the same thing as retriever.invoke() from the previous post — finds the 2 most similar documents. Just a more direct way to call it inside a tool function.

It returns a plain string. The agent needs to read the result, so we join the documents into a single readable string.

Step 3: Creating the Agent

from langchain.agents import create_agent
from langchain_openai import ChatOpenAI

free_llm = ChatOpenAI(
    model="nvidia/nemotron-nano-9b-v2:free",
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ.get("OPENROUTER_API_KEY"),
)

agent = create_agent(
    model=free_llm,
    tools=[search_docs],
    system_prompt=(
        "You are a helpful assistant. If you don't know the answer, "
        "use the search_docs tool to look it up."
    ),
)

The agent gets:

The LLM — its brain for reasoning
A list of tools — what it's allowed to use
A system prompt — its personality and instructions

The system prompt is important here: "If you don't know the answer, use the search_docs tool." This tells the agent to rely on its own knowledge first and only search when it genuinely needs to.

Step 4: Running the Agent

queries = [
    "What is RAG and how does it work?",
    "Hello! How are you?",
    "What is LangGraph?",
]

for q in queries:
    print("\n" + "=" * 60)
    print(f"Q: {q}")
    result = agent.invoke({"messages": [{"role": "user", "content": q}]})
    print(f"A: {result['messages'][-1].content[:300]}")

What Actually Happens for Each Query

Q: "What is RAG and how does it work?"
   ↓
   Agent thinks: "I should look this up"
   ↓
   Calls search_docs("What is RAG")
   ↓
   Gets back relevant documents
   ↓
   Answers using those documents

Q: "Hello! How are you?"
   ↓
   Agent thinks: "This is just a greeting, I know how to handle this"
   ↓
   Skips search_docs entirely
   ↓
   Replies directly

Q: "What is LangGraph?"
   ↓
   Agent thinks: "I should look this up"
   ↓
   Calls search_docs("LangGraph")
   ↓
   Gets back relevant documents
   ↓
   Answers using those documents

That middle query is the key difference from 2-step RAG. No unnecessary search, no wasted tokens.

Full Code

import os
from dotenv import load_dotenv

from langchain.agents import create_agent
from langchain_huggingface import HuggingFaceEmbeddings
from langchain.tools import tool
from langchain_community.vectorstores import FAISS
from langchain_openai import ChatOpenAI
from langchain_core.documents import Document

load_dotenv()

# 1. Knowledge base
docs = [
    Document(page_content="LangChain is a framework for building LLM applications. It provides tools for chaining, agents, and retrieval."),
    Document(page_content="Agents in LangChain use an LLM to decide which tools to call. They can reason step-by-step."),
    Document(page_content="RAG stands for Retrieval-Augmented Generation. It combines retrieval with LLM generation for grounded answers."),
    Document(page_content="Vector stores like FAISS and Chroma store embeddings and enable semantic search over documents."),
    Document(page_content="LangGraph is a library for building stateful, multi-actor applications with LLMs."),
]

# 2. Vector store
print("Building vector store...")
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(docs, embeddings)

# 3. Search tool
@tool
def search_docs(query: str) -> str:
    """Search the knowledge base for relevant information. Use this when you need to look up facts about LangChain, RAG, agents, or related topics."""
    results = vectorstore.similarity_search(query, k=2)
    return "\n\n".join(d.page_content for d in results)

# 4. Agent
free_llm = ChatOpenAI(
    model="nvidia/nemotron-nano-9b-v2:free",
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ.get("OPENROUTER_API_KEY"),
)

agent = create_agent(
    model=free_llm,
    tools=[search_docs],
    system_prompt=(
        "You are a helpful assistant. If you don't know the answer, "
        "use the search_docs tool to look it up."
    ),
)

# 5. Run
queries = [
    "What is RAG and how does it work?",
    "Hello! How are you?",
    "What is LangGraph?",
]

for q in queries:
    print("\n" + "=" * 60)
    print(f"Q: {q}")
    result = agent.invoke({"messages": [{"role": "user", "content": q}]})
    print(f"A: {result['messages'][-1].content[:300]}")

The Key Insight

The @tool decorator and the docstring together are what make this work. The agent doesn't have hardcoded rules like "always search" or "never search." It reads the tool's description, reads the question, and makes a judgment call — just like a person would.

This is why the docstring on search_docs matters so much:

"""Search the knowledge base for relevant information. 
Use this when you need to look up facts about LangChain, 
RAG, agents, or related topics."""

That description is literally the agent's instruction manual for when to use this tool. Write it vague and the agent will use it at the wrong times. Write it clearly and the agent will use it exactly when needed.

What to Keep in Mind

More power, more responsibility. Because the agent decides when to search, a poorly written system prompt or tool description can lead to it searching when it shouldn't, or skipping search when it should. Test your queries.

It costs more per query — sometimes. When the agent searches, it makes an extra LLM call to decide whether to search in the first place. For simple, high-volume Q&A over a fixed dataset, 2-step RAG might actually be the smarter choice.

You can give it multiple tools. Nothing stops you from adding a second tool — say, a web search or a calculator. The agent will pick whichever one fits the question. That's where Agentic RAG really starts to shine.

When to Use Which

Use 2-step RAG when every query needs retrieval and you want simple, predictable behavior
Use Agentic RAG when questions are mixed — some need retrieval, some don't, some might need multiple searches

Start with 2-step. Move to agentic when you hit its limits. That's the practical path.

DEV Community