Syeed Talha

Posted on Jul 21

Hybrid RAG in LangChain

#ai #rag #agents #langchain

The two RAG approaches we've covered so far both have a silent assumption baked in — that the first retrieval attempt will return good enough documents, and that the generated answer will be good enough to return directly.

Most of the time that works fine. But sometimes the original query is vague, the retrieved documents are only loosely related, or the generated answer drifts away from what the context actually says.

Hybrid RAG adds validation loops to catch those failures before they reach the user.

The Core Idea

2-Step RAG has one shot at retrieval. Agentic RAG decides whether to retrieve at all. Hybrid RAG does both and then asks: "Was that actually good enough?"

If the retrieved documents can't answer the question — try again with a better query. If the generated answer isn't grounded in the context — regenerate it. Only return the answer when it passes both checks.

The Workflow

Original Query
      ↓
  Step 1: Query Enhancement
  LLM rewrites the query to be more search-friendly
      ↓
  Enhanced Query
      ↓
  Step 2: Retrieve (k=2)
  FAISS similarity search
      ↓
  Step 3: Validation Loop — "Are these docs sufficient?"
  ┌─────────────────────────────────────────────────┐
  │  LLM checks: can these docs answer the query?   │
  │                                                 │
  │  NO → refine query → retrieve again (k=3) ──┐  │
  │  (max 3 attempts)                           │  │
  │                         ↑───────────────────┘  │
  │  YES → continue                                 │
  └─────────────────────────────────────────────────┘
      ↓
  Step 4: Generate Answer
  LLM answers using retrieved docs as context
      ↓
  Step 5: Answer Validation — "Is this answer accurate?"
  ┌─────────────────────────────────────────────────┐
  │  LLM checks: is the answer grounded in context? │
  │                                                 │
  │  NO → regenerate answer ────────────────────┐  │
  │                                             │  │
  │  YES → done                  ───────────────┘  │
  └─────────────────────────────────────────────────┘
      ↓
  Final Answer

Walking Through the Code

The Knowledge Base and Vector Store

docs = [
    Document(page_content="LangChain is a framework for building LLM applications..."),
    Document(page_content="Agents in LangChain use an LLM to decide which tools to call..."),
    Document(page_content="RAG stands for Retrieval-Augmented Generation..."),
    Document(page_content="Vector stores like FAISS and Chroma store embeddings..."),
    Document(page_content="LangGraph is a library for building stateful, multi-actor applications..."),
]

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(docs, embeddings)

Same as before — embed the documents and store them in FAISS. Nothing new here.

Step 1: Query Enhancement

def enhance_query(original: str) -> str:
    prompt = f"Rewrite this question to make it better for searching a LangChain documentation knowledge base. Return ONLY the rewritten question.\n\nOriginal: {original}\n\nRewritten:"
    result = free_llm.invoke(prompt)
    return result.content.strip()

Before searching, we ask the LLM to rewrite the query into something more likely to match the documents.

For example:

Original:  "What is RAG?"
Enhanced:  "Explain Retrieval-Augmented Generation and how it works in LangChain"

The enhanced version has more overlap with the actual document text, which means FAISS is more likely to return relevant results.

Step 2: Retrieve

def retrieve(query: str, k: int = 2) -> list[Document]:
    return vectorstore.similarity_search(query, k=k)

A straightforward similarity search. Takes the (now enhanced) query, embeds it, and finds the k most similar documents in FAISS.

Step 3: Retrieval Validation Loop

def is_sufficient(query: str, docs: list[Document]) -> bool:
    prompt = f"Can the following documents answer this question? Answer ONLY 'yes' or 'no'.\n\nQuestion: {query}\n\nDocuments:\n{chr(10).join(d.page_content for d in docs)}\n\nAnswerable:"
    result = free_llm.invoke(prompt)
    return result.content.strip().lower().startswith("yes")

This is the first validation gate. We show the LLM the question and the retrieved documents and ask it one thing: "Can you actually answer this question from these documents?"

If the answer is no, the retrieval loop kicks in:

retrieved = retrieve(enhanced)
attempt = 1

while not is_sufficient(q, retrieved) and attempt < 3:
    print(f"  Attempt {attempt}: not sufficient, refining...")
    refined = enhance_query(f"{q} (more specific)")
    retrieved = retrieve(refined, k=3)
    attempt += 1

On each failed attempt:

The query is rewritten again, this time with (more specific) appended to push the LLM toward a more targeted rewrite
We retrieve again, this time with k=3 to cast a slightly wider net
We try a maximum of 3 times before moving on regardless

Step 4: Generate

def generate(query: str, context: str) -> str:
    prompt = f"Answer the question using ONLY the context below.\n\nContext:\n{context}\n\nQuestion: {query}\n\nAnswer:"
    result = free_llm.invoke(prompt)
    return result.content.strip()

Same as 2-step RAG — build a prompt with the retrieved context and ask the LLM to answer from it. The ONLY constraint keeps the LLM from pulling in outside knowledge.

Step 5: Answer Validation

def is_answer_good(query: str, answer: str, context: str) -> bool:
    prompt = f"Is the following answer accurate and based ONLY on the context? Answer ONLY 'yes' or 'no'.\n\nQuestion: {query}\n\nContext:\n{context}\n\nAnswer:\n{answer}\n\nAccurate:"
    result = free_llm.invoke(prompt)
    return result.content.strip().lower().startswith("yes")

The second validation gate. After generating the answer, we ask the LLM to review its own output: "Is this answer actually grounded in the context we retrieved?"

if is_answer_good(q, answer, context):
    print(f"  Validation: PASSED")
else:
    print(f"  Validation: FAILED, regenerating...")
    answer = generate(q, context)

If it fails, we regenerate once more with the same context. The idea is that the first generation might have drifted — a second attempt with the same context tends to stay more grounded.

Full Code

import os
from dotenv import load_dotenv

from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_openai import ChatOpenAI
from langchain_core.documents import Document

load_dotenv()

# 1. Knowledge base
docs = [
    Document(page_content="LangChain is a framework for building LLM applications. It provides tools for chaining, agents, and retrieval."),
    Document(page_content="Agents in LangChain use an LLM to decide which tools to call. They can reason step-by-step."),
    Document(page_content="RAG stands for Retrieval-Augmented Generation. It combines retrieval with LLM generation for grounded answers."),
    Document(page_content="Vector stores like FAISS and Chroma store embeddings and enable semantic search over documents."),
    Document(page_content="LangGraph is a library for building stateful, multi-actor applications with LLMs."),
]

print("Building vector store...")
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(docs, embeddings)

free_llm = ChatOpenAI(
    model="nvidia/nemotron-nano-9b-v2:free",
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ.get("OPENROUTER_API_KEY"),
)

# 2. Pipeline functions
def enhance_query(original: str) -> str:
    prompt = f"Rewrite this question to make it better for searching a LangChain documentation knowledge base. Return ONLY the rewritten question.\n\nOriginal: {original}\n\nRewritten:"
    result = free_llm.invoke(prompt)
    return result.content.strip()

def retrieve(query: str, k: int = 2) -> list[Document]:
    return vectorstore.similarity_search(query, k=k)

def is_sufficient(query: str, docs: list[Document]) -> bool:
    prompt = f"Can the following documents answer this question? Answer ONLY 'yes' or 'no'.\n\nQuestion: {query}\n\nDocuments:\n{chr(10).join(d.page_content for d in docs)}\n\nAnswerable:"
    result = free_llm.invoke(prompt)
    return result.content.strip().lower().startswith("yes")

def generate(query: str, context: str) -> str:
    prompt = f"Answer the question using ONLY the context below.\n\nContext:\n{context}\n\nQuestion: {query}\n\nAnswer:"
    result = free_llm.invoke(prompt)
    return result.content.strip()

def is_answer_good(query: str, answer: str, context: str) -> bool:
    prompt = f"Is the following answer accurate and based ONLY on the context? Answer ONLY 'yes' or 'no'.\n\nQuestion: {query}\n\nContext:\n{context}\n\nAnswer:\n{answer}\n\nAccurate:"
    result = free_llm.invoke(prompt)
    return result.content.strip().lower().startswith("yes")

# 3. Run
queries = [
    "What is RAG?",
    "Tell me about vector stores",
]

for q in queries:
    print("\n" + "=" * 60)
    print(f"Q: {q}")

    # Step 1: Enhance
    enhanced = enhance_query(q)
    print(f"  1. Enhanced: {enhanced}")

    # Step 2: Retrieve with validation loop
    retrieved = retrieve(enhanced)
    attempt = 1
    while not is_sufficient(q, retrieved) and attempt < 3:
        print(f"  2. Attempt {attempt}: not sufficient, refining...")
        refined = enhance_query(f"{q} (more specific)")
        retrieved = retrieve(refined, k=3)
        attempt += 1
    print(f"  2. Retrieved {len(retrieved)} doc(s) after {attempt} attempt(s)")

    # Step 3: Generate
    context = "\n\n".join(d.page_content for d in retrieved)
    answer = generate(q, context)
    print(f"  3. Generated: {answer[:150]}...")

    # Step 4: Validate
    if is_answer_good(q, answer, context):
        print(f"  4. Validation: PASSED")
    else:
        print(f"  4. Validation: FAILED, regenerating...")
        answer = generate(q, context)
        print(f"     Regenerated: {answer[:150]}...")

    print(f"\n  Final: {answer}")

Sample Output

============================================================
Q: What is RAG?
  1. Enhanced: Explain Retrieval-Augmented Generation in LangChain
  2. Retrieved 2 doc(s) after 1 attempt(s)
  3. Generated: RAG stands for Retrieval-Augmented Generation...
  4. Validation: PASSED

  Final: RAG stands for Retrieval-Augmented Generation. It combines
  retrieval with LLM generation for grounded answers.

============================================================
Q: Tell me about vector stores
  1. Enhanced: How do vector stores work in LangChain for semantic search?
  2. Retrieved 2 doc(s) after 1 attempt(s)
  3. Generated: Vector stores like FAISS and Chroma store embeddings...
  4. Validation: PASSED

  Final: Vector stores like FAISS and Chroma store embeddings and
  enable semantic search over documents.

How It Compares to the Others

	2-Step RAG	Agentic RAG	Hybrid RAG
Query enhancement	No	No	Yes
Retrieval validation	No	No	Yes
Answer validation	No	No	Yes
Retries on failure	No	No	Yes
LLM calls per query	2	2-4	4-8
Complexity	Low	Medium	High
Answer reliability	Basic	Better	Highest

The tradeoff is obvious — more reliability costs more LLM calls. For a small knowledge base and simple questions, 2-step RAG is perfectly fine. Hybrid RAG makes sense when wrong answers have real consequences.

What to Keep in Mind

Each validation step is an LLM call. A single query can trigger 4 to 8 LLM calls depending on how many retries happen. Be mindful of latency and cost if you're running this at scale.

The LLM is grading its own work. is_answer_good() asks the same LLM that generated the answer to validate it. This is better than no validation, but it's not foolproof. A model that generated a bad answer might also approve it. In production systems, you'd want a separate, stronger model doing the validation.

The retry loop has a hard limit. The attempt < 3 cap prevents infinite loops when the knowledge base genuinely doesn't have the answer. After 3 attempts, the pipeline moves forward with whatever it has rather than looping forever.

When to Use Hybrid RAG

When answer accuracy matters more than speed
When your queries are varied and sometimes vague
When you can afford the extra LLM calls per query
When you want a self-correcting pipeline rather than a one-shot system

Start with 2-step RAG. If you find it returning wrong or incomplete answers, Hybrid RAG is the natural next step.

DEV Community