Building rag-critic: Factual Grounding & Adversarial Hallucination Auditing in LangGraph

Shaheer Ul Hassan — Fri, 23 Oct 2026 20:58:29 +0000

When building production Retrieval-Augmented Generation (RAG) pipelines, the primary failure mode isn't retrieval latency or vector index size—it's silent hallucination.

LLMs frequently generate fluent, highly plausible-sounding claims that are completely unsupported by the retrieved reference chunks. In multi-agent systems and mission-critical workflows, accepting an ungrounded draft can corrupt downstream decision-making.

To solve this in cyclic multi-agent graphs, I built and published rag-critic on PyPI—a lightweight, adversarial hallucination auditor and citation grounding engine architected specifically for LangGraph.

🔍 The Architectural Bottleneck

Standard RAG architectures evaluate answers at a holistic paragraph level using an LLM-as-a-judge. This approach has three major flaws:

Coarse-Grained Feedback: Telling an agent "this answer feels inaccurate" doesn't specify which sentence was hallucinated.
High Latency: Re-prompting large frontier models for full evaluations adds 3–5 seconds per step.
Lack of Determinism: Without an entailment matrix, confidence scores drift across model runs.

📦 Project Links & Open-Source Repositories
PyPI Package: pypi.org/project/rag-critic
GitHub Repository: github.com/samjanjua6/rag-critic
Author Engineering Portfolio: https://shaheerulhassan.com

💻 Quickstart & Python Usage

Install directly from PyPI:


bash
pip install rag-critic


from rag_critic import RAGCritic

critic = RAGCritic()

# Evaluate a draft answer against retrieved source chunks
result = critic.evaluate(
    draft_response="Python was released in 1991 by Guido van Rossum. It was originally named Monty.",
    retrieved_chunks=[
        "Python is an interpreted programming language released in 1991 by Guido van Rossum."
    ]
)

print(f"Status:              {result.status}")              # REVISE / APPROVED
print(f"Grounding Score:      {result.grounding_score:.2f}") # e.g. 0.50
print(f"Hallucination Index:  {result.hallucination_index:.2f}")

# Inspect claim-level verification
for claim in result.claims:
    print(f"[{claim.status.value}] {claim.text}")
    # [SUPPORTED] Python was released in 1991 by Guido van Rossum.
    # [UNGROUNDED] It was originally named Monty.


from langgraph.graph import StateGraph, END
from rag_critic import create_critic_node, critic_router

builder = StateGraph(MyRAGState)

# Add standard generator and retriever nodes
builder.add_node("retrieve", retrieve_docs)
builder.add_node("generate", generate_draft)

# Add rag-critic as an adversarial validation checkpoint
builder.add_node("critic", create_critic_node())

builder.add_edge("retrieve", "generate")
builder.add_edge("generate", "critic")

# Dynamic cyclic routing based on hallucination thresholds
builder.add_conditional_edges(
    "critic",
    critic_router,
    {
        "approved": END,              # Output is grounded; return to user
        "rewrite": "generate",        # Minor gaps; instruct re-writer
        "re_retrieve": "retrieve",    # Heavy hallucination; expand search
    }
)

DEV Community: Shaheer Ul Hassan

Building rag-critic: Factual Grounding & Adversarial Hallucination Auditing in LangGraph

🔍 The Architectural Bottleneck

💻 Quickstart & Python Usage