DEV Community

Vishal Anand
Vishal Anand

Posted on

How to ensure citation accuracy in LangGraph supervisor responses using structured RAG tool outputs?

#agents #tooling #architecture #llm

I’m building a customer support agent that helps users troubleshoot tech issues. The supervisor agent can call a tool called researcher, which returns:

  • A plain-English diagnostic summary (research text)
  • The sources it consulted (citations)
  • The exact places in the diagnostic summary where each source should be cited (citation placement)
  • The original text snippets pulled from each source for the citation (verbatim quoted text)

Once the troubleshooter tool finishes its job and hands off to the the main agent (or supervisor), it needs to:

  1. Analyse that information to either ask the user follow-up questions or suggest next steps,
  2. But it must do this in a way that:
    • When supervisor responds it should include proper inline citations tied to the right parts of the explanation from the research info by the troubleshooter
    • Is 100% citation accurate

Code for my current agent architecture 

import logging
from collections.abc import Sequence
from typing import (
    Annotated,
    TypedDict,
)

from langchain_core.messages import AIMessage, BaseMessage, SystemMessage
from langchain_core.runnables import RunnableConfig
from langchain_core.tools import tool
from langgraph.graph import END, StateGraph
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode

from agents.prompts import (
    SUPERVISOR_SYSTEM_PROMPT,
)
from agents.utils import enrich_prompt
from core import get_model
from schema.models import AnthropicModelName

class Citation(TypedDict):
    id: str
    source: str
    url: str | None

class Quote(TypedDict):
    id: str
    text: str

class ResearcherOutput(TypedDict):
    diagnostic_summary: str  # "The issue is X. <citation id=1>Try solution A</citation>. <citation id=2>Check B</citation>."
    citations: list[Citation]  # [{"id": "1", "source": "Microsoft Docs", "url": "..."}]
    verbatim_quotes: list[Quote]  # [{"id": "1", "text": "Exact quote from source"}]

@tool
def researcher(research_query: str) -> ResearcherOutput:
    """Analyze and research technical issues with source citations."""
    # RAG/RESEARCH LOGIC HERE that uses research_query to provide the below response
    # ......
    return {
        "diagnostic_summary": "Issue analysis with <citation id=1>inline citation title</citation>",
        "citations": [{"id": "1", "source": "Source Name", "url": "example.com"}],
        "verbatim_quotes": [{"id": "1", "text": "Exact quote from source"}]
    }

class AgentState(TypedDict):
    """The state of the agent."""
    messages: Annotated[Sequence[BaseMessage], add_messages]

# Tools that the supervisor can call when needed
tools = [researcher]

tools_by_name = {tool.name: tool for tool in tools}

model = get_model(AnthropicModelName.SONNET_4).bind_tools(tools)

tool_node = ToolNode(tools)

# Define the node that calls the model
def call_model(
    state: AgentState,
    config: RunnableConfig,
):
    supervisor_prompt = SUPERVISOR_SYSTEM_PROMPT
    system_prompt = SystemMessage(enrich_prompt(supervisor_prompt))
    response = model.invoke([system_prompt, *state["messages"]], config)
    return {"messages": [response]}

# Define the conditional edge that determines whether to continue or not
def should_continue(state: AgentState):
    messages = state["messages"]
    last_message = messages[-1]
    if not isinstance(last_message, AIMessage):
        return "end"
    # If there is no function call, then we finish
    if not last_message.tool_calls:
        return "end"
    # Otherwise if there is, we continue
    else:
        return "continue"

# Define a new graph
workflow = StateGraph(AgentState)

workflow.add_node("agent", call_model)
workflow.add_node("tools", tool_node)

workflow.set_entry_point("agent")

workflow.add_conditional_edges(
    "agent",
    should_continue,
    {
        # If `tools`, then we call the tool node.
        "continue": "tools",
        # Otherwise we finish.
        "end": END,
    },
)

workflow.add_edge("tools", "agent")
agent = workflow.compile()
Enter fullscreen mode Exit fullscreen mode

✅ Expected End-to-End Example

🧠 User Query sent by the supervisor to the researcher tool

“My Windows laptop is overheating a lot recently and the fan runs constantly. What can I do?”

🛠️ Researcher Tool Response 

{
  "diagnostic_summary": "This issue is often caused by dust buildup in the vents or outdated drivers. <citation id='1'>Try cleaning the air vents with compressed air.</citation> <citation id='2'>Also, update your BIOS and graphics drivers to ensure proper thermal management.</citation>",
  "citations": [
    {
      "id": "1",
      "source": "Microsoft Support",
      "url": "https://support.microsoft.com/laptop-overheating"
    },
    {
      "id": "2",
      "source": "Dell Knowledge Base",
      "url": "https://www.dell.com/support/kb/article/BIOS-and-thermal"
    }
  ],
  "verbatim_quotes": [
    {
      "id": "1",
      "text": "Use compressed air to clean the vents and fans to prevent overheating caused by dust."
    },
    {
      "id": "2",
      "text": "Keeping your BIOS and drivers updated can improve system cooling and fan control."
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

🧑‍⚖️ Supervisor Agent’s Response to user After Reviewing the Research (Expected Output)

Just to check — have you noticed whether the fan runs loudly even when you’re not doing anything intensive, like browsing or idling?

That helps narrow things down, but based on what you’ve described, your laptop overheating is likely due to dust buildup in the vents or outdated drivers/BIOS firmware.

  • Step 1: Clean your air vents [1] using compressed air — dust can clog vents and reduce airflow, making the fan work harder and heat build up.
  • Step 2: Update your BIOS and GPU drivers [2] — this often improves fan behavior and thermal management.

Let me know if you want help with either step!

References:
[1] - https://support.microsoft.com/laptop-overheating | Microsoft Support

[2] - https://www.dell.com/support/kb/article/BIOS-and-thermal | Dell Knowledge Base

✅ Inline citations match source URLs

✅ Suggestions trace back to verbatim quotes

✅ Clear, friendly explanation while staying citation-accurate

QUESTION

Prompting alone can lead to hallucinated or misplaced citations. Even if I instruct the supervisor model to tag citations explicitly, it’s not always reliable. So how do I architect my agentic setup to guarantee 100% citation accuracy? I want to make sure that:

  1. The citations point to actual, verifiable text spans from the source documents provided by researcher tool.
  2. The supervisor agent can still learn from the research, so it can assist the user meaningfully, without distorting or guessing.

What’s the best way to enforce that architecturally/structurally, not just through prompts?

Top comments (0)