Graph as Architecture, Not Graph as Data: Why I Stopped Reaching for GraphRAG First

I assumed GraphRAG was the smarter option until I actually checked the numbers behind that assumption

I need to confess something before I get into any code. For most of this year, if you’d asked me which retrieval approach I’d reach for on a new agent project, I would have said GraphRAG without much hesitation. Not because I’d benchmarked it against plain vector search on my own workloads, but because it felt more advanced. A knowledge graph sounds like the grown-up version of a vector store, entities and relationships and structured reasoning over connections instead of blind similarity search, the way a normalized relational schema feels more correct than a spreadsheet even when the spreadsheet does the job fine.

I’d read pieces like Graph Engineering with Claude and the walkthrough on turning millions of documents into an agentic knowledge graph, and the framing in both, reasonably, since that’s what they were about, was that graphs were the more capable substrate for an agent’s memory layer. I filed that away as a general truth about retrieval rather than a claim scoped to specific use cases, and I started every project’s retrieval design by asking “do I need a graph database” instead of “do I need retrieval beyond similarity search at all, and if so, what kind.”

Then I read two things back to back that made me actually go check my assumption instead of repeating it.

The first was an evaluation paper, “How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG” (arXiv:2506.06331), which did something that should have been done from the start of the GraphRAG hype cycle: it audited the LLM-as-judge evaluations that a lot of the original win-rate claims were built on. LLM judges, it turns out, carry the same biases anyone who has run a pairwise eval has probably noticed and not fully accounted for. Position bias, where swapping which answer appears first in the prompt can swing the reported win rate by more than 30 points on its own. Length bias, where the longer, more elaborated answer wins on style rather than substance. Trial bias, where the same exact comparison, re-run, disagrees with itself. After correcting for these, one popular method’s reported 66.7 percent win rate over baseline RAG fell to about 39 percent, which is below the 50 percent line where you’d expect a coin flip to land. That’s not “the gains were smaller than reported.” That’s “the gains, once you strip out judge bias, may not exist at all for that method on that evaluation.”

The second was GraphRAG-Bench (arXiv:2506.02404, accepted at ICLR 2026), a benchmark built specifically to stop relying on LLM-judge win rates and instead test GraphRAG methods against plain text-chunk retrieval on graded, verifiable questions across sixteen disciplines. And the result that stopped me was the simple fact retrieval category, the “what is X” lookup questions that make up a large share of what production RAG systems actually get asked. On that category, plain text chunks scored 60.9 and graph-based retrieval scored 60.1. Effectively a tie, with graph construction, storage, and query complexity added on top for no measurable benefit on that question type.

GraphRAG-Bench results by question category (approximate, from published results)
+----------------------------+---------------+---------------+------------+
| Question category | Text chunks | Graph retrieval| Winner |
+----------------------------+---------------+---------------+------------+
| Simple fact retrieval | 60.9 | 60.1 | Tie |
| Complex reasoning | 42.9 | 53.4 | Graph +10.5|
| Contextual summarization | 51.3 | 64.4 | Graph +13.1|
+----------------------------+---------------+---------------+------------+

I want to be careful about what this does and doesn’t prove, because overcorrecting is exactly as lazy as the original overcorrection toward graphs. Neither finding is the final word. Benchmarks in this space have a shelf life measured in months, GraphRAG-Bench tests nine specific methods against one chunking baseline, and a method that loses on this benchmark’s simple-lookup category could still be the right call for a corpus this benchmark doesn’t resemble. The bias audit doesn’t say GraphRAG never helps, it says a specific evaluation methodology was inflating specific claims, which is narrower and more useful than “graphs don’t work.” What both papers do establish, and what actually changed how I build things, is that graph-based retrieval is not a categorical upgrade over vector search. It wins clearly on complex reasoning and summarization tasks that require synthesizing across multiple pieces of context. It doesn’t reliably beat plain retrieval on the simple lookups that dominate a lot of real usage. That’s a more boring, more useful fact than “graphs are the advanced option,” and boring facts are the ones you can build a decision process around.

But here’s the thing that took me embarrassingly long to notice while I was busy re-litigating GraphRAG: the word “graph” was doing two completely unrelated jobs in my head, and conflating them is most of why I’d been thinking about this wrong in the first place.

Two things called “graph,” doing nothing alike

Graph as data is what GraphRAG means: a knowledge graph, entities and relationships extracted from your corpus, stored as nodes and edges, used as the substrate you retrieve from instead of, or alongside, chunked text and embeddings. It’s an answer to the question “where does the agent’s information live and how does it find the relevant piece.”

Graph as architecture is something else entirely, and it’s what tools like LangGraph mean when they use the word. It’s a state machine describing how control flows through your agent: which step runs next, under what condition, whether execution pauses for a human, and how the whole thing survives a crash and picks back up where it left off. It’s an answer to the question “how does the agent’s own execution move from step to step,” which has nothing to do with where facts are stored.

The two meanings of "graph" in agent engineering
+------------------+---------------------------------+---------------------------------+
| Aspect | Graph as architecture | Graph as data |
+------------------+---------------------------------+---------------------------------+
| Answers | How does control flow move | Where do facts live and how |
| | between agent steps | do I retrieve the right ones |
+------------------+---------------------------------+---------------------------------+
| Nodes represent | Units of work: classify, fetch, | Entities: people, orders, |
| | decide, respond | products, concepts |
+------------------+---------------------------------+---------------------------------+
| Edges represent | Transitions and conditions | Relationships: "works for", |
| | between steps | "purchased", "caused by" |
+------------------+---------------------------------+---------------------------------+
| Typical tool | LangGraph, a hand-rolled state | Neo4j, a Cosmos DB graph API, |
| | machine, an orchestration library | a GraphRAG pipeline |
+------------------+---------------------------------+---------------------------------+
| Cost to adopt | Low. A dependency and some | High. Extraction pipeline, |
| | Python classes | graph store, query layer |
+------------------+---------------------------------+---------------------------------+
| When it's wrong | Almost never, once you have | When your questions are simple |
| to skip it | more than a couple of steps | lookups a vector or keyword |
| | and any branching | search already answers |
+------------------+---------------------------------+---------------------------------+

Once I separated these in my head, the shape of a sane decision process became obvious. Graph as architecture is cheap and useful almost immediately for any agent with more than one or two steps, so there’s little reason not to reach for it early. Graph as data is expensive, and its payoff depends entirely on whether your questions need relationship traversal, which you can check before you build anything. I’d been treating the expensive, conditional thing as the default starting point, and the cheap, near-universal thing as an afterthought. That’s backwards. Here’s what building it the right way around looks like.

Building the architecture: a real LangGraph agent with checkpointing and a human interrupt

I’m going to build a small support-triage agent. It classifies what a user is asking for, fetches relevant account data, decides whether the request is risky enough to need a human’s sign-off before it proceeds, and then responds. Refund requests need a human. Balance and order-status lookups don’t. This is a deliberately small example, but every mechanism in it, the state schema, the conditional routing, the checkpointing, the interrupt, is the real mechanism you’d use in a production agent, just with more nodes and more realistic tool calls.

Install the dependencies first:

pip install langgraph langgraph-checkpoint-sqlite

Start with the state schema. This is the shared data structure every node reads from and writes into as execution moves through the graph:

from typing import TypedDict, Literal
class AgentState(TypedDict):
    user_message: str
    intent: str
    risk_level: str
    account_data: dict
    human_decision: str
    final_response: str

Now the four nodes. classify_intent looks at the incoming message and tags it with an intent and a risk level:

def classify_intent(state: AgentState) -> dict:
    message = state["user_message"].lower()
    if "refund" in message or "cancel" in message:
        intent, risk = "refund_request", "high"
    elif "balance" in message or "order status" in message:
        intent, risk = "account_lookup", "low"
    else:
        intent, risk = "general_question", "low"
    return {"intent": intent, "risk_level": risk}

fetch_data stands in for whatever your real system talks to, a CRM, an order database, a billing API:

def fetch_data(state: AgentState) -> dict:
    fake_db = {
        "refund_request": {"order_id": "ORD-4471", "amount": 249.00, "status": "delivered"},
        "account_lookup": {"balance": 1200.50, "last_order": "ORD-4402"},
        "general_question": {},
    }
    return {"account_data": fake_db[state["intent"]]}

The routing function decides whether the graph needs to detour through a human review step. This isn’t a node itself, it’s the conditional edge that picks the next node:

def needs_human(state: AgentState) -> Literal["human_review", "respond"]:
    return "human_review" if state["risk_level"] == "high" else "respond"

human_review is where the interrupt actually happens. Calling interrupt() inside a node pauses the whole graph right there, mid-execution, and returns control to whatever called graph.invoke, without losing any state:

from langgraph.types import interrupt
def human_review(state: AgentState) -> dict:
    decision = interrupt({
        "question": f"Approve {state['intent']} for order {state['account_data'].get('order_id')}?",
        "amount": state["account_data"].get("amount"),
    })
    return {"human_decision": decision}

And respond produces the final answer, taking the human's decision into account if one was needed:

def respond(state: AgentState) -> dict:
    if state["risk_level"] == "high":
        if state.get("human_decision") == "approve":
            text = f"Refund approved for {state['account_data']['order_id']}."
        else:
            text = "This request needs manual follow-up before we can proceed."
    else:
        text = f"Here's what I found: {state['account_data']}"
    return {"final_response": text}

Now wire the graph together. This is the part that’s actually the point of this whole exercise, an explicit, readable description of control flow instead of a pile of nested if-statements inside one giant function:

from langgraph.graph import StateGraph, START, END
builder = StateGraph(AgentState)
builder.add_node("classify_intent", classify_intent)
builder.add_node("fetch_data", fetch_data)
builder.add_node("human_review", human_review)
builder.add_node("respond", respond)
builder.add_edge(START, "classify_intent")
builder.add_edge("classify_intent", "fetch_data")
builder.add_conditional_edges(
    "fetch_data",
    needs_human,
    {"human_review": "human_review", "respond": "respond"},
)
builder.add_edge("human_review", "respond")
builder.add_edge("respond", END)

The last piece is checkpointing, and this is the part I actually cared about most when I built this the first time, because a graph that can’t survive a restart isn’t meaningfully different from a script. LangGraph ships a SQLite-backed checkpointer that’s genuinely fine for a single-process deployment or local development, no external database required:

from langgraph.checkpoint.sqlite import SqliteSaver
with SqliteSaver.from_conn_string("support_agent_checkpoints.sqlite") as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)
    config = {"configurable": {"thread_id": "ticket-8821"}}
    result = graph.invoke({"user_message": "I want a refund for my last order"}, config)
    print(result)

Run that and the graph walks through classify_intent and fetch_data, hits the conditional edge, routes into human_review, and stops there, because interrupt() halted execution. result at this point contains an __interrupt__ entry describing what the graph is waiting on, not a final_response, since the graph genuinely didn't finish. This is the moment that matters: the process can crash here, the machine can reboot, and because every step's state was written to support_agent_checkpoints.sqlite as it happened, nothing is lost. When the process comes back up, you reopen the same checkpoint file and the same thread ID, and the graph is exactly where it left off:

with SqliteSaver.from_conn_string("support_agent_checkpoints.sqlite") as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)
    config = {"configurable": {"thread_id": "ticket-8821"}}
snapshot = graph.get_state(config)
    print(snapshot.next) # ('human_review',)
    from langgraph.types import Command
    final = graph.invoke(Command(resume="approve"), config)
    print(final["final_response"])

Command(resume="approve") is what feeds a value back into the paused interrupt() call, exactly as if a human had clicked an approve button in whatever review interface sits in front of this. Everything before the interrupt, the classification, the fetched data, didn't get recomputed. It was sitting in the checkpoint the whole time.

The first time I built something like this, I skipped the checkpointer entirely and just kept state in a Python dict in memory, reasoning that I’d add persistence “later.” Then I restarted the process during testing while a request was sitting mid-review, and the entire in-flight ticket vanished, no error, no trace, just gone, because nothing outside the process’s memory ever knew it existed. That’s the failure mode checkpointing exists to prevent, and it’s the reason I now treat it as part of the graph, not an optional add-on to the graph.

When graph as data actually earns its complexity

None of this means GraphRAG is a bad idea. It means it’s the wrong first idea, and it should show up once you have a concrete need, not because it sounds more sophisticated. The pattern in the benchmark results above, ties on simple lookup, real wins on complex reasoning and summarization, points at exactly what that concrete need looks like: questions that require traversing relationships between entities, not just finding the passage that’s semantically closest to the query.

Three question types where I’d actually reach for a knowledge graph now:

Multi-hop relationship questions, where the answer requires chaining across more than one connection. “Which suppliers does the manufacturer that made this recalled part also supply for other product lines” isn’t answerable by finding the single most relevant chunk of text, it requires walking from the part, to the manufacturer, to the manufacturer’s other supply relationships. A vector store returns the passage most similar to your query; it has no mechanism for chaining hops.

Questions where the relationships are the answer, not supporting context. “Who on the team has worked with both the payments service and the fraud-detection service” is fundamentally a graph traversal, person-connects-to-service-connects-to-person, and forcing it through similarity search means hoping some document happens to state the intersection directly, which it usually doesn’t, because the fact you want was never written down as a sentence, it only exists as the shape of the graph itself.

Root-cause and impact-chain questions in operational systems: “what upstream change could explain this cascade of failures across these three services” is a question about causal and dependency edges, not about which log line reads most similarly to “cascade of failures.” This is the context-graphs-for-agent-memory case I’ve seen argued well elsewhere, and it’s the one place I think that framing is exactly right: for genuinely relational operational knowledge, a graph isn’t a nice-to-have representation, it’s the only representation that actually contains the answer.

If you don’t have questions shaped like these, and a decent chunk of production RAG traffic doesn’t, GraphRAG is complexity with no corresponding payoff, per the benchmark numbers above.

The token-cost trap: don’t stack graph-as-data on an already-expensive setup

There’s a cost dimension to this decision that I think gets underweighted, and it compounds badly with a mistake a lot of teams are separately making with multi-agent architectures. Anthropic’s own writeup on building their multi-agent research system found that multi-agent setups use roughly 15 times more tokens than a single chat interaction, and single agents with tool use already run about 4 times more tokens than plain chat. More strikingly, when they analyzed what actually explained performance variance across runs, token usage by itself accounted for 80 percent of it, with the number of tool calls and model choice as the other two factors, all three together explaining 95 percent. Performance differences between runs, in other words, were mostly explained by how many tokens got burned, not by how “smart” the orchestration was.

That’s a reason for caution around multi-agent designs generally. But it becomes a specific, practical warning the moment you’re considering adding graph-as-data retrieval on top of a system already running multiple agents: a GraphRAG pipeline adds its own token overhead on both ends, extraction and graph construction up front, and more elaborate context assembly at query time as multi-hop traversal results get formatted back into a prompt. Stack that on a multi-agent setup already burning 15 times the tokens of a simple call, without measuring the delta, and you can end up with cost scaled by an order of magnitude while quality on your actual question mix hasn’t moved, because most of your traffic was simple lookups graph retrieval doesn’t help with anyway.

The fix isn’t complicated, it’s just a step people skip because it’s less fun than building the graph: measure the token delta of adding graph-as-data retrieval, on your actual question distribution, against your actual multi-agent baseline, before committing to it. If your traffic is mostly the simple-lookup shape where GraphRAG ties plain retrieval, you’ve just paid extraction and query-time overhead for nothing. If a meaningful share is genuinely multi-hop, you’ll see it in the measurement, and now you have a number to justify the added cost instead of a hunch.

A staged way to actually adopt this

Here’s the order I’d tell someone starting fresh today to build in, because it’s the order I wish I’d started with instead of the order I actually used.

Start with graph as architecture for control flow, immediately, regardless of whether you think you’ll ever touch a knowledge graph. It’s cheap, it’s a dependency and some state classes, and it pays for itself the moment your agent has more than a couple of steps or any branching logic, which is nearly every agent worth building. Checkpointing and interrupts aren’t advanced features to bolt on once you’re in production, they’re what makes an agent resilient to the crashes and the human-review pauses that happen constantly in real deployments, and retrofitting them onto a system that wasn’t built as an explicit state machine is far more painful than building it that way from the start.

Only add graph as data once you have concrete multi-hop questions that you’ve verified, not assumed, vector search actually fails on. That verification step matters more than it sounds like it should: run your actual candidate questions against your existing retrieval, look at where it genuinely falls short, and check whether the shortfall is a relationship-traversal problem or something else entirely, like bad chunking or a missing document. A lot of “our RAG isn’t finding the answer” problems turn out to be the second thing, and a knowledge graph doesn’t fix bad chunking.

And always, before you commit to standing up a graph database, benchmark against plain RAG on your own data and your own questions, using the same kind of graded, verifiable comparison GraphRAG-Bench uses rather than an LLM-as-judge pairwise win rate, given what the bias audit found about how unreliable that evaluation style can be. If a knowledge graph doesn’t clearly beat a well-tuned vector or keyword baseline on the questions you actually care about, you haven’t found a case for graph as data yet, whatever the architecture sounds like it should be capable of on paper.

I still use graphs constantly. I just stopped assuming which kind I meant before I’d checked what the problem in front of me actually needed.

Tags: langgraph, graphrag, agentic-ai, rag, llm-engineering