Plain RAG chops your documents into chunks, embeds them, and at query time returns the handful whose vectors sit nearest the question. That's great for "look up one fact" and useless for "how is A related to C?" — because that answer isn't in any single chunk. It's spread across many, none of which is individually close to the question. GraphRAG (Microsoft Research, 2024) fixes this by retrieving over a knowledge graph instead of a pile of text. Here's how it works.
Flat RAG returns nearest chunks — and misses the joins
Nearest-neighbour retrieval has no notion of connection. Ask a multi-hop question and each hop lives in a different chunk; the chunk that names the bridge between A and C may share almost no words with your query, so it never gets retrieved. The connective tissue simply isn't in the index — you can only ever get back the passages that already resemble the question, and a chain of relationships doesn't resemble anything you can type.
The indexing step: entities become nodes, relations become edges
GraphRAG adds an indexing pass before any query. An LLM reads every chunk and extracts (entity)–[relation]→(entity) triples — people, orgs, models as nodes, relationships as edges — and merges repeated mentions of the same thing into one node. That merge is the whole point: the same entity named across ten documents becomes one node with ten edges, so it builds the links across documents that flat RAG never has. The extraction is a one-time cost you pay at index time, not per query.
A useful way to picture the output: each chunk yields rows like (GraphRAG)–[introduced by]→(Microsoft) and (Microsoft)–[builds]→(Azure). Ten chunks that each mention Microsoft don't stay ten disconnected facts — they collapse onto one Microsoft node, and suddenly a path exists from GraphRAG to Azure that no single sentence ever wrote down.
Multi-hop = walk the graph
Now a "how is A related to C?" question becomes a graph traversal. Start at the seed entity, walk edge by edge, and surface the connecting path — A→B→C — even when no single chunk ever mentioned A and C in the same breath. The answer is reconstructed from the relationships along the path, not fetched from one lucky chunk. This is the "local" query mode, and it's where the graph earns its keep: five hops of reasoning that flat retrieval structurally cannot assemble.
Community summaries for the global questions
For big-picture questions — "what are the main themes across all of this?" — traversal isn't enough either, so the indexer clusters densely-linked nodes into communities and has the LLM write a short summary of each. A "global" query then map-reduces those community summaries into an answer about the whole corpus. No fixed-size chunk retrieval can do that, because the answer is about the shape of everything, not any single passage — there's no chunk to be "near".
vs flat vector RAG
The trade is cost and freshness: GraphRAG pays an upfront LLM pass to extract, merge and summarize the graph, where flat RAG just embeds and stores, and re-indexing when documents change is heavier. In return it answers the two question types flat RAG fumbles — multi-hop connections and global synthesis — while still handling simple local lookups. It's not really a replacement for vector RAG so much as the layer that adds the joins on top of it; many systems keep both and route each question to whichever fits — a simple lookup stays on flat retrieval, a connect-the-dots or big-picture question goes to the graph.
The one thing to remember
Flat RAG retrieves passages that resemble the question. GraphRAG retrieves relationships between the things the question mentions. That's the whole distinction, and it's why the demo can show a five-hop answer lighting up as a path across the graph while flat RAG, on the identical corpus, can only return the nearest chunk and shrug at the parts that aren't in it.
Watch a graph get built from six documents, then run a 5-hop query that lights up a path flat RAG can't see:
Top comments (0)