KristinZ

Posted on Nov 5

Does Graph RAG Actually Help? I Measured It on Postgres and TypeScript

#ai #webdev #rag #graphrag

Graph RAG is usually described with one claim: vector search misses answers that sit behind a chain of facts, and a knowledge graph fixes that.

That is plausible. It is also expensive to adopt, because every document goes through an LLM before it is searchable. So I built a small version on Postgres, ran it against a hybrid search baseline on 100 multi-hop questions, and counted what it cost.

The short result: graph retrieval helped, mostly when combined with hybrid search, and the one-time indexing bill was about twelve times the token cost of the whole hybrid evaluation. The details matter more than the headline, so this article walks through the setup, the numbers with their uncertainty, the questions where the graph made things worse, and what the experiment cannot tell you.

All code and data are in aiengineering-book/graph-rag-experiment. You can skip the expensive build step and reproduce the results with a 0.8 MB file.

What Was Tested

Four systems answer the same questions with the same answer prompt, so the only difference between them is which passages the model sees:

System	What it retrieves
Closed-book	Nothing. The model answers from what it already knows (a control).
Hybrid	Vector search and Postgres full-text search (`ts_rank`), merged by reciprocal rank fusion. Top 5 passages. No reranker. A BM25 version is tested later.
Graph	Passages found by walking a knowledge graph from the entities in the question. Top 5 passages.
Graph + hybrid	The two ranked lists merged by reciprocal rank fusion. Top 5 passages.

The questions come from MuSiQue (CC BY 4.0), a benchmark of questions that need two to four reasoning steps across different passages. For example:

Where does Ulysses Kay's birth city hold NASCAR races?

Answering it takes two passages: one that says Ulysses Kay was born in Tucson, and one that says what races Tucson hosts. The question names only the first.

I sampled 100 questions from the dev set with a fixed seed. Each question ships with 20 passages (19 for one of them), a few relevant and the rest distractors. I pooled all of them into one corpus of 1,847 passages, so every system searches the same collection, as it would in a real deployment.

The stack:

Postgres 16 with pgvector. Vectors, full-text search and the graph all live here. There is no graph database.
nomic-embed-text, running locally through Ollama, for all embeddings (768 dimensions).
DeepSeek (deepseek-flash, thinking disabled) for graph extraction, for finding entities in a question, and for answering. Any OpenAI-compatible chat API would work for the same roles.

Scoring is exact match and token F1 against the gold answer and its aliases, plus the share of each question's supporting passages that a system actually retrieved. Exact match is strict ("The Mother (Tracy McConnell)" does not match "Tracy McConnell"), so F1 is reported too.

How the Graph Is Built and Searched

Indexing adds one step to what a hybrid index does. An LLM reads each passage and returns the entities in it and the relations between them as JSON:

{
  "entities": ["Ulysses Kay", "Tucson, Arizona"],
  "relations": [
    { "source": "Ulysses Kay", "relation": "born in", "target": "Tucson, Arizona" }
  ]
}

The results go into three tables next to the passages. Entities are merged by normalized name, and each relation remembers which passage it came from:

CREATE TABLE entities  (id serial PRIMARY KEY, name text, key text UNIQUE, embedding vector(768));
CREATE TABLE mentions  (entity_id int REFERENCES entities, chunk_id int REFERENCES chunks,
                        PRIMARY KEY (entity_id, chunk_id));
CREATE TABLE relations (id serial PRIMARY KEY, src int REFERENCES entities,
                        dst int REFERENCES entities, label text, chunk_id int REFERENCES chunks);

Searching the graph (what GraphRAG calls local search) has four steps:

Ask the LLM for the named entities in the question.
Embed each name and find the nearest entity in the entities table. These are the seeds.
Walk the relations table outward from the seeds for two hops.
Score each passage by the entities it mentions, weighting each by 1 / (1 + hops from a seed), and keep the top 5.

For the Ulysses Kay question, the seed is "Ulysses Kay". One hop reaches "Tucson, Arizona", so the passage about Tucson is retrieved even though the question never mentions it.

This is a deliberately small graph. It has no community detection and no community summaries, which Microsoft's GraphRAG uses for questions about a whole corpus. That matters for what the results can and cannot say, and it is covered at the end.

Results

System	Exact match (95% interval)	F1	Supporting passages retrieved	Exact match vs hybrid, same questions
Closed-book	10%	0.19	n/a	n/a
Hybrid	28% [19, 37]	0.42	51%	baseline
Graph	35% [26, 45]	0.48	61%	+7 points [-3, 17]; right where hybrid was wrong 18 times, the reverse 11
Graph + hybrid	42% [33, 52]	0.53	68%	+14 points [5, 24]; 20 times against 6

The intervals are 95% bootstrap intervals over the 100 questions. The last column compares systems on the same questions, which is more informative than comparing two percentages.

What the table supports:

Retrieval helps at all. Without it, the model answers 10% of these questions. Hybrid search triples that.
Graph + hybrid beat hybrid. The interval for the difference, [5, 24] points, excludes zero.
Graph alone did not clearly beat hybrid. Its interval, [-3, 17], includes zero. Seven points is the best estimate, but 100 questions cannot separate it from noise.
Hybrid misses half the evidence. It retrieved 51% of the supporting passages. That gap is the room a graph has to work with.

Where the Graph Helped

The 18 questions the graph answered and hybrid did not share a shape: the answer sits behind an entity the question does not name.

For the Ulysses Kay question, hybrid search matched the words "NASCAR races" and returned passages about Richmond, Virginia, a NASCAR TV show and two NASCAR drivers, then answered "Richmond International Raceway". The graph started from "Ulysses Kay", followed the birthplace relation to Tucson, and answered "Tucson Raceway Park".

The same pattern appeared for "Who is the child of the actor who is a cast member of Love in Pawn?", for "In which county was James Finch born?", and for "Who was the spouse of the leading lady in Gone With the Wind?". In each case the second passage is reachable by following a relation from a named entity, and not by similarity to the question's words.

Where It Hurt

The graph answered 11 questions worse than hybrid (two of them are the same question, which appears twice in the sample). Two causes showed up.

The question has no entity to start from. "What was the percentage that the Dow Jones fell when the US had the highest unemployment rate?" contains no usable named entity, so the graph search had no seeds, returned unrelated passages, and the model answered "I don't know". Hybrid found it from the words alone.

The seed is a hub. For "In what part of Florida is Tom Denney's birthplace located?", the question names "Florida" as well as "Tom Denney". "Florida" has 41 neighbors in the graph, and walking out from it pulled in passages about other Florida cities, which crowded out the correct one.

The second cause suggested a fix: do not walk outward through entities with many neighbors. The median entity in this graph has one neighbor and the 99th percentile has 13, so I set the cutoff at 20, which catches 43 hub entities including "Florida". That fixed the Tom Denney question, and it raised graph + hybrid from 42% to 48%.

There is a catch, and it affects how much to trust that number. I chose the cutoff after reading the failures in this same sample of 100 questions. Even with the paired comparison, a gain from a setting tuned on the test set is optimistic, and its interval ([0, 12] points over graph + hybrid without the cap) barely clears zero. The Tom Denney question also appears twice in the sample under different ids, so that fix is counted twice. Treat the hub cap as a hypothesis to confirm on fresh questions, not as a result.

What Else Changed the Outcome

Each of these reuses the same graph, so there is no rebuild:

Variation	Exact match	Supporting passages retrieved
Graph + hybrid, 2 hops (default)	42%	68%
Graph + hybrid, 1 hop	36%	64%
Hybrid, 8 passages instead of 5	33%	60%
Graph + hybrid, 8 passages	48%	76%
Hybrid, BM25 for the keyword half	27%	59%
Graph + hybrid, BM25 for the keyword half	38%	68%

The second hop matters. With one hop, graph + hybrid falls to 36%, and its advantage over hybrid no longer clears zero. Most of the benefit comes from reaching passages two steps away.
A stronger keyword search did not rescue the baseline. The keyword half of the hybrid search above is Postgres's built-in full-text ranking, which is not BM25. Swapping in BM25 (same parameters as the one in the book: k1 = 1.5, b = 0.75) raised the share of supporting passages hybrid retrieved from 51% to 59%, but exact match stayed at 27%. Graph + hybrid with BM25 scored 38%, which is 11 points above hybrid with BM25 [3, 20], winning 16 questions to 5. That is a little smaller than the 14 points against the original baseline, and the interval still excludes zero. My guess for why the better retrieval did not help: an answer needs every hop's passage, and BM25 mostly found more of the first-hop ones.
The gain is not just more context. Giving hybrid search 8 passages lifts it from 28% to 33%. Graph + hybrid with 8 passages reaches 48%, ahead of hybrid with the same 8 passages by 15 points [7, 23], on 17 questions against 2. That extra context costs about 50% more prompt tokens (121,000 against 80,000 per 100 questions).

What It Costs

Item	Cost
Building the graph, once	1,827 LLM calls, about 964,000 tokens (450,000 prompt, 514,000 completion), about 14 minutes at 4 concurrent requests
Evaluating hybrid on 100 questions	about 80,000 prompt tokens
Evaluating graph + hybrid on 100 questions	about 82,000 prompt tokens, plus one extra LLM call per question to find the question's entities
Updating one edited passage, graph	about 255 prompt and 306 completion tokens, about 2 seconds
Updating one edited passage, hybrid index	about 70 to 90 ms, no LLM call

The build is about twelve times the token cost of the entire hybrid evaluation, and nearly all of the experiment's spend. At query time the graph adds one short LLM call and about the same prompt size as hybrid, so serving cost is modest. The cost lives in indexing.

Two costs are not visible in those numbers:

Extraction fails sometimes. 57 of 1,847 passages (3%) returned JSON that did not parse, so they have no graph entries at all.
Entity merging is crude. Entities merge by normalized name, so "Future Ted" and "Future Ted Mosby" ended up as separate entities. A split like that can break a path. Better entity resolution would cost more.

The update figure applies to this small graph, where re-extracting one passage is enough. A system that also keeps community summaries has to regenerate those after changes, which this experiment did not measure.

What This Does Not Show

One run on 100 questions, with one model. The intervals are wide. A different sample, or a different answering and extraction model, could move the numbers by several points. Stronger models might narrow the gap by making hybrid search better at using the passages it finds, or widen it by extracting a better graph. I did not test either.
Only multi-hop questions. The benchmark has no single-fact questions, so this says nothing about whether the graph helps or hurts them. It also has no whole-corpus questions of the "which risks recur across all these contracts" kind.
No community summaries. Those are what GraphRAG's global search relies on. This experiment tests the local half of graph retrieval only. Whether a graph helps on whole-corpus questions needs a different system and a different corpus.
No reranker. I tested BM25 against the built-in full-text ranking, but not a reranking step (a cross-encoder that rescores the top candidates). It could lift the baseline further. It only reorders passages that were already retrieved, so I would expect it to help less with passages found through a bridge entity, but I did not test that.
No comparison with multi-step retrieval. An agent that rewrites its query after each retrieval round can also follow a chain of facts, with no extra index. I did not compare against it.
Possible training-data overlap. The model answers 10% of the questions with no retrieval at all, which is modest but not zero. The control run is there so you can see it.

When It Is Worth Trying

The experiment points to a narrow case. A graph is worth evaluating when:

a large share of your questions need facts from several documents joined through an entity the question does not name;
your documents describe entities and relations that stay fairly stable, because every change means re-extraction;
you can afford an indexing bill that scales with corpus size and can measure the gain against your current hybrid search first.

If most of your questions are single-fact lookups, hybrid search with reranking is the cheaper path, and this experiment gives you no reason to change it. And before building a graph, measure how many of your questions are multi-hop at all. If that share is small, the cost is unlikely to pay back.

Reproduce It

Clone the repo and start Postgres with pgvector (docker compose up -d). Install Ollama and run ollama pull nomic-embed-text. Then:

npm install
cp .env.example .env          # add an OpenAI-compatible API key for answering
npm run import-graph          # loads the finished graph, about 2 minutes, no LLM calls
npm run eval -- --system hybrid
npm run eval -- --system graph+hybrid
npm run eval -- --system hybrid --keyword bm25    # the BM25 variants
npm run report -- results/hybrid-100.json results/graph-hybrid-100.json

import-graph copies the graph exactly but recomputes the embeddings locally. Embeddings come out slightly different on each import, so on the first 30 questions a few near-tied searches returned a different top 5, and exact match moved by two questions. Expect your numbers within a few points of the tables above, not identical to them.

To try a different answering model, the repo also ships the exact prompts each system sent (prompts/*.jsonl, retrieval already done). npm run run-prompts sends them to any OpenAI-compatible API, and npm run score-answers scores the result, with no database or Ollama needed. I only tested DeepSeek, so another provider may need small client changes.

To rebuild the graph yourself, download MuSiQue and draw the sample with scripts/sample-musique.py (the steps are in data/README.md), then run npm run ingest and npm run build-graph in place of import-graph. The build is the 964,000-token step.

DEV Community