One finding, and it is not about graphs. It is about how the result you are quoting was produced.
For breadth-oriented sensemaking questions, GraphRAG community summaries reached 72 to 83 percent comprehensiveness and 62 to 82 percent diversity win rates, and root-level summaries cut answer tokens by about 97 percent (arXiv:2404.16130). Those are the numbers the category quotes.
They were produced by an LLM judge, with no gold answers.
Score the same kind of comparison against ground truth and it turns over. On ROUGE-2 against gold answers, GraphRAG loses to plain RAG: 6.99 against 10.08 on SQuALITY, and 3.23 against 6.32 on QMSum. The same paper reports that summary-ordering position bias can flip the LLM judge's preference on its own (arXiv:2502.11371).
Both results are true
That is the part worth sitting with. Neither number is wrong and neither paper is sloppy.
One measures which answer a model prefers. The other measures overlap with a gold answer. They are different questions, so they are allowed to have different answers, and a comparison that looks decisive is often just one of the two questions asked loudly.
The same seam runs through the retrieval verdicts. On GraphRAG-Bench, "RAG (w rerank)" beat "GraphRAG (local)" 60.92 to 49.29 on fact retrieval, while "HippoRAG2" beat "RAG (w rerank)" 53.38 to 42.93 on complex reasoning (arXiv:2506.05690). One table, two verdicts, depending on which row of it you were looking for.
Cost has the same problem
"Graph equals expensive" is too crude to act on. On the Novel split of the same benchmark, query cost spans more than two orders of magnitude within the graph methods themselves: about 331,375 total query tokens for "MS-GraphRAG(global)", about 100,832 for LightRAG, about 4,204 for Fast-GraphRAG, about 1,008 for "HippoRAG2", against about 879 for vanilla RAG.
Index build shows the same spread: roughly 9.2 million tokens for "HippoRAG2" against roughly 115.5 million for GraphRAG, about twelvefold (arXiv:2502.14802, Table 12).
Those two figures come from different primaries with different method sets, and no single primary puts index-build cost and query cost for the same method on one axis. Which is itself the lesson: the tidy cost model you want does not exist yet, and assembling one from two papers gives you a useful heuristic, not a controlled comparison.
The question that survives
Before accepting any retrieval win, mine included:
Was this judged by a model, or scored against ground truth?
It takes ten seconds to check and it changes what the number means. A model-judged win says an evaluator preferred that answer. A ground-truth score says the answer overlapped with a known correct one. Quoting the first while implying the second is how a benchmark verdict gets mistaken for a decision framework.
Disclosure: I work on Mnemoverse, a memory engine for AI agents. Every number above comes from a linked paper, none of them ours. The longer version, with the full decision matrix, is on the original page.
Top comments (24)
The third question behind yours is which layer the number is about.
Both your instruments score the emitted answer. A retrieval metric scores the evidence. Run both on one run and the gap locates the loss.
My ATM-Bench row: Recall@10 92.89, Quality Score 68.43. I decomposed the 31.57 lost points against the evidence each question actually received. 204 of 1,013 questions, worth 19.90 points, had at least 80% of the gold answer's content tokens inside the model's own prompt. That is 63% of the entire loss, occurring after retrieval had already succeeded: 87 refusals, 117 wrong answers.
For a GraphRAG versus RAG table: if graph context changes how willing a model is to commit, and a community summary and a raw chunk have no reason to produce equal willingness, part of the measured graph win is a reader effect wearing a retriever's name. Neither instrument separates them, because both score a refusal and a wrong answer as the same zero.
Two caveats. The 80% token overlap is a lexical proxy for "answerable from what was on screen", and I have measured it wrong in both directions. And the refusals are not gateable on the retrieval score: P(a correct refusal scoring below a wrong one) is 0.493, medians 0.5192 against 0.5229.
Disclosure: I work on RE-call, an open retrieval engine. The ATM row is mine.
Giulio, the layer split is the correction I needed, and 63 percent of the loss landing after retrieval already succeeded is the number I will be carrying around.
The part I want to push on is your refusal and wrong-answer collapse, because I think it is worse than symmetric. Take your 204. Against a gold answer, a refusal shares almost no bigrams with it, while a confident wrong answer shares the domain vocabulary, so ROUGE ranks the wrong answer above the refusal. Now score the same pair with an LLM judge on comprehensiveness and diversity, which is what the GraphRAG headline numbers were produced with. A fluent wrong answer is comprehensive and diverse. A refusal is neither. So the judge ranks the wrong answer above the refusal too, and by a much wider margin.
Both instruments agree in direction and disagree in magnitude, and they agree on the wrong side. Your 87 refusals are being scored below your 117 wrong answers by every instrument in the comparison, which means the field is systematically rewarding the more dangerous of the two failures.
On the gating result I would state it harder than you did. 0.493 is not "not gateable", it is indistinguishable from a coin flip, and your medians differ by 0.0037. The retrieval score carries essentially no information about which failure you got. That makes the layer split mandatory rather than useful: no refinement on the retrieval side recovers something the retrieval side does not contain.
And your lexical proxy caveat is the honest one. It is the problem interpretability has one layer down: a token being on screen is not the same as it being load-bearing, and overlap can only stand in for that.
Agreed on gating, and 0.493 with medians 0.0037 apart is the right way to say it. My correct-refusal arm is n=17, so it is thin evidence of nothing rather than thick.
The collapse splits on whether the rubric has a correctness term, not on judge versus gold.
ATM's judge scores correctness. My 117 wrong answers with evidence on screen average 0.0208, my wrong refusals exactly 0.0000. I also ran the conversion: 55 wrong open_end refusals pushed into commitments gave 31 commitments, 54.84% judged true, arm score 0.3091, which is exactly the 17 true ones. The 14 confident wrong answers scored what the refusals they replaced scored.
Comprehensiveness and diversity have no correctness term, so there the fluent wrong answer takes the full margin. Two of your three instruments cannot see correctness, and the third prices the safe failure against the dangerous one at 0.02.
The part I cannot get around: no instrument in common use ranks a refusal above a confident wrong answer. Best case is a tie at zero.
Nor does bolting on Garfield's axis fix it. ATM has 23 questions in 1,013 where refusing is right. I registered a criterion on the 17 in scope and it failed at 0.938 against 0.95, where one question moves the ratio by 0.0588. There is no honest place to put a threshold on a stratum that size.
Giulio, the axis correction lands, and it is a better cut than mine. Judge versus gold was the wrong split. Whether the rubric carries a correctness term at all is the split, and comprehensiveness and diversity carry none by construction, so the fluent wrong answer takes the full margin there without any judge misbehaving.
The conversion arm is the part I would put in front of people, and it says something sharper than it reads. 55 refusals scored 0.0000. The same 55 forced into commitments scored 0.3091, and the 14 confident wrong answers cost exactly what the refusals they replaced cost. That is not a scoring quirk, it is a gradient. The price of guessing is zero and the payoff is 0.3091, so anything optimising against these numbers learns to commit, and it learns it from the benchmark rather than from the data.
Your best case is a tie at zero. A tie at zero plus a free lottery ticket is a strict preference, which is why the tie is the whole problem rather than a mild version of it.
On the threshold I think your negative result is the more useful half. 23 questions in 1013, one of them moving the ratio by 0.0588, and a registered criterion that failed at 0.938 against 0.95. That says the axis cannot be bolted on. A stratum that size cannot carry a threshold no matter who writes the rubric, so this is a construction problem rather than a scoring one, and n of 17 is the evidence for that reading rather than a weakness in it.
That benchmark exists, I have run it, and it does not rescue the gradient.
MTRAG (IBM, TACL 2025) labels tasks ANSWERABLE, PARTIAL, UNANSWERABLE or CONVERSATIONAL, and a correct "I do not have that information" on an UNANSWERABLE task scores a full 1.0 on all three metrics.
I refuse correctly on 16 of 55, second of ten systems. The system above and on pair with me, end to end refuse 12.7% and 5.5% of what they cannot answer, and the best refuser in the table is last on answer score.
The arithmetic is why. 55 UNANSWERABLE against 709 ANSWERABLE, so going from my 16 to a perfect 55 is worth at most 4.6% of the task set, while a false abstention on the answerable side costs 0.82 of a point, 0.0726 against 0.8901 when it answers. I measured that trade: swapping my generator prompt for the paper's gave up about 14 correct refusals to rescue about 28 answerable tasks, and moved Task B from 6th to 3rd of ten. A correctness term is necessary and not sufficient. While the stratum where refusing is right is 6.5% of the benchmark, committing stays dominant and the rubric never enters into it.
The same run prices the layers against each other, and it converges with the 63%. MTRAG separates Task B, generation from gold contexts, from Task C, full RAG. Holding prompt and generator fixed and swapping the entire retriever moves end to end by +0.0011 with the official prompt and +0.0099 with mine. Holding contexts fixed and swapping only the prompt moves it by +0.0200 and +0.0288, and by +0.0282 on Task B where retrieval is perfect by construction. The prompt outweighs the whole retrieval stack in all four comparisons, on a benchmark built to compare retrievers. Same cause: my prompt produced 83 false abstentions on 709 answerable tasks. All those rows were recomputed by me through one codebase, so they compare with each other and not with the published leaderboard.
Since your reading is that this is a construction problem, here is what I am building against it, including the part that failed. "Agent Memory Bench" grades memory for coding agents by execution only: checkers run the produced artifact against oracles the sandbox never contained, so the endpoint is whether the tests pass and no judge is in the loop. Its first calibration run, 630 sessions over five arms, mostly measured the instrument. 7 of 11 tasks were solved by the memory-free arm every time, so those tasks carry no headroom and no arm difference can appear on them.
Which is your construction problem from the other end. Your 23 in 1,013 has no room below the ceiling, my 7 of 11 has no room above the floor, and an aggregate score reports neither
Giulio, the MTRAG row corrects me, and it corrects the strongest thing I agreed to. An instrument that scores a correct refusal at a full 1.0 exists, so the line about no instrument ranking a refusal above a confident wrong answer was too broad. What survives is narrower: the rubric can carry the term and still not decide anything.
Your arithmetic is the part I would keep. 55 against 709 means the whole refusal stratum is worth at most 4.6 points of the task set, while a false abstention on the majority side costs 0.82 each. That is a base rate, not a scoring property. Any suite where the refuse-correct stratum is small rewards committing no matter what the rubric says, because the expected loss on the majority swamps the gain available. So the fix cannot be another axis. The strata have to be reported separately and never summed.
The prompt against retriever numbers land in the same place. 0.0011 and 0.0099 for swapping the entire retriever against 0.0200, 0.0288 and 0.0282 for swapping only the prompt, on a suite built to compare retrievers, says what the aggregate can see rather than anything about retrieval.
Your closing symmetry is the general form and I would put it first rather than last. 23 in 1013 with no room below the ceiling and 7 of 11 with no room above the floor are one defect with the sign flipped: the arms cannot differ there, and the score is reported without the headroom that would say so. Publishing headroom per stratum would make both visible before anyone ran an arm.
On the 630 sessions, I would not file that under the part that failed. Measuring the instrument before trusting it is the result, and most suites never produce one.
Fair point: authority/freshness/ACL validity is not an abstention metric, and bolting it onto an aggregate benchmark will not fix class imbalance. I’d treat it as a separate policy-validity stratum with cost-sensitive outcomes: correct answer, correct abstention, unsafe answer, and false abstention, reported per slice rather than averaged. The construction must deliberately include enough unanswerable, superseded, conflicting, and permission-denied cases to estimate each rate with confidence. For enterprise use, the unsafe-answer cost is domain-specific and often asymmetric, so the benchmark should publish a risk curve across operating thresholds, not choose one universal score.
The four-outcome frame does the work an aggregate never could, because unsafe answer and false abstention are not the same failure and no single number prices both. Two additions from the measurement side. First, a published risk curve is only reusable if the quadrant counts behind it are published too, per slice, so a deployer can re-derive the curve under their own cost matrix instead of trusting the benchmark's. The moment a benchmark chooses the operating threshold it has silently chosen someone's asymmetry. Second, the curve is also the honest answer to the instrument problem this article started from: a threshold sweep states the question it answers at every point, while an averaged score answers a question nobody wrote down. And one requirement I would add to your construction list: the unanswerable, superseded, and permission-denied cases need to carry the reason they are unanswerable, because a correct abstention for the wrong reason will pass a slice metric and still fail in deployment.
The indexing cost spread gets nastier when your corpus updates weekly instead of sitting static.
With standard chunk-and-embed RAG, updating or dropping a source file is a single delete and re-insert in pgvector. The moment you rely on hierarchical community summaries, an updated source requires either re-running the clustering extraction across the neighborhood or accepting orphaned nodes in the higher-level summaries.
That maintenance tax rarely shows up in the initial benchmark papers because they treat index generation as a one-time setup cost rather than a recurring operational loop.
Reid, this is the axis I left out, and it is the one that actually bites in production.
The article treats build cost as a one-time charge and query cost as a per-request charge, which is how the papers treat them. Update cost is a third thing and no benchmark in that set measures it at all. Index generation gets reported as setup because in an evaluation harness it happens once.
The asymmetry you describe has a compact form worth stating. In a flat index the unit of update is the chunk, and the blast radius of changing one file is that file. In a hierarchical index the unit of update is the neighborhood, and the neighborhood is defined by the clustering, which is itself a function of the whole corpus. So the cost of a one-file change is not a property of the change. It is a property of the index, and you cannot read it off the diff.
The orphaned-node branch is the part I would push on hardest. An orphaned summary does not fail. It answers, and it answers with content that was true before the update. That is worse than a stale chunk, because the summary is the layer a reader is most likely to trust: it reads as the considered view rather than the raw source.
So a weekly corpus does not just make the graph more expensive. It makes the graph's most authoritative layer the one most likely to be wrong, and the failure is silent.
The thread has the instrument question well covered, so here is a different one that applies to every number in the post, including the ones that support your argument.
None of them carry an interval. "6.99 against 10.08" on SQuALITY and "60.92 to 49.29" on fact retrieval are point estimates on benchmarks with a few hundred items, and a gap of that size can be inside the sampling noise depending on the item count and the per-item variance. I am not claiming these particular ones are, because the papers would have to give per-item scores to check, and mostly they do not. I am claiming that a reader cannot tell, and the tables are formatted as though they could.
There is a subtler version aimed at your own framing. "72 to 83 percent comprehensiveness" reads as a confidence interval to most people scanning quickly. It is a range across conditions, which is a different object, and a wide across-condition range with unknown within-condition variance tells you less than a single number with an interval would.
The practical version of your ten-second check, as an addition rather than a correction: after asking whether it was judged by a model or scored against ground truth, ask how many items, and whether any dispersion is reported at all. If the answer is a few hundred items and no dispersion, the verdict is a direction, not a measurement, and it should not be the thing that decides an architecture.
You are right, and the sharpest sentence in your comment is the object distinction: a range across conditions is not an interval, and formatting one to look like the other invites exactly the misread you describe. Guilty as charged on the 72 to 83: that is across-condition spread from the cited survey, and the post should have said so in the same breath. Your addition to the ten-second check survives contact with every number in the post: ask the item count, ask whether any dispersion is reported, and if the answer is a few hundred items and none, the verdict is a direction, not a measurement. I would add one cheap proxy for the common case where a paper reports neither per-item scores nor spread: run the judge twice on identical outputs and publish its agreement with itself. A judge that cannot repeat its own verdict caps how much any gap it reports can mean, and test-retest costs one more inference pass, not a new benchmark. Direction, not measurement, is going into my working vocabulary for this, with your name attached.
The biggest takeaway for me is that an evaluation score is only meaningful when you know what question the instrument was designed to answer. Two systems can genuinely produce different “winners” because one metric rewards answer preference while another rewards overlap with a reference. I’d be careful about combining those numbers into a single ranking without first understanding what each metric can and cannot observe.
You have named the discipline the post was reaching for: a score is an answer to the instrument's question, and the question is part of the result. I would go one step further on combining. The problem is not the composite itself, it is that averaging two metrics erases the fact that they disagreed, and on the run in the post the disagreement was the most informative output we had. Two instruments agreeing is weak evidence you measured something real. Two instruments disagreeing is strong evidence you measured two different things, and a single ranking number hides exactly that. If a leaderboard must combine, the honest way is to publish the per-instrument results next to the composite, so the reader can see what was collapsed.
The query set question is the one I'd add to your list. We used synthetic queries in one project and real user logs in another, and the retrieval ranking flipped completely between them. Both were reasonable eval sets, they just indexed different things. On cost, the range within graph methods surprised us more than the absolute numbers did, because Fast-GraphRAG and HippoRAG2 are both "graph" but not in remotely the same cost class.
The measurement instrument constructs the reality it purports to evaluate.
An LLM judge measures preference and rhetorical fluency. A ground-truth score (such as ROUGE) measures factual overlap. Both statements are true, yet they answer fundamentally different questions.
Rob, your split is cleaner than mine, and "rhetorical fluency" is the part I left implicit. But I would not let the ground-truth side off as easily as the phrasing suggests.
ROUGE-2 scores bigram overlap against a reference answer. That rewards answers phrased like the reference, not answers that are correct. Two responses can be equally true and score far apart because one of them happened to reuse the reference's construction. So it is not that one instrument measures preference and the other measures fact. It is that one is biased toward fluency and the other toward phrasing, and neither of them is asking whether the claim holds.
The detail that made me stop trusting the judge side specifically: the same paper reports that summary-ordering position bias can flip the LLM judge's preference on its own. Not change the margin, flip it. An instrument that is unstable under permutation of its inputs is not measuring the thing you named, whatever else it is doing.
Which leaves the uncomfortable version of your sentence. It is not only that the instrument constructs the reality it evaluates. It is that in this field we usually have two instruments, they disagree, and the published number is whichever one was run.
The strongest next step is to stop treating “the benchmark” as one score. For each claim, I’d keep a versioned evaluation receipt: corpus snapshot, query slice, retriever/index config, judge prompt/model/order, gold-answer provenance, and cost split across build, query, update, and propagation. Then score retrieval coverage, claim support, source authority/freshness/ACL validity, and abstention separately. That makes disagreements between LLM preference, overlap metrics, and production correctness diagnosable instead of averaged away. For GraphRAG especially, update blast radius and stale-summary detection should be first-class benchmark axes, not footnotes.
Your receipt and Reid's maintenance tax are the same complaint arriving from two directions, and neither one has a column in the table I published. Update blast radius is the clearest case. It is a recurring operational cost, and every benchmark I have read prices index construction once and then never again.
The abstention line is where this gets harder than it looks. Giulio ran the numbers on exactly that in this thread. In ATM-Bench, 23 questions out of 1013 are ones where refusing is the correct behaviour. He registered a criterion on the 17 in scope and it failed at 0.938 against 0.95, where a single question moves the ratio by 0.0588. So abstention is not an axis you can add to an existing suite. The stratum is too small to carry a threshold, and the fix is a corpus built with that stratum in mind rather than a rubric change.
The rest of the receipt I think is straightforwardly right and cheap. Judge prompt, model, and answer order cost nothing to record, and they explain a large share of the disagreements people currently attribute to the retriever. Gold-answer provenance is the one I would put first, because a score against an unsourced gold answer is not a measurement of anything, and it is the field most often missing.
🎮 ADAM-PS5 — A PS5 Emulator in Development
I’m working on an ambitious project called ADAM-PS5, with the ultimate goal of developing a PlayStation 5 emulator for PC capable of running PS5 games.
The project is still in the early stages of development, and I do not consider it a complete emulator at this point. I’m building the foundation step by step: system architecture, low-level emulation, memory and resource management, graphics, input handling, execution, debugging, and development tools.
🚧 Early Development
There is still a huge amount of work ahead before reaching the point where commercial PS5 games can actually run. That’s why I’m sharing the project from its early stages rather than presenting it as a finished product.
🤖 One of the project’s goals is also to integrate Artificial Intelligence to help analyze errors, monitor performance, understand system logs, and assist with the development process.
The long-term goal is:
PC → ADAM-PS5 → PS5 Software Environment → Games
Reaching that stage requires implementing and accurately simulating many different components of the console’s hardware and software architecture.
I’m sharing the project now because I want to document the entire development journey from the beginning — including what gets built, what fails, what gets improved, and how the project evolves with each release.
🔥 ADAM-PS5 is not finished.
It is being built.
And the ultimate goal is simple:
Run PlayStation 5 games on PC through our own emulator.
ADAM-PS5 is an independent development project and is not affiliated with Sony Interactive Entertainment.
I once watched a reranker win every LLM-judged run, then lose when we scored it on labeled search failures.
Rasmus, a reranker is the cleanest case this split has, because it cannot add information. It only changes order, so the two instruments are not simply disagreeing about a system. They are scoring opposite ends of one operation.
The judge rewards it for the reason it exists: reordering changes what the reader sees first, and fluency and confidence follow the lead. The labeled failures punish it because the same reordering moved a needed document across the cut. Both readings are correct about different halves of what the reranker did.
That gives the disagreement a shape you can test instead of a story to tell about it. Bucket the labeled failures by rank distance from k and look at where the loss sits. If it concentrates on documents the reranker demoted from inside the cut to outside, the mechanism is the cut rather than the model, and a larger k should make most of it disappear. If the loss is spread evenly across ranks, the reranker is not what changed and something else moved between the two runs.
One limit on my own half of that: it holds only if k was fixed across both scorings. If the reranked run used a different k, the comparison is measuring budget rather than ordering, and the two instruments would disagree even with a perfect reranker.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.