title: "PageRank vs RAG on a Real Codebase: Corrected Numbers, and What I Almost Got Wrong Twice"
published: true
description: "Second correction to this experiment. The Hit@Gold numbers are now independently verified and reproducible. But my own 'gold standard is 100% valid' claim wasn't — here's the gap between validating a file and validating the file that was actually used."
tags: machinelearning, python, ai, devtools
cover_image:
TL;DR: RAG (BM25) beats PageRank at retrieving the exact file that answers a query — 50% vs 36% Hit@Gold on a dense code graph (n=50). These numbers are now independently reproduced, not just computed once and trusted. But getting here took two rounds of correction: first I found my gold-standard labels were stale, then — after "fixing" them — I found I'd validated a different file than the one the experiment actually used. 4 of 50 labels were still broken in the script that produced the headline numbers, even after I'd published a "100% valid" validation report for a sibling file that nobody was running.
The Setup
Project: MSCodeBase Intelligence (50K LOC Python, 129 files)
Methodology:
- Gold Standard: 50 queries → manually curated target file for each
- 3 Selection Methods: PageRank (varying graph density), Random baseline, RAG (BM25)
- Metric: Hit@Gold — did the selection include the exact file that answers the query?
- Token budget: ~70K tokens (top 20% of files) for a fair comparison across methods
Tools: NetworkX, tiktoken (cl100k_base), Python AST
This is the second revision of this post. If you read the first correction: the direction hasn't changed, but I'm now more careful about what "verified" actually means.
Round One: The Gold Standard Was 20% Stale
I originally hand-wrote a GOLD dictionary mapping 50 queries to target files. I never checked, before running the experiment, whether those file paths still existed — the codebase had been refactored since I wrote the labels. When I finally checked: 7 of 28 unique target files (25%) pointed at paths that no longer existed, and one query's target (tests/test_search_code.py) was outside the directory my scanner even walked. That's roughly 10 of 50 queries (20%) that were unwinnable by any method, for reasons that had nothing to do with retrieval quality.
I fixed the paths, reran, and got new numbers. I also published a validate_gold.py script and a gold_validation.json reporting 100% of gold paths valid. That felt like closing the loop.
It wasn't.
Round Two: I Validated the Wrong File
Here's the mistake, stated plainly: I created a clean, corrected gold-standard dictionary in experiments/gold_standard.py, and wrote validate_gold.py to check it. It came back 50/50 valid. I took that as confirmation that the experiment was now sound.
But run_experiment_e2e_v2.py — the actual script that produces the Hit@Gold numbers in this post — has its own separate, inline GOLD_STANDARD dictionary, hand-duplicated instead of imported from the "fixed" module. Nobody kept the two in sync. When I diffed them: 11 of 50 entries differ, and 4 of those still point at nonexistent files — the exact same stale paths from round one (src/providers/reranker.py, src/core/intelligence/engine.py), still sitting in the script that actually runs.
So the true state, verified by independently re-running both scripts against the real repo:
| Result | |
|---|---|
experiments/gold_standard.py (validated, unused by the experiment) |
50/50 valid (100%) |
run_experiment_e2e_v2.py's inline GOLD_STANDARD (the one that actually produced the numbers below) |
46/50 valid (92%) |
I published a green validation report for a file that wasn't wired into anything. That's the same failure pattern I complain about in AI agent output all the time — a status that says "done" without checking it against the artifact that matters. Doing it to myself, in a post about being more rigorous, was a useful reminder that "I wrote a validation script" and "I validated the right thing" are not the same claim.
There's also at least one gold label that passes a file-existence check but is still wrong on the merits: "what tests exist" maps to src/__init__.py — a real file, but not one that answers the question. Path validation catches missing files; it doesn't catch wrong-but-existing ones.
The Numbers (Independently Reproduced)
I re-ran run_experiment_e2e_v2.py myself against a fresh checkout, end to end, rather than trusting the last run's output file. It reproduced exactly:
Dense Graph (imports + class refs + function calls, 388 edges, 128 files)
| Method | Hit@Gold | SUFFICIENT | Avg Tokens |
|---|---|---|---|
| RAG (BM25) | 50% | 25/50 | ~40,200 |
| PageRank | 36% | 18/50 | ~35,900 |
| Random | 12% | 6/50 | ~27,000 |
Sparse Graph (imports only, 110 edges)
| Method | Hit@Gold |
|---|---|
| RAG (BM25) | ~50% |
| PageRank | 18% |
| Random | 12% |
These are the numbers I'll stand behind — not because they're perfectly clean (4 of 50 gold labels are still wrong, as detailed above), but because I've now actually reproduced them from a fresh run instead of trusting a cached result, and I know precisely which and how many labels are still bad. Real effective sample size for reliable inference: 46/50, not 50/50.
On confidence: a 7-point gap (18 vs 25 hits) on n≈46-50 is a real, repeatable effect in this codebase — it holds across both graph densities and has a mechanistic explanation (below), so I trust the direction. I do not have the sample size to defend "50% vs 36%" to the point, and you shouldn't take the exact percentages more seriously than "RAG is clearly ahead, PageRank is clearly ahead of random, by a moderate but not enormous margin."
Why My Earlier "Keyword Accuracy" Numbers Were Also Misleading
Before Hit@Gold, I measured "keyword accuracy" — does the selected context contain the query keyword anywhere? That produced inflated, near-meaningless numbers: PageRank scored ~78-80% keyword accuracy on both graph densities, while its actual Hit@Gold was 18-36%. Keywords like search, error, lock, sql appear in dozens of files — a random 25-file selection covers most queries by keyword presence alone, which is exactly why the random baseline (12% Hit@Gold) looks so much weaker than its keyword-accuracy score would suggest. Keyword presence tells you nothing about whether you found the file that actually answers the question.
I also previously tested a "Smart Summary" approach (a compressed 2K-token repo overview fed to the LLM). It looked like 90% accuracy on 10 hand-picked easy queries. On the full 50-query set it dropped to 26%. Ten easy queries and fifty real ones are not the same benchmark.
What Graph Density Actually Does
| Graph | Edges | PageRank Hit@Gold |
|---|---|---|
| Random | 0 | 12% |
| Import-only | 110 | 18% |
| Imports + class refs + func calls | 388 | 36% |
Denser graphs roughly double PageRank's Hit@Gold. On a sparse import-only graph, PageRank mostly just surfaces the biggest files, which aren't necessarily the most relevant ones for a given query. Adding class-reference and call-graph edges breaks that coupling. Even at its best, though, dense-graph PageRank doesn't catch RAG.
Why RAG Wins (and Why the Comparison Was Never Fully Fair)
RAG (BM25) is query-aware — it scores files against the literal terms in the question. PageRank is query-agnostic — it ranks files by global structural importance once, then returns the same top-N regardless of what's asked.
For "where is DebounceBatch defined": RAG matches the term directly and finds rate_limiter.py. PageRank, working off a precomputed graph, ranks structural hubs like engine.py or runtime_coordinator.py highly, with no mechanism to notice neither one mentions DebounceBatch.
Worth saying directly: comparing a query-agnostic ranking against a query-aware retrieval method and reporting "RAG wins" is a bit like reporting that a road atlas loses to GPS navigation at finding a specific address — true, but not really a fair contest. The more useful question isn't "which wins" (RAG will, structurally, on any query-specific task) but "does PageRank add anything on top of RAG" — an experiment I still haven't run.
Honest Corrections to the Record (Now at Version 2)
| Claim | Status | Why |
|---|---|---|
| "Top 20% = -2% savings" | Sparse-graph artifact | Density matters a lot |
| "Smart Summary = 90% accuracy" | 26% on the full query set | 10 easy queries ≠ 50 real ones |
| "PageRank doesn't work" | 36% Hit@Gold, +24pp over random | Works, modestly |
| "PageRank beats RAG" | Still false | RAG 50%, PageRank 36% |
| "My gold standard is 100% valid" (round-one fix) | False when it mattered | Validated a file the experiment didn't use; the used file was 92% valid |
| Current numbers (50%/36%/18%/12%) | Independently reproduced | Re-run from a fresh checkout, not trusted from a cached file |
What This Actually Means
For AI code tools:
- Use PageRank as a prior to blend with RAG, not a replacement — a hypothesis based on the mechanism, still unmeasured.
- Sparse import-only graphs underserve PageRank; if you use it, build the denser graph.
- Validate your gold standard against the file that actually runs the experiment — not a nicely validated sibling copy that nothing imports from.
For anyone benchmarking retrieval on their own codebase:
- Don't trust "X% accuracy" without knowing exactly what's measured — keyword-presence accuracy and exact-file Hit@Gold can differ by 40+ points on the same run.
- Always include a random baseline.
- A green validation script only tells you about the file it checked. If your pipeline has two copies of the same data structure, a passing check on one proves nothing about the other — check that yourself before publishing, because I didn't, twice.
- Report sample size next to any percentage. "50%" and "50% (n=50, direction robust, magnitude uncertain)" are different claims.
The Math (Corrected, Reproduced Independently)
Total: 129 files (128 after filtering), ~406K tokens
Top 20% budget: ~25 files
Hit@Gold (E2E metric, dense graph, 388 edges):
RAG (BM25): 50% (25/50)
PageRank: 36% (18/50) ← +24pp over random
Random: 12% (6/50)
Gold-standard integrity (verified by independent re-run):
gold_standard.py (validated, unused by experiment): 50/50 valid
Inline GOLD_STANDARD in run_experiment_e2e_v2.py
(the one that actually produced these numbers): 46/50 valid
Related Work
- Aider uses symbol-level elision — needs a dense graph plus query-aware retrieval to work well.
- CodeGraph does on-demand, query-conditioned retrieval — the direction this points toward.
- Codebase-Memory reports honest comparative metrics (83% vs. 92%) rather than a single flattering number.
Open Questions
- Does PageRank-as-a-prior (blended with BM25 scores) measurably beat RAG alone? Still unmeasured.
- Does the RAG advantage hold on a codebase with less descriptive file/function naming? This project's names are unusually aligned with what they implement.
- What happens at n=200 queries with proper confidence intervals, and a gold standard that's imported once from a single source of truth instead of copy-pasted?
Reproduce this yourself: scripts in experiments/. If you do: check whether the gold-standard dictionary the experiment script actually imports is the same one your validator checked. It wasn't, for me, and I'd already published a "100% valid" report before I noticed.
Part of my research on MSCodeBase Intelligence — an MCP server for codebase intelligence.
Top comments (2)
Codebase context needs ranking, not just more tokens. The interesting part is deciding which files deserve attention before the model reads them, because context quality usually beats context volume.
I subscribe to your words, and I also conducted additional experiments and corrected errors.