62% Fewer Tokens? Not in My Code Reviewer: Testing CodeGraph Three Ways

CodeGraph promises 62% fewer tokens for coding agents. I tested it three ways in my multi-agent code reviewer, and it saved nothing in any of them. When it replaced the reviewer's own code map, the reviewer found 18% fewer real issues.

The reviewer is a Claude Code plugin I built that reviews a whole backend instead of one PR (I wrote about it in my previous post). CodeGraph is an open-source MCP server that indexes a repository into a graph of symbols and calls, so an agent can ask "how does X reach Y" in one call instead of a grep-and-read loop. My reviewer reads a lot of code, so a code graph should make it cheaper. I measured that on the same four private projects as last time, with blind judges checking every finding against the code.

What CodeGraph promises

CodeGraph parses the repository with tree-sitter, stores functions, classes, imports and calls in a local SQLite database, and keeps it in sync with a file watcher. The main MCP tool, codegraph_explore, takes a question or a list of symbol names and returns the relevant source, the call paths between the symbols and their callers.

Its README reports a benchmark: seven open-source repositories, one architecture question per repository, Claude Code headless with and without the MCP server, median of four runs. The headline is "88% fewer tool calls, 53% faster, 62% fewer tokens, 44% cheaper". The README is also candid about a trap: without blocking the CLI in the control arm, 26 of 28 control runs found CodeGraph through Bash anyway. That's why all my baseline runs happened before CodeGraph was on the machine at all.

Indexing is fast. My largest project, 583 Python files, became 9,798 symbols and 26,368 edges in about two seconds.

What my reviewer already does

My reviewer doesn't wander. Before any agent starts, a collect step builds the map it works from:

• Modules. The code is split into modules of at most 10,000 lines; one agent per module and aspect reads every line of its module.
• An import graph. Per-file fan-in and fan-out, edges and cycles between modules and between top-level layers.
• Dead-code candidates from per-language tools: staticcheck and deadcode for Go, knip and tsc for TypeScript, vulture for Python, cargo check for Rust.
• Scanner and linter hints: vulnerable dependencies, secrets, malicious packages, performance and security lint.

Module, security and performance reviewers work per module, an architect looks across modules, a global security pass looks at the attack surface, and a verifier tries to refute every finding. So the question was whether CodeGraph's map beats the one my reviewer already has, or adds something on top of it.

The setup

The same four projects as in the model comparison, at the same commits:

Project	Language	Reviewed
A	Go	~830 lines
B	TypeScript	~1,030 lines
C	Rust	~1,220 lines
D	Python	~11,250 lines (2 modules)

Each project was cloned into a scratch directory, and collect ran once per project, so every run in a comparison started from the identical inventory. All baseline runs happened before CodeGraph was installed at all: no CLI on the PATH, no MCP server, no index. Then I installed it the way the README says:

npm i -g @colbymchenry/codegraph@1.6.2
codegraph telemetry off
codegraph install --target claude --location global --yes
cd <repo> && codegraph init

The installer registers the MCP server in ~/.claude.json (loaded from the first prompt), allows its tools in ~/.claude/settings.json, adds a prompt hook and writes a section to ~/.claude/CLAUDE.md telling agents to reach for CodeGraph before grep or reading files when a repository has an index.

Test 1: install it and let the agents decide

Two full Opus 5.5 runs without CodeGraph, two with it, all four projects.

The agents saw it. They listed the .codegraph directory, excluded it from their greps, and carried on reading code with sed and grep exactly as their prompts tell them to. In 8 project runs there were 3 real codegraph_explore calls, all from the global security pass on the Python project. A general hint in CLAUDE.md loses to a specific role prompt.

The bill went up, not down: $32.19 on average against $29.06, and 51.2M tokens against 43.3M. CodeGraph can't be the cause; it was barely called. The CodeGraph runs kept more findings on the Python project (59 and 48 against 40 and 42), and more findings means more verifier work. I read this as run-to-run noise.

Test 2: make it mandatory where it should help

Module reviewers must read every line of their module, so a graph can't spare them much. The roles that trace across files can benefit: the architect, the global security pass and the verifiers of their findings. I added one section to those three prompts:

This repository is indexed by CodeGraph. For every question that crosses files
(who calls X, where a value comes from, how a request reaches Y, what depends
on a module, whether a guard exists upstream), call the `codegraph_explore` MCP
tool FIRST, with `projectPath` set to the repository root, instead of
grep/sed/cat loops.

Then I ran only those agents, two runs per project, Opus 5.5, and compared them with the same agents from the full runs. Now they used it: 15 codegraph_explore calls per run across the four projects, 8 to 10 on Python. Tool calls dropped from 190 to 178 per run. Tokens and cost didn't move: 14.6M tokens and $11.21 per run, against 14.9M and $11.26 without CodeGraph.

The reason is in the output. One codegraph_explore answer on the Python project was about 25,000 characters: the full source of every relevant symbol, plus call paths. It replaces three or four sed -n reads of the right line ranges, but the agent was going to read those lines anyway, and the answer then stays in the context for every following turn. The agents made fewer round-trips and read the same amount of code.

Test 3: replace the reviewer's map with CodeGraph's

The fairest test of CodeGraph as a tool, not as a hint: take out what my collect step computes about structure, put in the same fields computed from CodeGraph's index, and make every agent use the MCP tool. Arm A keeps the plugin's map, arm B gets CodeGraph's.

• Kept in both arms: the module partition, the vulnerability, secret and malicious-package scanners, and the linter hints. CodeGraph has no equivalent for those.
• Replaced in arm B: module and layer edges, cycles, per-file fan-in and fan-out, and the dead-code candidates. A short script read them from .codegraph/codegraph.db. Dead-code candidates are functions, methods and types with no incoming call, reference, import or inheritance edge.
• Arm A ran on clones without an index, so the CLAUDE.md section told its agents to skip CodeGraph. They did: zero calls.

Both graphs agreed on structure. On the Python project CodeGraph produced the same 33 module edges and the same two cycles as my import graph. They disagreed on dead code. In the two reviewed Python modules my tools reported no candidates; CodeGraph's "no incoming edge" reported 41 (229 across the whole repository), even after I dropped variables and Rust test functions. Getters, static members and framework-wired methods have no visible caller.

All eight runs on Sonnet 5.5 at the same time, then one blind Opus judge per project with both finding lists in shuffled order and every mention of CodeGraph scrubbed from the findings. The judges found 86 distinct real issues and no false ones in either list.

Cost was identical, $17.37 against $17.41. The CodeGraph arm finished the Python project faster, 8.4 minutes against 11.8, and used CodeGraph 31 times. And it found fewer real issues: 65 against 79.

The costly issues came out even: 30 of 31 against 29 of 31. Each arm also caught one or two costly issues the other missed, so neither setup is a strict superset of the other.

The gap is in maintainability findings: 49 against 36, most of it on the Python project. I can't prove the cause from one run. My best guess is attention: 41 dead-code leads that mostly didn't hold up, plus the instruction to go through CodeGraph first, pulled the reviewers away from reading. The CodeGraph arm also raised 4 more findings that its own verifier refuted (27 against 23). What it missed were mostly the quiet issues that reviewers pick up while reading: duplication, unused code and leftovers.

All three tests in one picture

The vendor's benchmark and my tests measure different jobs. Its agent answers one question about an unfamiliar repository and would otherwise explore it with grep and file reads, so the graph replaces most of the work. My reviewer gets a map before it starts and is told to read every line of its module, so there is little exploration left to remove.

Limits

• Few runs. Two runs per arm in tests 1 and 2, one in test 3. The two baseline runs of test 1 differed by 3% in cost; differences inside that band are noise.
• The judges were Opus. They were blind to which list came from which setup, but one model family judged both.
• My dead-code heuristic. "No incoming edge in the graph" is my reading of CodeGraph's data, not a feature it ships. A better filter would cut the noise, and the maintainability gap with it.
• Small projects. About 14,000 reviewed lines across four projects. On a large monolith, where the architect and the security pass have more to trace, the cross-file roles might profit more.
• One plugin. My reviewer front-loads structure on purpose. An agent that works without a prepared map is the case CodeGraph's own benchmark measures.
• The MCP server needs projectPath. It starts in the session's working directory. My runs started elsewhere, so the agents had to pass the repository path, which CodeGraph's server instructions tell them to do.

What I'm doing with it

My take: CodeGraph stays out of the reviewer. It cost the same as my own map, and replacing that map lost maintainability findings without adding costly ones. The two graphs agreed on structure, so for this job CodeGraph's index adds nothing my collect step doesn't already compute.

Where I'd still use it: an interactive session in a large repository I don't know yet, where the alternative really is an agent grepping its way through it. That's the job it was benchmarked on.

The general lesson is older than either tool: measure the claim on your own workload before you wire it in. What's in your agent setup that you added because a benchmark said so?