I benchmarked a code graph against grep across 37 repositories
I've been working on Kivgraph, a local open-source code graph for coding agents.
There are already plenty of code search and code graph tools around, so I wasn't very interested in proving that "graphs are better than grep".
grep is extremely good when you know what you're looking for.
What I wanted to test was something narrower:
Can a resolved code graph answer structural questions with the same accuracy while making an agent read significantly less code?
The benchmark
I put together 29 questions across 37 repositories written in Go, TypeScript, Rust, Python and Dart.
The questions had hand-written ground truth and covered things like:
- who calls this symbol?
- what can reach this within two hops?
- what would break if I changed this?
- which other repository consumes it?
- where is this declared?
- show me the relevant source
The baseline was intentionally boring: grep for the relevant identifiers, then read the matching files.
The results:
| Kivgraph | grep + reading | |
|---|---|---|
| Exact answers | 28/29 | 28/29 |
| Returned tokens | 35,961 | 267,980 |
| Median context reduction | ~5.95x | — |
The accuracy was basically identical.
The context usage wasn't.
Kivgraph returned around 36k tokens in total versus roughly 268k for grep + reading.
But grep still won on 5 of the 29 questions, usually when the identifier was rare and already known.
That result actually changed how I think about the tool.
A graph shouldn't replace grep.
If you already know the name of the thing you're looking for, grep is often exactly the right tool.
The graph becomes useful when the question is structural.
A name isn't an edge
One of the things I wanted to avoid was building relationships by matching identifiers.
Two unrelated methods called Handle shouldn't become connected just because they happen to share a name.
For Go, TypeScript and Rust, Kivgraph resolves relationships using:
go/types- the TypeScript type checker
rust-analyzer
Dart uses the Dart Analysis Server.
Python is intentionally more conservative. When Kivgraph can't prove a relationship using a semantic analyzer, the built-in fallback can report it as CANDIDATE rather than presenting it as an EXACT relationship.
That distinction matters when an agent is asking questions such as:
"Who actually calls this?"
or:
"What could break if I change this symbol?"
Text search can show occurrences. A resolved graph can tell you which occurrences represent actual relationships.
The problem before graph traversal
While testing this, I kept running into another problem.
Sometimes the coding agent knows exactly what it's looking for conceptually, but has no idea what the codebase calls it.
For example:
"Where is the code that decides whether a failed request should be retried?"
The implementation might use retry.
Or it might call the same concept requeue, backoff, reschedule, or something project-specific.
That's what find_by_intent is for.
The agent asks the graph what the code does, Kivgraph ranks the symbols and files that are likely to implement it, and once it has an entry point the agent can switch to the resolved graph for callers, references, dependencies or blast radius.
There are no embeddings and no model calls inside Kivgraph.
The coding agent itself already has a pretty good idea of which implementation words might represent a concept, so it can optionally pass those as hints.
I ran a separate 24-question benchmark where the question deliberately didn't contain an identifier from the answer file:
| Approach | Correct |
|---|---|
| grep | 7/24 |
| intent only | 6/24 |
| intent + likely code vocabulary | 11/24 |
| intent + vocabulary + repository | 17/24 |
The interesting part for me wasn't that intent search "beat grep".
Plain intent didn't.
What mattered was combining what the model knows about the problem with what Kivgraph knows about the codebase.
The workflow I've ended up with is roughly:
natural-language question → find a likely entry point → traverse resolved relationships
Cross-repository code
Kivgraph indexes multiple repositories into the same graph.
Every result includes the repository, path, qualified name and line range, so an agent can use the output of one query directly in another.
That means questions like:
"Which other repository consumes this?"
don't require manually searching every repository in the workspace.
This was one of the main reasons I started building it in the first place.
What Kivgraph doesn't try to do
It's not a replacement for grep.
It's not a replacement for the language server.
It's not another model sitting between your coding agent and the codebase.
And it doesn't turn uncertain relationships into confident ones just to make the graph look more complete.
The goal is mostly to give coding agents a smaller, structured view of the codebase when reading a pile of files would be wasteful.
Everything runs locally and Kivgraph itself doesn't require an API key or external model.
Current language support is Go, TypeScript, Rust, Python and Dart. I'm currently adding Java and C#.
Benchmark caveat
The benchmark harness, ground truth and captured responses are public.
The corpus itself isn't.
So the exact benchmark can't be independently reproduced without running the harness against another corpus.
That's the biggest limitation of the numbers above, and one I'd rather state explicitly than hide behind the benchmark.
If anyone wants to run it against another large codebase, I'd be very interested in seeing where the results fall apart.
Kivgraph is open source:
https://github.com/Luqueee/kivgraph
Docs and full benchmark:
I'm also curious how other people are handling this with coding agents: where do you draw the line between grep/file reading and a persistent structural index?
Top comments (1)
Your approach of comparing Kivgraph's structural querying capabilities with grep's identifier-based searches is insightful, particularly in how you highlight the strengths of each tool. I appreciate how you’ve incorporated semantic analysis to avoid misleading relationships, which is crucial for accuracy in complex codebases. As you refine the
find_by_intentfeature, consider adding more contextual hints based on common patterns in the repositories you’ve analyzed—it might enhance retrieval accuracy even further. If you're seeking additional development support as you iterate on Kivgraph, I’d be interested in discussing ways I could contribute to this project.