I benchmarked a code graph against grep across 37 repositories

luqueee — Fri, 28 Aug 2026 14:43:00 +0000

I benchmarked a code graph against grep across 37 repositories

I've been working on Kivgraph, a local open-source code graph for coding agents.

There are already plenty of code search and code graph tools around, so I wasn't very interested in proving that "graphs are better than grep".

grep is extremely good when you know what you're looking for.

What I wanted to test was something narrower:

Can a resolved code graph answer structural questions with the same accuracy while making an agent read significantly less code?

The benchmark

I put together 29 questions across 37 repositories written in Go, TypeScript, Rust, Python and Dart.

The questions had hand-written ground truth and covered things like:

who calls this symbol?
what can reach this within two hops?
what would break if I changed this?
which other repository consumes it?
where is this declared?
show me the relevant source

The baseline was intentionally boring: grep for the relevant identifiers, then read the matching files.

The results:

	Kivgraph	grep + reading
Exact answers	28/29	28/29
Returned tokens	35,961	267,980
Median context reduction	~5.95x	—

The accuracy was basically identical.

The context usage wasn't.

Kivgraph returned around 36k tokens in total versus roughly 268k for grep + reading.

But grep still won on 5 of the 29 questions, usually when the identifier was rare and already known.

That result actually changed how I think about the tool.

A graph shouldn't replace grep.

If you already know the name of the thing you're looking for, grep is often exactly the right tool.

The graph becomes useful when the question is structural.

A name isn't an edge

One of the things I wanted to avoid was building relationships by matching identifiers.

Two unrelated methods called Handle shouldn't become connected just because they happen to share a name.

For Go, TypeScript and Rust, Kivgraph resolves relationships using:

go/types
the TypeScript type checker
rust-analyzer

Dart uses the Dart Analysis Server.

Python is intentionally more conservative. When Kivgraph can't prove a relationship using a semantic analyzer, the built-in fallback can report it as CANDIDATE rather than presenting it as an EXACT relationship.

That distinction matters when an agent is asking questions such as:

"Who actually calls this?"

or:

"What could break if I change this symbol?"

Text search can show occurrences. A resolved graph can tell you which occurrences represent actual relationships.

The problem before graph traversal

While testing this, I kept running into another problem.

Sometimes the coding agent knows exactly what it's looking for conceptually, but has no idea what the codebase calls it.

For example:

"Where is the code that decides whether a failed request should be retried?"

The implementation might use retry.

Or it might call the same concept requeue, backoff, reschedule, or something project-specific.

That's what find_by_intent is for.

The agent asks the graph what the code does, Kivgraph ranks the symbols and files that are likely to implement it, and once it has an entry point the agent can switch to the resolved graph for callers, references, dependencies or blast radius.

There are no embeddings and no model calls inside Kivgraph.

The coding agent itself already has a pretty good idea of which implementation words might represent a concept, so it can optionally pass those as hints.

I ran a separate 24-question benchmark where the question deliberately didn't contain an identifier from the answer file:

Approach	Correct
grep	7/24
intent only	6/24
intent + likely code vocabulary	11/24
intent + vocabulary + repository	17/24

The interesting part for me wasn't that intent search "beat grep".

Plain intent didn't.

What mattered was combining what the model knows about the problem with what Kivgraph knows about the codebase.

The workflow I've ended up with is roughly:

natural-language question → find a likely entry point → traverse resolved relationships

Cross-repository code

Kivgraph indexes multiple repositories into the same graph.

Every result includes the repository, path, qualified name and line range, so an agent can use the output of one query directly in another.

That means questions like:

"Which other repository consumes this?"

don't require manually searching every repository in the workspace.

This was one of the main reasons I started building it in the first place.

What Kivgraph doesn't try to do

It's not a replacement for grep.

It's not a replacement for the language server.

It's not another model sitting between your coding agent and the codebase.

And it doesn't turn uncertain relationships into confident ones just to make the graph look more complete.

The goal is mostly to give coding agents a smaller, structured view of the codebase when reading a pile of files would be wasteful.

Everything runs locally and Kivgraph itself doesn't require an API key or external model.

Current language support is Go, TypeScript, Rust, Python and Dart. I'm currently adding Java and C#.

Benchmark caveat

The benchmark harness, ground truth and captured responses are public.

The corpus itself isn't.

So the exact benchmark can't be independently reproduced without running the harness against another corpus.

That's the biggest limitation of the numbers above, and one I'd rather state explicitly than hide behind the benchmark.

If anyone wants to run it against another large codebase, I'd be very interested in seeing where the results fall apart.

Kivgraph is open source:

https://github.com/Luqueee/kivgraph

Docs and full benchmark:

https://kivgraph.dev

I'm also curious how other people are handling this with coding agents: where do you draw the line between grep/file reading and a persistent structural index?

DEV Community: luqueee

I benchmarked a code graph against grep across 37 repositories

I benchmarked a code graph against grep across 37 repositories

The benchmark

A name isn't an edge

The problem before graph traversal

Cross-repository code

What Kivgraph doesn't try to do

Benchmark caveat