How a 45% hit rate turned into 100% recall@10.
The moment
Someone asked me where a change belonged in my own project. Not a big project. Fifty-odd files, four layers, every line written by me.
I opened three files before I found it.
That is the part nobody writes a blog post about. Not "I inherited a legacy monolith" — I wrote this last month and I still had to go looking. And if I had to go looking, the AI assistant I keep pasting files into had no chance. It was working from whatever I happened to remember to give it.
So I did the obvious thing first. I did not build anything.
What I tried before building anything
I used graphify. It parses your repo into a real AST knowledge graph — symbols, calls, imports, communities — in under a second. No LLM, no embeddings bill, no waiting.
It is a good tool. I still use its parser underneath everything I built afterwards.
But when I actually asked it questions — "where does this bug live?", phrased the way a GitHub issue is phrased — it kept handing me plausible files that were not the file. So I stopped trusting my impression of it and ran a benchmark: 40 real GitHub issues from SWE-bench Lite, each with a known ground-truth file. Give the retriever the issue text, see if the right file comes back.
45% at rank one. 82.5% in the top ten.
Read that second number again, because it is the one that matters. Nearly one in six times, the file I needed was not in the list at all. Not ranked low — absent. And you cannot tell the difference from the outside: a confident list of ten wrong files looks exactly like a confident list with the right one in it.
That is when it stopped being a preference and became a bug worth fixing.
The idea, in one paragraph
A call graph knows what exists. It does not know what matters for the question you asked.
So I gave the graph an architecture first. TLDRGraph works out your project's real layers — CLI surface, ingestion, engine, storage, UI — from evidence in the repo, then assigns every symbol to one. When a question arrives, it works out which layer the question lives in before it ranks anything, and searches there.
That is the whole trick. Not a better embedding model. Not a bigger context window. Just refusing to rank a database helper and a CLI flag against each other as if they were the same kind of thing.
The number
Same 40 issues. Same harness. Same machine.
| Retrieval method | Recall@1 | Recall@10 | MRR | Tokens |
|---|---|---|---|---|
| BM25 keyword search | 60.0% | 85.0% | 0.671 | 28,500 |
| Chunked dense RAG | 70.0% | 95.0% | 0.797 | 22,400 |
| Graphify (AST graph) | 45.0% | 82.5% | 0.591 | 9,500 |
| TLDRGraph | 75.0% | 100.0% | 0.823 | 8,000 |
100% recall@10. Across all 40 issues, the file that actually needed changing was in the top ten every single time.
I want to be precise about what that is and is not. It is file-level localization on 40 SWE-bench Lite tasks — finding where, not writing the patch. Every baseline in that table is my own implementation inside one harness, so it is an honest comparison between them on my machine, not an official score for anyone else's project. Run it yourself; the harness is in the repo.
But the practical meaning is simple, and it is why I stopped tuning and shipped: paste the top ten and the answer is in there. At 8,000 tokens instead of 22,000, which is the other half of the point.
(This number is newer than my README, which still shows the older 0.682 MRR run. Fixing that today.)
What it looks like when you use it
Two things come out of it.
A map — your codebase drawn as layers, modules and the calls between them, so you can see the shape of what you built instead of remembering it.
And flows — the end-to-end journeys through the code, drawn as a line of steps with the decisions hanging off each one. Where the branches come from the actual parsed source, not from a summary. That view exists because a diagram of what-calls-what still could not answer "what happens if this check fails?" — the graph had thousands of edges and not one of them was a decision.
Both are for the same purpose: stop re-deriving your own architecture, whether the one doing the re-deriving is you or the assistant you paste into.
Try it
pip install tldrgraph
tldrgraph init # parse the repo, work out the layers
tldrgraph ui # open the map
I built this because I could not find a file in a codebase I wrote myself. The fix was not a better memory. It was giving the machine the same thing I use when I know where something lives: a sense of which part of the system the question belongs to.
https://github.com/vikrand/TLDRGraph
I build TLDRGraph — architecture maps and flow tracing that humans and coding agents read the same way.
Top comments (0)