A question I can't stop chewing on, and I want your answer more than I want your agreement.
How would you test whether an AI actually understands your codebase?
Not whether it writes good code. That's mostly settled, and benchmarks for it are everywhere. I mean understanding in the sense a senior engineer has it: knowing what depends on what, what breaks if you change this, where the bodies are buried. The thing you'd want before letting an agent touch a load-bearing model.
It's a slippery thing to measure, and I want to lay out why, show you the test I built, point at where I think it's wrong, and then hand it to you.
Why "understands" is hard to pin down
The obvious tests don't work.
→ Ask it to explain a file? It'll explain it beautifully. Reading a single file is local, the answer is in the text, and models are great at it. Proves nothing about structure.
→ Ask it trivia about the architecture? It'll recite plausible-sounding answers, and on a famous repo it'll recite memorized ones that may not match how the code is wired today. You're testing recall, not comprehension.
→ Ask it to make a change and see if tests pass? Now you're testing the test suite as much as the model, and green tests miss exactly the silent breaks you care about.
The hard part is that a confident, fluent, wrong answer looks identical to a correct one until production tells you otherwise. Any test worth running has to defeat that. It has to catch the answer that reads as understanding but isn't.
My attempt: the change-impact teardown audit
What I landed on, described so you can attack it.
Pick the hub model of a real app, the one half the system leans on. Give the agent a real maintainer task:
Before you change how this model is torn down, find every place in the codebase that depends on it.
That's a pure structure question. It has no answer inside any one file. To get it right you have to know the edges, the dependents that reach the model through a concern, a polymorphic association, a worker that loads it by id three calls deep, a config-string registry. The ones with no shared token to grep for.
Three design choices do the real work:
-
A hand-built answer key. Before any run, the real dependents get pinned by hand against the source, to an exact
file:line, weighted toward the scattered non-obvious ones. The agent never sees it. - Grade against the key, not the prose. A reference-aware judge checks each cited dependent against the key. No credit for a vague gesture at the right neighborhood. The citation lands on the line or it doesn't count. (I learned this one the hard way, an earlier reference-blind judge called a 44%-complete audit "exhaustive" because it had nothing to compare against.)
- Run it twice per arm. Because inference isn't deterministic, and a single run hides the variance. One repo went 2 then 0 on the identical prompt.
I ran it across thirteen real Ruby codebases, with and without a structural map the agent could query. The map swung the big apps hard and tied on small colocated ones, which felt like a real signal: it's tracking structure that outgrows what the model can read, not repo size or luck.
Where I think it's flawed
This is the part I actually want help with. I can see holes:
→ One model archetype. The "find dependents before a teardown" task is one shape of understanding. Is it representative, or did I pick the shape that flatters a graph-based answer? What's the task that would break my test?
→ Static structure only. It catches call-and-dependency edges. It misses a pub-sub event bus, dynamic dispatch, runtime data flow. An agent could ace my test and still not understand a heavily event-driven system. How would you test understanding of the dynamic wiring?
→ The answer key is mine. I built it by hand, which means it inherits my blind spots. If I missed a dependent, the bench can't know. How do you build a ground truth you don't trust yourself to have gotten complete?
→ Recall, not judgment. I measure how much of the dependent set it finds. I don't measure whether it knows which dependents are risky. Finding everything and prioritizing nothing is its own failure.
I think the test is good enough to have taught me something real. I don't think it's the last word, and I'd rather hear why than defend it.
So, genuinely, how would you do it?
A few specific forks I'd love comments on:
→ What's the task you'd give an agent to prove it understands your codebase? Not a toy, the real one you ship.
→ How do you build a ground truth for "understanding" without it just being your own opinion written down?
→ Has an agent ever given you a confident answer about your architecture that was flat wrong? What was it, and how did you find out?
→ Is "find what breaks if I change this" even the right proxy for understanding, or is there a better one?
I'll be in the comments, and I'll publish whatever I learn from yours. If your test is better than mine I'll say so and steal it for the next round.
And if you'd rather just run mine
The cheapest way to have an opinion here is to run the experiment on a repo you know cold. Pick the model you'd schedule a careful afternoon for. Ask your agent the teardown question above and read the audit. Then map the codebase and ask again:
→ curl -fsSL https://luuuc.github.io/sense/install.sh | sh
→ sense scan in the repo you know best
→ sense setup to connect your agent
Diff the two answers. The gap is what your agent currently can't see, and it'll sharpen whatever you come back to argue in the comments. (The map's a structural index, not a model, so it's measuring your code's structure, not Claude's mood, which is most of why I trust the diff.)
My full setup, the answer keys, the judge prompts, every transcript.
I build that map, so I'm not a neutral party, which is exactly why I want your test design and not just my own. Everything is open to pick apart.
PS. The best benchmark is the one that embarrasses the person who built it. Mine has, twice. Tell me how to make it embarrass me a third time. That's the comment I most want.
Top comments (0)