DEV Community: Luc B. Perussault-Diallo

How would you test whether an AI understands your codebase?

Luc B. Perussault-Diallo — Sun, 12 Jul 2026 10:54:18 +0000

A question I can't stop chewing on, and I want your answer more than I want your agreement.

How would you test whether an AI actually understands your codebase?

Not whether it writes good code. That's mostly settled, and benchmarks for it are everywhere. I mean understanding in the sense a senior engineer has it: knowing what depends on what, what breaks if you change this, where the bodies are buried. The thing you'd want before letting an agent touch a load-bearing model.

It's a slippery thing to measure, and I want to lay out why, show you the test I built, point at where I think it's wrong, and then hand it to you.

Why "understands" is hard to pin down

The obvious tests don't work.

→ Ask it to explain a file? It'll explain it beautifully. Reading a single file is local, the answer is in the text, and models are great at it. Proves nothing about structure.
→ Ask it trivia about the architecture? It'll recite plausible-sounding answers, and on a famous repo it'll recite memorized ones that may not match how the code is wired today. You're testing recall, not comprehension.
→ Ask it to make a change and see if tests pass? Now you're testing the test suite as much as the model, and green tests miss exactly the silent breaks you care about.

The hard part is that a confident, fluent, wrong answer looks identical to a correct one until production tells you otherwise. Any test worth running has to defeat that. It has to catch the answer that reads as understanding but isn't.

My attempt: the change-impact teardown audit

What I landed on, described so you can attack it.

Pick the hub model of a real app, the one half the system leans on. Give the agent a real maintainer task:

Before you change how this model is torn down, find every place in the codebase that depends on it.

That's a pure structure question. It has no answer inside any one file. To get it right you have to know the edges, the dependents that reach the model through a concern, a polymorphic association, a worker that loads it by id three calls deep, a config-string registry. The ones with no shared token to grep for.

Three design choices do the real work:

A hand-built answer key. Before any run, the real dependents get pinned by hand against the source, to an exact file:line, weighted toward the scattered non-obvious ones. The agent never sees it.
Grade against the key, not the prose. A reference-aware judge checks each cited dependent against the key. No credit for a vague gesture at the right neighborhood. The citation lands on the line or it doesn't count. (I learned this one the hard way, an earlier reference-blind judge called a 44%-complete audit "exhaustive" because it had nothing to compare against.)
Run it twice per arm. Because inference isn't deterministic, and a single run hides the variance. One repo went 2 then 0 on the identical prompt.

I ran it across thirteen real Ruby codebases, with and without a structural map the agent could query. The map swung the big apps hard and tied on small colocated ones, which felt like a real signal: it's tracking structure that outgrows what the model can read, not repo size or luck.

Where I think it's flawed

This is the part I actually want help with. I can see holes:

→ One model archetype. The "find dependents before a teardown" task is one shape of understanding. Is it representative, or did I pick the shape that flatters a graph-based answer? What's the task that would break my test?
→ Static structure only. It catches call-and-dependency edges. It misses a pub-sub event bus, dynamic dispatch, runtime data flow. An agent could ace my test and still not understand a heavily event-driven system. How would you test understanding of the dynamic wiring?
→ The answer key is mine. I built it by hand, which means it inherits my blind spots. If I missed a dependent, the bench can't know. How do you build a ground truth you don't trust yourself to have gotten complete?
→ Recall, not judgment. I measure how much of the dependent set it finds. I don't measure whether it knows which dependents are risky. Finding everything and prioritizing nothing is its own failure.

I think the test is good enough to have taught me something real. I don't think it's the last word, and I'd rather hear why than defend it.

So, genuinely, how would you do it?

A few specific forks I'd love comments on:

→ What's the task you'd give an agent to prove it understands your codebase? Not a toy, the real one you ship.
→ How do you build a ground truth for "understanding" without it just being your own opinion written down?
→ Has an agent ever given you a confident answer about your architecture that was flat wrong? What was it, and how did you find out?
→ Is "find what breaks if I change this" even the right proxy for understanding, or is there a better one?

I'll be in the comments, and I'll publish whatever I learn from yours. If your test is better than mine I'll say so and steal it for the next round.

And if you'd rather just run mine

The cheapest way to have an opinion here is to run the experiment on a repo you know cold. Pick the model you'd schedule a careful afternoon for. Ask your agent the teardown question above and read the audit. Then map the codebase and ask again:

→ curl -fsSL https://luuuc.github.io/sense/install.sh | sh
→ sense scan in the repo you know best
→ sense setup to connect your agent

Diff the two answers. The gap is what your agent currently can't see, and it'll sharpen whatever you come back to argue in the comments. (The map's a structural index, not a model, so it's measuring your code's structure, not Claude's mood, which is most of why I trust the diff.)

My full setup, the answer keys, the judge prompts, every transcript.

I build that map, so I'm not a neutral party, which is exactly why I want your test design and not just my own. Everything is open to pick apart.

PS. The best benchmark is the one that embarrasses the person who built it. Mine has, twice. Tell me how to make it embarrass me a third time. That's the comment I most want.

Your next model upgrade won't close this gap

Luc B. Perussault-Diallo — Thu, 09 Jul 2026 06:31:34 +0000

There's a comfortable thing people say when they see an AI agent query a code map.

"Nice crutch. For now."

The logic underneath it is reasonable. Coding agents are young. Context windows are small and getting bigger. Models are dumb today and will be smart tomorrow. So a structural index, the thing that hands the agent a dependency graph it would otherwise have to reconstruct, looks like a patch over a temporary weakness. Wait two releases. The model will just hold the whole repo in its head and the map becomes a quaint workaround, like a spellchecker for someone who learned to spell.

I build one of those maps Sense. I went looking for the data that would kill it.

I didn't find it. I found the opposite.

What a map hands an agent is a computed fact. What a better model hands you is a more confident guess. No amount of model progress turns the second into the first, because the difference between them isn't a quality gap that closes with scale. It's a difference of kind. The rest of this piece is the two findings that forced me there.

The belief, stated fairly

The claim at full strength, because a weak version is easy to knock over.

A code map exists to compensate for what the model can't do yet. Today's agent greps, samples, and guesses at structure because it can't read the whole codebase at once. Tomorrow's agent reads all of it, reasons over all of it, and the guessing stops. Bigger windows plus better weights equal no more blind spots. The map is scaffolding you'll tear down once the building stands.

If that's true, the right move is to skip the tool and wait.

Both findings, in order.

Proof one: the best model available was still blind

The benchmark ran the same task on thirteen real Ruby repos. Pick the hub model of an app, the Inbox, the MergeRequest, the Spree::Order, and ask the agent to find every place that depends on it before a teardown change. The non-obvious dependents, the ones scattered through concerns and workers and config-string registries, were pinned by hand against the source and kept hidden. The agent gets graded against that key, every citation checked to a real file:line.

The headline arm ran on Claude Code with Opus 4.8. Not a budget model. The best coding model money could buy when this ran.

Cold, on Chatwoot, it found 2 of the 11 scattered dependents. On a rerun of the identical prompt, 0. It missed nine that a maintainer has to check before touching that model. On GitLab, the largest open-source Rails monolith there is, it ground for five minutes per run and surfaced 2 of 16. The audit it wrote each time read as finished. It wasn't. It was two-sixteenths of the truth in a tidy, confident report.

Handed the map, the same model, same prompt, same pinned commit, caught 11 of 11 on Chatwoot and 13 of 16 on GitLab.

You can wave that off as one model having a bad day. The next part you can't. The same lift showed up on GPT-5.5, from a different lab, on different weights. Chatwoot baseline 0.47, with the map 0.82. Across the whole bench, every agent gained: Opus +0.26, Devstral +0.24, Qwen +0.18, Kimi +0.13, GPT-5.5 +0.13. Five models, three model families, US-frontier to China-open to EU-open. Same direction every time.

This isn't a weak-model artifact. It isn't a single-lab artifact. The frontier needed the map too.

Proof two: a computed fact and an inferred guess are different categories

A better model can't fix this one.

When the map answers "what depends on MergeRequest," it isn't being clever. It walked the call-and-dependency graph and returned the resolved set. 932 symbols, the same 932 every time you ask. It's a computation over a closed world. The answer is deterministic, and dead-code reasoning works because the world is closed. If it isn't in the graph, it isn't a caller.

When the LLM answers the same question, it's inferring. It reads some files, pattern-matches against every Rails app in its training set, and produces its best guess. Run it twice on Chatwoot and you get 2, then 0. Same prompt. Different answer. That's not a bug in the model. That's what inference is.

Now turn a better model loose on each side.

A better model computes nothing new, because it was never computing. It infers. And a stronger inference engine infers more confidently, not more correctly. Watch where that goes wrong. On Discourse, the plain agent found a dependent of Upload and described its role backwards, named the class that listens for an upload as the one that renders it. On Solidus it called a promotion calculator the rule that decides eligibility. Both audits read beautifully. Both would send a maintainer to change the wrong thing.

A smarter model states that kind of mistake with more authority, not less. The confident-wrong answer is the dangerous one, because you only catch it after you've trusted it. Scaling up the inference makes the prose better and the confidence higher. It does not turn a guess into a computation.

That gap, between a thing computed and a thing inferred, does not shrink as inference improves. It's a category difference, not a quality difference. You don't close it by buying more inference.

The scene that made this concrete for me

The moment it landed: the best model I could pay for, pointed at a famous open-source Rails app, writing a clean audit of a model's teardown, and quietly blind to nine of the things that would break. No flailing. No "I'm not sure." Just a composed, wrong-by-omission report, the kind you'd ship.

The model wasn't failing at reasoning. It was reasoning perfectly over the half of the structure it could see. The other half never lived in the files it read. It lived in the edges between them, and you can't reason your way to an edge you never loaded.

So the map isn't a model bet. It's infrastructure.

Whether this is worth adopting turns on one thing. If a code map were just a stopgap for weak models, you'd be betting on a losing horse and you should wait. It isn't, for three reasons that have nothing to do with which LLM is winning this quarter.

→ It's repo-specific. The map is built from your code at its current commit. No model memorized your private monolith. It can recite the public Rails API from training, sure, but the win on Rails sat entirely on the non-memorized query-compiler internals, the part that never made it into the weights. Your code is all non-memorized. The map is the only thing that has read it.

→ It's always current. It re-indexes as files change, as you switch branches, as a git pull lands. A model's knowledge is frozen at its training cutoff. The map's knowledge is frozen at your last save. One of those is months stale by definition. The other never is.

→ It's LLM-agnostic. It serves the graph over MCP, the wire your editor already uses. One index, every agent. Switch from Opus to GPT-5.5 to whatever ships next quarter and the map doesn't care. You are never stranded by a model swap, because the thing that knows your codebase isn't the model.

A bet rides on one outcome. Infrastructure outlasts the outcomes. This is the second kind.

Where the boundary actually is

The honest version of all this: the map's value isn't fixed. On a small, readable, colocated gem, a strong model reads the whole dependent set in one pass and the map adds nothing. One repo in the bench was a flat tie for exactly that reason. The boundary is real, and the small-repo half of the board is a tie because models keep getting better at reading.

But the apps you actually ship aren't gems. They grow. And they grow faster than context windows do. The dependents that matter spread further apart as the codebase gets larger, which is precisely when the model can no longer hold them all. The boundary moves, every model release nudges it. It never closes, because the codebase is always on the other side of it, getting bigger.

That's why "wait for the next model" isn't a plan. The next model moves the line a little. Your codebase moved it back further while you were reading this.

Test the claim on your own code

Don't take the bench's word, or mine. The cheapest possible check is one you run yourself, and it takes a few minutes.

Pick the model in your own app you'd schedule a careful afternoon for. Ask your agent cold, "before I change how this model is torn down, find every place that depends on it." Watch it grep and guess. Note the count.

Then give it the map.

→ curl -fsSL https://luuuc.github.io/sense/install.sh | sh
→ sense scan in the repo you know cold
→ sense setup to connect your agent

Ask again. Diff the two answers. If your repo is small and clean, you'll watch a tie, and that's a real result you can trust. Point it at the biggest app in your org and watch the gap open instead. The gap is the part of your own codebase your model can't currently see, and no upgrade on the calendar closes it for you.

The benchmark, the methodology, the raw data for all thirteen.

I build Sense. That's exactly why I went hunting for the data that would sink it, and why all of it is open, the code, the harness, the pinned commits, the transcripts, the judge prompts. Disbelieve this and rerun it.

PS. If you think the next model release closes this, name the release and the repo and I'll run the teardown audit on it, cold then mapped, in public. I want to be wrong about this more than you want me to be. So far the data hasn't let me.

"Ruby is the most AI-friendly stack" is half true

Luc B. Perussault-Diallo — Mon, 06 Jul 2026 15:55:55 +0000

You've seen the claim in every Ruby thread for the past year. Ruby and Rails are the most AI-friendly stack. Fewer tokens, less hallucination, the model just writes it cleanly.

Half of that claim I'll concede without a fight. The other half I measured, across thirteen real Ruby codebases, and that's where a line shows up, sharp enough to put every repo on one side or the other. Including yours.

The half that's true: writing Ruby is solved

Start with the part that holds up, because it really does.

A model that has seen ten thousand Rails apps knows where the model lives, where the job goes, what a concern does, what has_many implies, before it reads a line of yours. Convention over configuration was always written partly for the next human reading the code. It turns out the model is the next reader too, and the conventions answer half its questions before it asks them.

So "write me a service object," "add a scope," "refactor this controller"? The stack carries the model. Fewer wrong guesses, tighter loops, less to hallucinate because the shape is already known. Anyone who builds on Rails has lived this, and the AI-friendly reputation earned it. I'm not here to take that away.

I'm here to point out it answers a question nobody dangerous is asking.

The half that isn't: navigating Ruby at scale

"Can AI write Ruby" is settled. The question that ships broken deploys is different: can AI navigate Ruby? What breaks if this model changes, who depends on it, where the blast radius ends.

Reading and navigating feel like the same skill when you're fluent. They are not the same skill for an agent. Reading a file is local, the answer is right there in the text. Navigating is structural, the answer lives in the edges between files, what calls what, what breaks what, and no single file contains it.

So I ran the structural question on all thirteen repos. Same task each time: take the hub model, the Inbox, the MergeRequest, the Spree::Order, and find every dependent before a teardown change. The scattered, non-obvious dependents were pinned by hand and kept hidden. Two arms, same model, same commit: a plain agent that greps and infers, and the same agent handed a structural map it can query. Graded against the hidden key, every citation checked to a file:line.

Now look at the shape of the board. Baseline recall, then with the map:

→ Chatwoot, Inbox → 0.29 → 0.97
→ Mastodon, Status → 0.28 → 0.83
→ GitLab, MergeRequest → 0.26 → 0.67
→ Discourse, Upload → 0.35 → 0.75
→ Solidus, Spree::Order → 0.40 → 0.68
→ Forem, Article → 0.35 → 0.63
→ Rails, ActiveRecord::Relation → 0.67 → 0.92

Then the gap collapses:

→ Lobsters, Story → 0.68 → 0.79
→ Redmine, Issue → 0.67 → 0.78
→ RubyLLM, provider/Chat → 0.76 → 0.80
→ raix → 0.60 → 0.73
→ llm.rb, provider → 0.48 → 0.48

That bottom group is the friendly reputation, intact. Small, readable, colocated repos where Ruby names its dependents literally and keeps implementations close. A strong agent reads app/models whole, finds the set, and the map adds a dependent or two. On llm.rb it added nothing to the answer at all, it just reached the same one on a fifth fewer tokens. That half of the claim is true with no asterisk.

One repo sits in neither group. langchainrb, 0.24 → 0.41: a small gem that scored like a monolith. The model "knows" langchainrb. It knows a version that no longer ships, because the gem moved faster than the training data. Hold that thought, it comes back.

The top group is the same stack at a size where legibility stops scaling. The hub model's dependents reach it through associations, concerns, polymorphism, and config-string registries, scattered across directories that share no token to grep for. The plain agent finds a fraction and stops, confidently. It's not Chatwoot being messy. It's the opposite, it's decomposed the way the books tell you to decompose, and that's what spreads the dependents out of grep's reach.

It's one problem at two scales. Below the line, the model reads the whole repo and the map is redundant. Above it, the model works from a sample of your codebase, and it doesn't tell you which sample.

Two things on that board I didn't expect

The smallest model got the second-biggest lift. I ran the bench across five models, two frontier, three open. The lifts: Opus +0.26 at the top, and right behind it Devstral, the smallest open model in the set, at +0.24, ahead of everything else I ran, including the other frontier model. That looks backwards until you see why. The map stands in for what the model can't hold in its own head. The less it holds, the more the map is worth. Model size and repo size are the same lever pulled from two ends.

The most famous repo was one of the hardest. Forem runs dev.to itself, one of the most-read Ruby codebases on the internet. You'd guess a model handles it best. It's the cautionary one. A frontier model has partly memorized it and can sometimes recite a hub model's dependents from training, with no tools at all. The win held, but the variance is the lesson: a model reciting a repo from memory can still miss how it's wired today, and you can't tell which you're getting.

Rails is the clean version of the same effect. Every model has the Rails source memorized, and the map's whole margin there sits on the query-compiler internals, the part that never made it into the weights. And langchainrb, the gem from earlier that scored like a monolith, is the stale version: the model recites a langchainrb that no longer ships. Fame makes a repo more recited, not better understood.

The exit you're reaching for, and why it's closed

Everyone reading a piece like this quietly exempts themselves. My code is clean. My app is well factored. I'd know if something hid. I'm in the bottom group.

Maybe you are. Today. The line isn't fixed.

The day your repo grows past what the model can hold in one pass, you move from the half where grep is enough to the half where it isn't. There's no deprecation warning for outgrowing your agent. You find out the way you always do, from a diff that sailed through review, three weeks later, in production.

And the line is moving the wrong way for the comforting story. Models keep getting better at reading, which is why the small-repo half is a tie and that tie only gets safer. But the apps you actually ship grow faster than context windows do, and their dependents spread further apart as they grow. The half of the board where the map wins is the half that describes real production work, and it's the half getting larger.

This is also why "wait for a better model" doesn't resolve it. A better model reads more, so it pushes the line. Your codebase grows, so it pushes the line back further. The map isn't a bet on weak models being weak forever. It's repo-specific (built from your code at this commit, which no model memorized), it stays current as your code changes, and it serves any agent over MCP, so a model swap never strands you.

You can't reason your way to which half you're on. Test it.

It takes a few minutes on your own code.

Then give it the map.

→ curl -fsSL https://luuuc.github.io/sense/install.sh | sh
→ sense scan in the root of the app you know best
→ sense setup to connect your agent

Ask again, and diff. Small and clean? You'll watch a tie, and that's a real, trustworthy answer, your repo is on the friendly half and you can stop worrying. Big app? Watch the gap open. Either way you walk out with the number the reputation can't give you: which half you're actually on.

The whole board, the methodology, the raw data for all thirteen.

I build the map in this experiment, so I'm not a neutral narrator here. The code, the harness, the pinned commits, and the transcripts are all open, so rerun it and trust your own number over mine.

PS. The engineers most certain their repo is on the clean half are right about half the time. The other half send me a screenshot a week later. Run it before you're sure which one you are.

I recorded my agent auditing a 36k-file Rails app: the play-by-play

Luc B. Perussault-Diallo — Sat, 04 Jul 2026 17:05:41 +0000

I gave a coding agent a real maintainer chore on the largest open-source Rails monolith there is, and recorded the whole session. Then I gave it the same chore with a map of the codebase and recorded that one too.

What follows is the play-by-play. The agent's moves are real, lightly trimmed for length; my commentary is the indented notes between them. Watch where the first run stops, and why it doesn't know it stopped.

The repo is GitLab, indexed at a pinned commit:

gitlabhq @ 1f9c256f0
tracked files / ruby       68,289 / 29,784
files indexed (coverage)   36,829 (100%)
symbols / graph edges      177,929 / 1,121,147

A million edges in one Rails app. Nobody holds that in their head. The question is whether the agent can rebuild enough of it from files to do the job.

The task, identical on both runs:

You are about to rework how a MergeRequest, and everything attached to it, behaves when it changes or is torn down. Before touching the model, audit every place that depends on the MergeRequest contract.

The gold set was built by hand against the source first: 16 scattered dependents, planted across app/services, app/workers, app/models, app/graphql, app/serializers, and lib/, so no single grep rounds them all up. The agent never sees the list. It gets graded against it.

Same model both runs (Claude Code, Opus 4.8). Same prompt. Same commit. The only variable is the map.

Run 1, cold. The grep grind.

[00:00]  me → audit every dependent of MergeRequest before the rework
[00:06]  agent → grep -rin "merge_request" app/ lib/ ee/ | wc -l
         => 41,800+

First move, the only move it has without structure: search the token. Forty-one thousand hits. That's not a list of dependents, it's a second codebase. The agent has a token budget, not a weekend.

[00:31]  agent → grep -rinE "belongs_to :merge_request|has_many :merge_requests"
         => 380+ named associations

This part it does well. The named associations come back fast, and the Rails conventions earn their reputation here, a service in app/services, a worker in app/workers, named for what it is. The agent reads these correctly. The easy third of the answer.

[01:10]  agent → grep -rinE "merge_request_id|\.merge_request\b|MergeRequest\." | wc -l
         => 9,200+

Now the real work, and the dead end. Nine thousand hits, and the agent has to decide which are dependents and which are an incidental local variable or a log line. It samples. It can't read all of them.

It walks straight past things like this:

# Issuable is mixed into both Issue and MergeRequest.
# Code that resolves notes on "the issuable" depends on MergeRequest
# without ever typing the string a grep could match.
module Issuable
  def resolvable_discussions
    @resolvable_discussions ||= discussion_notes.resolvable
  end
end

The notes-resolution service depends on MergeRequest through the Issuable concern. It never types the class name. No grep for MergeRequest reaches it. The dependents that matter are exactly the ones with no shared token to match on.

[04:50]  agent → [writes the audit]
         "Audited the dependents of MergeRequest. Found: the
          named associations, the diff serializer, the pipeline
          link. The contract is depended on primarily through..."
[05:12]  agent → done.

Five minutes, near the harness ceiling. It ground honestly the whole way and produced a calm, finished-looking report. It never invented a citation, every dependent it named is real. It just found 2 of the 16 that matter. On a rerun, 1. It does not know it stopped at two. Nothing in the output could tell it.

Cold result: 2 then 1 of 16. Scattered recall 0.09. The audit looked complete.

That last sentence is the whole danger. On a small repo you'd notice two-sixteenths. On a monolith you ship it, and the fourteen you missed are the ones that page someone at 3am.

Run 2, same task, with the map.

[00:00]  me → [identical prompt]
[00:03]  agent → sense_blast MergeRequest

         MergeRequest  (app/models/merge_request.rb)
           932 symbols in blast radius
           app/workers/merge_requests/process_auto_merge_worker.rb
           app/services/discussions/resolve_service.rb
           app/serializers/merge_request_diff_entity.rb
           app/graphql/types/issuable_type.rb
           ee/app/services/ee/milestones/promote_service.rb
           lib/gitlab/url_builder.rb
           ... 926 more

One call. 932 symbols, the resolved set, returned in a single shot instead of reconstructed from 41,800 grep hits. The notes-resolution service that the Issuable concern hid? It's right there in the list. The map walked the edge the grep couldn't see.

[00:18]  agent → [reads each candidate, pins to file:line]
[03:40]  agent → [writes the audit]
         "16-item dependency audit of MergeRequest, each pinned
          to file:line: auto-merge worker, notes-resolution
          service, cycle-analytics builder, API discussions,
          GraphQL issuable, Jira integration, milestone
          promotion, ghost-user handler, timelog..."

Same model. Same budget, spent differently, on reading and pinning instead of hunting. It caught 13 of 16 on its best run, a floor of 10. Twelve of the dependents it found are ones the cold runs reached in no attempt:

auto-merge worker · notes-resolution service · cycle-analytics builder · API discussions · GraphQL issuable · Jira integration · milestone promotion · ghost-user handler · timelog · timeline event · URL builder · enterprise discussion

Mapped result: 10 then 13 of 16. Scattered recall 0.72, full audit 0.26 → 0.67.

Two things the recording taught me that the scores don't

The map cost more here, and I'm not hiding it. On this run the map billed about 9% more tokens, not fewer (27,604 → 30,128). On other repos it came in cheaper. Token cost is task-dependent and I'd never compare it across agents. What didn't move is reach: 2.6x more of the real dependent set, plus the twelve silent breaks. Nine percent more tokens to go from two of sixteen to thirteen is a rounding error against the incident you didn't have.

The first time I ran this, the map lost, and that's why I trust it now. Early runs scored 12, then 8, then 1, all over the place. The lazy read was to blame the scenario. The transcripts said otherwise: sense_blast was returning a different set of callers each call. On a hub this size almost every dependency is a plain method call sharing one confidence score, and the index capped that tied list with an unstable sort, even evicting direct callers for distant ones. A non-reproducible impact analysis, handed to any large-repo user, silently. The fix made the cap deterministic, ties broken by confidence then direct-over-indirect, and it ships for everyone now. The benchmark was supposed to score the tool. It kept fixing it instead.

That determinism is also the quiet argument for why this is structure, not a model trick. The map computes the same 932 every time. A model infers a different answer every run, you watched it go 2 then 1. A better model infers more confidently, not more reproducibly. The map reads this repo at this commit, not a training snapshot, and any agent can call it over MCP. None of that rides on which model you run next quarter. It's the part that doesn't change when the model does.

Record your own

This is worth watching on your own code, because your monolith has a MergeRequest too.

Pick it, the model half your services reach into and nobody fully tracks. Ask your agent cold, "before I change how this model is torn down, find every place that depends on it." Watch the grep grind. Count the answer.

Then give it the map.

→ curl -fsSL https://luuuc.github.io/sense/install.sh | sh
→ sense scan in the root of the app that pages your team at night
→ sense setup to connect your agent

Ask again and diff the two transcripts. On a tree this size, the dependents you couldn't find cold are exactly the ones the change would have broken.

The full session logs, the answer key, every transcript for thirteen repos.

I build the map in this recording. Everything you'd need to call me wrong is public, the transcripts, the harness, the pinned commit, the judge, so check the session instead of taking my read of it.

PS. The scariest frame in that whole recording is the cold run writing a composed, confident audit of two dependents out of sixteen and signing off. No flailing. That composure is the thing to be afraid of on a big repo.

Audit your AI agent's blind spots in 4 commands

Luc B. Perussault-Diallo — Thu, 02 Jul 2026 16:01:06 +0000

Don't take my word for any of it. Run this on a repo you know cold and read the diff yourself. Four commands, about five minutes, and you'll know exactly what your coding agent can and can't see in your own codebase.

No theory below this line. Just the steps.

Pick the model first

Open the app you know best. Pick the one model you'd schedule a careful afternoon to refactor, the hub half the app leans on. Inbox, Order, Account, Booking, MergeRequest. The one nobody volunteers to touch on a Friday because nobody's sure what depends on it.

That's your test subject. Hold it in mind for all four steps.

Step 1, ask your agent cold

In your normal agent (Claude Code, Cursor, Copilot, whatever you drive), ask the structural question with no tools beyond what it has today:

before I change how the Inbox model is torn down,
find every place in the codebase that depends on it.

Swap Inbox for your model. Then watch how it answers, not just what it says. It'll grep a token, open what it lands in, sample a few neighbors, and rebuild the relationships in its head. You'll see a chain that looks like this:

grep -rin "inbox" app/ lib/        # hundreds of hits, most irrelevant
grep -rinE "belongs_to :inbox|has_many :inboxes"   # the named associations, found fast
grep -rin "inbox_id"               # and now the guessing starts

Save the answer it gives you. Count the dependents it lists. It'll read as complete, confident, well-structured. That feeling is the thing we're about to test.

Step 2, install the map

curl -fsSL https://luuuc.github.io/sense/install.sh | sh

One binary. It builds a local index of your codebase's call-and-dependency graph and serves it to your agent over MCP, the same wire your editor already uses to reach the model. Nothing leaves the machine. It never edits a line.

Step 3, scan the repo

sense scan

In the root of your app. This is the one-time build. When it finishes it'll print the shape of your codebase, files, symbols, edges between them, the graph your agent was trying to reconstruct from greps a moment ago. On a large app that's a number no human holds in their head, which is the whole reason the cold run missed.

Step 4, connect your agent and ask again

sense setup

This wires the structural tools into your agent so it reaches for them on its own. (That last part matters, an agent that never calls the map scores like one that never had it.)

Now ask the identical question from Step 1 again. This time the agent makes one structural call instead of a hundred greps. Something like:

> sense_blast Inbox
Inbox  (app/models/inbox.rb)
  110 symbols in blast radius
  app/services/...    app/workers/...    lib/...

It gets the resolved dependent set back in one shot and spends its budget reading and pinning each file instead of hunting for it.

Now diff the two answers

Put the cold audit from Step 1 next to the mapped one. The gap between them is the answer.

It won't be the obvious associations, the agent found those cold, the named ones grep catches. It'll be the dependents that reach your model sideways, through a concern, a service, a worker that loads it by id three calls deep, a config-string registry. The ones with no shared token to search for. On the benchmark that produced this experiment, a frontier agent found 2 of 11 of those cold on one well-built Rails app and 11 of 11 with the map. Your numbers will be your own. The shape will be familiar.

Two things to notice in your own diff:

→ The cold agent almost certainly didn't invent anything. Everything it listed was real. Its failure is omission, not invention. Nothing in the output warns you it stopped early.

→ If your repo is small and colocated, the diff might be tiny or zero. That's not a failed experiment. That's a real, trustworthy answer: your agent can already see your whole structure, and you can stop wondering. The gap opens on the big apps, which is exactly where you couldn't afford it.

What you just proved

The map didn't make your model smarter. It gave it something to read that was never in any single file, the edges between them. And it'll keep doing that across model upgrades, because it's built from your code at this commit (which no model memorized), it re-indexes as your code changes, and it serves whatever agent you point at it. It's not a bet on today's model. It's the structure your agent was missing under any model.

The diff you just ran is the part of your codebase your agent has been quietly guessing at. Now you've seen the number.

The benchmark behind this, the methodology, the raw data for thirteen real repos.

Disclosure: I build the map (Sense). It's all open, so check it instead of trusting me. But the only check that matters here is the one you just ran on your own code.

The AI judge that called a half-finished audit 'exhaustive'

Luc B. Perussault-Diallo — Tue, 30 Jun 2026 00:00:00 +0000

If you're building anything with an LLM judge in the loop, this is the failure mode that will get you, and you won't see it happen. I didn't, until I went looking for the opposite.

The story, in the order it happened.

The thing I was building

I wanted to measure something specific: can an AI coding agent navigate a real codebase, not just read one file, but answer "what depends on this model, everywhere, before I change it." That's the question a maintainer answers in their head before a risky refactor, and it's the one an agent tends to get confidently wrong.

So I built a benchmark. Pick the hub model of a real app, the Inbox in Chatwoot, and ask the agent to find every dependent before a teardown change. Run it two ways: a plain agent that greps and infers, and the same agent handed a structural map of the codebase it can query. Same model, same prompt, same pinned commit. Measure the difference.

The catch with any benchmark like this is the grading. You can't eyeball a hundred audits. So, like everyone does now, I reached for an AI judge. Hand the model the agent's audit, ask it to score how complete and correct the analysis is. Cheap, fast, scales.

That decision is the one that nearly sank the whole thing.

The run that looked like a tie

First real scenario, Chatwoot. Two arms ran. The judge scored them.

It called them a tie.

That stopped me, because I'd watched both transcripts. The plain agent had found a couple of the scattered dependents and stopped. The mapped agent had walked the whole set. They were not the same audit. One was clearly more complete than the other. The judge couldn't tell.

I pulled up what the judge actually wrote about the plain run. The word it used was "exhaustive."

The audit it called exhaustive was 44% complete.

Why it got fooled

The mechanic is worth sitting with, because it generalizes to every eval you'll ever build.

The judge was grading each answer on its own merits. It read the plain agent's audit and found it well-written, internally consistent, confident, structured like a thorough piece of work. Every dependent it listed was real. Nothing was fabricated. By every signal available inside the text, it looked like a complete job.

The judge had no idea what a complete job contained. It was never told. It had nothing to compare against, so "looks thorough" was the only axis it could score on, and the plain agent's audit looked extremely thorough. It was a beautifully written half.

That's the trap under every AI-grades-AI setup. The judge inherits the same blind spot as the thing it's grading. Both are reading the same text. Neither knows what isn't there. A confident, fluent, half-finished answer doesn't read as half-finished. It reads as done. And "reads as done" is the exact signal a judge with no reference latches onto.

This is the same failure I was building the whole benchmark to expose in coding agents, and it had crept into my own scoring. The bench was about to flatter itself, and I'd have published the tie with a straight face.

The fix that flipped it

The fix wasn't a better judge model. A smarter model with no reference makes the same mistake more eloquently.

The fix was a reference.

Before any run, I'd already built an answer key by hand: every real dependent of the Inbox, pinned to an exact file:line, assembled from the source. The scattered ones a grep sweep slides past, eleven of them, kept hidden from the agents. I had the list of right answers the whole time. I just wasn't giving it to the judge.

So I changed the judge's job. Stop assessing whether this reads as complete. Take this verified list of what an honest audit must contain, and check each item off. Found it, pinned to the right line, or didn't.

The tie evaporated on the spot. The mapped agent covered everything. The plain one covered half. Same audits, same judge model. The only thing that changed was that the judge now knew what the answer was supposed to be.

The reference-blind judge was retired for the entire series after that run. Every score I've published since is reference-aware, graded against a fixed key, with no credit for a vague gesture at the right neighborhood. The citation lands on the line or it doesn't count.

The rule, for anyone building evals

If you take one thing from this:

An LLM judge with no reference doesn't measure correctness. It measures confidence and fluency, and then it rewards them.

That's fine if confidence is what you want to measure. It's a disaster if you think you're measuring whether the answer is right. The two come apart exactly when the answer is wrong but well-written, which is the case you most need to catch.

Concretely, the things that made my scoring trustworthy after this:

→ A hand-built answer key, made before the runs, so the bench can't quietly redefine "correct" to match what the tool produced.
→ The judge grades against that key, not against the vibe of the prose.
→ Every claim checked to a real file:line. No partial credit for "mentions the right area."
→ The key stays hidden from the agents and only the judge sees it, so it discriminates instead of leaking.

None of that is exotic. It's just the difference between an eval that can embarrass you and one that can't.

Why this matters past my benchmark

The reason I keep telling this story isn't the benchmark. It's that the same blindness shows up wherever you let a model judge work without a ground truth, and that's a lot of places now: AI grading support tickets, AI reviewing AI-written code, AI scoring its own RAG answers. In every one of them, "the output reads as good" is doing more of the work than anyone admits, and a fluent wrong answer scores like a correct one.

It's also, not by accident, the exact thing the benchmark went on to measure in coding agents. A plain agent's audit reads as finished and is missing half. A structural map is what gives the agent, and the judge, a ground truth to check against instead of a story to believe. The map computes what depends on what. It doesn't infer it and hope. That's the whole reason it's worth wiring into your agent and not just waiting for a smarter model: a smarter model is a more convincing storyteller, and convincing is precisely the problem.

Run the version that caught me

The cleanest way to feel this is to reproduce the half I was almost fooled by.

Pick a model in your own codebase you'd be nervous to refactor. Ask your agent cold, "before I change how this model is torn down, find every place that depends on it." Read the audit it gives you. It'll read as complete. That's the trap.

Then give it a real reference to check against.

→ curl -fsSL https://luuuc.github.io/sense/install.sh | sh
→ sense scan in the repo you know cold
→ sense setup to connect your agent

Ask again, and diff the two. The gap between them is the part the first audit was confidently silent about, the same gap my judge scored as a tie. Counting it yourself is more convincing than any benchmark I could show you.

The benchmark, the answer keys, the judge prompts, every transcript.

Disclosure: I build the map in that experiment. All of it is open, including the judge that embarrassed me, so you can check the work instead of trusting me.

PS. I still have the log of the judge writing "exhaustive" under a 44%-complete audit. It's the most useful wrong answer I've ever been handed, and it's the reason every score I've published since has an answer key sitting behind it.

How do you benchmark an MCP server you built?

Luc B. Perussault-Diallo — Fri, 15 May 2026 13:34:59 +0000

I built a code-intelligence MCP server. Then I built a benchmark for code-intelligence MCP servers. Then my tool placed first on every scenario.

I didn't believe it.

So I threw the harness out and rewrote it from scratch. Same result. I built three held-out scenarios with hand-graded reference scores, ran each iteration of the bench against them, and only trusted a number when it correlated above 0.85 with my own judgment.

That's the part I want to write about. Not which tool won. The methodology decisions you have to make when you are also one of the contestants.

Benchmarking AI tools is harder than benchmarking models because the variable is the tool, not the model.

Disclosure first. I am the author of Sense, one of four MCP servers in the bench, scored alongside a no-MCP baseline (Claude Code with grep, find, and Read). Everything below is in the open: code, scenarios, rubrics, transcripts, judge prompts, and the analysis of where my own tool loses. Repo here. Don't take my word for any of it.

The result

Rank	Tool	Fairness	Quality	Tokens	Time	Cost
#1	sense	81.3%	85.4%	10,896	141s	$6.22
#2	probe	77.7%	84.8%	12,119	162s	$6.23
#3	baseline	77.2%	84.2%	12,716	185s	$7.57
#4	serena	75.2%	83.4%	14,800	191s	$7.57
#5	gitnexus	74.9%	84.5%	12,964	173s	$6.87

Two of the four MCP servers scored below the no-MCP baseline. Serena and Gitnexus, by default, made the agent worse than just letting Claude use grep, find, and Read.

Why an eval framework wouldn't cut it

That result is the question worth defending. Two MCP servers below baseline is a claim I need to back up, which is why most of my time went into the bench's methodology, not the bench's tools.

I looked at the obvious frameworks first (Inspect AI from UK AISI is the closest fit for agent evals; Promptfoo, DeepEval, OpenAI Evals all have their place). They handle maybe a third of the work: the run loop, the LLM-as-judge calls, the report rendering. The other two-thirds had to be custom.

What a framework would handle	What I had to build
Run loop, prompt construction	MCP server orchestration per cell
LLM-as-judge calls	Citation grounding vs pinned repo
Report rendering	`answer_text` / `tool_input` split
	Fairness vs adoption layering
	Held-out + Spearman anchor
	Judge variance characterization

The reason is shape. An eval framework is built around "model produces output, output gets graded against expected output." That's a single-turn paradigm. What I needed was: given an MCP server attached to Claude Code, does an agent reach a useful answer faster, cheaper, and with fewer hallucinated file paths than the same agent with no MCP?

The variable isn't the model. The model is fixed (Opus 4.7, 1M context). The variable is the tool the model has access to. None of those custom pieces exist as off-the-shelf primitives.

1. Citation grounding against a pinned commit

Most evals grade text against text. I needed to grade text against a filesystem.

When an AI agent says "the dispatch logic lives at app.py:1625," that claim is either true or it isn't. The number is either inside the file or beyond EOF. The file is either at that path or it isn't.

So every file:line and file:Symbol reference in the answer gets extracted by regex, then verified against the repo at the pinned commit. Three buckets:

Grounded: the file exists and the line is in range, or the symbol resolves within ±5 lines of the cited line
Unresolved: the file is not at the cited path (usually a basename-only reference where the agent dropped the directory)
Hallucinated: the file exists, but the line is past EOF. Outright fabrication.

Hallucinated is the hard signal. When you flag a line that doesn't exist in a file that does, the agent isn't paraphrasing. It's inventing.

The numbers across all five were stark:

Tool	Citation grounding
sense	89.2%
baseline	80.8%
gitnexus	76.9%
probe	72.8%
serena	61.9%

That gap between 89% and 62% is the difference between trusting the answer and manually verifying every line reference. The LLM-as-judge alone would never have surfaced this. The judge reads the answer; it doesn't crawl the repo.

The naive extractor handles ~95% of cases at a fraction of the complexity. It misses some edge cases (basename-only paths it correctly cannot resolve, symbols at unusual offsets) but it doesn't lie.

2. The `answer_text` vs `tool_input` split

This one almost killed the bench before it was real.

A keyword check asks: did the answer mention TopicCreator? Pre-fix, the scorer searched the entire transcript, including tool calls. So when the agent ran Grep("TopicCreator"), the keyword TopicCreator got a "hit" inside the grep invocation. Even if grep returned nothing.

That's not a measurement. That's a tax on tools that don't use grep. Sense, which uses semantic search instead of grep, would lose keyword points purely because its tool calls didn't contain English-language symbol names. Probe, baseline, anything grep-flavored would win keyword points just for grepping for the right string.

The fix sounds simple: keyword checks search the assistant's prose only. Tool inputs and results live in a separate audit_text field, available for diagnostics but never scored against.

# Before: scorer searched the whole transcript
hits = count_keyword_in(transcript_text, keyword)

# After: scorer searches assistant prose only
hits = count_keyword_in(answer_text, keyword)
# Tool calls live in audit_text, never scored

A bench cannot be honest if its first instinct rewards using the tools the bench's author dislikes.

3. Fairness vs adoption: two layers, never folded

Two questions that look similar are not the same question:

Did the developer get a better answer? That's fairness.
Did the agent fluently use the MCP tools available? That's adoption.

If you measure how often the agent uses an MCP server, Sense looks great. But baseline (no MCP attached) gets a structural zero on that axis. Through no fault of its own. The agent literally cannot call a server that isn't there.

So if "adoption" feeds into the headline score, the headline answers the second question, not the first. That's not a benchmark. That's an MCP-adoption survey.

The two-layer model:

fairness  = 0.10 * keyword_coverage
          + 0.55 * llm_quality
          + 0.15 * citation_grounding
          + 0.20 * efficiency

adoption  = 0.60 * tool_fluency
          + 0.40 * discoverability

Adoption is computed, reported, and never folded into fairness. It's there for code-intel-vs-code-intel comparisons only. The headline number is fairness, and baseline can beat any MCP server on it.

Two of the four MCP servers I benchmarked did score below baseline on fairness. They added friction without offsetting it with answer quality.

If adoption had been in the formula, that finding would have been invisible.

4. Held-out scenarios + Spearman correlation

The anti-Goodhart move.

A self-improving benchmark can drift away from human judgment. You tune the rubric, the rubric tunes the scores, the scores look better, and at some point you're optimizing the metric instead of the underlying capability. Goodhart's Law applies to your own measurement.

The defense is an anchor the loop cannot touch.

Three held-out scenarios, frozen. Hand-graded reference scores in gold.json. SHA256-pinned in a lockfile:

# bench/locked/held-out.lock
flask-blueprints:
  transcript_sha256: 7f3a...
  rubric_sha256: 91e2...
  gold_sha256: 8d4c...

The improvement loop refuses to start if any hash has drifted. Each iteration re-judges the frozen transcripts against the current rubric, then computes Spearman correlation between the current llm_quality and the gold scores.

Drop below 0.85, convergence fails, the loop must stop or be re-anchored.

It's the line between we tuned the bench and we tuned the bench until it agrees with our own grading. The smallest part of the codebase. Possibly the most important.

5. Judge variance: what's reliable, what isn't

You don't get to claim the LLM-as-judge is consistent. You measure it.

I ran the judge twice over the same 12 transcripts. Same prompt. Same model. Default sampling.

Layer	Max stdev	Target	Verdict
Per-criterion (raw 0–1 scores)	0.071	<0.05	Fails
Per-step (4-criterion weighted sum)	0.048	<0.05	Passes
Per-scenario (mean of 4 steps)	max \|Δ\| 0.014	<0.05	Rock-solid

The judge is jittery at the criterion level, especially on a fuzzy criterion like "uncertainty." It averages down at the step level. At the scenario level, the number that actually matters for ranking tools, it's stable enough.

So I use scenario_quality to rank. I don't gate decisions on a single criterion-level delta of 0.05 between two tools. That's inside the noise floor. Per-criterion rationales are commentary, not data.

The win here isn't proving the judge is perfect. It's knowing where it's reliable and where it isn't, so I know which numbers I can defend and which ones I shouldn't quote at all.

Reproduce it yourself

The bench is open. Adding a new MCP server to test is one shell script in bench/tools/.

git clone https://github.com/luuuc/sense
cd sense/bench

# Run the full bench (12 sessions = 5 tools × 6 repos, sense + baseline at minimum)
bash bench/bench.sh

# Or run a single (tool, repo) cell
bash bench/run.sh --tool sense --repo flask

Cost: ~$40 in Opus 4.7 tokens for the full 5×6 matrix. Time: ~20 minutes wall-clock. Every transcript ends up under bench/results/<tool>/<repo>/ with transcript.json, scored.json, judged.json, and run_meta.json pinned to the commit the agent worked against.

To add a new MCP server:

# 1. Drop a config script
cat > bench/tools/yourtool.sh <<'EOF'
#!/bin/bash
# MCP server invocation for yourtool
exec your-mcp-binary --your-flags
EOF
chmod +x bench/tools/yourtool.sh

# 2. Run it
bash bench/run.sh --tool yourtool

If you find a methodology hole, the bench is open and replayable. I'll patch what's real.

What the bench does not claim

Not a benchmark of AI model quality. The model is fixed.
Not a real-world end-to-end task benchmark. Each scenario is bounded, scripted, rubric-anchored.
Not a cost benchmark in production terms. Cost is computed from public API pricing for comparability, not from anyone's actual invoice.
Not a measure of MCP-tool fluency in isolation. Adoption is a secondary signal, not the headline.

It is, narrowly, this: given a six-scenario, four-step exploration script across six real repos, how does each tool affect the agent's answer quality, citation grounding, and efficiency.

A smaller claim than "this is the best code-intel MCP." Also a more useful one.

PS: The hardest part wasn't building the tool. It was building a benchmark I'd still trust after it favored me. Most benchmarks die at that step.

Links:

Bench (code, scenarios, rubrics, replay): github.com/luuuc/sense/tree/main/bench
Full results report: report-for-humans.md
Sense: github.com/luuuc/sense

I built AI tools that roast your startup ideas so your investors don't

Luc B. Perussault-Diallo — Sun, 22 Feb 2026 09:50:46 +0000

I got tired of pitching bad ideas to polite friends.

You know the drill, you share your latest "revolutionary" concept, and everyone nods. "Sounds cool." "Interesting." "You should totally build that." Then six months later you realize nobody ever told you it was a terrible idea.

So I built StartupCheck - a set of free AI tools that deliver the painfully honest feedback founders rarely get.

What it does

Five tools, each targeting a different founder blind spot:

Idea Funeral
Submit your startup idea. Get an honest eulogy instead of polite encouragement.

Corporate Translator
Paste your corporate speak. Find out what it actually means. ("Right-sizing the organization" = layoffs start next week.)

Startup Therapist
Share your founder optimism. Get the reality check your co-founder is too nice to give you.

Pitch Without Slides
Deliver your pitch in plain text. See if it holds up without pretty graphs and stock photos.

Build or Die
Describe your project scope. Find out if your "2-week MVP" is actually a 6-month odyssey.

Why I built it

Every founder has blind spots. The best feedback I've ever received was the most uncomfortable. But most people won't give you that feedback unprompted, it's socially awkward, it risks the relationship, and honestly, it's easier to just nod.

AI has no social incentive to spare your feelings. That's a feature, not a bug.

Details

Free to start, no signup required. Builder plan at 7 EUR/month for higher limits
Built with AI that has zero reason to be polite

Check it out: star.tupcheck.me

Would love to hear your thoughts, and yes, you're welcome to roast StartupCheck using its own tools. Fair game.

DEV Community: Luc B. Perussault-Diallo

How would you test whether an AI understands your codebase?

Why "understands" is hard to pin down

My attempt: the change-impact teardown audit

Where I think it's flawed

So, genuinely, how would you do it?

And if you'd rather just run mine

Your next model upgrade won't close this gap

The belief, stated fairly

Proof one: the best model available was still blind

Proof two: a computed fact and an inferred guess are different categories

The scene that made this concrete for me

So the map isn't a model bet. It's infrastructure.

Where the boundary actually is

Test the claim on your own code

"Ruby is the most AI-friendly stack" is half true

The half that's true: writing Ruby is solved

The half that isn't: navigating Ruby at scale

Two things on that board I didn't expect

The exit you're reaching for, and why it's closed

You can't reason your way to which half you're on. Test it.

I recorded my agent auditing a 36k-file Rails app: the play-by-play

Run 1, cold. The grep grind.

Run 2, same task, with the map.

Two things the recording taught me that the scores don't

Record your own

Audit your AI agent's blind spots in 4 commands

Pick the model first

Step 1, ask your agent cold

Step 2, install the map

Step 3, scan the repo

Step 4, connect your agent and ask again

Now diff the two answers

What you just proved

The AI judge that called a half-finished audit 'exhaustive'

The thing I was building

The run that looked like a tie

Why it got fooled

The fix that flipped it

The rule, for anyone building evals

Why this matters past my benchmark

Run the version that caught me

How do you benchmark an MCP server you built?

The result

Why an eval framework wouldn't cut it

1. Citation grounding against a pinned commit

2. The answer_text vs tool_input split

3. Fairness vs adoption: two layers, never folded

4. Held-out scenarios + Spearman correlation

5. Judge variance: what's reliable, what isn't

Reproduce it yourself

What the bench does not claim

I built AI tools that roast your startup ideas so your investors don't

What it does

Why I built it

Details

2. The `answer_text` vs `tool_input` split