I published a result that said my tool scored 34.5% and another scored 5%. A reader on Reddit asked two questions about how I counted. I went back, fixed the method, tested 40 more commits, and the result didn't hold.
This is what went wrong, and how I'd run this kind of test now.
The test
I build EngramGraph, a code graph. I wanted to know if a code graph can tell an AI agent which other files need to change when a function changes. So I compared it with code-review-graph, an open source tool with about 32k stars on GitHub.
The method was simple. Take a real pydantic commit. Go back to the parent commit and index it. Ask each tool who calls the changed functions. Compare the files it names with the files the commit really changed.
I froze the method in a file before running it, and I ran 20 commits. That felt rigorous. It wasn't enough.
What the reader asked
The first question: when a tool declines to answer, is that a miss, or is it dropped from the denominator?
It was a miss. An empty prediction scored 0 for that commit, and I averaged across commits. code-review-graph declined 18 of 42 lookups, so most of its score came from not answering. My headline was mostly measuring how often each tool answered, not whether its answers were right.
The second question: what's the ceiling? Many files change together without any call between them, so 34.5% means something different against a ceiling of 40% than against 100%. I hadn't measured one.
They also suggested showing per-commit numbers next to the pooled ones. I'm grateful for all three. None of this needed graph expertise, just someone reading the method carefully.
What a decline really was
I had described those 18 lookups as "returns nothing". Reading the source showed that was wrong.
With a bare function name, code-review-graph searches names, paths and signatures as full text. Underscores count as word breaks, so rebuild_model_fields also matches test_rebuild_model_fields_preserves_description. When more than one node matches, it stops and returns a ranked candidate list, asking you to retry with a full name. That's a reasonable design for an agent. My script treated the reply as empty.
What I changed
- I froze a new method, including every number I'd report.
- I used 40 new commits, chosen by the same rule, none from the first 20.
- I kept the same tool versions, so the rounds could be combined.
- I made the retry part of the method, not an analysis added afterwards.
- I re-ran the first commit from round one and got identical EngramGraph results, so the setup hadn't drifted.
What happened
My first run said one tool was far ahead. Across all 60 commits, with 171 files to find:
- Without a retry, EngramGraph found 29 files and code-review-graph found 7.
- With a retry, code-review-graph found 21, and 51% of its answers were right. EngramGraph's were right 35% of the time.
- On the 17 commits where both answered, each found 21 files, EngramGraph from 50 guesses and code-review-graph from 41.
So the honest summary depends on whether your agent retries, and I haven't measured that. If it does, the two are about level and code-review-graph is more accurate.
Calls also explain less than I assumed. Only about 30% of the co-changed files mention a changed function at all, so that's a rough ceiling for any tool working from calls.
What I'd do differently
- Report how often a tool answered, separately from how often it was right.
- Never let an empty answer quietly count as a wrong one, or the reverse.
- Estimate a ceiling before reading a score.
- Show per-commit numbers. Round one's median was 0 for both tools, and the average hid that.
- Read the other tool's source before describing its behaviour.
- Treat a result from 20 commits as a question, not a finding.
- Don't compare your own product with someone else's and call it a finding. I set the method, chose the data and wrote the conclusion, so I wasn't a neutral judge.
I've stopped using these tests for that kind of comparison. I still run them, because they're the best way I have to find what to fix.
What I'm fixing
The most useful thing that came out of this was a signal neither tool uses: files that historically change together. Added to EngramGraph's callers on the 40 new commits, it found 29 files instead of 19, at a cost of 6 points of accuracy. It's only been checked on one repository.
The numbers
All tables use the same scoring: files predicted, files correct, share correct when it answered, share of the files to find, and F1 pooled across commits. "Answered" counts commits with at least one prediction.
Round one, as first published (20 commits, averaged per commit, empty answers score 0):
| Precision | Recall | F1 | |
|---|---|---|---|
| EngramGraph | 0.345 | 0.345 | 0.337 |
| code-review-graph, bare name | 0.05 | 0.05 | 0.05 |
| code-review-graph, retry (added after the run) | 0.15 | 0.145 | 0.139 |
Round one, pooled (45 files to find):
| Answered | Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|---|
| EngramGraph | 11 / 20 | 18 | 10 | 56% | 22% | 0.32 |
| code-review-graph, bare name | 1 / 20 | 1 | 1 | 100% | 2% | 0.04 |
| code-review-graph, retry | 4 / 20 | 8 | 5 | 63% | 11% | 0.19 |
Round two, frozen in advance (40 new commits, 126 files to find):
| Answered | Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|---|
| EngramGraph | 24 / 40 | 65 | 19 | 29% | 15% | 0.20 |
| code-review-graph, bare name | 10 / 40 | 16 | 6 | 38% | 5% | 0.09 |
| code-review-graph, retry | 13 / 40 | 33 | 16 | 48% | 13% | 0.20 |
All 60 commits (171 files to find):
| Answered | Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|---|
| EngramGraph | 35 / 60 | 83 | 29 | 35% | 17% | 0.23 |
| code-review-graph, bare name | 11 / 60 | 17 | 7 | 41% | 4% | 0.07 |
| code-review-graph, retry | 17 / 60 | 41 | 21 | 51% | 12% | 0.20 |
Git history, new 40 commits only (top 2 files per commit, using only commits before the parent):
| Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|
| EngramGraph callers | 65 | 19 | 29% | 15% | 0.20 |
| Git history alone | 80 | 16 | 20% | 13% | 0.16 |
| Both | 125 | 29 | 23% | 23% | 0.23 |
Retries in round two: code-review-graph declined 47 of 78 lookups. Retrying found caller files for 24, found no callers in other files for 8, and had no usable candidate for 15. My script made those retries, picking the candidate in the seed file. A real agent might not retry, or might pick wrong.
Ceiling: 51 of 171 co-changed files mention a seed function by name, from a text search on the parent commit. That's an estimate.
Limits: Python only, one repository, direct callers only. "Changed in the same commit" stands in for "affected", so a real caller that didn't need changing counts as wrong. Tool versions were code-review-graph 2.3.9 and EngramGraph 0.13.0. Its main feature, impact analysis from a diff, wasn't tested. In round one the pooled numbers and retry were added after the run; in round two everything was frozen first. I build EngramGraph, so weigh that.
This article was written with the help of AI, based on my experiment notes and the scoring output.
Top comments (1)
답변률과 답한 경우의 정확도를 분리한 정정이 특히 유용합니다. 후보 목록을 받은 뒤의 재시도 횟수·도구 호출 비용도 고정해 두면, 그래프 자체의 품질과 에이전트의 탐색 정책이 점수에 미치는 영향을 더 분리해서 볼 수 있겠네요.