- Book: Observability for LLM Applications
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Give two agents the same failing test. One comes back with a new helper module, a retry wrapper and a config flag nobody asked for: 400 new lines. The other changes one condition and deletes 12 lines. Both turn the test green.
On a coding leaderboard, those two are the same result. One more resolved task each.
Parsimony is built to count that difference. Its author posted it as a Show HN on October 13, 2026: an open-source benchmark that counts how much code each model adds on the tasks it solves. The pitch, in the author's own words on the thread: "Existing benchmarks measure solve rate and token output/cost, but don't punish excess code."
The net code an agent adds is the maintenance bill you sign for when you merge its branch. It costs nothing to read, and most of us never look at it. Read it on every agent branch, next to the green checkmark.
What a leaderboard row records
Parsimony reads patches from several benchmarks (DeepSWE, SWE-bench Verified and SWE-bench Live, per the Show HN post). This post sticks to SWE-bench Verified: 500 real GitHub issues, manually curated. The score is % Resolved, the share of those issues a model fixed so the hidden tests pass (Simon Willison's write-up explains the setup). The "Bash Only" board runs every model through the same minimal agent, mini-SWE-agent, with the same system prompt, so the model is the main variable.
That's a fair way to measure whether a model can fix a bug. It says nothing about what the fix weighs. A 4-line patch and a 400-line patch move % Resolved by exactly the same amount.
The weight matters once the code is yours. GitClear's analysis of 211 million changed lines found that 2024 was the first year copy-pasted lines outnumbered moved lines, and that blocks of five or more duplicated lines rose eightfold that year. GitClear sells code-analysis tooling and measured AI assistants in general, not any one model, so treat it as a trend rather than a verdict. Still, the direction is clear: generating code got cheap, and the usual scorecards don't charge for it.
What Parsimony counts instead
Parsimony doesn't count lines. It counts "coding units": one syntax element each, such as a statement, a call, a comparison, a name or a literal. Comments, formatting, punctuation and name length don't count (README). The score is net: units added minus units removed. The README's worked example replaces a summing loop with sum(items), which removes 13 units and adds 3, for a net of −10.
It runs nothing. It takes the patches benchmarks already publish, parses each changed file before and after, and takes pass or fail from the published results. Test files, docs and generated files are left out (method). The SWE-bench Verified board measures Python only.
The Verified board's README ranks 35 configurations by "a model's net coding units added on the tasks it solved, divided by what the other ranked models added on those same tasks". Only tasks at least three other models solved count. Two rows from that table:
| Model | Solved | Net units per solved task (mean) | Vs. others, same tasks | Rank |
|---|---|---|---|---|
| Claude Opus 4.5 (high) | 76.8% | +15.5 | 0.67× | 3 of 35 |
| GPT-5.2 (high) | 72.8% | +36.6 | 1.67× | 35 of 35 |
Four points apart on solve rate. More than twice the net code per solved task on average. On a % Resolved table these two look like neighbours. On Parsimony they sit at opposite ends.
The same table has a plain GPT-5.2 row, with no effort setting listed, solving 69.0% with a mean of +28.5 units (1.22×), so the high-effort run solved more and added more. The README also says the agent version affects results, and not every row ran on the same mini-SWE-agent release, so don't lean hard on that gap.
Two limits before you quote any of this. The README labels the board an "Experimental footprint comparison, not an official coding-ability ranking." And it only ranks models with public patch exports. In the Show HN post, the author says GPT-6 and Claude 5.5 "have no public patches yet", so they ran them on 3 to 10 tasks on their own subscriptions. The site shows those runs, on the first few DeepSWE Python tasks, faded and unranked.
Smaller isn't the score
Parsimony's own page pushes back: "Models that write more code also tend to solve more tasks (Spearman 0.33–0.60 on the boards with 26+ models)." Extra code often buys extra solves. The page also warns: "Smaller isn't always better: units measure size, not readability or design." A model whose fixes mostly delete code can rank first with a negative net.
The Hacker News thread found the other edge fast. One commenter, noting that a model called Astra is "winning" the bench, called its output hard to read ("Astra plays code golf") and said they'd rather have Claude's extra comments and tests. Parsimony already leaves comments and test files out of the count, so a model isn't punished for those. But a dense one-liner that nobody can review is a bill too, just paid in reading time.
So the point is to see the number, not to minimise it. A 400-line fix might be the right fix. It should have to make that case to a reviewer who can see it's 400 lines.
Read the bill on every agent branch
You don't need Parsimony to start. Git already knows. git diff --numstat prints added and deleted lines per file, and for binary files it "outputs two - instead of saying 0 0" (git docs). The three-dot form main...branch diffs from the merge base, so you see only what the agent's branch changed, not what landed on main in the meantime.
# net lines one agent branch adds,
# leaving out tests/ and lockfiles
git diff --numstat main...hermes/fix-login -- . \
':(exclude)tests' ':(exclude)*.lock' \
| awk '$1 != "-" { a += $1; d += $2 }
END { printf "+%d -%d net %+d\n", a, d, a - d }'
The :(exclude) pathspec drops matching paths from the diff (gitglossary). Adjust the excludes to your layout, since tests here means the top-level folder only. For a quick look without the script, git diff --shortstat main...hermes/fix-login prints one summary line.
Then hold the number against the size of the ask. A one-condition bug that comes back at +400 isn't automatically wrong, but it has earned a question: what is each new file for, and would a smaller fix pass the same tests? Ask the agent. It costs one prompt.
Lines are a crude unit: a reformat or a renamed variable inflates them. If you want Parsimony's units on your own work, the repo exposes measure(patch, get_source=...) in parsimony.analysis, which takes a unified diff plus a function that returns each file's source before the patch, and returns net_units (README). The site has an "Open benchmark JSON" button: "Download the example file, fill in your results and open it to get the same chart and table." It also says "The file stays in your browser." I haven't run Parsimony on my own branches, so start from the README's example.
Where Hermes puts the number
I build Hermes IDE, so weigh this section with that in mind. Hermes runs each coding agent in its own terminal, on its own hermes/ branch and worktree, and as of 2.3.1, released October 13, 2026, the strip under each session is labelled "Agent replies". Each chip is one reply from the agent and the lines it changed, shown as +added −removed. Click one to see its changes or restore your files to that point. When replies pile up, a "+N earlier" chip lists every reply with its time, files and lines changed.
That's the numstat check, run per reply, without you remembering to run it. The +400 shows up reply by reply, before the branch is done.
The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows the same work by turn or by file, with risk flags, so you can read the heavy turn first.
The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns listed (T1: 3 files +69 −1, T2: 1 file +1 −0), turn 1's diff of src/db.ts adding a refunded status, the changed-files list, and a commit message drafted from the turns.
Know its limits. Hermes counts lines, not Parsimony's syntax units, so a reply that reformats a file looks heavier than it is. In a task with extra folders, the strip tracks changes in the main project only. Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It isn't affiliated with Nous Research, Anthropic, OpenAI or any agent vendor. Download it or read the code on GitHub.
Would you put it next to solve rate?
Leaderboards will keep ranking by % Resolved, and they should: a fix that doesn't pass is worth nothing. The question is what you look at after the checkmark.
Would you choose an agent partly on how much code it adds per fix? Or is a terse fix you can't read a worse bill than a long one you can? Tell me in the comments.
If this was useful
If you ship LLM features of your own, the trap is the same one: an eval that only scores pass or fail will happily approve the bloated, expensive answer. Observability for LLM Applications is about putting traces, cost and quality next to the pass rate, so you can see what each green check cost you. And if you want the +added −removed number in front of you on every agent reply, Hermes IDE is free.
Top comments (0)