DEV Community

CryptoPunk Dz2
CryptoPunk Dz2

Posted on

Measured: 7.76%, Not x5. Your Judge Does Not Know When It Is Wrong

Anton says you cannot build a product in a week, that the honest speedup is x5 to x10 rather than x100, and that for a small flow you can put an LLM where the backend would go. He also says out loud that somebody should go and get the numbers on how much faster engineers actually got. So I went and got them, and they make his case harder than he makes it himself.

The measured speedup is not x5. It is about eight percent.

DX measured 400+ companies from November 2024 to February 2026. AI tool usage rose roughly 65 percent. Median pull-request throughput rose 7.76 percent. The mean was 13.1 percent, pulled up by outliers, and even the 90th percentile only reached about 44 percent. Most organisations sit between 5 and 15 percent.

The one randomised trial is worse than that. METR took 16 experienced open-source developers, 246 tasks in their own large repositories, randomly allowing or forbidding AI. Developers were 19 percent slower with AI. They had predicted a 24 percent speedup beforehand, and afterwards they still believed AI had sped them up by 20 percent. The follow-up on newly recruited developers landed at minus 4 percent, confidence interval minus 15 to plus 9.

CircleCI, across 28 million CI workflows: workflow throughput up 59 percent, while main-branch throughput for the median team fell 7 percent. More runs, less shipped.

And the self-reported number across 349 technical workers: 1.4x to 2x. So the gap is not between slow and fast, it is between measured and felt.

Which means the x5-x10 in the post is an estimate, and it is more generous than any measurement. The honest reading: for one person on a fresh small project x5 is plausible, because there is no legacy, no review queue and no other people. For an engineer inside a large mature codebase the measured figure is eight percent. Both numbers are real, they just describe different work, and the post's conclusion survives either way: development is still months.

There is also a clean reason why, and it is exactly the construction analogy from the post. Coding is around 16 percent of how an engineer spends their time. Speed that 16 percent up tenfold and you buy back roughly 15 percent of the whole. The ground, the water, the utilities and the suppliers do not get faster because the bricklaying did.

Now the part I would push on: "the LLM will be expensive."

We measured that. On typed decisions a cheap model scored 59 out of 60; our Opus scored the same 59 out of 60, grok 59, codex 59, at a sixtyfold price gap: $0.0152 against $0.90 per thousand decisions. The barista in the coffee machine does not have to be on a six-figure salary. Where the task is typed, the cheap one pours the same coffee.

And the part the post gets right in a way worth putting numbers on: "it will work in some number of cases out of 100."

We run an LLM exactly as a backend, a judge deciding whether a mined candidate is gold or noise, with a human marking the judge afterwards. The ledger: 138 judged items, 111 confirmed, 27 rejected. 80 percent.

But the spread is the real finding, and it is not one number:

task hit rate
recurring patterns 7/7 = 100%
contradictions 2/2 = 100%
prompt design 22/23 = 96%
identity statements 20/21 = 95%
bridges between notes 8/9 = 89%
open questions 21/30 = 70%
lead signals 3/7 = 43%
orphan detection 3/8 = 38%

Typed, bounded, text-shaped work: 95 to 100 percent. Judgement about the outside world: 38 to 43 percent. Same model, same prompt discipline, same week. So "an LLM instead of a backend" is not a general bet, it is a bet on how typed the task is, and the failure mode is twice as likely on exactly the judgement calls you would most want to automate.

One more thing from the same ledger, and it is the uncomfortable part. The judge labels its own confidence. Where it said "ok", the human agreed 71 times out of 89. Where it hedged with "partial", the human agreed 40 out of 49. 80 percent against 82 percent. The model's own confidence carried no information about whether it was right. So the barista cannot tell you which cups came out wrong, which is the whole reason the business processes have to close around it rather than inside it.

⚠️ Caveats, named rather than buried: the judge ledger is from 5 July 2026, n=138, and the live review database is gone, so the marking stopped. The 59-out-of-60 probe is n=60 and blind to a one-point difference. The external measurements are throughput proxies, not quality, and METR's authors explicitly limit their result to experienced developers on large mature repositories with early-2025 models.

The full story, in two versions:
📖 For humans, the longread: https://github.com/tonydzi/clawrush/blob/main/longreads/20261110.md
🤖 For machines, the devlog: https://github.com/tonydzi/clawrush/blob/main/devlog/20261110.md. Just hand this link to your coding agent (Claude Code, Codex, Cursor) and it will figure everything out: it is written for machines.

🔗 All our channels and contacts in one place: https://linktr.ee/PaloAltoAI

Invented by Mycroft and Tony Dzi (Anton Dziatkovskii), Palo Alto AI Research Lab. Proudly made in Silicon Valley.

Top comments (0)