Microsoft measured our thesis, and we still cannot quote ours

#ai #testing #benchmarking #programming

Microsoft's .NET team published the benchmark behind their polyglot unit testing agent. We have been
arguing its conclusion for months. We cannot cite our own version of it, because we withdrew our
number on 2026-08-06 and have not earned it back.

Both halves of that are worth writing down.

What they measured

The setup is unusually clean. One tool, one model, 152 tasks from real repositories. The only variable
is a plugin that makes the agent research the repository, plan, implement, and then verify before it
claims to be done. A task passed only if the repository built, all tests passed, the agent added at
least one test, and it removed none.

Arm	Completed	Rate
With the workflow	140 / 152	92.1%
Same model, stock	120 / 152	78.9%

63% fewer failures, same model, same tool.

The breakdown is the interesting part.

Prompt type	With workflow	Stock
Vague, 89 tasks	79 (88.8%)	59 (66.3%)
Detailed, 63 tasks	61 (96.8%)	61 (96.8%)

On detailed prompts the two arms tie exactly. Every point of the gap comes from vague prompts, where
the agent has to work out for itself what to test, which framework is in use, and where tests belong.
The scaffold is not making the model smarter. It is removing the guessing.

Then the row that matters to anyone choosing a model:

Model	With workflow	Stock	Failure reduction
Claude Opus 4.8	43/45 (95.6%)	35/45 (77.8%)	80%
GPT-5.5	41/45 (91.1%)	36/45 (80.0%)	56%
Claude Haiku 4.5	34/45 (75.6%)	25/45 (55.6%)	45%

GPT-5.5 with the workflow completed 91.1%. Opus 4.8 without it completed 77.8%. Same 45 tasks. The
mid-tier model inside a system beat the frontier model outside one, and it was not close.

That is the sentence we have been writing on our own site, in our own words. Someone with a real
benchmark budget just ran it.

Two caveats we would want applied to us

Their coverage barely moved: 72.4% versus 72.2% line, 49.8% versus 49.1% branch. The workflow did not
produce better tests. It produced tests that exist, build, and pass, more often. That is a completion
result, not a quality result, and they say so.

And the effect shrinks on harder work. On a polyglot set of 44 tasks it was 36.4% versus 27.3%. Same
direction, much lower ceiling.

We would hold our own numbers to exactly that. So we will.

Why we cannot quote ours

We had this result. On our own harness, with the verification layer switched off, correctness fell
from 100% to 75%. With the guards off as well it halved to 50%. Small n, our harness, not a public
benchmark, and we say that every time we cite it.

The bigger claim was a lift figure: a cheap model alone, then the same model inside our cascade. We
published it. Then we audited our own truth layer and found that our two baselines for the same
quantity, the same cheap model alone on the same benchmark, disagreed by 3.1 points. One page computed
the lift from one baseline. Another page computed it from the other. Each traced back to a real
measurement, which is exactly why nobody noticed.

Both lift figures are now removed from every public surface, and they stay removed until one baseline
is re-run.

So the honest position is this. Microsoft has a clean, large, well-controlled measurement of the thing
we believe. We have an unresolved 3.1 point disagreement with ourselves about ours. Their result does
not repair ours. Being right is not the same as having measured it.

The part we did measure, and it points the same way

There is one result we can put next to theirs, because it survived a second independent run and an
adversarial pass.

We ran three models over the same 160 HumanEval+ problems, same prompts, same scorer. They finished
92.5%, 91.2% and 90.0%. A 2.5 point spread that reads as interchangeable.

Partition by task instead of by column total and the picture changes. 83% of the set is uncontested,
all three models get it right, and the models disagree on roughly 15 to 18 percent of the work across
two runs. The identity of the contested problems reproduces: 17 tasks are contested in both runs,
against 3.6 expected by chance.

Which problems they split on is a property of the models. Which model scores highest is a property of
the run.

Put that beside Microsoft's table and you get one argument instead of two. They measured that the
workflow around the model carries most of the reachable gain. We measured that the ranking you would
use to pick the model is mostly noise. Both point at the same place. The leverage is not in the choice
of model.

What we are taking from it

One technique, immediately.

Before their agent declares a task complete, it performs lightweight mutation testing. It considers
small changes to the code that should make the tests fail. Not a full mutation framework. A cheap
heuristic that catches an assertion which cannot fail.

We know this rule. We wrote it down after a scorer of ours passed its positive control perfectly while
being structurally incapable of returning a negative. We wrote it down again when a guard printed OK
at every boot and then fired on zero of seven deliberately injected defects. We apply it from memory,
and memory is not a source. They made it a step in the loop.

So we built it. Every guard in our tree now declares, in its own header, an injected defect that must
make it exit non-zero, and a runner executes each one in a fresh directory and reports whether the
guard can still say no. Three grades: PROVEN, BROKEN, and UNPROVEN for the ones that have not declared
a control yet. UNPROVEN is a counted warning rather than a failure, because a gate that is red on its
first day gets overridden reflexively, and a reflexive override has stopped being a signal.

We watched it fail before we trusted it. A stub guard that always exits zero grades BROKEN.

Then we pointed it at the guard from the story above, the one that had been printing OK at every boot.
We injected a wrong percentage where a real measured one belongs. Exit 0. We injected a cost multiple
we had already struck from our own public copy. Exit 0. A third run, with the correct value in place,
confirmed the guard really was reading two live surfaces against the real ledger, so a trivial pass
could not be mistaken for a catch.

It is now marked BROKEN in our build output, by its own declaration, with a note saying not to delete
the control to make the board green.

Current score: 2 proven, 1 broken, 34 unproven, out of 37. The unproven number is the honest one, and
it is the one we expect to be asked about.