In a public Favur Evals run on October 3, an agent team shipped a small Java desktop app with 38 of 38 tests passing and a clean, reproducible build. A reviewer then tried to launch it the way a user would, with java -jar. It failed. The jar had no Main-Class, the line in the jar's manifest that tells Java which class to start, so the build that passed every test could not be opened.
That reviewer was one of the AI critics in a review loop that now ends every phase of a Favur run. Each critic starts cold, with no access to the build's plan or the team's own account of what it finished, and it may only score what it can prove. The pattern is the Gauntlet Loop, which Matt Shumer wrote up in July. We wrote it into our harness in September and spent the month fixing it. In October it ran on 74 public runs, and every critic's score, reason and evidence is on the board.
Passing tests are a claim about the tests
Tests check what the agents decided to check. On the Java run they checked the app's logic and passed, and none of them checked whether the packaged app started. The critic was a desktop application engineer, and starting the app is the first thing that specialty does. Its score for packaging was 4 out of 10. After one round of fixes the jar had a Main-Class, the suite had grown from 38 tests to 94, and the score rose to 6.
The same thing happened on other runs in other ways. On a Kimi K2.6 run in Go on October 9, the spec asked for a large stationary circle with a small circle orbiting it. The first build showed two circles of the same size, both moving, with no label. A critic reviewing as a first-time user scored comprehensibility 2 out of 10 and wrote that there was "no stationary circle, no size differentiation, and no direction label." After the fix the circles were distinct, and the score rose to 6.
On a Gemini 3.8 Flash run on October 4, the first build worked, but the only way to find the control that reverses the orbit was to press keys until something happened. The first-time-user critic scored time to first success 4. The fix put the controls on screen, and the score rose to 8.
None of these are failures a test suite reports, because the suite was written by the same team that missed them.
The critic has to run the thing
Every one of those catches came from a critic doing what a user does. That is the rule doing the work. In our harness a critic's score counts only with evidence attached, which means a file and line, a screenshot of the running app, or the output of a command the critic ran. A score with no evidence is rejected before it is saved. When the evidence cannot be had, because the app will not open or the window cannot be captured, that part is marked unscorable and reported. It is never guessed from the source.
The proof rule is what turns "check the app" into something a critic cannot skip. A reviewer allowed to score from reading code would have read a correct-looking build file and moved on. A reviewer that must attach the output of java -jar finds out the jar does not start.
The other rules point the same way. Each critic is a specialist with its own bar, written down before it opens the code. The lowest score on the panel decides whether the phase passes, so one critic who found a hole is never outvoted by two who looked elsewhere. And every round after a fix gets new critics who have never seen the build, the last scores or the list of fixes, so a fix is judged by someone with no reason to believe it worked.
What a month of runs shows
The catches above are single runs. The board lets us look across them. We matched October runs that used the loop against runs with the same spec, the same language and the same models that did not. Six sets matched.
| What the board measures | Runs with the loop, against matched runs without it |
|---|---|
| How complete the shipped project is | more complete in 4 of 6 sets |
| Wall-clock time | slower in 5 of 6; the median set took about 3.4 times as long |
| Tokens | more in 5 of 6; the median set used about 1.9 times as many |
The loop makes a run more complete and makes it take longer. That is the trade. The spread is wide. The Muse Spark 1.3 set in C# took about the same time with and without the loop, and the Kimi K2.6 set took more than six times as long. One set is an outlier, its run without the loop done in 19 minutes, and leaving it out moves the median only to 3.3.
The sets are small, mostly one or two runs a side, and nearly every run without the loop predates October's harness, so a gap between the sides is not only the loop. Read the table as direction, not size.
Inside the loop the signal is cleaner, because each review is compared with the one before it on the same build. In October a build was reviewed again after fixes 93 times. The lowest critic score rose in 48 of those, held in 18 and fell in 27. Because every round starts with new critics, a fall can mean a fresh reviewer found something the last one never looked at, which is the point of starting cold.
Why the single score drops
The board's overall score blends eight subjects, and it weights speed and effort at 13 points each and completeness at 6. A loop that trades time for completeness therefore lowers the overall number while it raises the subject it is built to raise. On the six matched sets, runs with the loop scored lower overall in five of them.
That is a reason to read the subjects, not the total. If what you need from an agent team is the most complete project it can ship, the overall score is the wrong number to sort by. The completeness column and the critics' findings are the right ones, and every finding links to its evidence.
What it costs to run
Every critic is a full agent session that launches the app, runs commands and captures the window. A panel of three critics over three rounds adds nine sessions to the build. But the critics are not where most of the time goes. A finding sends the team back to fix the build, rebuild it and wait for a new panel, and on the Java run that second pass added a Main-Class line and grew the suite from 38 tests to 94. Each catch buys a round of real work.
If you try this, the panel size and the moment you run it are the two dials. A small panel at the end of a phase catches the app that does not start without reviewing every change, and a phase can stop early once the lowest score stops moving.
Every score, reason and piece of evidence from the runs above is on the public board. You can open the Java run with the jar that would not start, the Kimi K2.6 run with the two same-size circles, and the Gemini 3.8 Flash run with the hidden control. Favur itself is closed-source and invite-only, and the Favur site has a real run you can drive.
Top comments (0)