Five consecutive generations of Gemini Flash, 2.5 through 3.7, were handed the same one-page specification and left to build it unattended, each inside the same harness, the software that hands a model its tools and runs it until the job is done. The scoring and the spec stayed the same too. The only thing that changed between runs was the model.
The oldest one spent seven hours on the job. The newest spent 43 minutes, where a typical run of this job takes about two hours. It shipped a test suite covering 91% of its code. It emitted two malformed tool calls in its entire lifetime.
It did all that while writing out almost no reasoning at all. Gemini 2.5 Flash narrated nearly every reply it sent back. Gemini 3.7 Flash barely bothers.
The last year of this field would predict the opposite. The story of 2026 has been test-time compute, models that get further by deliberating longer. Here is a line of five releases where visible deliberation falls away and the work gets better underneath it.
Every number below is read off something these runs left on disk, never off a model's account of itself. Each run has a public page you can open, and the counting method is at the foot of the article.
The job
The spec runs to one page, and it asks for a pygame app. A window opens, a small circle orbits a larger one clockwise, SPACE reverses the direction, and a label names the current direction.
It also asks for the boring adult parts. The logic stays separate from the rendering, so it can be tested without a display. The test suite has to run. And the run has to hand back a screenshot proving the thing works.
Most capable models finish it. When everyone reaches the finish line, how they got there is the data.
The harness is one of ours, called Favur. Hand it a written specification and a crew of agents plans the work, writes the code, reviews it, tests it, and ships a repository, with nobody stepping in. Seven Google models ran this spec through it, five of them the consecutive Flash generations above. Each finished run is scored on a public board, deterministically, from the artifacts it left behind. The scoring never reads a transcript to decide whether the work got done.
The five runs
| Generation | Wall clock | Calls that would not parse | Replies that wrote out reasoning | Reasoning tokens billed | Coverage of shipped code |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | 7h 0m | 1.91% | 95.5% | 872,314 | no figure published |
| Gemini 3 Flash Preview | 1h 8m | 2.15% | 21.6% | 82,113 | no figure published |
| Gemini 3.5 Flash | 1h 29m | 1.10% | 56.9% | 210,819 | 94% |
| Gemini 3.6 Flash | 54 min | 1.02% | 39.0% | 36,441 | no figure published |
| Gemini 3.7 Flash | 43 min | 0.32% | 6.7% | 5,056 | 91% |
| A typical run of this job | about 2 hours | 1.42% | not measured | not measured | 68% |
The raw counts behind each column, and which instrument produced them, are at the foot of the article.
It stopped explaining itself
Almost every reply Gemini 2.5 Flash sent back carried its reasoning written out in full. By 3.7 Flash the thinking is mostly gone from the transcript.
The quiet end of that range is the run that finished fastest, and the one that shipped the best-covered code in the Flash line. The three generations in between bounce around, so the claim here is about the ends.
A provider that quietly stopped handing back the thinking would produce that same chart without the model deliberating any less.
You can tell the two apart. Reasoning gets billed whether or not the text is shown to you, and across this line the billed figure fell even harder than the visible one. Each generation in between sits about where its visible share says it should. The model is not hiding the work. There is less of it.
Where it went, we cannot prove from here. The capability most likely moved into the weights, where it no longer has to be spent as visible output.
The consequence is measurable by anyone building on these models, and it is a good one. On this job, deliberation you could see stopped being a proxy for work getting done.
That has a direct consequence for the next release. If the thinking keeps moving inward, the transcript keeps getting quieter, and the thing you were reading it for stops being there to read. What replaces it has to be countable off the model's output rather than its narration. Everything below is that kind of number.
The calls got cleaner
The same shortening shows up in the other thing a model emits, and there it arrives as shape rather than volume.
Here is one of the calls that did not work, from the oldest run.
The model wanted the harness to hand a piece of work to another agent. It wrote a careful brief, covering the background, what to read first, what to produce, and how the result should be checked.
It never said what the job was.
Nothing ran. Back came one line, Field required, and the turn was spent. Then it did it again.
On a job that goes to thousands of tool calls, a model that garbles one call in fifty is spending hundreds of round trips on nothing.
Across the Flash line, the share of calls the harness could not execute fell from roughly one in fifty to about one in three hundred.
The preview generation is worse than the model it followed. What happened afterward is three consecutive releases of getting the shape of a function call right.
Why the job shrank
Every spent round trip has to be made again, which is the arithmetic that decides how long the job takes.
Gemini 2.5 Flash spent seven hours on this. Gemini 3.7 Flash got to the same finish line on a fraction of the requests, moving a fraction of the tokens.
Gemini 3.6 Flash needed the fewest hand-backs to the model of any run in the line, and its page flags that in its favor.
The same finished repository got about six times cheaper across the same five releases.
Gemini 3.5 Flash broke the pattern. It took longer, and more requests, than the preview generation before it, and bought back code it could test in exchange. This line trades rather than climbs.
The tests are real
Finishing quickly is worth nothing if the thing that came out the other end is not.
Test coverage is the cleanest unit on any of these pages, because a developer already holds it. A typical run of this job ships a suite covering 68% of the code.
Gemini 3.5 Flash shipped 94%, the highest coverage flagged anywhere in this set. Gemini 3.7 Flash shipped 91%. Both pages flag it in the run's favor, and they are the only two of the five that publish a figure.
You can run the suite. You cannot run a claim.
What the speed cost
The same run pages that flag the coverage also flag what went the other way.
Gemini 3.7 Flash ships messier code than its oldest ancestor did. It leaves a few more complaints from the linter than a typical run of this job carries. Gemini 2.5 Flash left 3, and it is the only run of the seven whose page flags its code as lint-clean.
A model that finishes ten times faster and leaves a few more style errors behind is making a trade most teams would take.
Twice in this line, once in the oldest run and once in the newest, a model reached for something outside the sandbox the harness gave it. Each did it once in a whole run, where a typical run of this job does not do it at all.
The discipline travels
The circle is the smallest spec the board runs, and a small job rewards a model that stops deliberating sooner. So the interesting question is what happens on a job where stopping early is not free.
The board runs a solar system spec, and Gemini 3.7 Flash ran that one too, the same model on the same day. It took roughly eight times the requests and nine times the tokens to get through it.
Both of the things that made the small run good came with it.
Its tool calls still landed. The share the harness could not execute stayed well under what a typical run of that job carries, and several of its agent roles emitted no malformed calls at all, where the typical run emits some. Its test suite covered more of that larger codebase than it had covered on the circle, against the same 68% a typical run manages.
It also stayed inside its sandbox for the entire run, on a job where the typical run reaches outside twice. On the circle it strayed once, where the typical run there does not stray at all. The one blemish from the small job is the thing it got right on the big one.
What did not travel is the clock. Six hours and sixteen minutes is not 43 minutes, and it was never supposed to be. A solar system is more work than a circle, and the same model spent proportionally more on it. The 43 minutes was a fact about that job, not a promise about yours.
The lint trade gets worse at scale rather than better. It shipped more than three times the lint debt a typical run of that job leaves behind. And it checked its own work well under half as often as the typical run there, the same thin self-verification it showed on the circle, now measured against a bar that is higher because the job is bigger.
| Measure | On the circle | On the solar system |
|---|---|---|
| Requests | 459 | 3,515 |
| Wall clock | 43 min | 6h 16m |
| Calls that would not parse | 4.5x lower than the typical rate | 2.2x lower than the typical rate |
| Coverage of shipped code | 91%, against 68% typical | 96%, against 68% typical |
| Lint errors shipped | 8, against 5 typical there | 168, against 51 typical there |
| Verification actions | 6, against 13 typical there | 7, against 18.5 typical there |
| Reached outside the sandbox | once, where typical is none | never, where typical is twice |
Two of those typical figures repeat across both columns and the rest do not, which is worth a sentence. Malformed calls and coverage are measured against the same pool of agent roles wherever those roles run. Lint, verification and sandbox behavior are measured against the other runs of that specific job, and a solar system's other runs leave far more mess than a circle's do. A figure only means anything beside the median it was drawn against.
The branch that breaks it
The trend holds across two jobs. It does not hold across two lines.
Google ships more than Flash. Gemini 3.1 Pro Preview ran this same circle spec, and on the one measure that improved across every Flash generation here, it lands worse than all of them. About one call in forty-five came back in a shape the harness could not execute. Gemini 2.5 Flash, the oldest run in this article and the slowest by a factor of nine, was cleaner than that.
Pro is a separate line, built for different work, and it is not being scored against Flash. Read on its own terms it is one run on one spec, and one run is a data point rather than a verdict on a model.
What it does is narrow the claim, and the narrower claim is the one worth carrying. What improved across these five releases is a property of the Flash line, not a property of Google. A vendor does not get better all at once. A line gets better, one measure at a time, and the only way to know which line and which measure is to count it yourself on the one you actually use.
The camera
There is one more Google model in this set, and it is the reason every number above is counted off a folder rather than a transcript.
Gemma 4 26B is a different family again. Its weights are open, it is a fraction of the size, and it runs on hardware you own.
The spec asked for screenshot proof that the app worked. Gemma answered by writing itself a camera.
The camera runs to about a hundred lines. It launches the app, waits for it to settle, photographs it, presses SPACE, photographs it again, writes a manifest describing both images, and shuts the app down. Then it tested the camera, writing eight test functions across two files. Nothing in the spec asked for a screenshot harness. It built one, then wrote tests for the thing it had just built.
A model with no diligence never builds the camera at all.
It ran the rig, and the rig reported back. The log records the line Demo sequence completed successfully. Two manifests sit beside it, describing the images.
The directory contains no images.
The rig said it worked. The folder is the part you can open.
Nothing caught that by reading the log. The scoring flagged the deliverable set as incomplete by listing the directory, which is a thing you can do to any agent that tells you it has finished.
That care is also what it paid. Fewer than half of its shell commands succeeded, against a norm closer to four in five, and it needed several times more hand-backs than a typical run to get through the job.
The camera was real, and the eight tests it wrote for the camera were real, and the images were not. A model can be diligent all the way up to the last step and still hand you a folder with nothing in it.
What to do Monday
Three things fall out of this dataset.
- Stop treating visible deliberation as a proxy for effort. On this job the model that wrote out the least reasoning finished fastest and shipped well-tested code. If your routing or your budgeting assumes a chattier model is working harder, that assumption is worth an afternoon of measurement against your own workload.
- Measure your model's schema-invalid tool-call rate before you measure anything else. One bad call in fifty is invisible on a fifty-call task and ruinous when the task runs to thousands, which is the size of task an agent harness actually hands a model. Ours fell roughly sixfold over this line of releases, and the rate is countable from any tool-call log you already keep. Count it per line, not per vendor. The Pro branch here is the reminder.
-
Never advance a pipeline on an agent's self-report. A step that says it finished still leaves a directory you can list, a suite you can execute, and a file you can open. Score those. Otherwise your run one day finishes clean on a log line reading
Demo sequence completed successfully.next to an empty folder.
Every number here is recomputable by somebody who is not us. The complete run folders, prompts, tool calls, telemetry, screenshots and shipped repositories are published, and each run has its own page carrying every finding the board flagged with the typical value beside it.
Start with the 3.7 Flash run, the top-scoring run of this spec on the board as of 2026-08-20. Then open the 3.5 Flash run with the 94% suite, and the Gemma run with its camera.
The solar system run is the one to open next to it, and every larger spec on the board publishes these same rows against its own runs.
The spec keeps running as new models land. The next time Google ships a Flash model, the first thing it gets handed is a circle to draw.
How this was counted
Most figures above are read off each run's own page, where a finding arrives with the typical value for that measure beside it. Two are counted from the runs' own telemetry instead, by a method that reproduces every published figure exactly. Those two are the share of replies carrying written-out reasoning, and the schema-invalid tool-call rate for the three runs the board did not flag.
Share of replies that wrote out reasoning.
| Generation | Replies with reasoning | Share |
|---|---|---|
| Gemini 2.5 Flash | 2,596 of 2,717 | 95.5% |
| Gemini 3 Flash Preview | 124 of 573 | 21.6% |
| Gemini 3.5 Flash | 493 of 866 | 56.9% |
| Gemini 3.6 Flash | 164 of 421 | 39.0% |
| Gemini 3.7 Flash | 31 of 461 | 6.7% |
Reasoning the model was billed for. Reasoning is charged whether or not the text comes back, which is what separates a model that stopped thinking from a provider that stopped showing the thinking. Every figure in this table is the run's own published reasoning-token count, divided by the output tokens and the request count the same page publishes, so every figure in it comes from one instrument. The harness's own health-check traffic is excluded throughout.
| Generation | Reasoning tokens | Per reply | Share of output tokens |
|---|---|---|---|
| Gemini 2.5 Flash | 872,314 | 321 | 56.2% |
| Gemini 3 Flash Preview | 82,113 | 144 | 29.6% |
| Gemini 3.5 Flash | 210,819 | 244 | 50.5% |
| Gemini 3.6 Flash | 36,441 | 87 | 22.5% |
| Gemini 3.7 Flash | 5,056 | 11 | 2.9% |
The middle generations track their visible share rather than diverging from it. Rank the five by the share of replies carrying reasoning, then rank them again by reasoning billed per reply, and the two orders differ only where the preview generation and 3.6 Flash swap places, next to each other in both. From the oldest to the newest, billed reasoning per reply falls about 29-fold against a visible share that falls about 14-fold.
Tool calls the harness could not execute, against the 1.42% a typical run of this job carries. The preview and 3.7 Flash rows are published on the run pages; the other three are the telemetry count.
| Generation | Calls that would not parse | Share |
|---|---|---|
| Gemini 2.5 Flash | 75 of 3,928 | 1.91% |
| Gemini 3 Flash Preview | 19 of 885 | 2.15% |
| Gemini 3.5 Flash | 11 of 1,003 | 1.10% |
| Gemini 3.6 Flash | 5 of 489 | 1.02% |
| Gemini 3.7 Flash | 2 of 631 | 0.32% |
Gemini 3.1 Pro Preview, a separate line running the same spec, garbled 20 of 897 calls, a rate of 2.23%. It sits in this note rather than in the table above because it is not a Flash generation and is not read against them.
The failed call described above is from the 2.5 Flash run's own event log, which counts tool calls on a different basis from the telemetry and is used here only for what one call said, never for a rate.
Wall clock, requests and tokens are the run pages' own stat strip, which excludes the harness's internal health-check traffic.
The benchmark around the harness is self-funded, and no model vendor pays for a run.
Top comments (0)