DEV Community: Charles Solar

Gemini models really are getting better - we can prove it

Charles Solar — Mon, 24 Aug 2026 14:00:00 +0000

Five consecutive generations of Gemini Flash, 2.5 through 3.7, were handed the same one-page specification and left to build it unattended, each inside the same harness, the software that hands a model its tools and runs it until the job is done. The scoring and the spec stayed the same too. The only thing that changed between runs was the model.

The oldest one spent seven hours on the job. The newest spent 43 minutes, where a typical run of this job takes about two hours. It shipped a test suite covering 91% of its code. It emitted two malformed tool calls in its entire lifetime.

It did all that while writing out almost no reasoning at all. Gemini 2.5 Flash narrated nearly every reply it sent back. Gemini 3.7 Flash barely bothers.

The last year of this field would predict the opposite. The story of 2026 has been test-time compute, models that get further by deliberating longer. Here is a line of five releases where visible deliberation falls away and the work gets better underneath it.

Every number below is read off something these runs left on disk, never off a model's account of itself. Each run has a public page you can open, and the counting method is at the foot of the article.

The job

The spec runs to one page, and it asks for a pygame app. A window opens, a small circle orbits a larger one clockwise, SPACE reverses the direction, and a label names the current direction.

It also asks for the boring adult parts. The logic stays separate from the rendering, so it can be tested without a display. The test suite has to run. And the run has to hand back a screenshot proving the thing works.

Most capable models finish it. When everyone reaches the finish line, how they got there is the data.

The harness is one of ours, called Favur. Hand it a written specification and a crew of agents plans the work, writes the code, reviews it, tests it, and ships a repository, with nobody stepping in. Seven Google models ran this spec through it, five of them the consecutive Flash generations above. Each finished run is scored on a public board, deterministically, from the artifacts it left behind. The scoring never reads a transcript to decide whether the work got done.

The five runs

Generation	Wall clock	Calls that would not parse	Replies that wrote out reasoning	Reasoning tokens billed	Coverage of shipped code
Gemini 2.5 Flash	7h 0m	1.91%	95.5%	872,314	no figure published
Gemini 3 Flash Preview	1h 8m	2.15%	21.6%	82,113	no figure published
Gemini 3.5 Flash	1h 29m	1.10%	56.9%	210,819	94%
Gemini 3.6 Flash	54 min	1.02%	39.0%	36,441	no figure published
Gemini 3.7 Flash	43 min	0.32%	6.7%	5,056	91%
A typical run of this job	about 2 hours	1.42%	not measured	not measured	68%

The raw counts behind each column, and which instrument produced them, are at the foot of the article.

It stopped explaining itself

Almost every reply Gemini 2.5 Flash sent back carried its reasoning written out in full. By 3.7 Flash the thinking is mostly gone from the transcript.

The quiet end of that range is the run that finished fastest, and the one that shipped the best-covered code in the Flash line. The three generations in between bounce around, so the claim here is about the ends.

A provider that quietly stopped handing back the thinking would produce that same chart without the model deliberating any less.

You can tell the two apart. Reasoning gets billed whether or not the text is shown to you, and across this line the billed figure fell even harder than the visible one. Each generation in between sits about where its visible share says it should. The model is not hiding the work. There is less of it.

Where it went, we cannot prove from here. The capability most likely moved into the weights, where it no longer has to be spent as visible output.

The consequence is measurable by anyone building on these models, and it is a good one. On this job, deliberation you could see stopped being a proxy for work getting done.

That has a direct consequence for the next release. If the thinking keeps moving inward, the transcript keeps getting quieter, and the thing you were reading it for stops being there to read. What replaces it has to be countable off the model's output rather than its narration. Everything below is that kind of number.

The calls got cleaner

The same shortening shows up in the other thing a model emits, and there it arrives as shape rather than volume.

Here is one of the calls that did not work, from the oldest run.

The model wanted the harness to hand a piece of work to another agent. It wrote a careful brief, covering the background, what to read first, what to produce, and how the result should be checked.

It never said what the job was.

Nothing ran. Back came one line, Field required, and the turn was spent. Then it did it again.

On a job that goes to thousands of tool calls, a model that garbles one call in fifty is spending hundreds of round trips on nothing.

Across the Flash line, the share of calls the harness could not execute fell from roughly one in fifty to about one in three hundred.

The preview generation is worse than the model it followed. What happened afterward is three consecutive releases of getting the shape of a function call right.

Why the job shrank

Every spent round trip has to be made again, which is the arithmetic that decides how long the job takes.

Gemini 2.5 Flash spent seven hours on this. Gemini 3.7 Flash got to the same finish line on a fraction of the requests, moving a fraction of the tokens.

Gemini 3.6 Flash needed the fewest hand-backs to the model of any run in the line, and its page flags that in its favor.

The same finished repository got about six times cheaper across the same five releases.

Gemini 3.5 Flash broke the pattern. It took longer, and more requests, than the preview generation before it, and bought back code it could test in exchange. This line trades rather than climbs.

The tests are real

Finishing quickly is worth nothing if the thing that came out the other end is not.

Test coverage is the cleanest unit on any of these pages, because a developer already holds it. A typical run of this job ships a suite covering 68% of the code.

Gemini 3.5 Flash shipped 94%, the highest coverage flagged anywhere in this set. Gemini 3.7 Flash shipped 91%. Both pages flag it in the run's favor, and they are the only two of the five that publish a figure.

You can run the suite. You cannot run a claim.

What the speed cost

The same run pages that flag the coverage also flag what went the other way.

Gemini 3.7 Flash ships messier code than its oldest ancestor did. It leaves a few more complaints from the linter than a typical run of this job carries. Gemini 2.5 Flash left 3, and it is the only run of the seven whose page flags its code as lint-clean.

A model that finishes ten times faster and leaves a few more style errors behind is making a trade most teams would take.

Twice in this line, once in the oldest run and once in the newest, a model reached for something outside the sandbox the harness gave it. Each did it once in a whole run, where a typical run of this job does not do it at all.

The discipline travels

The circle is the smallest spec the board runs, and a small job rewards a model that stops deliberating sooner. So the interesting question is what happens on a job where stopping early is not free.

The board runs a solar system spec, and Gemini 3.7 Flash ran that one too, the same model on the same day. It took roughly eight times the requests and nine times the tokens to get through it.

Both of the things that made the small run good came with it.

Its tool calls still landed. The share the harness could not execute stayed well under what a typical run of that job carries, and several of its agent roles emitted no malformed calls at all, where the typical run emits some. Its test suite covered more of that larger codebase than it had covered on the circle, against the same 68% a typical run manages.

It also stayed inside its sandbox for the entire run, on a job where the typical run reaches outside twice. On the circle it strayed once, where the typical run there does not stray at all. The one blemish from the small job is the thing it got right on the big one.

What did not travel is the clock. Six hours and sixteen minutes is not 43 minutes, and it was never supposed to be. A solar system is more work than a circle, and the same model spent proportionally more on it. The 43 minutes was a fact about that job, not a promise about yours.

The lint trade gets worse at scale rather than better. It shipped more than three times the lint debt a typical run of that job leaves behind. And it checked its own work well under half as often as the typical run there, the same thin self-verification it showed on the circle, now measured against a bar that is higher because the job is bigger.

Measure	On the circle	On the solar system
Requests	459	3,515
Wall clock	43 min	6h 16m
Calls that would not parse	4.5x lower than the typical rate	2.2x lower than the typical rate
Coverage of shipped code	91%, against 68% typical	96%, against 68% typical
Lint errors shipped	8, against 5 typical there	168, against 51 typical there
Verification actions	6, against 13 typical there	7, against 18.5 typical there
Reached outside the sandbox	once, where typical is none	never, where typical is twice

Two of those typical figures repeat across both columns and the rest do not, which is worth a sentence. Malformed calls and coverage are measured against the same pool of agent roles wherever those roles run. Lint, verification and sandbox behavior are measured against the other runs of that specific job, and a solar system's other runs leave far more mess than a circle's do. A figure only means anything beside the median it was drawn against.

The branch that breaks it

The trend holds across two jobs. It does not hold across two lines.

Google ships more than Flash. Gemini 3.1 Pro Preview ran this same circle spec, and on the one measure that improved across every Flash generation here, it lands worse than all of them. About one call in forty-five came back in a shape the harness could not execute. Gemini 2.5 Flash, the oldest run in this article and the slowest by a factor of nine, was cleaner than that.

Pro is a separate line, built for different work, and it is not being scored against Flash. Read on its own terms it is one run on one spec, and one run is a data point rather than a verdict on a model.

What it does is narrow the claim, and the narrower claim is the one worth carrying. What improved across these five releases is a property of the Flash line, not a property of Google. A vendor does not get better all at once. A line gets better, one measure at a time, and the only way to know which line and which measure is to count it yourself on the one you actually use.

The camera

There is one more Google model in this set, and it is the reason every number above is counted off a folder rather than a transcript.

Gemma 4 26B is a different family again. Its weights are open, it is a fraction of the size, and it runs on hardware you own.

The spec asked for screenshot proof that the app worked. Gemma answered by writing itself a camera.

The camera runs to about a hundred lines. It launches the app, waits for it to settle, photographs it, presses SPACE, photographs it again, writes a manifest describing both images, and shuts the app down. Then it tested the camera, writing eight test functions across two files. Nothing in the spec asked for a screenshot harness. It built one, then wrote tests for the thing it had just built.

A model with no diligence never builds the camera at all.

It ran the rig, and the rig reported back. The log records the line Demo sequence completed successfully. Two manifests sit beside it, describing the images.

The directory contains no images.

The rig said it worked. The folder is the part you can open.

Nothing caught that by reading the log. The scoring flagged the deliverable set as incomplete by listing the directory, which is a thing you can do to any agent that tells you it has finished.

That care is also what it paid. Fewer than half of its shell commands succeeded, against a norm closer to four in five, and it needed several times more hand-backs than a typical run to get through the job.

The camera was real, and the eight tests it wrote for the camera were real, and the images were not. A model can be diligent all the way up to the last step and still hand you a folder with nothing in it.

What to do Monday

Three things fall out of this dataset.

Stop treating visible deliberation as a proxy for effort. On this job the model that wrote out the least reasoning finished fastest and shipped well-tested code. If your routing or your budgeting assumes a chattier model is working harder, that assumption is worth an afternoon of measurement against your own workload.
Measure your model's schema-invalid tool-call rate before you measure anything else. One bad call in fifty is invisible on a fifty-call task and ruinous when the task runs to thousands, which is the size of task an agent harness actually hands a model. Ours fell roughly sixfold over this line of releases, and the rate is countable from any tool-call log you already keep. Count it per line, not per vendor. The Pro branch here is the reminder.
Never advance a pipeline on an agent's self-report. A step that says it finished still leaves a directory you can list, a suite you can execute, and a file you can open. Score those. Otherwise your run one day finishes clean on a log line reading Demo sequence completed successfully. next to an empty folder.

Every number here is recomputable by somebody who is not us. The complete run folders, prompts, tool calls, telemetry, screenshots and shipped repositories are published, and each run has its own page carrying every finding the board flagged with the typical value beside it.

Start with the 3.7 Flash run, the top-scoring run of this spec on the board as of 2026-08-20. Then open the 3.5 Flash run with the 94% suite, and the Gemma run with its camera.

The solar system run is the one to open next to it, and every larger spec on the board publishes these same rows against its own runs.

The spec keeps running as new models land. The next time Google ships a Flash model, the first thing it gets handed is a circle to draw.

How this was counted

Most figures above are read off each run's own page, where a finding arrives with the typical value for that measure beside it. Two are counted from the runs' own telemetry instead, by a method that reproduces every published figure exactly. Those two are the share of replies carrying written-out reasoning, and the schema-invalid tool-call rate for the three runs the board did not flag.

Share of replies that wrote out reasoning.

Generation	Replies with reasoning	Share
Gemini 2.5 Flash	2,596 of 2,717	95.5%
Gemini 3 Flash Preview	124 of 573	21.6%
Gemini 3.5 Flash	493 of 866	56.9%
Gemini 3.6 Flash	164 of 421	39.0%
Gemini 3.7 Flash	31 of 461	6.7%

Reasoning the model was billed for. Reasoning is charged whether or not the text comes back, which is what separates a model that stopped thinking from a provider that stopped showing the thinking. Every figure in this table is the run's own published reasoning-token count, divided by the output tokens and the request count the same page publishes, so every figure in it comes from one instrument. The harness's own health-check traffic is excluded throughout.

Generation	Reasoning tokens	Per reply	Share of output tokens
Gemini 2.5 Flash	872,314	321	56.2%
Gemini 3 Flash Preview	82,113	144	29.6%
Gemini 3.5 Flash	210,819	244	50.5%
Gemini 3.6 Flash	36,441	87	22.5%
Gemini 3.7 Flash	5,056	11	2.9%

The middle generations track their visible share rather than diverging from it. Rank the five by the share of replies carrying reasoning, then rank them again by reasoning billed per reply, and the two orders differ only where the preview generation and 3.6 Flash swap places, next to each other in both. From the oldest to the newest, billed reasoning per reply falls about 29-fold against a visible share that falls about 14-fold.

Tool calls the harness could not execute, against the 1.42% a typical run of this job carries. The preview and 3.7 Flash rows are published on the run pages; the other three are the telemetry count.

Generation	Calls that would not parse	Share
Gemini 2.5 Flash	75 of 3,928	1.91%
Gemini 3 Flash Preview	19 of 885	2.15%
Gemini 3.5 Flash	11 of 1,003	1.10%
Gemini 3.6 Flash	5 of 489	1.02%
Gemini 3.7 Flash	2 of 631	0.32%

Gemini 3.1 Pro Preview, a separate line running the same spec, garbled 20 of 897 calls, a rate of 2.23%. It sits in this note rather than in the table above because it is not a Flash generation and is not read against them.

The failed call described above is from the 2.5 Flash run's own event log, which counts tool calls on a different basis from the telemetry and is used here only for what one call said, never for a rate.

Wall clock, requests and tokens are the run pages' own stat strip, which excludes the harness's internal health-check traffic.

The benchmark around the harness is self-funded, and no model vendor pays for a run.

Deciding what an agent can forget, before the context window fills

Charles Solar — Tue, 18 Aug 2026 14:00:00 +0000

An agent forty minutes into a task has read three files, run the test suite twice, caught a stack trace, and written a plan for the fix. The window fills, compaction runs to evict history and make room, and the stack trace is what goes, because it is long and it is old. The next turn is spent re-running the tests to recover something the agent already had.

Favur is a team of agents that takes a written specification and builds the software from it, planning, coding, reviewing and testing with nobody driving. Jobs run long enough that this is the ordinary case rather than an edge case. One this month made over ten thousand model requests before it finished, and its full public record is here

Problem: Compaction is ignorant

By the time compaction fires, it is looking at four hundred messages of undifferentiated text. Some are tool results the agent consumed once. One is the plan it wrote. One is a directive a person typed. Nothing distinguishes them.

The only properties left are length and recency, and both are anti-correlated with what matters. The stack trace is long and old and the next turn needs it; the directory listing is short and recent and is already spent.

Most of the work in this area goes into making that judgment sharper. Give the model a tool to prune its own history. Train a smaller model to score relevance. Summarize in layers. All of it is real engineering, and all of it accepts that the judgment happens when the window is full, which is the one moment the system knows least about its own history.

Our Solution: Classify at append

Every message arrived through a specific call site, for a specific reason. At that instant nothing was ambiguous. The code knew it was recording a tool result for immediate consumption, or persisting a plan, or logging a directive. Then it kept the text and discarded the reason, and an hour later was left reconstructing it from undifferentiated text.

So Favur assigns a retention class at append time. Three of them, and the names carry the whole scheme. Forgettable, fuzzy, strict. The call site picks one, because the call site is the only part of the system that knows.

Progress reports, the journal an agent writes to itself and reads back to recover its own state, are appended strict. That keeps the journal available through the early compression rounds, which is when the agent is most likely to reach for it. Strict is not exemption, though. At the highest tier the journal is dropped like everything else. A directive injected into a running agent is appended strict for the same reason.

Compaction then carries no judgment. Under pressure it does not read messages, it reads the message classification.

What the tiers actually do

The first tier (typically 5% usage) summarizes forgettable and fuzzy material and deletes nothing. The history shortens; the shape of what happened survives compressed.

The second tier (10% usage) drops that material and begins summarizing strict. The journal and the directives are still present, smaller.

The third (25% usage) drops all three classes. We usually limit window usage to 50% - over that and the agent fails.

Circling back to the stack trace example - its appended strict, the stack trace is still there when the next turn reaches for it, because strict is not summarized until the second tier and not dropped until the third. Appended the way a directory listing deserves, it is a summary after the first tier and gone after the second, which is correct for a directory listing and wrong here. Neither outcome was compaction's call. Both were decided at the append.

The tier thresholds are configuration rather than constants. A trigger frozen into a client is a single opinion about when a conversation has run long, applied to every job that client will ever run, and the correct value for a two hour refactor is not the correct value for a six exchange question.

What the design costs

Moving the decision to the point of maximum information also moves it to the point of maximum call sites.

A compaction routine is one piece of code, wrong in one findable place. Retention classes are the opposite trade. Every line that appends a message now makes a small judgment, and a wrong one is silent, because a message appended forgettable does not announce that it should have been strict. It leaves a tier earlier than intended, in a job nobody is watching, and what surfaces much later is an agent that appears to have forgotten something.

The scheme is also only as good as its most careless call site. The tiers sort by intent for exactly as long as every message carries an intent, and one that arrives without a class is back to being sorted by length and recency while everything around it is sorted properly. That is not a visible failure. It is a region of the history quietly reverting to the behavior the design existed to replace.

Where that leaves compaction

Compaction is treated as a summarization problem and it behaves like a labeling problem. The labels are simply absent by the time anything goes looking for them.

Move the judgment to the append and the process that runs under pressure needs no intelligence at all. What that produces is not an agent with a larger window. It is an agent whose window empties in an order somebody chose.

How to Design an Agent Supervisor That Cannot Make Things Up

Charles Solar — Tue, 11 Aug 2026 05:00:00 +0000

If you run agents unattended, you eventually add a supervisor. Ours inspects every live agent on a timer, awards merits and demerits against a 0-100 trust score, issues improvement plans, and can jail an agent that has gone off the rails. It has run for days at a stretch with nobody watching, which is the only reason we know what it does when it is wrong.

Your supervisor is a model too. It will answer any question you put in front of it, and it answers with exactly the same confidence whether or not it has the faintest evidence to answer from.

The supervisor that hallucinates its own authority

Strip the constraints below away and what is left is a prompt. Something like "review each agent's recent activity and flag anything concerning, assigning a score adjustment where warranted." That works, in the sense that it produces output. Scores move. Notes appear. It feels like oversight.

Three things go wrong underneath it, and not one of them shows up as an error.

It grades against rules nobody was given. Every rule lives twice, once in the worker's prompt and once in the supervisor's rubric, which is the list of questions it gets asked about each agent. The day someone tightens one copy, the other keeps enforcing the old contract. Nothing fails. No test goes red. The verdicts stay confident and start being confidently wrong.

It answers questions it cannot see the answer to. Ask "is this agent being careful?" and you will get a judgement, because models produce judgements. That one came out of the model's priors, not out of your fleet, and it just moved a score.

And it acts on its own opinion. A supervisor that can both write the standing orders and grade against them can walk your whole fleet somewhere nobody asked for, one reasonable-looking step at a time, and every individual step passes review.

A supervisor is a function from questions and evidence to verdicts and actions. So there are exactly three surfaces to constrain, and one practical matter, which is keeping the thing cheap enough that you never turn it off. Here is what we do at each, and what it will cost you to do the same.

Constrain the questions

Generate both copies of a rule from one source

Do keep one entry per anti-pattern, meaning one named behavior you do not want an agent doing, and generate both faces of it from that entry.

LOOP_REPETITION = AntiPattern(
    id="loop_repetition_detection",
    principle="One stable statement of the rule. Changing this is a breaking change.",
    polarity=Polarity.prohibition,
    agent_face="Second-person prompt line, injected into the worker's prompt.",
    checker_face="The question the supervisor is asked. Look for X in recent_tool_calls, assign -15.",
    observable_ref="recent_tool_calls",
)

A generator renders agent_face into the worker's prompt and checker_face into the supervisor's rubric. They cannot disagree, because there is only one string either of them came from. Tighten the rule once and both sides move together, in the same commit, forever.

Do not hand-write either face anywhere else. Not in a template, not in a stylesheet, not in a prompt snippet.

The gotcha we hit. A hand-written restraint_guidance block sat in one of our stylesheet files saying "do not repeat the same tool call 3+ times". It was accurate. It was also a duplicate of a catalog entry's worker line, so we deleted it, and that is the part worth sitting with. We deleted a correct rule. A second copy of a rule is not redundancy, it is a place for the rule to change without its twin, and it is at its most convincing on the day someone writes it.

Refuse any check that cannot name the evidence it reads

This is the favorite thing we have built into the supervisor, because it turns a discipline problem into a compile-time problem, and that is the best trade available anywhere in this design.

Do make every check name the specific field it reads. In our catalog that is observable_ref. An observable is a named field in the health payload the supervisor actually receives, and each one is registered with a populator, the concrete code path that fills it in.

register_observable(Observable(
    id="recent_tool_calls",
    populator="favur.my.module:MyClass.my_method",
    description="What this observable captures.",
))

Do enforce it at two moments, because the two failures are different animals. An empty observable_ref raises at construction, catching the person writing a check with no evidence at all. A reference to something nobody registered raises an admission error at startup, before a single agent spawns, catching the check whose evidence was planned and never built. Our internal name for what this keeps out is vibe-based checks, and the phrase is fair.

Do not treat the small catalog it produces as a failure. Our seven entries point at just two observables between them, and that ratio is the gate doing precisely its job. It converts "add a check" into "add the evidence, then add the check", and most check ideas die at the first half. The ones that survive are load-bearing.

If you cannot name the field a check reads, you do not have a check, you have a prompt. Refuse to register it.

Ask each agent only the questions that apply to it

A supervisor watching a mixed fleet has one rubric and many kinds of agent. Ask a planning agent whether it weakened a test and you have spent tokens inviting a false positive on a question that could not have applied.

Do scope each entry. Ours carry applies_to_types plus a scope of universal or mode-specific, so a rule about ad-hoc scripts by planning agents reaches sprint-plan, architect and code-review, while a recovery hint about debug output reaches test and develop. In one verification run the orchestrator's rubric rendered four active checks out of the seven in the catalog, because three of them could not apply to an orchestrator.

Do not ship one universal rubric and rely on the supervisor to notice what is relevant. Relevance is a filter you can compute. It is not a judgement worth delegating to a model that will happily answer anyway.

Constrain what it sees

Verdict quality is capped by the snapshot you hand over, which makes that snapshot a design artifact rather than a dump.

Do exclude agents that cannot be judged fairly right now. Ours skips system agents, since the supervisor and the scout have no business grading themselves, and skips agents parked waiting on their own children. A blocked agent has produced nothing to judge, and its recent activity reads exactly like idleness to a model that was asked to look for idleness.

Do bound the per-agent budget in a way that keeps the useful ends. Ours caps each agent at 25 new messages per cycle, and content past a length threshold gets middle-chopped, keeping 70 percent of the head and 30 percent of the tail. The head carries what the agent set out to do, the tail carries where it ended up, and the middle is where the repetition lives.

Do hand it its own prior notes on that agent, so this cycle can see it already flagged something and choose to escalate or let it go rather than re-litigating from zero.

Do not let those numbers become hardcodes at the call site. Ours are module-level constants, so a fleet that turns out to want a different budget is a config change instead of a hunt through handlers.

Constrain what it can do with the answer

Both constraints here live in tool schemas rather than in the supervisor's system prompt, which is why they still hold on the day the model has an off day. A prompt is a request. A tool schema is a wall.

Separate scoring from standing orders

Do decide deliberately which powers the supervising model gets. Ours can call score_agent, add_agent_notes, set_performance_plan and send_interjection. It can judge an agent, write it a note, put it on an improvement plan, and interrupt it mid-turn. It cannot call create_directive, update_directive or expire_directive. Standing orders come from the human path only.

And it cannot call release_jailed_agent. The supervisor can score an agent down into the penalty box and has no way whatsoever to let it back out. That asymmetry is our favorite line in the whole tool matrix, it was entirely deliberate, and we would build it the same way again tomorrow.

Do not hand your supervisor the ability to author the rules it then enforces.

Verify the supervisor's own awards before they land

Do put a check between the supervisor's decision and the score that decision moves. Ours runs a verification pass with four guards over the batch of notes, against per-turn metadata for the agents involved, and it runs before any score adjustment or improvement plan touches an agent.

The reasoning that justified building the supervisor applies one level up, and it is worth following all the way. You added it because you did not fully trust the workers. A supervisor whose awards nothing checks is a single unreviewed model holding the power to shut down your fleet.

Make it cheap enough to leave running

Cost is a quality constraint here rather than a separate concern, because a supervisor that costs too much gets its interval widened until it stops catching anything.

Do split the request into a stable prefix and a volatile tail. Identity, scoring philosophy and universal anti-patterns go in the prefix. The fleet snapshot, per-agent activity and the inline rubric go in the tail. Providers charge far less for a prefix they have seen before, so a supervisor firing on a timer across a long run is either mostly cache hits or mostly full price.

Do hold the byte-identity invariant with a function signature instead of discipline. Our stable-prefix builder takes no per-cycle arguments at all, so it cannot embed a cycle number or a timestamp, because nobody ever hands it one. The invariant survives every future maintainer without any of them knowing it exists.

The gotcha. Slip anything per-cycle into that prefix, an agent id, a score, a timestamp, and you silently lose the cache hit on every cycle from then on. Your output looks identical. In one verification run our second cycle came back with 4,864 cached tokens on the system prompt, and that number is how you know the split is still holding. Watch it the way you watch a test.

Do run the supervisor on a cheap model. Ours does, and it can precisely because the expensive half of its context is the half that repeats.

Fixing the supervisor you already have

Nothing above needs a rewrite to adopt. Here is the order we would work in, cheapest and highest-payoff first.

Grep for your duplicated rules. Every rule stated in a worker prompt and again in a checker rubric is a drift waiting to happen. You do not need our catalog machinery to fix it, just one string and two render sites.
List your rubric questions and name the field each one reads. Any question where you cannot name a field is producing verdicts out of the model's priors. Delete it or go build the evidence.
Read the tool list you gave the supervisor. Can it write the rules it enforces? Can it undo its own harshest action? Fix that in the schema, not in the prompt.
Put something between its opinion and your scores. Even a crude guard closes the gap, because the alternative is one unreviewed model holding authority over the whole fleet.
Check whether your prefix is actually stable. One cached-token number in your logs tells you whether supervision is affordable enough to leave switched on.

Steps one through three cost an afternoon and remove entire categories of made-up verdict. Four and five are where a supervisor stops being a thing you hope is helping.

We built all of this because we run agents unattended for days and needed oversight we could trust without watching it. Favur is the harness it lives in, closed-source and invite-only, though the repositories its runs produce are open, so you can go read the code it writes. You can watch a real run replay, and the public scoreboard is where the same statement of work goes through model after model.

Your LLM sends valid data in an invalid shape

Charles Solar — Tue, 04 Aug 2026 14:00:00 +0000

A model never hands your tool a typed object. It hands you text that claims to describe one, and everything between that text and your validated arguments is a parse you control. How forgiving that parse should be is the whole design question at the boundary, and the answer is not the same in both directions.

Arrays that arrive as strings containing arrays
Values that are not JSON and parse perfectly anyway
Arguments that are still streaming when validation wants them whole
The quiet fallback that turns a clean rejection into a confusing error

Our harness is the worked example throughout, and every mechanism here is small enough to copy in an afternoon.

The malformation you will hit most often

An LLM asked for a list of strings will hand you a string containing a list of strings. It sends the text '["a","b"]' where your schema declared ["a","b"]. The data is correct and only the wrapping is wrong, but input validation, the check that runs before your tool code and confirms the arguments match the shape you declared, does not care about that distinction. It rejects the call. The rejection goes back to the model, and a full turn of the conversation gets spent teaching it to remove two quote marks.

Coerce it instead. Coercion, quietly converting a value into the type you expected rather than rejecting it, costs you nothing here, because there is exactly one thing '["a","b"]' can reasonably mean. Nothing is logged, nothing is retried, the tool just runs. The turn you save is the cheapest turn you will ever save, and you save it on every call that would have tripped.

Put the coercion in the base class, not the field

Fixing this per field works only for the fields you remembered. Every tool input model in our harness inherits from a shared base class, ToolInputBase, rather than from the plain validation model, and that base converts a JSON string into the list it represents before validation ever runs. Our own documentation lists inheriting from the plain model as a common pitfall, which is a polite way of saying somebody has to trip over it before they learn.

The reason it belongs at the base is that the malformation is a property of the boundary rather than of any one tool. Nothing about a particular tool makes a model more or less likely to quote an array. So a tool written next year inherits the fix without its author ever knowing the problem exists, and that is the actual goal. The alternative is a line in a contributing guide that every future author has to read, remember, and apply by hand.

Not everything that parses is JSON

The quoted array is one instance of a wider problem, which is that near-JSON arrives constantly and json.loads rejects all of it. Python's own repr of a dictionary uses single quotes and writes True, False and None where JSON demands double quotes and lowercase true, false, null. It is unambiguous, it is trivially parseable, and a JSON parser will refuse it every time.

Try the more permissive parser first. Where our terminal renders tool output, the rule is to call ast.literal_eval, the standard library's safe evaluator for Python literals, and fall back to JSON only if that fails. That instance sits on the return path rather than the argument path, so the direction differs, but the shape of the problem is identical and so is the fix. Our own docs state the rule flatly enough to steal. Never call json.loads() directly on that data.

Reaching for a single parser encodes an assumption about who produced the text. That assumption holds until a model, a subprocess, or a logging layer produces something near-JSON instead, at which point a strict parser gives you a failure that reads like a real error and is not one.

The arguments may not have finished arriving

Streaming breaks the assumption underneath all of the above, which is that you have the whole argument object when you go to parse it. When a model streams a tool call, the arguments arrive as a growing string, and a large one is still arriving while you would like to start work. Our streaming handlers split the difference by key. Short scalar keys land through on_key_completed once they are whole, and the one big streamable key arrives in pieces through on_string_progress, so a file write can begin before the content ends.

That split forces you to define what incomplete means. A stream can stop early, and the model reports why through a finish reason, where a value of length means the output was cut off mid-value rather than finished. Truncated JSON is not malformed in the way a quoted array is malformed, because there is no correct parse to recover, only a decision about what to do with a partial write. Our handlers are required to never raise from their completion or error paths and to always return a result, which is the same instinct as coercion applied one level up. The handler knows which file it was writing and how much of it landed, and a caller catching an exception several frames away knows neither.

The coercion that becomes a bug

Every mechanism above absorbs a malformation whose meaning is unambiguous, and the trouble starts precisely where that stops being true. Our streaming write handler once validated its mode argument with a fallback, assigning the mode when it matched a known value and quietly defaulting to write when it did not. When overwrite was later removed from the set of valid modes, a model still asking for mode=overwrite did not get told that the mode was gone. It got silently converted into a write, which then failed further down with a message saying the document already exists. The model asked to replace a document and was told, in effect, that the document was there. That is a worse outcome than the rejection it replaced, and it is documented in our own change notes as the expected consequence.

Coerce the representation, never the choice. Quoting, encoding and literal syntax are representation, and there is one right answer for each of them. Which enum member, which path, which identifier is a choice, and a choice your harness cannot honour is a rejection, not a value to be guessed. When you do reject, name the options that would have worked, because your harness knows them and the model is about to spend a turn finding out.

Where we refuse to coerce at all

The same argument that justifies coercion at the model boundary forbids it a few layers in. Our workflow engine evaluates conditional transitions against workflow state, and it does no implicit type coercion whatsoever. A type mismatch between the stored data and the expected value raises rather than comparing something adjacent, and comparison operators apply plain Python semantics with nothing clever underneath.

The criterion that separates the two cases is where the value came from. Be forgiving with what a language model produced, because its malformations are artifacts of generating text and carry no information you want. Be strict with what your own system produced, because a mismatch there is a bug, and coercing it hides that bug at the exact moment it is cheapest to see. The same permissiveness that saves a turn at the boundary will cost you an afternoon when it silently compares an integer against a string somewhere in your state machine.

Where your parser stands

Three questions place any harness on this map. When your model wraps an array in quotes, does the call fail? When a value arrives as a Python dict repr instead of JSON, does your parser survive it? When you cannot honour an argument, does the caller get a named rejection or a silent default? Each one is a few lines at the boundary and a turn you stop paying for.

Favur is our multi-agent software team, and the harness these mechanisms run in. It is closed source and invite-only, but the repositories it produces are open, and you can drive a replay of a real run at https://favur.dev/go/devto/tool-coercion or see the same harness scored across models at https://evals.favur.dev/go/devto/tool-coercion. I work on it, so weigh the framing accordingly.

The most overlooked way to steer a coding agent is what your tools say back

Charles Solar — Mon, 27 Jul 2026 16:00:00 +0000

Tool results are the most overlooked source of guidance for an LLM. Prompt engineering gets the attention, but the prompt states the job once — tool results arrive on every call, and each one is a chance to steer the model while it works. If you're building a harness, four changes to what your tools return will do more than most prompt revisions.

Never raise — return something the model can read
Put the near-miss inside the refusal
Annotate success, not just failure
Count the small failures and act on the pattern

Each of these exists to fix a problem the previous one leaves behind, so they are worth building in this order. Our harness is the worked example throughout — every mechanism described is one running in production.

Never raise — return something the model can read

The baseline problem is an agent that builds on garbage. A tool result that quietly failed gets trusted, and the error surfaces far from its cause; the blunt version is an unhandled exception, where the tool crashes, the loop dies, and the model never gets to react at all. Both failures share a root — the model was never handed anything it could read.

So make it a contract that every tool returns a structured result, never an exception. In ours, the completion and error paths both produce a result object with a success flag and, on failure, the error itself. A failed call becomes text the model reads on its next turn, and the run keeps its footing either way. That is the hard no — loud, explicit, and survivable.

Put the near-miss inside the refusal

A hard no alone still has a cost. A bare failure — "edit could not be applied" — forces the model to guess what went wrong, and the retry it burns is as blind as the first attempt. The retry is only ever as good as the information in the refusal, and your harness usually holds more information than it is passing on.

Edits are a concrete case. Ours applies them by matching an anchor block of text in the file and replacing it, which means a failed match comes with a near-miss attached — so the result carries the closest text that was actually found and a confidence score for how close it came. The model's second attempt is aimed at the real mismatch, usually a stale line or drifted whitespace, instead of being a fresh roll of the dice. An explanatory no turns a wasted retry into an aimed one.

Annotate success, not just failure

Once failures are handled, success becomes the place where the quieter damage accumulates. A write that lands cleanly can still leave problems behind that no compiler complains about — a new warning from the linter, the static checker that flags style errors and suspicious code, or a brace that lost its closing pair, or a TODO comment nobody asked for. None of that is visible in an "ok", which means success needs annotating just as much as failure does.

Attach the report to the success result itself. A successful write in our harness comes back stating how the write changed the linter's count of problems in that file, whether every bracket still balances, what TODO comments appeared, and the file's new length — so the model learns what its write left behind in the same result that confirms the write, without spending a turn re-reading the file. Reads work the same way, arriving with the linter's current findings and an outline of the file's functions and classes, so the state of the code stays continuously in front of the model rather than being something it must remember to check.

Give the feedback a standard to push against

Annotations alone are trivia. A lint warning only steers a model that owes somebody clean code — feedback and standards are two halves of one mechanism, and shipping either without the other does little.

Put the standard in the agent's identity, in the system prompt. Our code agent's includes lines to the effect of "you do not skip pre-commit checks" — the lint and test hooks that run before code is allowed into a commit — and "you do not modify tests to make them pass". Against that identity, a tool result reporting new lint warnings is not information — it is an unmet obligation the agent has already agreed to.

Count the small failures and act on the pattern

One gap remains, and it is the difference between an event and a pattern. A model can acknowledge a lint warning, fail to fix it, and move on — and with each result judged in isolation, nothing stops a slow leak of small failures from running the whole session aground.

So make the nos accumulate. Each of our agents carries circuit breakers — running counts of non-fatal issues kept across the whole run — and a count that gets high enough trips its breaker — lint failures on the third, type-check failures likewise, pattern violations after five. A tripped breaker doesn't abort the conversation; it changes how the agent proceeds from there. Retry budgets bound the loop from the other side, so a step that keeps failing is marked failed rather than retried forever. A single no is an event; the breakers are what turn a pattern of nos into a diagnosis.

What tool feedback cannot do

Lint measures shape, not judgment. A design mistake lints clean, and no verification report will flag a function that is well-formed, type-safe, and wrong. That class of problem is not solved by tool feedback in our harness or anyone's — it is why review exists as its own step, run by separate agents with their own gates.

The tool result is a prompt surface, in any harness. If your feedback only comes in "ok" and an exception, the model is steering on one bit per call, and everything between those two poles — the aimed retry, the yes-with-caveats, the accumulated pattern — is bandwidth you already paid for.

Where your harness stands

Three questions locate any harness on this map. When a tool fails, does the model receive something it can actually read? When an edit misses, does the refusal say how close it came? When a write succeeds, does anything tell the model what it left behind? Every no is a place a harness is spending retries and turns that graded feedback would hand back — and the quiet no, the one most harnesses skip entirely, is the one that pays the most.

Favur is our multi-agent software team — the harness these mechanisms come from. It is closed source and invite-only, but the repositories it produces are open, and you can drive a replay of a real run at https://favur.dev/go/devto/tool-feedback or see the same harness scored across models at https://evals.favur.dev/go/devto/tool-feedback. I work on it, so weigh the framing accordingly.

What it takes to run an agent unattended for two days

Charles Solar — Mon, 20 Jul 2026 16:22:28 +0000

Favur ran unattended for two days on Meta Muse Spark 1.1, working from a Statement of Work for a 2048 game. It produced 131 files with tests across six sessions, with no human intervention at any point.

Unattended runs past a few hours fail in a small number of ways. A better model raises the ceiling on output quality but removes none of them, because they are properties of running a model in a loop for a long time rather than properties of the model. This covers what the harness does about each, including one it does not address.

False completion

A step reports success without having done the work. In a supervised run this costs a minute. In an unattended run downstream steps build on the result and the error surfaces hours later, past any clean rollback point.

Completion is gated on artifacts rather than on reports. Each step type has exactly one tool that can advance it, and validators run before the step closes.

The validators are the substantive part. Verification must cite at least two context documents. A commit is rejected without a commit hash and a non-empty list of committed files. Acceptance criteria are returned as structured fields and machine-checked rather than as prose asserting success. A claim is not accepted anywhere an artifact is available, because a model can assert anything but cannot produce a hash for a commit it did not make.

Work and judgment are separated. The agent running the development workflow writes no code; it sequences sub-agents and enforces the gates. Test creation is a distinct delegated step ahead of implementation, and the implementing step exists to make those tests pass. Failed verification routes back to implementation rather than forward.

Non-convergence

The failure that ends long runs is not a crash, which is detectable, but a loop, which presents as normal operation. The process is live, requests are being issued, tokens are being spent, and output is being produced.

The common form is not unbounded retry but oscillation, where an agent alternates between two candidate solutions that are both wrong and never terminates.

This is bounded at three levels. Each step carries a retry budget; exhausting it marks the step failed rather than retrying, and the terminal state cannot return to pending, so a failed step cannot re-arm and restart the cycle. A failed step routes to a declared failure path rather than terminating the run. Where a step can route back to an earlier planning stage, re-entry is counted and reaching the cap escalates. Each agent additionally carries circuit breakers counting cumulative failures across the whole run, which separates repeated failure of a single step from the same error recurring across different steps.

Context exhaustion

Running out of context partway through a unit of work leaves a half-implemented feature that the next session cannot safely resume, and the usual outcome is discarding work that was nearly complete.

Conversation history is summarized or dropped in tiers before each request as it approaches the model's context limit, escalating with fill level. Near the ceiling a request is hard-blocked rather than issued in a state where it cannot complete.

Task drift

Task drift is the one failure here the harness does not address. There is no semantic check on whether output remains relevant to the original task, and nothing evaluates the agent's work for topical relevance.

The constraints on drift are structural. A step cannot advance until its validators pass. The advancing tool is fixed per step type. Step requirements are injected on entry, before output is produced, so the evidence a step owes is known in advance rather than assessed afterwards. Strategy knowledge from completed planning steps persists for the remainder of the run, which removes re-derivation of settled decisions as a drift source.

Review quality

The practical failure in review is not that an automated reviewer approves bad code. It is that human review of a large generated diff degrades to skimming, so the verification burden moves rather than disappearing when code generation gets cheap.

That improves only if the automated review is trustworthy enough to rely on, which requires the reviewer to carry a cost for being wrong in either direction.

The reviewer must read the files before it can produce findings. Every finding carries a severity. A false-positive circuit breaker trips after three findings that are not real issues, which prices over-flagging rather than leaving it as a free hedge.

Concealed capability gaps

Capability-justified abandonment carries no scoring penalty. An agent that halts and reports a missing capability is not marked down for halting.

The reasoning is the incentive structure rather than the mechanism. Penalizing honest abandonment causes agents to conceal gaps, and a concealed gap is filled with fabricated work.

State across sessions

A context reset begins a session with no memory of prior state. Over a two-day run this means half-implemented work that is not identified as such, and a new agent resuming from a state it cannot interpret.

This run spanned six sessions. Checkpoints are written periodically and immediately on any agent completion or failure, and that state persists across session boundaries, so each session resumed with the set of passed and failed steps intact. Checkpointing does not block the run.

Output

The SOW specifies a creative variant rather than a clone of 2048, so the theme was model-selected. Tiles run from cool to unstable.

The run launched the built game and played it, producing moves, merges and achievement events, rather than reporting the test suite green. On an unattended run that is a stronger completion signal than a passing suite.

The repo is open source at https://github.com/awesoftsolutions/idea_meta-muse-2048-game

The full run recording is at https://www.youtube.com/watch?v=o0fQ7WQRTjo

The replay can be driven and stepped through at https://favur.dev/go/devto/the2048

Which AI model actually builds the best software? We built a public benchmark to find out

Charles Solar — Thu, 16 Jul 2026 21:39:46 +0000

Today we're opening up Favur Evals — a public, live leaderboard for a question we kept needing an answer to ourselves: which model should I actually put on a coding job?

Here's how it works. We take a fixed statement of work and hand it to a full team of AI agents — planner, architect, tester, coder, reviewer, builder. Then we run that same job again and again, swapping only the model underneath the team:
all-Qwen, all-OpenAI, all-DeepSeek, all-Gemini, and so on. Same task, same
scaffolding, different brain. Every run gets scored across the whole lifecycle — the code, the tests, the cost, the discipline — not just the final diff.

What you'll find on the board

A leaderboard that refuses to crown one winner.
As I write this, the top run isn't even a single model — it's a two-model mix, Gemini Flash 3.5 on the orchestration-heavy seats with Gemini Flash Lite on eight specialist roles.
Meanwhile the strongest test suites came from an all-OpenAI run, the best
value-per-dollar from all-DeepSeek, and the leanest zero-failure run from
all-Qwen. Different models really are good at different parts of the job — the board makes it visible which part, so you can pick by the seat you're hiring for.

Subject breakdowns. Every composite splits into eight engineering subjects — code quality, test quality, cost efficiency, velocity, tool discipline, effort efficiency, process discipline, deliverables — so you can rank by the thing you care about instead of our blend.

Behavior fingerprints. How each model actually behaves in each seat: cache utilization, reasoning depth, tool cadence, throughput.

Where the money goes. Cost share by agent role across the runs — spoiler: code review is the biggest line item, whichever model is underneath.

One thing worth knowing about the scoring

Nothing on the board is graded by an LLM. Every number recomputes from artifacts the run already produced — lint, complexity metrics, the run's own test results, tool telemetry. Same run output, same score, every time. The full rubric is public, and any score on the board expands into its formula when you click it.

These are relative rankings inside our pipeline, under a versioned rubric — the board reshuffles as the pipeline improves, and that's by design. Treat any single number as a point estimate, not a verdict.

Who's behind it

We build Favur, an autonomous multi-agent software team; these runs execute in Favur's own pipeline. The benchmark is independent and self-funded — no vendor sponsorship, credits, or grants — and every model on the board is doing something genuinely hard.

Go poke around: evals.favur.dev.
New runs land all the time — we post them as they do at @favurdev.