I expected Claude Opus 5 high to produce a better design than Opus 5 medium.
It did not.
I had given both configurations the same repository, feature request, and design template. The task was to add dependency-aware execution to Galley, an open-source local runtime for supervised, multi-model AI coding. A task could declare other tasks as dependencies, and the daemon would claim it only after all of them had succeeded.
In the blind evaluation, Opus medium's design received 84 points from Claude and 82 from Codex. Opus high received 68 and 50. They took almost exactly the same amount of time (21 minutes and 12 seconds for medium, 21 minutes and 25 seconds for high), while high processed 2.29 million tokens to medium's 1.56 million.
My first reaction was that something had gone wrong in the evaluation. I read both critiques again, then the designs and session logs. The scores were not the most interesting part. The designs were.
High explored waiting deadlines, persisted archive history, additional configuration, more state transitions, recovery behavior for those transitions, and tests for the machinery it had introduced. Much of it was technically plausible. Some of it addressed real failure modes. Taken together, those additions made the requested feature larger and harder to operate.
Medium explored less. It stayed closer to Galley's existing lifecycle and reused more of the current claim and failure machinery.
High found more things to think about.
That was the problem.
I had treated effort as a quality setting. In these runs, it behaved more like an exploration multiplier.
What I actually measured
The feature was not a self-contained coding exercise. Galley stores tasks as YAML files and moves them through lifecycle directories. Multiple daemon processes may compete to claim work. Adding dependencies affected parsing, scheduling, failure recovery, CLI output, backward compatibility, filesystem error handling, and tests that had to prove execution order without calling an external model.
I ran seven model-and-effort configurations through four separate phases:
- technical design;
- design review;
- implementation from a common approved design;
- code review of a common implementation.
The configurations were GPT-5.6 Sol medium and high, GPT-5.6 Terra xhigh, GPT-5.6 Luna max, Claude Sonnet 5 high, and Claude Opus 5 medium and high. Within each phase, every configuration received the same repository state and the same inputs. These were four controlled phase comparisons around one feature, not seven end-to-end projects in which an early design changed the later implementation.
For design review, all seven configurations reviewed the unmodified Opus medium design. Between phases, I worked with the orchestrating LLM to prepare controlled inputs from the original requirements and the recorded artifacts. For implementation, all configurations received the same deliberately revised design. For code review, I fixed the target to the Sonnet high implementation from the implementation phase so every reviewer inspected the same production code and tests.
I deliberately did not tell the models to produce the smallest possible change or prescribe a convergence method. The request fixed the outcome, constraints, and exclusions; deciding how much machinery they justified was part of the engineering task I wanted to observe. An explicit subtraction rule would test compliance with a supplied convergence strategy, which is a different question.
The outputs were anonymized and evaluated independently by Claude and Codex against a common rubric. I treat the scores as measurements attached to the artifacts, not as verdicts on which model “won.” Concrete mechanisms, defects, preserved boundaries, and demonstrated failures carry more weight than small differences in averages, so I preserved both critiques when the evaluators disagreed.
There is one important defect in the design scoring that needs to be stated here rather than hidden in the notes. The original design request said that only runnable queued tasks could be claimed. The prompt used for the later blind scoring weakened this to saying that tasks with unfinished dependencies must not be executed; it omitted the prohibition on claiming them. This affected the later blind scoring of the design artifacts, not the design-review runs discussed below, which received the original requirements. Both Opus designs proposed claiming some unresolvable tasks and failing them afterward, so the omission did not obviously favor only one of them, and the evaluators separated the designs on broader scope and compatibility grounds as well. The design scores are still not absolute measures of requirement compliance.
In the design phase, both evaluators independently placed Sol high first and Opus high last, so the main direction was not explained by simple provider loyalty. Family and style preference did appear elsewhere, especially in design-review scoring. What I have is a structured case study from one repository, not a population estimate or a model leaderboard.
What high effort changed in Opus
Opus high did not misunderstand the repository. It inspected the implementation extensively and found real questions around concurrency, archive behavior, timing, recovery, and tests. The difference appeared after discovery: once high found a plausible concern, it was more likely to preserve that concern in the final design.
A rare race suggested another check. The check introduced a state transition. The transition needed persistence and recovery. The recovery path then needed tests and an operational explanation. Each step followed reasonably from the previous one, but the chain had little pressure to return to the current requirement and ask which mechanisms could be removed.
On a quick read, parts of the high-effort design looked reassuring. It had considered more failure modes. The overreach became clearer only when I traced each mechanism back to the requested behavior and asked what would stop working without it.
Opus medium showed many of the same instincts at a smaller scale. It still investigated failure handling and concurrent claims, but it generated fewer branches than it had to rank, combine, or reject. In this run, that amount of exploration fit its ability to turn the result into a bounded design.
I think of the observed pattern as an exploration-convergence mismatch. Exploration discovers files, contracts, alternatives, side effects, and failure conditions. Convergence decides what matters now, merges issues that share one correction, rejects optional hardening, and produces a result that fits the approved scope. Increasing effort can expand the first activity faster than the second.
The failure does not look careless. It looks like serious engineering applied to a larger problem than the one currently being solved. In a real codebase, someone still has to own every new state, setting, branch, test, and recovery path after the impressive design document is gone.
Twenty-two million tokens missed one sentence
The design-review phase showed the same imbalance from another direction.
Every reviewer received the original requirements, the same design, and the same repository state. The requirement explicitly said that a queued task with unresolved dependencies must not be claimed or executed. The design under review proposed claiming some such tasks first, then recording a failure afterward.
All seven configurations understood and summarized that mechanism. All seven found real problems elsewhere. Between them they processed 22.64 million tokens and produced 115 findings.
None identified the direct contradiction with the requirement.
The reviewed artifact was the Opus medium design from the opening. It was more bounded than the high-effort design and scored better in the blind design evaluation, but it still violated this governing condition. Better scope control and a stronger score did not make it implementation-ready.
They found partial-write hazards, race conditions, duplicate IDs, stale state, archive behavior, unnecessary scans, failure propagation, notification problems, and recovery gaps. Luna max uniquely followed a dependency cycle outward to a task that was not itself in the cycle. Opus medium found a subtle conflict between stale-claim recovery and a task that had waited for a long time.
The missing check was simpler than most of what they found. More independent exploration created diversity around the technical consequences of the design; it did not guarantee that anyone returned to the governing request and compared the selected mechanism with it directly.
The contradiction did not propagate into implementation. Between phases, I worked with the orchestrating LLM to prepare a revised common design from the original requirements and the review evidence. That controlled input kept both waiting and blocked tasks queued. The correction came from dataset preparation at the orchestration boundary, not from any of the seven review artifacts. This separation was deliberate: each phase compared model behavior against one controlled input instead of letting an early artifact determine every downstream result.
This is why I do not treat a high-scoring review as an approval decision. A review can be relatively strong, contain valuable findings, and still miss the condition that determines whether implementation may proceed.
The same behavior helped in code review
The code-review phase changed the value of wide exploration.
The common implementation contained defects outside its normal path. A lightweight task reader interpreted some valid YAML differently from the standard loader. Duplicate task IDs could cause one dependency result to be reused for another task. A partial failure while reading the task store could allow a dependent task to run before success had been established. Some CLI output described a different state from the one used by the daemon.
Opus high inspected production code, user-visible behavior, performance, and tests. More importantly, it changed a central dependency guard in a safe copy and showed that the main integration test still passed. The test looked like evidence for execution ordering. The mutation demonstrated that it was not sensitive to the behavior it claimed to prove.
No other configuration established that point as clearly. The blind evaluations reflected this: Opus high received 91 and 93, while medium received 88 and 91.
The high-effort review still carried the same weakness seen in design. It produced nine findings, six of them minor, split related problems, and did not always rank incorrect execution above a presentation issue. But a review has a downstream consumer. Candidate findings can be merged, reranked, or declined before they become production structure. Missing a defect is often more expensive than discarding an extra review item.
Design has no equivalent cleanup stage once unnecessary machinery becomes the architecture. The same search behavior that expanded a design helped expose a test that was green for the wrong reason.
Sol showed why this is not a medium-versus-high story
The Sol results moved in a different direction. Sol high received higher blind averages than Sol medium in all four phases, although the size and cost of the difference varied.
In design, Sol high explored the existing code deeply, initially considered a path that would add work to tasks without dependencies, and then corrected itself before completing the document. It avoided introducing a new persistent lifecycle state and allowed blocked tasks to recover when dependency state changed. Both evaluators placed it first, with scores of 89 and 93.
The added exploration was useful there. It was also useful in implementation, where the average moved from 66 for medium to 80.5 for high and the resulting implementation covered more meaningful behavior.
In design review and code review, substantially higher recorded token use accompanied much smaller differences in the measured result. Sol high used 3.24 million tokens in design review compared with medium's 1.13 million, while their averages differed by three points. In code review, high used 4.90 million tokens compared with 3.18 million, with a 2.5-point difference.
The important observation is not that one effort level beat another. Effort expanded the search frontier in both families. Opus high carried more of that frontier into the final artifact; Sol high more often produced a smaller final result from the broader search. The usefulness of the remaining exploration then depended on whether the phase needed selection, implementation coverage, or review recall.
The implementations separated at the edges
All seven implementations handled much of the ordinary dependency flow. They could add the YAML field, wait for unfinished tasks, recognize successful dependencies, display reasons, and pass broad test suites. The models separated where the new feature met existing behavior:
- What happens when only part of the task store can be read?
- Does an alternate or relative path use the same interpretation as the daemon?
- Can malformed input change behavior in an existing command?
- Does a feature that is not being used add a full-store scan to an old path?
- Is a cycle result stable, or does it depend on evaluation order?
- Does a green integration test observe the ordering guarantee, or merely execute two tasks that would have succeeded anyway?
Opus medium produced the implementation with the fewest observed gaps. It connected more of these boundaries without changing unrelated behavior. Luna max implemented the ordinary feature at a much lower estimated API cost, but replaced broader search and display paths, changed the treatment of damaged tasks, and added work even when dependencies were unused. Sol high kept the implementation comparatively small but missed alternate-path behavior and direct daemon-level proof.
These differences are hard to compress into a pass rate. The feature works until the repository is incomplete, an older path reads the same YAML differently, or the test suite gives confidence without observing the protected behavior. That is not benchmark trivia. It is where software maintenance happens.
Resources describe activity, not quality
Across all 28 runs, the logs recorded 167.52 million total tokens. Most were cached input repeatedly read across tool turns, not fresh input or hidden reasoning tokens. The summed active execution time was just under eight hours, excluding human approval waits but including tests and builds; the configurations ran in parallel, so this is not the experiment's wall-clock duration. Applying the standard API prices published on August 3, 2026 produced an estimated cost of $104.58, not an actual CLI subscription charge.
Those resource figures describe activity, not quality. Luna max processed the most tokens in design review and remained inexpensive because cached tokens were cheap, but it was also the slowest configuration in that phase. Token volume, price, exploration breadth, latency, and artifact quality were separate variables.
Effort belongs to the phase
I no longer choose one effort level for a model and carry it through an entire workflow. The useful exploration budget depends on what the phase must produce and whether another stage can remove excess work.
| Phase | What additional exploration must earn |
|---|---|
| Design | A smaller justified decision after inspecting the real constraints |
| Implementation | Coverage of another real contract or execution path without changing unrelated behavior |
| Review | A supported candidate finding that can still be merged, ranked, or declined downstream |
This is related to a conclusion I reached while simplifying an older agent workflow: be strict about boundaries and evidence, but flexible about the route between them. More capable models made detailed route instructions age badly. These measurements add a second problem. Giving the model more room to search is not sufficient when the phase also requires aggressive subtraction.
The operational question is therefore not “How important is this task?” Important work does not automatically deserve high effort. The better question is “What will happen to the additional possibilities this configuration finds?”
If the answer is that they become architecture immediately, uncontrolled breadth is expensive. If they remain review candidates that another step can verify and consolidate, higher recall may be worth buying.
What I can and cannot conclude
This comparison used one run per configuration per phase. Another task or another run could change close measurements. Claude and Codex sometimes disagreed by more than twenty points, and the overlap between evaluator families and candidate families leaves room for style preference. The design scoring prompt omitted one original claim constraint. The tools and caching behavior also differed between providers, so raw token totals are useful descriptions of these sessions but imperfect provider-level comparisons.
The experiment did not isolate an internal mechanism inside Opus or Sol, and I cannot show that a particular latent “convergence ability” caused the outputs. What I can show is the external pattern in this setup: higher effort expanded the work explored; the configurations differed in what they retained; and the value of that retained work changed across design, implementation, and review.
I also cannot conclude that medium effort is generally better, or that high effort causes overengineering. Sol contradicts both claims. An explicit convergence rule might change the outputs, but testing that would measure the models under a new intervention. Here, I intentionally left engineering selection to the models because that judgment is part of how I use them in real development.
The result is strong enough to change how I assign effort without pretending to be a universal model ranking.
Exploration is not the finished product
Recent coding models are much better at searching a repository, following effects across components, and generating technically plausible responses to what they find. That creates a new failure mode: a model can discover more work than it can economically judge.
Reasoning effort pushes on that boundary. Sometimes it exposes the missing execution path that makes an implementation safe. Sometimes it produces the review finding nobody else could prove. Sometimes it turns a small design into a system for managing the risks introduced by the design itself. The setting does not know which one is happening.
Calling it “high” invites the wrong intuition. High effort is not a higher position on a quality scale. It is a larger search budget handed to a particular model inside a particular phase.
The practical limit of a coding agent may no longer be how much it can explore. It may be how much of that exploration it can turn back into one coherent change.
Complete phase scores
These are the simple averages of the two blind evaluations. They summarize the recorded critiques; they do not override evaluator disagreement or the artifact-level evidence discussed above.
| Configuration | Design | Design review | Implementation | Code review |
|---|---|---|---|---|
| GPT-5.6 Sol medium | 82.0 | 80.5 | 66.0 | 86.5 |
| GPT-5.6 Sol high | 91.0 | 83.5 | 80.5 | 89.0 |
| GPT-5.6 Terra xhigh | 71.5 | 68.0 | 66.0 | 74.0 |
| GPT-5.6 Luna max | 72.5 | 84.5 | 79.0 | 61.5 |
| Claude Sonnet 5 high | 70.5 | 71.0 | 78.5 | 68.0 |
| Claude Opus 5 medium | 83.0 | 76.5 | 94.5 | 89.5 |
| Claude Opus 5 high | 59.0 | 72.5 | 91.5 | 92.0 |
Methodology notes
- Human approval wait time was excluded from active elapsed time. Test and build execution time was included.
- Token totals include normal input, cached input, cache writes where applicable, and output. Cached input accounted for most of the total.
- Cost estimates apply the standard API rates available on August 3, 2026 to recorded usage. They are not actual CLI subscription charges.
- Artifacts were anonymized before evaluation. Scores are averages of one Claude evaluation and one Codex evaluation; original disagreements were preserved rather than adjudicated into new scores.
- The candidate and evaluator families overlap, so the blind process removes model-name knowledge but not possible preference for a family-specific style.
- The comparison used one run per configuration per phase and should be read as an instrumented case study rather than a population estimate.
Top comments (1)
This is one of the most rigorous breakdowns of AI agent reasoning I've read. Calling effort an "exploration multiplier rather than a quality setting" completely nails why so many developers get burned when cranking Opus or Sol to High on everyday tasks. The "exploration-convergence mismatch" you observed is the exact trap: when an agent is given a massive exploration budget without strict boundaries, it inevitably invents hypothetical edge cases, over-engineers state transitions, and builds complex recovery machinery just to justify its search path. In my everyday CLI workflows, I've found that the only way to tame high-effort models is enforcing strict context isolation and structural subtraction. Giving the model fewer files to "explore" via aggressive ignore patterns and scoped task briefs forces that search budget to converge on the actual requirement instead of expanding the architectural frontier. "Effort belongs to the phase" should be printed on the desk of anyone building multi-agent pipelines. Brilliant write-up!