DEV Community: Yohji Sakamoto

Is GPT-5.6 Sol Max Worth It? High fixes. Max builds.

Yohji Sakamoto — Sun, 19 Jul 2026 13:06:48 +0000

Is GPT-5.6 Sol Max Worth It?

Disclosure: I maintain Tura. This is the complete public analysis and its benchmark artifacts; it is not an independent review.

Short answer: High fixes. Max builds.

Max buys more search, revision, and agent rounds. That helps when the model must discover the path through a rewrite, migration, or new project. A bounded bug with a failing test has far less uncertainty left to buy. The data supports no universal default: task and harness change the return while the bill grows quickly.

The task decides whether Max has work to do

The clearest split comes from two datasets: the public DeepSWE v1.1 task and trial records, and Tura's eza Rust-to-Python rewrite.

Task shape	High	Max	Score change	Cost	Output tokens	Rounds / steps
Scoped repair, DeepSWE (7 tasks)	64.3%	57.1%	-7.1 pp	2.53x	2.18x	1.70x
Feature implementation, DeepSWE (95 tasks)	70.2%	74.6%	+4.4 pp	2.43x	2.11x	1.66x
Repository rewrite, eza (3 harnesses)	78.8-89.4%	92.3-94.2%	+4.8 to +13.5 pp	2.27-3.27x	1.75-3.13x	1.47-2.40x

Max spends more in every task class. The score return depends on the work.

The DeepSWE classification is intentionally mechanical:

Group	Inclusion rule	Tasks	Scored attempts
Scoped repair	Title starts with `Fix`, `Restore`, `Preserve`, or `Reconstruct`	7	28 High + 28 Max
Feature implementation	Title starts with `Add` or `Implement`	95	379 High + 378 Max
Excluded	Title matches neither rule	11	Not used in the split

Three verifier timeouts follow DeepSWE's exclusion rules. Seven repair tasks cannot prove that Max generally hurts fixes; they can disprove the assumption that it always helps.

For eza, High is the mean of two runs per harness; Max is one selected run. The artifacts are public.

eza rewrite harness	High	Max	Gain	Cost	Output	Reasoning	Rounds
Tura Balanced	89.4%	94.2%	+4.8 pp	2.39x	3.13x	4.83x	2.40x
Tura Direct	79.8%	92.3%	+12.5 pp	3.27x	3.02x	5.31x	2.25x
Codex CLI	78.8%	92.3%	+13.5 pp	2.27x	1.75x	2.34x	1.47x

Max run	Checks	Cost	Output	Reasoning	Rounds	Duration
Tura Balanced	49/52	$14.55	218.4k	113.1k	72	98.9 min
Tura Direct	48/52	$6.26	114.7k	81.9k	18	53.7 min
Codex CLI	48/52	$12.01	69.2k	30.2k	92	30.9 min

Direct and Codex move from about four checks in five to more than nine in ten. A rewrite leaves enough unresolved behavior and compatibility work for Max to reach the artifact.

DeepSWE shows the average price

DeepSWE is the cleanest broad comparison because it holds the 113 tasks and mini-swe-agent harness fixed across four full runs per effort level.

GPT-5.6 Sol	Pass@1	Cost / task	Output	Input	Time	Steps
High	69.4%	$3.47	28.5k	2.71M	9.9 min	36.9
Max	72.7%	$8.39	60.0k	7.91M	18.8 min	61.3
Max / High	+3.3 pp	2.42x	2.11x	2.91x	1.90x	1.66x

Max adds 3.3 points while every measured resource rises much faster.

That works out to about one additional passing attempt per 31 attempts. Cost per expected pass rises from roughly $5.00 at High to $11.54 at Max.

The intermediate setting shows the curve flattening before Max:

Effort	Pass@1	Cost/task	Gain vs previous	Cost increase
High	69.4%	$3.47	-	-
XHigh	70.7%	$4.70	+1.3 pp	+35%
Max	72.7%	$8.39	+2.0 pp	+78%

DeepSWE is not a small bug-fix benchmark. Its own report says bug localization and refactoring are under-represented. The published task statistics make the scope visible:

Benchmark	Mean prompt	Reference lines added	Files edited	Repositories
SWE-bench Verified	1,700 chars	10	1	12
SWE-bench Pro Public	4,614 chars	120	5	11
DeepSWE	2,158 chars	668	7	91

DeepSWE gives a long-horizon average, not a verdict on every bug. Its aggregate hides routing information.

OpenAI proves strength, not marginal value

OpenAI's GPT-5.6 launch report establishes that Sol Max is a top coding configuration:

OpenAI-published coding result	Sol Max	Evaluation surface
Artificial Analysis Coding Agent Index v1.1	80	Composite coding-agent index
DeepSWE v1.1	72.7%	Long-horizon repository implementation
Terminal-Bench 2.1	88.8%	Complex command-line work in containers

OpenAI also reports Sol Max beating Fable 5 on the Coding Agent Index by 2.8 points while using less than half the output, less than half the time, and about one-third lower estimated cost.

OpenAI's Sol Max vs Fable 5 claim	Reported result
Coding Agent Index advantage	+2.8 points
Output tokens	Less than 0.5x
Completion time	Less than 0.5x
Estimated cost	About 33% lower

The official API price also explains why output discipline matters:

GPT-5.6 Sol API usage	Price per 1M tokens
Uncached input	$5.00
Cache read	$0.50
Cache write	$6.25
Output	$30.00

These cross-model results prove strength, not the economics of moving Sol from High to Max. DeepSWE measures that: +3.3 points for 2.42x cost.

Artificial Analysis provides an independent within-model comparison:

Artificial Analysis	XHigh	Max	Max change
Intelligence Index	58	59	+1 point
Time to first token	44.58s	145.61s	3.27x

One index point can matter. So can 3.27x longer startup latency.

What third-party data actually says

Third-party numbers cover different tasks and cannot be merged into one score.

Third-party source	Dataset	Published numbers	What it supports
ARC Prize	ARC-AGI-3	Sol High 2.1%; Max 7.8%; +5.7 pp / 3.7x relative	Direct evidence that Max can cross a hard abstraction threshold
FutureSearch	150+ web-research tasks	GPT-5 Low 49.6%, $0.25, 230s; High 48.1%, $0.39, 217s	More effort can cost more and score less when reasoning is not the bottleneck
Bai et al.	8 models on SWE-bench Verified	Agent tasks use about 1000x code-chat tokens; same-task runs vary up to 30x; self-prediction correlation at most 0.39	Token volume is stochastic and does not reliably predict success
SWE-Atlas QnA	Real-repository comprehension	Frontier ceiling about 35%; GPT-5.4 used more than 2x Opus operations; top answers exceed 1,200 words	Long trajectories can be useful when evidence gathering is the deliverable
CodeRabbit	100+ long-horizon coding tasks	Sol 63.7% and 20,968 output/task; Terra 40.7% and 55,594 output/task	Cheaper token pricing does not guarantee cheaper solved tasks

FutureSearch is especially useful because it publishes the whole operating point rather than only the winning score:

Model	Effort	Score	Cost/task	Time
Claude 4.6 Opus	Low	53.1%	$0.24	73s
Claude 4.6 Opus	High	55.0%	$0.55	183s
Claude 4.6 Sonnet	Low	50.4%	$0.27	130s
Claude 4.6 Sonnet	High	54.9%	$0.46	262s
GPT-5	Low	49.6%	$0.25	230s
GPT-5	High	48.1%	$0.39	217s
Gemini 3 Flash	Low	49.9%	$0.05	96s
Gemini 3 Flash	High	47.9%	$0.14	182s

Here, effort helps Claude 4.6 and hurts GPT-5 and Gemini. Evaluate the setting in its actual harness.

CodeRabbit's review harness adds a second warning about raw output:

CodeRabbit review result	Sol	Terra
Actionable passes	69/99 (69.7%)	53/101 (52.5%)
Actionable precision	31.6%	35.7%
Raw comments	231	143

More output can raise recall while lowering precision. This is product data, not a Sol High-to-Max comparison.

The harness decides what an extra token means

In Tura's 277 usage-available runs, output predicts success in one harness family and not the other:

Output vs success	Runs	Task-adjusted relationship	Interpretation
Tura family	137	r=0.44, p<0.000001	Positive association, driven by Direct
Tura Direct	70	r=0.51, p<0.00001	More output often tracks implementation work
Tura Balanced	67	r=-0.10, p=0.41	No useful association
Codex family	140	r=0.03, p=0.73	No association
Codex High, raw rank	70	rho=-0.13, p=0.29	Negative direction, not statistically conclusive

The bill also has different composition:

Harness family	Output share of tokens	Output share of modeled cost
Tura	1.17-1.77%	31.98-37.64%
Codex	0.34-0.41%	12.80-17.00%

Output is a small part of volume and a much larger part of cost.

Rounds make the problem compound. Across 278 analyzed runs, total token volume grows superlinearly with round count:

Configuration	Token exponent	Cost exponent
Tura Balanced	1.47	0.94
Tura Direct	1.40	0.88
Codex Medium	1.24	0.95
Codex High	1.38	1.05

Doubling rounds tends to more than double tokens; caching keeps cost nearer linear.

In eza, Balanced grew from 30 to 72 rounds, Direct from 8 to 18, and Codex from 62.5 to 92. Max magnifies the loop: tests or second-guessing, depending on the harness.

Greenfield is plausible, not proven

Public data supports GPT-5.6 on new projects, but there is no matched Sol High-to-Max greenfield benchmark yet.

Source	Greenfield-related data	Limitation
OpenAI / Base44	30 app-building conversations; GPT-5.6 used 22% less input and 23% less output than GPT-5.5 while staying competitive	Model-generation comparison, not High vs Max
CodeRabbit	Sol passed 63.7% of 100+ long-horizon tasks	Sol vs Terra, not effort ablation
SWE-Atlas	Native scaffolds improved frontier models; top model ran hundreds of commands	Codebase comprehension, not app construction

Greenfield has more architectural decisions, so Max has somewhere to spend search. This is a routing judgment, not a measured High-to-Max gain.

The rule I would use

Task	Start with	Escalate when
Bounded bug or scoped change	High	Localization remains uncertain or High fails
Feature implementation	High	Failure is expensive enough to justify about 2.4x cost
Rewrite or migration	Max is often justified	Compatibility surface is broad and externally verified
Greenfield product	High or Max by scope	One agent owns architecture, implementation, and verification

The conclusion is not "Max is too expensive." It is more specific:

High closes known gaps. Max is worth paying for when the agent still has to build the path.

Token-Saving Plugins Are Mostly Stupid Idea

Yohji Sakamoto — Sun, 19 Jul 2026 12:30:27 +0000

Token-Saving Plugins Are Mostly Stupid Idea

Disclosure: I maintain Tura. This is the complete public analysis and its benchmark artifacts; it is not an independent review.

I am tired of people believing every "save 90% of tokens" claim attached to a coding-agent plugin.

In simple sentence: They are useless if not worse.

The trick is embarrassingly simple: None of them benchmarked against real long-horizon tasks.

Confusing those statements is not optimism...

We ran the plugins on a repository rewrite

FULL BENCHMARK

This was not a toy prompt asking for one function. The task was to rewrite the Rust eza repository as a behavior-compatible Python implementation. The agent had to inspect the reference project, reproduce the CLI in another language, and face 52 harness assertions.

Every published run used GPT-5.6-sol, High reasoning, and Codex CLI 0.144.1. The comparison contains exactly two runs per arm:

Ponytail r2/r3, both with full hook + skill activation;
RTK r2/r3, both with isolated RTK activation; and
two previously published no-plugin runs with the same task, model, reasoning level, and CLI version.

Arm	n	Harness score	Total tokens	Modeled cost	Rounds	Duration
No plugin	2	78.85%	6.660M	$5.281946	62.5	895s
Ponytail, full hook + skill	2	80.77%	-7.56%	-8.87%	-9.60%	+13.51%
RTK	2	76.92%	+13.20%	+7.18%	+44.00%	+40.69%

The mean movement is smaller than the ordinary two-run swing. This is a matched n=2 comparison, not a causal estimate.

The sanitized per-run data, computed summary, methodology, and 293-round activation audit are public. All six Codex processes exited 0 and produced complete usage and evaluator data. A run can still miss harness assertions; that is the score, not a crashed experiment.

Ponytail looks 8.87% cheaper. RTK looks 7.18% more expensive. If this were a plugin landing page, this is where somebody would choose the flattering row, enlarge the percentage, and quietly send the error bars on vacation.

The "saving" is smaller than ordinary run variance

The same agent, model, task, and configuration did not produce remotely stable bills across two repetitions:

Arm	Cost in the two runs	Cost range / mean	Token range / mean	Round range / mean
No plugin	$4.139647 - $6.424245	43.25%	53.02%	40.00%
Ponytail	$3.569452 - $6.057281	51.69%	57.36%	47.79%
RTK	$4.789893 - $6.532388	30.78%	39.75%	26.67%

Here, "range / mean" is the gap between the two runs divided by their mean. It is not a confidence interval; with n=2, pretending to have one would be statistical cosplay.

But the scale still matters. Ponytail's apparent 8.87% cost saving sits inside a 51.69% within-arm cost swing. RTK's apparent 7.18% cost increase sits inside a 30.78% swing. Even the no-plugin pair moves 43.25% without any plugin to praise or blame.

These data therefore do not identify a plugin effect. They show that natural trajectory variance is a more economical explanation for these small mean differences until a much larger repeated experiment separates signal from noise. Declaring victory from two runs while ignoring a within-group swing four to six times larger is not benchmarking. It is numerology with a README.

What the experiment does establish is simpler: a local compression claim does not reliably predict the complete-task bill. Ponytail's mean moved modestly down; RTK's moved up. Neither result resembles the giant percentage printed on the local optimization.

Here is the actual coding-agent bill

The broader repository dataset contains 140 Codex CLI Medium and High runs: 10,365 agent rounds, 901,608,531 tokens, and $680.34 in modeled API cost. No Tura runs are included.

What Codex consumed	Share of all tokens	Share of cost
Cached input	96.46%	63.91%
New uncached input	3.16%	20.94%
Model output	0.38%	15.14%

The complete calculation is in the repository's 140-run summary. Under the repository pricing model, uncached input costs $5/M, cached input $0.50/M, and output $30/M. Cached input is one tenth the price of new input.

The four published plugin runs had the same shape: cached input was 96.74% of Ponytail tokens and 97.27% of RTK tokens. Apparently the denominator did not install the plugin.

A coding agent repeatedly carries prompt, history, commands, and command results into later rounds. Shortening one fragment can produce an impressive local percentage while barely touching the expensive complete trajectory.

Prompt and LOC savings are especially good comedy

Ponytail's Codex rules contain about 569 tokens. Give the claim every advantage: put those rules in every one of the 10,365 rounds and shorten them by 90% with zero quality loss. The modeled saving across all 140 runs is about $2.98, or 0.44% of total cost.

That is roughly two cents per task. Please alert the finance department.

The LOC argument is worse. Recoverable final production code in the 140 runs contains 512,412 tokens, only 0.0568% of all tokens consumed. Suppose Ponytail magically removes 80% of every functional code token, never deletes behavior, and never causes another reasoning step. Even valuing every removed token at the expensive output rate, the saving is 1.81% of total task cost.

Less code can be better engineering. But using LOC reduction as evidence of a huge inference-cost reduction is like shortening item names on a restaurant bill and announcing that dinner is cheaper.

RTK's 90% still belongs to a tiny slice

Across the 140 runs, we could uniquely classify 1,082 RTK-supported shell calls containing 1,458,927 returned tokens. That payload is just 0.1618% of all task tokens. Apply a perfect 90% reduction to every eligible return and the directly attributable modeled saving is 0.96% of total cost.

To manufacture a larger ceiling, we also assumed every compressible output remains in context until the task ends and gets reread on every later round. Under that deliberately generous fantasy, universal lossless 90% compression reaches 5.72%.

So the marketing number can be 90% while the complete-task saving stays below 1%. It approaches 5% only after we grant permanent retention, perfect classification, perfect compression, and zero information loss. The rabbit is real; the hat is doing most of the work.

Real Token Saving

Tura is a local, open-source coding agent for developers who are tired of vague skill claims, token-saving extensions with no evidence, and agents without judgment wreck their repos.

Across 20 DeepSWE v1.1 tasks, tested 60 sessions with GPT-5.6 SOL at High reasoning effort, Tura creates a substantial token-budget advantage by reducing repeated context and model round trips. You can spend that advantage in two ways. Direct turns most of it into lower cost: 83.5% fewer aggregate tokens than the official Codex CLI High configuration, with a verifier success rate of 65.0% versus 60.0%. Balanced puts more of the saved budget back into reasoning, investigation, and verification. It reached an 80.0% success rate—20 percentage points higher than Codex CLI High—while still using 49.6% fewer tokens

REAL BENCHMARK

One outside paper is enough

Bai et al., How Do AI Agents Spend Your Money? analyze trajectories from eight frontier models on SWE-bench Verified. They report that agentic coding consumes about 1,000x more tokens than code reasoning or code chat, that input rather than output drives total consumption, and that runs on the same task can differ by up to 30x. Higher token use also did not reliably mean higher accuracy.

That is the only external paper needed here. Coding-agent cost is a trajectory problem with huge run-to-run variance. A local compression ratio is not a task-level economic result. It is a numerator looking for an unsuspecting denominator.

Dataset

The matched-run package lives in blog_data. The broader distribution and scenario report lives in assets/plugin-token-savings.

Ponytail may be useful as an anti-overengineering discipline. RTK may be useful as a terminal-output formatter. Test those benefits honestly. Just stop waving "90%" around as if percentages are transferable between denominators.

They are not. The calculator is free.

I open-sourced a macro execution layer to reduce coding-agent turns (60-task benchmark)

Yohji Sakamoto — Sat, 18 Jul 2026 22:44:18 +0000

Disclosure: I maintain Tura.

A coding agent often spends a separate model turn on each part of a routine workflow: inspect the environment, edit package files, patch the implementation, update tests, build, run lint/tests, then inspect Playwright media.

Tura experiments with a simpler execution model: describe that sequence as a macro and let the runtime execute the steps together, while still returning structured results to the agent. The goal is to reduce orchestration turns without hiding failures.

On our current 60-task DeepSWE set:

Configuration	Passes	Pass rate	Observed tokens	Rounds
Macro + backward reasoning	48/60	80.0%	229,695,477	2,017
Macro Direct	39/60	65.0%	75,108,167	969
Codex CLI Medium	38/60	63.3%	333,538,349	3,140
Codex CLI High	36/60	60.0%	455,742,296	6,074

The implementation, task data, and benchmark methodology are public:

The important caveat is that fewer turns do not automatically mean lower end-to-end cost. Retries, failures, cache behavior, and completed-task rate all belong in the denominator. I would especially appreciate feedback on the benchmark design and cases where macro execution makes debugging worse rather than better.