Attributed compile + two small offline scripts
Primary source: Dex Horthy (with Gabe Orlanski), SlopCodeBench update 10/2026 - Opus 5.5 update, Astra improves by 1.5% over gpt-5.5, xhigh considered harmful (HumanLayer, 2026-10-15).
Benchmark paper: Gabriel Orlanski et al., SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (arXiv 2603.24755).
Second data point: paddo, The Default Was Right: Opus 5.5, Sonnet 5.5 and GPT-6 Sol Effort Levels on Real Work (2026-09-29).
Discussion: Hacker News thread (283 points on 2026-10-21).
I did not run SlopCodeBench or any coding agent. Everything below re-analyzes the published numbers offline, and every number is credited to its source.
HumanLayer's October SlopCodeBench (SCB) write-up made the Hacker News front page this week with a provocative subtitle: "\"xhigh considered harmful.\" The finding behind it: Opus 5.5 at medium effort got 28 of 196 checkpoints strictly passing, and at xhigh it got 25, at 4.22× the estimated cost ($193.47 vs $817.04). GPT-6 SOL went 30 → 26 at 2.11×. Only GPT-6 Astra improved, 29 → 32, at 2.91×."
To his credit, Horthy hedges right in the post: one campaign per setting, so "this doesn't prove that more reasoning hurts coding." The HN thread split along familiar lines: "I leave it on max," "medium is my sweet spot," and "run it five times and put a confidence interval on it." This post takes the last commenter's side. I ask what the numbers can support, re-rank the leaderboard by cost per pass, and give you a small script to run the same A/B on your own repo.
What SCB measures (and why it's harder than SWE-style benchmarks)
SCB gives an agent a problem in stages. It builds a CLI or API from a spec, then gets new requirements and must extend its own earlier code, with no fresh reference solution and no visible tests. The paper has 36 problems and 196 checkpoints. Hidden tests come in four groups: Core, Error, Functionality, and Regression (all earlier checkpoints' tests).
- Strict pass: every test category passes at that checkpoint, Regression included.
- Core pass: only the core tests for that checkpoint pass.
That gap is the whole story. In Horthy's run, the models score 65.8-74.0% on core but only 12.2-16.3% strict. Agents build the new feature far more often than they keep the old features working. The paper also found structural erosion rising in 77% of trajectories and verbosity in 75.5%, and agent code was 2.3× more verbose than a panel of 473 human Python repos.
Setup for anyone reproducing it (from the post): pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case pass policy, Docker with Python 3.12 and uv. Problems live at gabeorlanski/scb-problems.
Run 1: is 28 vs 25 a signal?
scb_effort_math.py takes the published table and runs standard proportion tests, stdlib only. One caveat applies to everything below: the tests treat the 196 checkpoints as independent coin flips. They aren't. Checkpoints in one problem share code, and a single regression can wipe out every later strict pass. So the intervals I report are too narrow, which means the real picture is even noisier.
From my run on this box (scb_effort_math.py, output trimmed):
Opus 5.5 strict 28->25 (-3), cost x4.22 (+$623.57); no extra strict passes bought
Fisher exact two-sided p (unpaired, indep. approx) = 0.768
paired best case: 3 discordant all one way -> exact sign-test p = 0.250
GPT-6 SOL strict 30->26 (-4), cost x2.11 (+$271.63)
paired best case: 4 discordant all one way -> exact sign-test p = 0.125
GPT-6 Astra strict 29->32 (+3), cost x2.91 (+$497.57); $165.86 per extra strict pass
paired best case: 3 discordant all one way -> exact sign-test p = 0.250
The "paired best case" line matters most. Medium and xhigh ran on the same 196 checkpoints, so the right test only looks at checkpoints where the two settings disagree. A net gap of 3 needs at least 3 disagreements. Even if all 3 went the same way, the exact two-sided p-value is 2 × 0.5³ = 0.25. No arrangement of these results can make the gap significant. The 95% Wilson interval for Opus Medium's strict rate (independence approximation) is 10.1%-19.9%, and all ten campaigns' strict counts (24-32) fall inside it.
How much data would settle it? Telling 14.29% from 12.76% at α = 0.05 with 80% power takes roughly 7,800 independent checkpoints per setting, about 40 full SCB campaigns each. Nobody is paying for that.
The honest reading: on strict passes, SCB can't tell medium from xhigh. "Harmful" overstates it, and so does "xhigh is better."
Run 2: cost per pass is where the signal is
Cost doesn't have that noise problem. The gaps are 2-4×, not ±3 checkpoints. Divide each campaign's estimated cost by its strict passes (same script, trimmed columns):
campaign strict $ $/strict
GPT-6.1 SOL Medium 29/196 178.51 6.16
Opus 5.5 Medium 28/196 193.47 6.91
GPT-6 SOL Medium 30/196 245.69 8.19
GPT-6 Astra Medium 29/196 261.05 9.00
GLM 5.3 XHigh 24/196 365.13 15.21
GPT-6 SOL XHigh 26/196 517.32 19.90
GPT-6 Astra XHigh 32/196 758.62 23.71
GPT-5.6 SOL XHigh 28/196 753.30 26.90
Opus 5.5 XHigh 25/196 817.04 32.68
Fable 5.1 XHigh 26/196 1550.07 59.62
All four medium campaigns are cheaper per strict pass than all six xhigh campaigns. Astra XHigh is the one setting that bought extra passes, at about $166 per extra checkpoint over Astra Medium. That price might be worth paying on a hard migration. It's a bad default for everyday edits.
paddo's independent bench points the same way. It used six real merged changes in a production TypeScript monorepo, 20 runs per level, graded by the original fix's tests. Opus 5.5 got 13/20 clean runs at medium, 12/20 at xhigh, and 11/20 at max, for $44.10, $139.79, and $307.16. His one exception: Sonnet 5.5 at max hit 13/20 versus 11/20 at medium, for about 35× the cost. My script puts that gap at Fisher p ≈ 0.75. paddo also warns that his Claude effort levels are inferred, since the requests don't record the setting.
Two benchmarks built differently, both noisy on pass counts, both clear on cost. Treat effort as a spending knob, not a quality knob. paddo's phrasing is "a spend dial."
Run 3: an effort A/B on your own tasks
Use your own repo as the benchmark. Pick 10-20 merged changes that each have tests, replay each change from its parent commit at two effort levels, and record whether the full suite passes. Two key facts from the docs (I did not run either command):
# Claude Code: per-session flag (also /effort, or CLAUDE_CODE_EFFORT_LEVEL)
claude -p "$(cat ticket.md)" --effort medium
# Codex CLI: per-run config override (persistent key lives in ~/.codex/config.toml)
codex exec -c model_reasoning_effort='"medium"' "$(cat ticket.md)"
Log one row per run, task_id,effort,run,passed,broke_existing,cost_usd, and pair the runs by task:
python effort_ab.py results.csv medium xhigh
effort_ab.py reports strict pass rate, regressions, cost per strict pass, and an exact paired sign test on the tasks where the two settings disagree. Its rule is deliberately conservative: switch to the pricier setting only if it wins on paired disagreements with p < 0.10. Otherwise keep the cheaper one as the default and save the pricier one for retries on failures, which is also paddo's conclusion. I smoke-tested the script only on clearly labeled synthetic data (demo_results_SYNTHETIC.csv). Those numbers say nothing about any model, so they aren't quoted here.
Three habits from the HN thread are worth stealing either way. Run a formatter and dead-code warnings as hard gates, because several commenters say Astra/SOL output comes back with odd or missing line breaks. Make the agent run the full suite before it finishes, since regressions are what separate core from strict. And read the diff size, because paddo saw Opus's diff on one change grow from about 480 added lines at medium to about 830 at max.
What I didn't test
- Any model, any harness, any effort level. No SCB campaign, no paddo replay.
- Whether SCB's slop metrics favor xhigh. Horthy reports Opus XHigh's flagged-code share at 77.75% versus 94.25% for Medium, and he notes the linters may not cover newer models' patterns. I didn't touch the quality data.
- The
claude -p ... --effortandcodex exec -c ...lines are taken from current docs, not executed. -
highandmaxaren't in the SCB effort comparison, which several HN commenters pointed out.
Why this matters
When most of your agent runs fail on regressions, more thinking mostly buys more tokens and bigger diffs. Default to medium, make the agent run the full test suite before it finishes, and escalate effort only as a retry. If you want to know whether xhigh helps on your codebase, a paired test on 20 of your own merged changes will tell you more than any leaderboard headline.
Written by YongBo Yu, AI engineer in Toronto. More at yongbo-yu.vercel.app and GitHub.
Top comments (0)