Your coding agent scores well on SWE-bench. You ship it into a real workflow where it has to extend its own code across multiple sprints. What happens?
SlopCodeBench just measured exactly that — and the results should change how you think about coding agent deployments.
The Benchmark Nobody Was Running
SlopCodeBench (UW-Madison, MIT, Washington State) tests 11 coding agents across 20 problems and 93 checkpoints. At each checkpoint, the agent receives an updated spec and must extend its own prior code — not a reference implementation. Its architectural decisions at checkpoint 1 become the foundation for checkpoint 2, and so on.
They tracked two quality signals at every step:
- Structural erosion — fraction of complexity mass concentrated in high-complexity functions (cyclomatic complexity > 10)
- Verbosity — redundant/duplicated code as a fraction of total lines
The Results
No agent solved any problem end-to-end. Zero. Across all 11 models.
- Best checkpoint solve rate: 17.2%
- Structural erosion rose in 80% of trajectories
- Verbosity rose in 89.8% of trajectories
- Agent code is 2.2x more verbose than 48 maintained open-source Python repos
- Human repositories stay roughly flat — agent code deteriorates with each iteration
The Prompt That Didn't Fix It
Quality-aware prompts reduce initial verbosity and erosion by up to a third. But they don't slow the degradation rate, improve pass rates, or reduce cost. Agents start cleaner and degrade at the same rate.
Why This Matters
Pass-rate benchmarks (SWE-bench, HumanEval) are single-shot. They don't give the agent's code a future it has to build on. If your agent is writing code that will be extended — by itself, by other agents, or by your engineers — those scores tell you almost nothing about how that code will age.
Production implications:
- Architectural review can't be automated away — complexity concentrating in hot functions is exactly what human architects catch
- The longer the horizon, the less benchmark scores predict real performance
- 2.2x verbosity isn't aesthetic — it's more tokens per turn, more bug surface, more context cost
- Quality prompting is a partial mitigation, not a fix
Pass-rate is what you optimize for when building demos. Extension robustness is what you need when building software. Only one is being measured.
Paper: https://arxiv.org/abs/2603.24755 · Leaderboard: https://www.scbench.ai
Originally published at vsebude.it
Top comments (1)
Extension robustness is the missing benchmark for real agent work. A one-shot pass rate does not tell you whether the code can survive the next requirement. I would love to see more evals where the agent has to live with its own earlier design decisions.