A program can pass every test in front of it and still become harder to change. A new branch lands inside an already crowded function. The next feature copies the branch and changes two conditions. A third feature adds a compatibility path around both. Each edit is locally reasonable. Together they turn the codebase into a record of patches rather than a design.
This failure mode matters more as coding agents produce larger changes. Correctness is unusually easy to verify: execute the program, compare its output with a hidden oracle, and reward the run that passes. Maintainability has no equally crisp oracle. It includes duplication, unnecessary indirection, coupling, names, module boundaries, and whether today’s structure can absorb tomorrow’s requirement. Those properties are partly measurable, but none reduces cleanly to one test result.
Sebastian at Earendil calls the accumulated result code sloppiness. His exploration points to SlopCodeBench, an experiment designed around a simple fact about real software: requirements arrive over time. Its results give us a better vocabulary for code that remains functional while its structure erodes.
A one-shot benchmark hides tomorrow
Most coding evaluations hand an agent a complete task and score the final repository. That setup measures whether the model can reach a destination when the destination is visible from the start. It does not measure whether an early choice leaves a good route to a destination disclosed later.
SlopCodeBench replaces one complete specification with a sequence of checkpoints. Its paper evaluates 20 problems across 93 checkpoints. Each problem starts small, then adds behavior that puts pressure on the implementation already in the workspace.
The code_search problem makes the design clear. Checkpoint one asks for a Python source searcher with exact and regular-expression rules. Later checkpoints add JavaScript and C++, AST patterns with metavariable capture, selector rules, automatic fixes, then Go, Rust, and Java. An early implementation that treats every rule and language as a special case can pass the first tests. A parser interface and a rule-dispatch layer may look unnecessary at that moment. Three requirements later, the difference becomes the whole problem.
Each checkpoint runs in a fresh container. The working directory carries forward, while installed packages, shell history, and agent session state reset. The next agent invocation sees the code its predecessor left behind and the new specification, much like a developer returning to an unfamiliar project after time away.
The benchmark specifies only observable CLI or API behavior. It avoids prescribing internal classes, function signatures, or modules because those would leak the design being evaluated. Tests remain hidden and interact through subprocesses or served APIs. This leaves the agent responsible for both behavior and architecture.
That is a harder and more useful contract than “make these visible tests green.”
Correct now is different from correct across time
SlopCodeBench separates four forms of success:
- Core tests cover behavior explicitly shown or stated in the current specification.
- Functionality tests probe more cases of the current behavior.
- Error tests check failure paths and invalid inputs.
- Regression tests repeat requirements from every earlier checkpoint.
A checkpoint is strictly correct only when all four groups pass. The benchmark also reports an isolated score without regression tests and a core score for the clearest part of the new requirement.
The gaps are revealing. Across the paper’s evaluated configurations, the best strict checkpoint solve rate was 17.2%. No agent completed any entire problem with all checkpoints correct. At the final checkpoint, strict correctness across configurations fell to 0.5%, while the chance of passing the explicit core behavior remained much higher. The ratio between core and isolated success widened from 1.4 times early in a problem to 13.3 times late in it.
The agents were still adding visible features. What they increasingly failed to do was preserve the complete contract around those features, especially error behavior and earlier requirements. Mean cost per checkpoint grew 2.9 times from the first part of a trajectory to the last, so more spending did not cancel the accumulated difficulty.
This distinction explains why a coding demo can feel excellent while a long-running repository becomes unstable. A demo rewards the newest path. Maintenance requires every old path to remain coherent while the new one joins it.
Two measurements for two kinds of decay
Lines of code are a useful warning light. If a modest feature adds thousands of lines, the change deserves inspection. But line count cannot distinguish a necessary protocol table from five copies of the same parser, and it becomes easy to game once it is a target.
SlopCodeBench uses two normalized signals instead: verbosity and structural erosion. They describe different failures and should be read together.
Verbosity measures the fraction of source lines that are redundant or unnecessarily elaborate:
verbosity = |AST-rule lines ∪ clone lines| / lines of code
The benchmark applies 137 structural-search rules with ast-grep to flag patterns that can be condensed. It combines those hits with detected clone lines, removes overlap, and divides by total logical lines. A line hit by several rules counts once.
The clone component matters because repeated structure drove most of the measured growth. Across agent trajectories, structural duplication rose by 66% in 72.1% of runs, while the density of AST-rule violations rose by a smaller 15.6%. The typical failure was not exotic bad syntax. It was another copy of logic that already existed.
Structural erosion asks where the codebase’s decision load lives. Each function receives a complexity mass:
mass(f) = cyclomatic_complexity(f) × √source_lines(f)
erosion = mass in functions with complexity > 10 / total mass
Cyclomatic complexity counts independent paths through a control-flow graph. Multiplying it by the square root of function length gives large, branch-heavy functions more weight without letting length dominate completely. Erosion is the share of all mass trapped in functions above the chosen complexity threshold.
This construction catches a common agent habit: extend the nearest function instead of reshaping the system. A codebase may gain helper functions and still erode if the important decisions keep accumulating inside one dispatcher.
Neither score claims to measure every part of good design. Together they make two expensive review questions cheap enough to ask at every checkpoint: how much code says the same thing twice, and how much of the system’s branching logic is collapsing into a few places?
The 1,099-line dispatcher
The paper’s circuit_eval trajectory provides a concrete picture. The program begins as a parser and evaluator for circuit files. Over eight checkpoints it gains cycle detection, optimization passes, equivalence checks, truth tables, linting, statistics, graph output, and more.
In one Opus 4.6 run, main() grows from 84 to 1,099 lines. Its cyclomatic complexity rises from 29 to 285. Nine command branches repeat argument-parsing patterns, and the largest command adds hundreds of lines inside the same dispatcher.
Every added command has an obvious local home: the existing command switch. That is precisely why the structure decays. The cost of creating a command abstraction is immediate, while the benefit arrives only with later commands. An agent optimizing for the current checkpoint repeatedly chooses the small local patch. The workspace then makes that choice the starting point for the next run.
The benchmark saw the pattern broadly. Mean high-complexity function count grew from 4.1 to 37.0, and mean maximum cyclomatic complexity grew from 27.1 to 68.2. Erosion increased through 80% of trajectories; verbosity increased through 89.8%.
Early architecture also predicted later correctness. All seven configurations in the paper’s code_search comparison passed the first two checkpoints, yet their internal designs already differed. When AST metavariable matching arrived, results split sharply. Implementations with extensible dispatch absorbed the new rule kind. Hardcoded implementations required invasive changes and lost tests. A benchmark that stopped at checkpoint two would have called them equal.
Human repositories are an imperfect but useful ruler
A sloppiness score needs scale. Is 0.3 normal for a growing project? Does every mature codebase drift upward?
The researchers measured 48 maintained Python repositories as a calibration panel. These are not matched human solutions to the benchmark problems; they differ in age, purpose, and development process. The comparison therefore cannot prove that a particular agent is worse than a human developer on an identical task. It can show whether agent output occupies the same structural range as maintained software.
The gap was large. The human panel averaged 0.15 ± 0.06 verbosity, while 990 agent checkpoints averaged 0.33 ± 0.10. Human erosion averaged 0.31 ± 0.17; agent erosion averaged 0.68 ± 0.20. Only one of the 48 repositories exceeded the agent mean for verbosity.
The trajectory comparison is stronger than a snapshot. For 20 human repositories, the authors sampled up to 30 source-changing commits, yielding 568 historical checkpoints. Human metrics tended to plateau. Agent metrics climbed. Median verbosity growth from first to last was 25% for the repositories and 43% for agent runs. Erosion rose in 55% of human repositories and 79% of agent trajectories.
Maintained human code is not clean by definition, and some sampled commits may include AI assistance. The useful finding is directional: ordinary repository history showed less consistent deterioration than repeated agent self-extension under the benchmark.
Better prompts move the start, not the slope
The obvious intervention is to tell the agent to plan and keep the code clean. The researchers tested baseline, anti-slop, and plan-first prompts on two models.
Those instructions helped the initial result. On GPT 5.4, the anti-slop prompt reduced verbosity in 19 of 20 problems and erosion in all 20. But the quality curves remained almost parallel to baseline as checkpoints accumulated. The starting point moved down; the rate of degradation did not change significantly.
Cleaner output also cost more. For GPT 5.4, anti-slop prompting raised run cost by 47.9%, from $304 to $450, without a consistent pass-rate improvement. Across the two tested models, large reductions in the structural metrics did not produce statistically detectable gains in the benchmark’s correctness categories.
This does not mean instructions are useless. It means a reminder in the prompt cannot replace feedback in the loop. If the reward sees only today’s tests, design quality remains secondary. The agent needs tools and policies that make structural change visible while it is still cheap.
A practical quality loop for agent-written code
The benchmark suggests a workflow that teams can apply without pretending any score is truth.
- Keep regression tests cumulative. A feature is complete only when old behaviors and error cases remain intact.
- Measure changes, not just snapshots. Record logical LOC, clone ratio, high-complexity functions, verbosity, and erosion for every merge. A slope is more informative than a universal threshold.
- Inspect the concentration behind the score. When erosion rises, list the functions contributing the most mass. The remedy should target those functions rather than chase a global number.
- Set a refactoring trigger. If a dispatcher gains its third parallel branch or clone growth jumps, pause feature work and extract the shared shape before another checkpoint builds on it.
- Give each session a map. Preserve architecture notes, decisions, module responsibilities, and rejected approaches in the repository. The working tree should carry more than code when session context resets.
- Use metrics as review queues. They should decide where a human looks, not whether a pull request is automatically “good.”
- Test the next likely requirement. During review, ask what change would make the current design painful. That counterfactual often reveals the shortcut hidden by today’s green suite.
The most useful unit is the trajectory. A large function can be justified. A repository can contain duplication for compatibility. What deserves attention is repeated movement toward more duplication, more concentrated branching, and weaker regression preservation.
What the numbers do not capture
Verbosity and erosion are proxies built from human choices. The 137 ast-grep rules encode judgments about needless patterns. Clone detection depends on thresholds. Cyclomatic complexity treats branches as costly even when a long decision table may be the clearest representation. The cutoff at 10 comes from a conventional tool boundary, not a law of software.
The metrics can also miss bad architecture spread across many small functions. Excessive coupling, unstable dependencies, poor cohesion, misleading names, and needless abstraction may leave both scores low. A codebase can game verbosity by compressing readable code or game erosion by slicing one tangled function into a web of tightly coupled helpers.
The benchmark itself evaluates Python implementations, uses synthetic staged problems, and resets more context than some real teams preserve. Its maintained-repository panel is a calibration reference rather than a controlled human baseline. These limits narrow the claims, but they do not erase the central observation: passing the newest tests says little about how well a design survives repeated extension.
The engineering opportunity is to make that survival visible. Correctness tests answer whether the current behavior works. Trajectory metrics show where maintenance cost is accumulating. Human review supplies the context neither can encode. Used together, they turn “this code feels sloppy” into a specific conversation about duplicated surface area, concentrated decisions, regression loss, and the next change the design must endure.
Top comments (0)