A benchmark that only checks the examples in the prompt is measuring the wrong thing.
I ran into this while building a small code-repair evaluation for the Kaggle Benchmarking Challenge. The goal was not to ask models whether they could spot an obviously broken line. It was to find out whether they could preserve behavior that was not directly demonstrated in the prompt.
The failure I wanted to catch
Many code bugs are quiet. The function returns something plausible, passes the examples in the issue report, and still breaks an invariant that matters in production.
The benchmark I built, invariant-repair-suite, contains eight Python repair cases:
- Empty-input boundaries
- Stable ordering
- Cursor pagination
- Deterministic conflict resolution
- Strict duration parsing
- Path containment
- Retry budgets
- Stateful LRU behavior
Each case gives a model three things: a contract, a deliberately buggy implementation, and two visible examples. The evaluator extracts the returned Python code, executes it, and checks both visible and hidden cases.
The score is intentionally boring:
case_score = 0.25 * visible_regression_rate + 0.75 * hidden_edge_case_rate
Invalid or unexecutable code gets zero. That makes the benchmark less interested in how persuasive an explanation sounds and more interested in what the code actually does.
A local sanity check is not a leaderboard
Before involving a model, I ran the same harness against reference fixes and the original buggy implementations.
The reference fixes scored:
- Overall score:
1.0 - Hidden edge-case accuracy:
1.0 - Visible regression rate:
1.0
The buggy starters scored:
- Overall score:
0.570312 - Hidden edge-case accuracy:
0.558824 - Visible regression rate:
0.5
These numbers are only a harness validation. They are not model results and should not be read as a claim about any model. The useful observation is that the benchmark has enough signal to distinguish a correct implementation from a plausible but incomplete one.
The per-case starter scores were also revealing:
- Strict duration parsing:
0.375 - Retry budgets:
0.15 - Path containment:
0.85
The path case is the one I would keep in a real evaluation. A prefix check can look correct and still allow a sibling path such as /srv/app_evil to escape /srv/app. Passing the happy path tells us very little about whether the security boundary is correct.
What this changes about benchmark design
The first design rule I would keep is: do not let the prompt contain every test that determines the score.
Visible examples are still useful. They make the task solvable and give the model a contract in concrete terms. But if every scoring condition is visible, the benchmark mostly measures instruction following and example copying.
The second rule is: make regressions part of the score.
A repair that fixes a hidden edge case by breaking ordinary behavior is not a successful repair. Weighting visible regression checks separately prevents a clever rewrite from winning by accident.
The third rule is: prefer executable evidence over a second model as a judge.
A model judge can be useful for open-ended tasks, but code has an advantage: the output can be run. Deterministic assertions make the benchmark easier to reproduce, easier to debug, and less vulnerable to judge drift.
What I would measure next
The next version should compare the same cases under different conditions:
- Zero-shot repair versus repair with a short execution trace
- Full-function rewrites versus minimal patches
- Models with and without access to a Python runner
- Mutation testing, where small changes to the contract create new hidden cases
I would also publish the raw transcripts and separate infrastructure errors from incorrect answers. A benchmark that cannot tell those apart is difficult to improve.
The broader lesson is simple: the interesting question is rarely whether a model can produce a convincing patch. It is whether the model notices the behavior that was left out of the prompt, and whether its fix keeps the rest of the system intact.
The benchmark is inspired by the Kaggle Benchmarking Challenge and uses the Kaggle Benchmarks SDK as the execution model.
Top comments (0)