DEV Community

Robin
Robin

Posted on

If 30% of Coding Tasks May Be Broken, Your Leaderboard Needs an Uncertainty Budget

#ai #testing #reliability #benchmarking

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial.

Primary source: OpenAI, “Separating signal from noise in coding evaluations”.

The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators.

Model task state separately from model result 

task validity: unreviewed | valid | broken | disputed
model result:  pass | fail | infrastructure_error | missing
Enter fullscreen mode Exit fullscreen mode

Never convert infrastructure_error to model failure without reporting that policy. Never delete broken tasks while retaining an old score label.

A row needs provenance:

{
  "task_id": "repo-issue-17",
  "dataset_revision": "sha256:...",
  "harness_revision": "git:...",
  "model_config": "immutable-config-id",
  "validity": "disputed",
  "result": "pass",
  "review_revision": 3,
  "evidence": ["fixture.log", "review.json"]
}
Enter fullscreen mode Exit fullscreen mode

Publish three denominators

Let:

  • P_v, N_v: passes and total among reviewed-valid tasks;
  • P_a, N_a: passes and total across all attempted tasks;
  • D: disputed tasks.

Report:

valid-only score = P_v / N_v
all-attempted score = P_a / N_a
uncertainty interval = score if every disputed task hurts conclusion
                       .. score if every disputed task helps conclusion
Enter fullscreen mode Exit fullscreen mode

This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity.

A tiny sensitivity calculator 

#!/usr/bin/env python3
import json, sys

rows = [json.loads(line) for line in open(sys.argv[1]) if line.strip()]
valid = [r for r in rows if r["validity"] == "valid"]
disputed = [r for r in rows if r["validity"] in ("unreviewed", "disputed")]
attempted = [r for r in rows if r["result"] in ("pass", "fail")]

rate = lambda passed, total: passed / total if total else float("nan")
valid_pass = sum(r["result"] == "pass" for r in valid)
all_pass = sum(r["result"] == "pass" for r in attempted)

# Bounds ask how unresolved validity could change the valid-only denominator.
low = rate(valid_pass, len(valid) + len(disputed))
high = rate(valid_pass + len(disputed), len(valid) + len(disputed))

print(json.dumps({
  "valid": {"pass": valid_pass, "n": len(valid), "rate": rate(valid_pass, len(valid))},
  "attempted": {"pass": all_pass, "n": len(attempted), "rate": rate(all_pass, len(attempted))},
  "disputed_n": len(disputed),
  "validity_sensitivity": [low, high]
}, indent=2))
Enter fullscreen mode Exit fullscreen mode

Run it against an immutable JSONL export:

python3 sensitivity.py results.jsonl
Enter fullscreen mode Exit fullscreen mode

The calculator deliberately makes a pessimistic simplification: every unresolved task can move the numerator one way or the other. A real audit can narrow the interval by classifying tasks and preserving reviewer disagreement.

Inject benchmark failures

Before trusting the pipeline, simulate:

  1. test requires an unavailable external service;
  2. repository revision cannot build;
  3. issue statement conflicts with tests;
  4. gold patch depends on hidden state;
  5. harness timeout occurs after a correct partial change;
  6. duplicate task appears under a new ID;
  7. review changes broken to valid after score publication.

Invariants:

- Raw attempts are append-only.
- Validity revisions never overwrite earlier reviews.
- Every published score names dataset, harness, and review revisions.
- Reclassification triggers deterministic recomputation.
- Infrastructure errors have their own count.
- Pairwise model comparisons use the same eligible task set.
Enter fullscreen mode Exit fullscreen mode

That last invariant matters. Comparing Model A on 600 valid tasks with Model B on 650 partly different tasks introduces a denominator change disguised as capability change.

Acceptance rule for model selection

Suppose two models differ by two percentage points, but either model's score moves by five points under plausible task-validity decisions. The benchmark cannot support the ranking on its own.

Use a rule such as:

accept_model_change_only_if:
  common_valid_tasks: ">= 500"
  infrastructure_error_rate: "< 1%"
  disputed_fraction: "< 5%"
  observed_margin: "> validity_sensitivity + run_variance"
  production_canary: "passes workload-specific gate"
Enter fullscreen mode Exit fullscreen mode

Thresholds are policy choices, not universal constants. Name the owner and revisit date.

Keep benchmark and deployment claims separate

A cleaner coding score does not prove lower incident rates, faster review, or better performance in your repository. Pair public benchmark evidence with a private, permission-safe canary containing your languages, tests, dependency graph, and review rules.

The audit's most useful lesson is architectural: a benchmark task is not merely an array element. It is a versioned claim that the input, oracle, environment, and scoring path can distinguish capability from noise.

If your preferred model changes when disputed tasks are removed, do you publish the new winner—or publish that the ranking is unresolved?

Top comments (0)