Coding Agents Fail Differently by Stratum

Coding-agent leaderboards still collapse unlike tasks into one pass rate, and that collapse is the product. A suite that mixes short Python katas with multi-file refactors is not one experiment. The mix of languages, difficulty, and hidden tests moves the number before any model runs. A protocol that freezes strata, sample size, and seeds can publish an interval another lab can rerun.

Sports desks would not average a marathon clock with a hundred-meter sprint and call the blend fitness. Coding evaluations still fold regex patches and schema migrations into one cheerful percent. Readers then treat that percent as a property of the model rather than of the basket. The honest unit of reporting is the stratum, with any overall figure labeled as a weighted summary only.

This article proposes a frozen evaluation protocol, not a vendor bake-off and not a claim about any named model. The protocol treats dataset mix, trial count, and seed list as controls that must exist before a percentage is allowed to print. The scripts below are a reproducible method other teams can run on their own traces, not unpublished private scores. No pass rates from a hidden leaderboard appear, because those numbers would recreate the original problem.

The dataset is the first control, and it is a mix rather than a pile of files. Each task needs a stable identifier, a language, a difficulty band, and a visibility flag for hidden tests. A holdout that never appears in prompt text belongs in a separate column so leakage can be audited later. If SQL migrations and Python katas share one average, the published figure mostly reports how the author weighted the homework.

A useful analogy is a kitchen scale that tares itself after every ingredient except salt. The displayed mass then tracks the salt habit of the cook, not the recipe. Coding scores behave the same way when easy autocompletes dominate the suite and rare multi-file repairs hide in the tail. Stratified sampling does not make models smarter; it stops the tail from being outvoted by trivia.

Metrics have to survive a rerun on another machine, which a naked percentage never does. Pass rate remains allowed, but only beside trial count, a confidence interval, and the per-stratum rates that produced it. Wall-clock timeout belongs in the same record because a slow agent can look identical to a wrong agent after the harness kills the process. Temperature and decoding seed belong there too, since they decide whether two labs even ran the same stochastic experiment.

The Wilson score interval is a boring choice on purpose, because coding trials are often small and extreme. A point estimate of eight successes in ten looks decisive until the interval reminds the reader that ten is a rumor. Binomial independence is an assumption, not a gift from the runtime, and shared caches can still correlate trials. The scorecard therefore prints the assumption in the same JSON object as the percentage, instead of burying it in a blog caveat.

Controls have to be frozen before the first generation, not edited after a disappointing cell in a spreadsheet. Sample size n is a budget, not a knob for hunting a prettier headline after seeing the traces. Seeds are a list committed to version control, not a random draw that changes when the author dislikes a failure. Timeout, temperature, and the stratum weights are likewise protocol fields; changing them midstream creates a new benchmark that should not keep the old name.

The proposed harness never calls a model. It grades recorded Bernoulli outcomes against a frozen protocol so the statistics can be checked without network access. Teams may fill outcomes.csv from local runners, continuous integration, or a hosted coding assistant, as long as the protocol file does not change under the data. The command surface is intentionally small so the methodology stays visible in review.

Create a protocol file that refuses to be vague. The weights must sum to one, and the seed list must match n exactly, or the loader should stop.

# PROTOCOL.toml — proposed frozen controls, not an executed vendor study
name = "coding-agent-strata-v1"
n = 10
timeout_s = 120
temperature = 0.0
seeds = [101, 102, 103, 104, 105, 106, 107, 108, 109, 110]

[weights]
python_easy = 0.40
python_refactor = 0.25
sql_migration = 0.20
ts_api = 0.15

Pair that file with a tiny task catalog. Hidden tests stay out of any prompt template that later models will see, and each row names exactly one stratum.

{
  "tasks": [
    {"id": "py-kata-017", "stratum": "python_easy", "hidden_tests": true},
    {"id": "py-ref-004", "stratum": "python_refactor", "hidden_tests": true},
    {"id": "sql-mig-009", "stratum": "sql_migration", "hidden_tests": true},
    {"id": "ts-api-021", "stratum": "ts_api", "hidden_tests": true}
  ]
}

Record one row per trial rather than a pre-averaged percentage. A later script can recompute intervals; a spreadsheet cell cannot reconstruct the seed that produced a lucky pass.

task_id,stratum,seed,passed,elapsed_ms
py-kata-017,python_easy,101,1,1840
py-kata-017,python_easy,102,1,1912
py-ref-004,python_refactor,101,0,120000
sql-mig-009,sql_migration,101,1,4022
ts-api-021,ts_api,101,0,8871

The grader loads those three artifacts, checks that every stratum in the catalog has a frozen weight, and refuses to print when trial counts drift. Wilson intervals are computed per stratum first. A weighted headline prints only after that gate, and it is labeled as a summary rather than as the result.

# protocol.py — proposed scorecard, not a published leaderboard
from __future__ import annotations

import csv, json, math, sys
from collections import defaultdict
from pathlib import Path

try:
    import tomllib
except ModuleNotFoundError:  # Python 3.10
    import tomli as tomllib  # type: ignore

Z = 1.959963984540054  # ~95% Wilson


def wilson(successes: int, n: int) -> tuple[float, float, float]:
    if n <= 0:
        raise ValueError("n must be positive before a rate is printed")
    p = successes / n
    denom = 1.0 + Z * Z / n
    center = (p + Z * Z / (2 * n)) / denom
    half = (Z / denom) * math.sqrt(p * (1 - p) / n + Z * Z / (4 * n * n))
    return p, max(0.0, center - half), min(1.0, center + half)


def load_protocol(path: Path) -> dict:
    proto = tomllib.loads(path.read_text())
    seeds = list(proto["seeds"])
    n = int(proto["n"])
    if len(seeds) != n or len(set(seeds)) != n:
        raise SystemExit("seed list must be unique and exactly length n")
    weights = proto["weights"]
    total = sum(weights.values())
    if abs(total - 1.0) > 1e-9:
        raise SystemExit(f"stratum weights must sum to 1.0, got {total}")
    return proto


def main() -> None:
    proto = load_protocol(Path("PROTOCOL.toml"))
    catalog = json.loads(Path("tasks.json").read_text())["tasks"]
    by_id = {row["id"]: row for row in catalog}
    wanted = set(proto["weights"])
    found = {row["stratum"] for row in catalog}
    if wanted != found:
        raise SystemExit(f"catalog strata {found} != protocol weights {wanted}")

    grouped: dict[str, list[int]] = defaultdict(list)
    with Path("outcomes.csv").open() as handle:
        for row in csv.DictReader(handle):
            task = by_id[row["task_id"]]
            if task["stratum"] != row["stratum"]:
                raise SystemExit(f"stratum drift on {row['task_id']}")
            if int(row["seed"]) not in proto["seeds"]:
                raise SystemExit(f"seed {row['seed']} is not in the frozen list")
            grouped[row["stratum"]].append(int(row["passed"]))

    strata = {}
    headline = 0.0
    for name, weight in proto["weights"].items():
        trials = grouped.get(name, [])
        if len(trials) != proto["n"] * sum(1 for t in catalog if t["stratum"] == name):
            raise SystemExit(f"{name} does not have n trials for every task")
        p, lo, hi = wilson(sum(trials), len(trials))
        strata[name] = {
            "n": len(trials),
            "pass_rate": round(p, 4),
            "wilson95": [round(lo, 4), round(hi, 4)],
            "weight": weight,
        }
        headline += weight * p

    scorecard = {
        "protocol": proto["name"],
        "controls": {
            "n_per_task": proto["n"],
            "timeout_s": proto["timeout_s"],
            "temperature": proto["temperature"],
            "seeds": proto["seeds"],
        },
        "independence_assumption": "trials treated as Bernoulli; shared caches can invalidate this",
        "strata": strata,
        "weighted_summary_only": round(headline, 4),
        "do_not_cite_without_strata": True,
    }
    Path("scorecard.json").write_text(json.dumps(scorecard, indent=2))
    json.dump(scorecard, sys.stdout, indent=2)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main()

Run the method against fixture traces before wiring any generator. The exit status is part of the protocol: a missing stratum is a failed experiment, not a blank cell to fill later.

python3 protocol.py
python3 -c "import json; print(json.load(open('scorecard.json'))['strata'])"

A scorecard that cannot be cited without strata is the point of the extra JSON noise. Reviewers can reject a pull request that quotes weighted_summary_only in a title while omitting wilson95. Continuous integration can store PROTOCOL.toml beside the outcomes so later authors cannot quietly reweight Python trivia after the run. The percentage is then a derived view, like a chart, rather than the scientific object.

Hosted runners become relevant when the protocol needs many independent trials and local machines are already booked. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option, which can replay the same frozen protocol without turning this text into a quota sheet. The method still requires recorded outcomes, frozen strata, and an interval; a free endpoint does not repair a blended task mix.

The method has sharp limits, and those limits are part of the result. Wilson intervals assume independent Bernoulli trials, which shared repositories, warm language servers, and leaked hidden tests can quietly violate. Stratification cannot save a catalog whose easy band is still drawn from public interview questions that models have already seen. Small n produces honest intervals that are too wide to rank two close systems, which is an answer, not a defect in the printer.

Procurement teams should not use this scorecard as a sole vendor gate, because mix design still encodes the buyer's taste. Marketing teams should not harvest weighted_summary_only for a banner, because the field is labeled as a summary on purpose. Classrooms that need a binary lab grade can keep pass or fail per student and skip intervals entirely. Researchers who cannot freeze seeds or hidden tests should not pretend the percentage is comparable across weeks.

Cheap generation has made it easy to emit code and even easier to emit leaderboards that look like evidence. The expensive part is now the boring bookkeeping: a versioned mix, a committed seed list, and an interval that survives contact with another lab. The next honest publication is a per-stratum record with n written beside the rate, not a single percent asked to represent five languages at once.