Stratify the Task Pack Before Averaging Agent Scores

A single pass rate across mixed coding tasks remains a marketing artifact rather than an engineering measurement. Honest agent comparison begins with a frozen stratified task pack, a paired control run, and metrics that refuse to collapse unlike work. Teams that publish one percentage after mixing refactors, bug fixes, and greenfield scaffolds are averaging bicycles with freight trucks. The method below treats dataset structure as the first control, not a footnote after the model call.

Recent discussion about coding agents often jumps from a demo transcript to a headline number as if those two objects were the same kind of evidence. A demo is a path through one prompt, one repository, and one reviewer mood, while a benchmark is a locked population of tasks with named families and a refusal to average them too early. When the population is undefined, the percentage is a slogan wearing a decimal point. The workflow here is a replay harness, not a leaderboard costume.

A useful analogy is vehicle testing on a mixed road that never labels the surface. Highway mileage, city mileage, and mountain mileage can each be honest inside their own family, yet a blended figure hides which surface produced the win. Coding-agent tasks behave the same way once refactors, test repairs, documentation edits, and scaffold generation share one mean. The candidate that looks strongest on the blend may be weakest on the family the team actually ships.

The dataset is therefore a pack, not a folder of leftover tickets. Each item carries a stable task identifier, a family label, a frozen fixture hash, a timeout, and an oracle that does not move during the comparison window. Families stay coarse enough to fill, such as refactor, bugfix, tests, and scaffold, because twelve tiny families with two items each recreate the original averaging problem under a fancier name. Operators who cannot name the family before the first agent call are not ready to report a score.

Metrics follow the pack instead of leading it. Per-family pass rate, median time to a green oracle, timeout rate, and a paired delta against a frozen control agent are the columns that survive review. A global mean is allowed only after a homogeneity check shows the families are not arguing with each other. When they argue, the table stays split, and the narrative stays split, even if that makes the blog post less dramatic.

Controls sit beside the candidate rather than in a later appendix. The same pack hash, the same seed, the same wall-clock budget, and the same oracle binary run twice: once for the control configuration and once for the candidate. Order is randomized inside a family so the second run does not inherit a warm cache as if it were skill. Any number that cannot be regenerated from the pack hash and the run manifest is treated as commentary, not as a result.

The artifact below is a proposed local harness, labeled as an unexecuted example rather than a claimed production study. It loads a JSON pack, refuses mixed-family averages by default, hashes the pack, and prints family rows that a later agent runner can fill. Teams can swap the stub run_agent function for a real runner without changing the reporting contract.

# bench_pack.py — proposed replay harness, not a published study
from __future__ import annotations

import hashlib
import json
import random
import statistics
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable, Dict, List


@dataclass(frozen=True)
class Task:
    task_id: str
    family: str
    fixture_sha256: str
    timeout_s: float


@dataclass
class Trial:
    task_id: str
    family: str
    passed: bool
    elapsed_s: float
    timed_out: bool
    agent_label: str


def load_pack(path: Path) -> List[Task]:
    raw = json.loads(path.read_text())
    tasks = [Task(**row) for row in raw["tasks"]]
    families = {t.family for t in tasks}
    if len(families) < 2:
        raise ValueError("pack must contain at least two families")
    counts = {f: sum(1 for t in tasks if t.family == f) for f in families}
    if min(counts.values()) < 3:
        raise ValueError("each family needs at least three tasks")
    return tasks


def pack_hash(tasks: List[Task]) -> str:
    blob = json.dumps([t.__dict__ for t in tasks], sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()


def stub_agent(task: Task, label: str, rng: random.Random) -> Trial:
    # Deterministic stub so the harness is testable without a network.
    start = time.perf_counter()
    bias = 0.72 if label == "candidate" else 0.64
    family_shift = {"refactor": 0.08, "bugfix": 0.00, "tests": -0.04, "scaffold": -0.10}
    p = bias + family_shift.get(task.family, 0.0)
    elapsed = rng.uniform(0.4, min(task.timeout_s, 8.0))
    timed_out = elapsed >= task.timeout_s
    passed = (not timed_out) and (rng.random() < p)
    return Trial(task.task_id, task.family, passed, elapsed, timed_out, label)


def run_pack(
    tasks: List[Task],
    runner: Callable[[Task, str, random.Random], Trial],
    seed: int,
) -> List[Trial]:
    rng = random.Random(seed)
    labeled = [(t, lab) for t in tasks for lab in ("control", "candidate")]
    rng.shuffle(labeled)
    return [runner(task, lab, rng) for task, lab in labeled]


def family_report(trials: List[Trial]) -> Dict[str, dict]:
    out: Dict[str, dict] = {}
    families = sorted({t.family for t in trials})
    for family in families:
        rows = [t for t in trials if t.family == family]
        def slice_label(label: str) -> List[Trial]:
            return [t for t in rows if t.agent_label == label]
        control, cand = slice_label("control"), slice_label("candidate")
        def pass_rate(xs: List[Trial]) -> float:
            return sum(t.passed for t in xs) / len(xs)
        out[family] = {
            "n": len(control),
            "control_pass": round(pass_rate(control), 3),
            "candidate_pass": round(pass_rate(cand), 3),
            "delta_pass": round(pass_rate(cand) - pass_rate(control), 3),
            "candidate_median_s": round(statistics.median(t.elapsed_s for t in cand), 3),
            "candidate_timeout_rate": round(sum(t.timed_out for t in cand) / len(cand), 3),
        }
    return out


def maybe_global_mean(report: Dict[str, dict]) -> str:
    deltas = [row["delta_pass"] for row in report.values()]
    # Crude homogeneity gate: mixed signs mean the mean is a costume.
    if any(d < 0 for d in deltas) and any(d > 0 for d in deltas):
        return "REFUSE_GLOBAL_MEAN: families disagree on sign"
    return str(round(statistics.mean(deltas), 3))


if __name__ == "__main__":
    pack = Path("task_pack.json")
    tasks = load_pack(pack)
    digest = pack_hash(tasks)
    trials = run_pack(tasks, stub_agent, seed=20260916)
    report = family_report(trials)
    print(json.dumps({
        "pack_sha256": digest,
        "seed": 20260916,
        "families": report,
        "global_delta": maybe_global_mean(report),
    }, indent=2))

A companion pack file keeps the contract visible. Fixture hashes are placeholders in this example and must be replaced with hashes of real tarballs before any public claim. The important property is that the pack is data, not a prompt stuffed into a README.

{
  "tasks": [
    {"task_id": "rf-001", "family": "refactor", "fixture_sha256": "ab", "timeout_s": 30},
    {"task_id": "rf-002", "family": "refactor", "fixture_sha256": "cd", "timeout_s": 30},
    {"task_id": "rf-003", "family": "refactor", "fixture_sha256": "ef", "timeout_s": 30},
    {"task_id": "bf-001", "family": "bugfix", "fixture_sha256": "11", "timeout_s": 30},
    {"task_id": "bf-002", "family": "bugfix", "fixture_sha256": "22", "timeout_s": 30},
    {"task_id": "bf-003", "family": "bugfix", "fixture_sha256": "33", "timeout_s": 30},
    {"task_id": "ts-001", "family": "tests", "fixture_sha256": "44", "timeout_s": 30},
    {"task_id": "ts-002", "family": "tests", "fixture_sha256": "55", "timeout_s": 30},
    {"task_id": "ts-003", "family": "tests", "fixture_sha256": "66", "timeout_s": 30},
    {"task_id": "sc-001", "family": "scaffold", "fixture_sha256": "77", "timeout_s": 45},
    {"task_id": "sc-002", "family": "scaffold", "fixture_sha256": "88", "timeout_s": 45},
    {"task_id": "sc-003", "family": "scaffold", "fixture_sha256": "99", "timeout_s": 45}
  ]
}

Running the stub locally should produce a family table and, quite often, a refused global mean. That refusal is the point of the method: mixed signs across families are information, not a rounding problem to hide inside one percentage. Replace the stub with a real agent only after the pack hash is recorded in the lab notebook or CI log.

python3 -c "import json,pathlib; p=pathlib.Path('task_pack.json'); json.loads(p.read_text()); print('pack json ok')"
python3 bench_pack.py | tee run_manifest.json

A cheap place to execute the same harness matters because environment drift is a second averaging sin. If the control ran on a laptop and the candidate ran on a crowded CI runner, the delta is partly hardware theater. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as a shared execution lane: free model access and a free server option let the control and the candidate share one runtime while the pack hash stays fixed. No model names, quotas, or hardware claims are added beyond that availability, because those details change and do not belong inside a methodology paper.

Numbers from this harness are not marketing when three conditions hold together. The pack hash is published beside the table, families remain visible, and the global mean is omitted whenever family deltas disagree in sign. A fourth informal check helps reviewers: if removing any one family reverses the winner, the headline was a blend, not a finding. That check is cheap, and it catches most accidental leaderboards before they leave the lab.

Limitations are part of the protocol rather than a closing apology. Twelve stub tasks cannot support a ranking of products, and the stub probabilities are instructional fakes that must never be quoted as measurements. The homogeneity gate is crude and can miss magnitude fights where every family moves the same direction but for different reasons. Timeout policy, flake retries, and human review of oracle mismatches still sit outside this script and will dominate real repositories.

This approach is a poor fit for live incident response, for single-ticket debugging, and for any team that needs a ship-or-not decision before a pack can be frozen. It is also the wrong tool when the product under test is allowed to mutate the fixtures, because a self-editing dataset is a moving target wearing a lab coat. Vendors who require a single public percentage as a procurement ritual will dislike the refused mean, which is evidence the method is working.

Readers who already keep agent evaluation notes can drop the harness beside those notes and require a pack hash on every new candidate row. The useful output is a family table that survives contact with a skeptical reviewer, not a slogan that could have been written before the run. Anyone who wants to try the same reporting contract on a shared runtime can start from the script, freeze a real fixture pack, and keep the mean split until the families agree.