Calibrate the Task Set Before You Scale the Run

Most of the budget in an agent benchmark goes to tasks that cannot separate two systems, and the fix is a cheap calibration sweep that runs before the scored one. A task that every candidate solves, or that none solves, costs the same tokens as a task in the informative band while contributing nothing but variance. Treating task selection as a measured step rather than an editorial one is what keeps a final number from becoming marketing. The workflow below produces a retained task set, a drop ledger, and a weighted score whose denominator a reader can reconstruct.

The signal lives in the middle

A binary-scored task carries the most information when its pass probability sits near one half, because that is where a small capability difference flips outcomes. Outside that band the task is still useful as a smoke test, but it cannot move a leaderboard in a defensible way. The practical consequence is that a 200-task suite can behave like a 40-task suite once the ceiling and floor tasks are counted honestly. Reporting the retained count next to the score is the cheapest integrity check available.

Run a pilot before the scored run

The pilot uses the same harness, the same system prompt, and the same scorer as the final run, at a smaller seed count. Its only job is to estimate per-task difficulty and to expose harness flake before those effects get baked into a headline. A short command is enough to launch it, and the output stays at line granularity for auditing.

# pilot sweep: 3 seeds per task, one model family, identical harness
for seed in 11 12 13; do
  agent-bench run \
    --tasks tasks.jsonl \
    --model "$MODEL" \
    --seed "$seed" \
    --temp 0 \
    --out "runs/pilot-$seed.jsonl"
done
cat runs/pilot-*.jsonl > runs/pilot.jsonl

The calibration script then reads the pilot records and applies three rules in a fixed order. Ceiling and floor tasks are dropped from the scored set, tasks that disagree with themselves under rerun are quarantined as harness noise, and the survivors are reweighted by family so a large family cannot dominate the aggregate. Every decision lands in a ledger row, including the rule that fired, so the drop is reviewable rather than silent.

#!/usr/bin/env python3
"""calibrate.py - retain discriminating tasks, reweight by family.

tasks.jsonl  {"id": str, "family": str, "prompt": str}
runs.jsonl   {"task_id": str, "model": str, "seed": int, "passed": bool}
"""
import json, csv, argparse
from collections import defaultdict
from math import sqrt


def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - m) / d, (c + m) / d)


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--tasks", required=True)
    ap.add_argument("--runs", required=True)
    ap.add_argument("--lo", type=float, default=0.15)
    ap.add_argument("--hi", type=float, default=0.85)
    ap.add_argument("--out", default="banded.jsonl")
    ap.add_argument("--ledger", default="ledger.csv")
    a = ap.parse_args()

    tasks = {r["id"]: r for r in map(json.loads, open(a.tasks))}
    runs = [json.loads(l) for l in open(a.runs)]

    by_task = defaultdict(list)
    by_task_model = defaultdict(list)
    for r in runs:
        by_task[r["task_id"]].append(r["passed"])
        by_task_model[(r["task_id"], r["model"])].append(r["passed"])

    kept, ledger = [], []
    for tid, task in tasks.items():
        flags = by_task.get(tid, [])
        n, k = len(flags), sum(flags)
        p = k / n if n else 0.0
        lo_ci, hi_ci = wilson(k, n)

        # harness flake: same model, same task, different seeds disagree
        flaky = any(len(set(v)) > 1 for v in
                    (by_task_model[(tid, m)] for m in
                     {r["model"] for r in runs if r["task_id"] == tid}))

        if n == 0:
            rule = "no-data"
        elif p >= a.hi:
            rule = "ceiling"
        elif p <= a.lo:
            rule = "floor"
        elif flaky:
            rule = "harness-flake"
        else:
            rule = "retained"

        row = {"task_id": tid, "family": task["family"], "n": n,
               "pass_rate": round(p, 4),
               "ci": f"[{lo_ci:.3f},{hi_ci:.3f}]", "rule": rule}
        ledger.append(row)
        if rule == "retained":
            kept.append({**task, **row})

    with open(a.out, "w") as f:
        for t in kept:
            f.write(json.dumps(t) + "\n")
    with open(a.ledger, "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=list(ledger[0]))
        w.writeheader()
        w.writerows(ledger)

    fam = defaultdict(int)
    for t in kept:
        fam[t["family"]] += 1
    print(f"retained {len(kept)}/{len(tasks)} tasks across {len(fam)} families")
    for f_, c in sorted(fam.items()):
        print(f"  {f_}: {c}")


if __name__ == "__main__":
    main()

Read the ledger before the score

The ledger is where the honesty of the number is decided, and it reads best as a short decision table rather than a paragraph. The right-hand column is the part reviewers actually argue about, so it should state the mechanism rather than a preference. Publishing this table alongside a score removes most of the ambiguity that makes benchmark numbers hard to trust.

Pilot signal	Action	Mechanism
Pooled pass rate ≥ 0.85	Drop from scored set, keep as smoke test	Little outcome variance left to explain
Pooled pass rate ≤ 0.15	Drop or split into smaller subtasks	Below the noise floor of the harness
0.15 < p < 0.85, seeds agree	Retain, weight by family	Near-maximal discrimination
Same model, same task, seeds disagree	Quarantine until harness is fixed	Measures flake, not capability

Stratified reporting follows directly from the retained families, and the weighting must be declared before the scored run rather than chosen afterward. A per-family pass rate with a Wilson interval is enough for most small suites, because intervals wider than the effect being claimed are themselves the finding. The aggregate should always travel with the retained task count, the dropped count, and the tokens spent per retained task.

Controls that make the number survive review

Four controls cover the common ways a suite quietly stops measuring capability. The harness version and scorer commit are pinned in the run record, because a scorer change mid-sweep is indistinguishable from a model change. Every candidate sees the identical prompt text and tool surface, since a wider toolset is a capability difference whether or not it is intended. Temperature and seed policy are fixed and reported, and a single model is run twice on a subset to estimate harness noise independently of model choice. Finally, the pilot and the scored run use separate output paths so a stale file cannot be mistaken for a fresh result.

Where a free tier fits, and where it does not

The calibration sweep is the token-hungry half of this workflow, because it spends three to five times the final run just to learn which tasks are worth scoring. That makes it a reasonable candidate for a free tier, and MonkeyCode advertises free model access plus a free server option for experiments of this shape. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator indicates a 10 million free token allowance on the current offer; treat that figure as operator-supplied and confirm it on the project's own page before planning a sweep around it, since promotions change.

The limits matter more than the allowance. A rate-limited or queued endpoint changes wall-clock timing, so a free tier must never be the source of a latency claim, and it should not carry the final scored run either. Use it for the pilot, for harness debugging, and for widening the seed count on tasks that sit near the decision boundary. Then move the scored run to a stable endpoint whose throughput is documented, and keep the pilot ledger as the record of what changed.

Limits and who should skip this

Suites below roughly thirty tasks cannot support stratification, and the calibration overhead will exceed the information it recovers, so a plain full run is the better choice. Teams whose tasks are already curated against a fixed difficulty target gain little from a second gate. Anyone who needs an absolute capability claim should not use this method at all, because a stratified pass rate on a retained subset describes the subset and nothing wider.

The method buys one thing: a score that comes with its own denominator, banding rule, and drop ledger, which is the difference between a measurement and a slide. If the ledger idea fits an existing harness, run the pilot against the free tier first and compare what it drops against what the current suite claims to measure.