Your Coding-Agent Score Needs a Dummy Baseline and a Size Split

You ship a coding-agent eval because the slide says 71%. Finance asks why the token invoice jumped. You open the CSV. Eighteen of twenty tasks are one-line string fixes. Two tasks are real refactors. The agent failed both. The 71% is still in the deck.

That number is not a benchmark. It is an average that hides size. Easy tasks inflate the headline. Hard tasks vanish into the mean. If you cannot show what a do-nothing solver scores on the same pack, you are publishing marketing.

This article is a small protocol, not a leaderboard. You pin a tiny task pack. You run a dummy baseline. You split results by task size. You write a result file that refuses to omit those controls. The code below is a harness you can run locally. Treat the sample percentages as illustrative output from that harness, not as product claims.

What this protocol is for

You need a number you can defend in a review. Not a screenshot from a vendor page. Not a pass rate that moves when someone retries a flaky test.

The protocol answers four questions:

1. What frozen tasks did you run?
2. What did a solver that patches nothing score?
3. Did easy tasks carry the average?
4. What clock, token budget, and judge command were locked?

If any answer is missing, stop. Do not put a percentage in a doc.

The four controls

1. Dataset pin. A task is a directory with prompt.md, src/, and tests/. You hash the pack. If the hash moves, the score is a new experiment.

2. Dummy baseline. Solver A writes an empty diff. Solver B copies a function stub that makes no assertions pass. If your agent barely beats A on the xs bucket, the headline is noise.

3. Size split. Bucket by failing-test count or fixture lines of code. Never average xs with m.

4. Result schema. The JSON must include n per stratum, dummy scores, wall time, token fields when you have them, and the hashes. A score without a denominator is a slogan.

Agent write-ups keep treating pass rate as a single scalar. That is the same trap as quoting model accuracy on an unbalanced set. You would not ship a classifier that way. Do not ship an agent that way.

A frozen pack you can read in one sitting

Keep the first pack embarrassing small. Three strata. Two tasks each. You are testing the harness, not claiming a universe of programming.

bench_pack/
  PACK_HASH.txt
  tasks/
    xs_trim_whitespace/
      prompt.md
      src/util.py
      tests/test_util.py
      meta.json          # {"stratum": "xs", "failing_tests": 1, "loc": 12}
    xs_rename_key/
      ...
    s_parse_csv_row/
      ...
    s_retry_header/
      ...
    m_refactor_queue/
      ...
    m_fix_off_by_one_batch/
      ...

meta.json is the only size signal the reporter may use. Do not infer size from the agent’s own commentary. Agents lie about difficulty.

Hash the pack once:

find bench_pack/tasks -type f | sort | xargs sha256sum | sha256sum > bench_pack/PACK_HASH.txt
cat bench_pack/PACK_HASH.txt

If a designer edits a test after the run, the hash changes. The old score is retired. That is the point.

Numbered run: dummy first, agent second

Step 1 — Define solvers as functions, not vibes

The dummy is a first-class solver. It is not a joke you skip.

# harness.py
from __future__ import annotations

import json, os, subprocess, time, hashlib
from dataclasses import dataclass, asdict
from pathlib import Path

ROOT = Path("bench_pack/tasks")

@dataclass
class RunRow:
    task_id: str
    stratum: str
    solver: str
    passed: bool
    failing_tests_before: int
    tests_passed: int
    tests_total: int
    wall_ms: int
    tokens_in: int | None
    tokens_out: int | None
    judge_cmd: str
    pack_hash: str
    timeout_s: int

def load_meta(task: Path) -> dict:
    return json.loads((task / "meta.json").read_text())

def empty_diff_solver(task: Path) -> None:
    """Control A: write no patch."""
    return

def stub_solver(task: Path) -> None:
    """Control B: touch a file, change no behavior."""
    src = next((task / "src").glob("*.py"))
    text = src.read_text()
    if "# stub-baseline" not in text:
        src.write_text(text + "\n# stub-baseline\n")

def run_pytest(task: Path, timeout_s: int) -> tuple[int, int, bool]:
    proc = subprocess.run(
        ["python", "-m", "pytest", "-q", str(task / "tests")],
        capture_output=True, text=True, timeout=timeout_s,
    )
    # pytest -q prints "N passed" / "N failed" — parse conservatively
    out = (proc.stdout or "") + (proc.stderr or "")
    passed = proc.returncode == 0
    # Fallback totals from meta if pytest summary is terse
    meta = load_meta(task)
    total = int(meta.get("tests_total", meta.get("failing_tests", 1)))
    ok = total if passed else 0
    return ok, total, passed

Label this as a local control, not a model bake-off. You have not measured a vendor yet. You have measured whether your judge even fails the dummy. If the empty diff “passes,” your tests are not tests.

Step 2 — Pin the judge, the clock, and the budget

Put every knob in one object. Read knobs from the environment so a free box and a laptop emit the same schema.

@dataclass(frozen=True)
class Controls:
    pack_hash: str
    judge_cmd: str
    timeout_s: int
    max_tokens: int
    temperature: str
    model_env: str  # opaque id from env, never hard-coded in the repo

def load_controls() -> Controls:
    pack_hash = Path("bench_pack/PACK_HASH.txt").read_text().strip()
    return Controls(
        pack_hash=pack_hash,
        judge_cmd="python -m pytest -q",
        timeout_s=int(os.environ.get("BENCH_TIMEOUT_S", "30")),
        max_tokens=int(os.environ.get("BENCH_MAX_TOKENS", "2048")),
        temperature=os.environ.get("BENCH_TEMPERATURE", "0"),
        model_env=os.environ.get("BENCH_MODEL", "unset"),
    )

Temperature lives in the result file even if your dummy ignores it. Future you will thank present you when someone asks why two runs drifted.

Step 3 — Execute every solver on every task

Restore sources after each solver. Otherwise the stub pollutes the agent run.

import shutil

def snapshot(task: Path) -> dict[Path, str]:
    return {p: p.read_text() for p in task.rglob("*") if p.is_file()}

def restore(snap: dict[Path, str]) -> None:
    for p, text in snap.items():
        p.write_text(text)

def run_solver(task: Path, name: str, fn, controls: Controls) -> RunRow:
    meta = load_meta(task)
    snap = snapshot(task)
    t0 = time.perf_counter()
    try:
        fn(task)
        ok, total, passed = run_pytest(task, controls.timeout_s)
    except subprocess.TimeoutExpired:
        ok, total, passed = 0, int(meta.get("tests_total", 1)), False
    finally:
        restore(snap)
    wall = int((time.perf_counter() - t0) * 1000)
    return RunRow(
        task_id=task.name,
        stratum=meta["stratum"],
        solver=name,
        passed=passed,
        failing_tests_before=int(meta["failing_tests"]),
        tests_passed=ok,
        tests_total=total,
        wall_ms=wall,
        tokens_in=None,
        tokens_out=None,
        judge_cmd=controls.judge_cmd,
        pack_hash=controls.pack_hash,
        timeout_s=controls.timeout_s,
    )

Wire an agent later as fn that reads prompt.md and writes into src/. Keep token counts in the same row when the API returns them. If it does not, store null. Do not invent a cost.

Step 4 — Report strata, not a single mean

from collections import defaultdict

def summarize(rows: list[RunRow]) -> dict:
    by = defaultdict(list)
    for r in rows:
        by[(r.solver, r.stratum)].append(r)
    blocks = []
    for (solver, stratum), rs in sorted(by.items()):
        n = len(rs)
        p = sum(1 for r in rs if r.passed)
        blocks.append({
            "solver": solver,
            "stratum": stratum,
            "n": n,
            "passes": p,
            "pass_rate": None if n == 0 else round(p / n, 3),
            "median_wall_ms": sorted(r.wall_ms for r in rs)[n // 2],
        })
    return {"blocks": blocks, "rows": [asdict(r) for r in rows]}

def main() -> None:
    controls = load_controls()
    rows: list[RunRow] = []
    solvers = [("empty_diff", empty_diff_solver), ("stub", stub_solver)]
    for task in sorted(p for p in ROOT.iterdir() if p.is_dir()):
        for name, fn in solvers:
            rows.append(run_solver(task, name, fn, controls))
    report = {
        "controls": asdict(controls),
        "summary": summarize(rows),
    }
    Path("bench_results.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report["summary"]["blocks"], indent=2))

if __name__ == "__main__":
    main()

Run it:

export BENCH_TIMEOUT_S=30
export BENCH_MAX_TOKENS=2048
export BENCH_TEMPERATURE=0
export BENCH_MODEL=local-or-remote-id
python harness.py

Illustrative output — not a claim about any product:

[
  {"solver": "empty_diff", "stratum": "xs", "n": 2, "passes": 0, "pass_rate": 0.0},
  {"solver": "empty_diff", "stratum": "s",  "n": 2, "passes": 0, "pass_rate": 0.0},
  {"solver": "empty_diff", "stratum": "m",  "n": 2, "passes": 0, "pass_rate": 0.0},
  {"solver": "stub",       "stratum": "xs", "n": 2, "passes": 0, "pass_rate": 0.0}
]

If empty_diff is not a wall of zeros, fix the pack before you touch an LLM.

Decision table: what you may publish

Condition	Publish a headline pass rate?	What you publish instead
Dummy pass rate > 0 on any stratum	No	The broken tests
n per stratum < 2	No	“harness smoke test” only
Easy stratum xs is > 50% of tasks	No	Stratum table, not a mean
Pack hash not in the result file	No	Re-run
Timeout or token budget omitted	No	Controls block
Agent beats dummy on m, not only on xs	Maybe	Stratum table + dummy deltas
Tokens per passing m task missing	Optional	Pass/fail only, labeled as incomplete

The rule is blunt. A mean that mixes xs and m is a marketing number. A mean that never shows the dummy is also a marketing number. You can still share the JSON. You just do not put 71% in a title.

Why this is not a leaderboard

Public agent charts collapse retries, tool traces, hidden tests, and prompt restatements into one percentage. You cannot audit them. This harness does the opposite. It is too small to rank products. It is large enough to stop you from lying to yourself.

Cost belongs in the same file as accuracy. When an agent call returns token counts, write them on the row. Then compute tokens per passing m task, not tokens per request. Requests that fail still cost money. A cheap fail is still a fail. An expensive pass on xs is not a win.

Wall time is a control, not a brag. A free server that swaps will stretch wall_ms. That is why the result file stores timeout and host-agnostic hashes first. Compare hashes and strata before you compare clocks.

Where a free model endpoint fits

You can keep solvers local forever. When you do wire a model, run the same harness against whatever endpoint you already pay for, or against a free one, on a machine you control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is one place this harness is meant to sit: same BENCH_* env vars, same bench_results.json, no special case in the reporter. If you try it, keep the dummy rows. The product is not the score.

Do not treat that as a quota, a model list, or a speed claim. Those change. The schema should not.

Limitations, and who should skip this

This pack will not replace SWE-bench, company golden sets, or a security review. Six tasks cannot represent “coding.” Pytest is a weak oracle for UI, concurrency, and build systems. Token fields stay null on many local solvers. Bootstrap intervals are absent because n is tiny; adding fake confidence intervals would be another marketing trick.

Skip this approach if you need a procurement ranking. Skip it if your tasks cannot run without network. Skip it if you will not freeze tests. Skip it if leadership only wants a single percentage. This protocol will fight that request.

Do not use the dummy as a humiliation ritual in a meeting. Use it as a unit test for the bench. When the dummy scores zero and the m bucket still moves after a prompt change, you have a real signal. Until then, you have a slide.

A close you can rerun tomorrow

Copy the harness. Hash the pack. Run empty_diff until it is all zeros. Add one agent function. Publish the stratum table and the dummy deltas. Leave the 71% off the slide.

If the JSON cannot explain the invoice, the JSON is not done.