Control Deltas Turn Agent Scores Into Evidence

Control deltas turn raw agent scores into evidence because they hold the dataset, the metric, and the runtime constant. A pass rate quoted without a frozen control is closer to marketing copy than to a measurement. The same agent can look strong on a laptop and weak in CI once the tool timeout or the retriever index changes. Publishing the signed difference against a pinned baseline removes that theatrical gap between demos and labs.

Think of a track meet that reports only finish times and never names the wind, the track, or the starter pistol. Runners would still be ranked, yet nobody could replay the race or trust the order. Agent leaderboards often work that way, treating the hidden prompt, the tool sandbox, and the judge as scenery. A benchmark methodology has to freeze those props before anyone averages a score.

The evaluation contract in this article is a small YAML file plus a runner that refuses to print a score without a control. The contract pins a dataset hash, a metric module, a control agent identifier, and a runtime pin. Those four fields are the difference between a blog number and a result another team can replay. What follows is a practical protocol for dataset, metrics, and controls, with a Python harness that other machines can execute.

When the dataset is a living folder of prompts, every quiet edit silently rebases the history of every published chart. The honest unit is a snapshot: one JSONL file, one SHA-256 digest, and a sampling rule that does not change mid-report. Each line should carry a task id, a prompt, and an expected artifact so the metric can fail closed instead of guessing. A pack of twenty well-specified tasks is enough to prove the harness; a larger pack should wait until the protocol itself is boring.

Metric functions belong in versioned source, not in a spreadsheet cell that someone might round for a slide. Exact-match strings, schema-valid JSON, and a bounded wall-clock flag are dull on purpose, because dull functions replay next week. Model-as-judge scores can sit beside those checks, but they cannot be the only column if the judge itself is an unpinned model. The report should print each metric under its module path and content digest so reviewers can see what was actually computed.

The control agent is the quiet half of the experiment, like a placebo arm in a trial that a launch blog would rather omit. It may be a scripted stub that returns empty tool calls, a frozen checkpoint, or last week's production prompt under the same sandbox. What matters is that it consumes the same tasks, the same timeout, and the same network policy as the candidate. The number that leaves the lab is then candidate minus control, with a simple spread over task-level paired differences.

Marketing wants a single percentage that can travel without footnotes, screenshots, or a disappointed control. Evidence wants a delta, a protocol hash, and a sentence about who is missing from the sample. If the control already scores 0.81, a candidate at 0.84 is a small lift rather than a revolution. If the control scores 0.10 because the sandbox was misconfigured, the candidate's 0.90 is a hardware anecdote wearing a lab coat.

The worked example below is a proposed harness, not a claim about any production fleet. It pins the protocol, hashes the task pack, runs control and candidate on each row, and prints a delta report. Teams can swap the call_agent function for their own HTTP client, CLI wrapper, or in-process stub. The important behavior is that the script exits nonzero when the control arm is missing or the dataset digest does not match.

# eval_protocol.yaml — proposed contract, not a live leaderboard
protocol_id: agent-bench-2026-09-21
dataset:
  path: tasks.jsonl
  sha256: REPLACE_AFTER_HASHING
metrics:
  module: metrics.py
  functions: [exact_match, json_schema_ok, within_deadline]
control:
  name: frozen-stub-v1
  kind: scripted
runtime:
  python: "3.11"
  timeout_s: 30
  seed: 20260921
candidate:
  name: agent-under-test

{"id": "t001", "prompt": "Return {\"ok\": true, \"n\": 3}", "expect": {"ok": true, "n": 3}, "deadline_s": 5}
{"id": "t002", "prompt": "Return {\"ok\": true, \"n\": 0}", "expect": {"ok": true, "n": 0}, "deadline_s": 5}

Save those two objects as tasks.jsonl, one object per line, then hash the file before any candidate run is allowed to print.

# bench.py — proposed evaluation harness (unexecuted example)
from __future__ import annotations

import hashlib, json, time, sys
from pathlib import Path
from statistics import mean, pstdev

import yaml

def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()

def exact_match(got, expect) -> float:
    return 1.0 if got == expect else 0.0

def json_schema_ok(got, expect) -> float:
    if not isinstance(got, dict):
        return 0.0
    return 1.0 if set(expect).issubset(got) else 0.0

def within_deadline(elapsed_s: float, deadline_s: float) -> float:
    return 1.0 if elapsed_s <= deadline_s else 0.0

def call_agent(kind: str, prompt: str) -> dict:
    # Proposed stub: replace with a real client. Control stays deterministic.
    if kind == "scripted":
        return {"ok": True, "n": 0}
    if 'n": 3' in prompt:
        return {"ok": True, "n": 3}
    return {"ok": True, "n": 0}

def run_arm(kind: str, tasks: list[dict]) -> list[dict]:
    rows = []
    for task in tasks:
        started = time.perf_counter()
        got = call_agent(kind, task["prompt"])
        elapsed = time.perf_counter() - started
        rows.append({
            "id": task["id"],
            "exact_match": exact_match(got, task["expect"]),
            "json_schema_ok": json_schema_ok(got, task["expect"]),
            "within_deadline": within_deadline(elapsed, task["deadline_s"]),
        })
    return rows

def paired_delta(control_rows, candidate_rows, key: str):
    by_id = {row["id"]: row for row in control_rows}
    pairs = [row[key] - by_id[row["id"]][key] for row in candidate_rows]
    mu = mean(pairs)
    sd = pstdev(pairs) if len(pairs) > 1 else 0.0
    return mu, sd

def main(protocol_path: str) -> int:
    proto = yaml.safe_load(Path(protocol_path).read_text())
    data_path = Path(proto["dataset"]["path"])
    digest = sha256_file(data_path)
    pinned = proto["dataset"]["sha256"]
    if pinned != "REPLACE_AFTER_HASHING" and pinned != digest:
        print(f"dataset digest mismatch: {digest} != {pinned}", file=sys.stderr)
        return 2
    tasks = [json.loads(line) for line in data_path.read_text().splitlines() if line.strip()]
    control = run_arm(proto["control"]["kind"], tasks)
    candidate = run_arm("candidate", tasks)
    print(f"protocol_id={proto['protocol_id']}")
    print(f"dataset_sha256={digest}")
    print(f"n_tasks={len(tasks)}")
    print(f"runtime_python={proto['runtime']['python']} timeout_s={proto['runtime']['timeout_s']} seed={proto['runtime']['seed']}")
    for key in proto["metrics"]["functions"]:
        c_mean = mean(row[key] for row in control)
        k_mean = mean(row[key] for row in candidate)
        delta, sd = paired_delta(control, candidate, key)
        print(f"{key}: control={c_mean:.3f} candidate={k_mean:.3f} delta={delta:.3f} sd={sd:.3f}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1] if len(sys.argv) > 1 else "eval_protocol.yaml"))

Hash the pack, write the digest into the YAML file, then run the harness on the same tree the protocol names.

python - <<'PY'
from pathlib import Path
import hashlib
print(hashlib.sha256(Path("tasks.jsonl").read_bytes()).hexdigest())
PY
pip install pyyaml
python bench.py eval_protocol.yaml

The printed line that matters is not candidate=0.850 sitting alone on a slide. The printed line that matters is delta=0.150 next to a dataset digest, a protocol id, and a runtime pin. Anyone who cannot reproduce that delta on a second machine is looking at a demo, even if the first number was real on Friday afternoon. Variance across tasks is part of the result; hiding the spread is how a thin lift becomes a press sentence.

A control delta can still lie if the candidate was trained or prompted on the same twenty tasks. The protocol does not detect leakage; it only detects protocol drift between two runs that claim to be comparable. Teams that publish should say whether the pack was held out, and they should rotate the pack when it becomes a study guide. Until that sentence exists, the delta is a replayable demo rather than a generalization claim.

The runtime pin is the least glamorous field and the one that breaks first in the wild. Python minor versions, JSON library quirks, and DNS timeouts all move scores without moving anything that looks like intelligence. Recording the interpreter version, the timeout, and the seed next to the delta keeps those ghosts visible. If two labs cannot match those three values, they are not yet arguing about agents.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A pinned protocol is only useful if a second machine can run it, and MonkeyCode's free model access and free server option are one way to give that machine a shared place to call a model and execute the runner. That is an availability convenience rather than a quality ranking, and it does not replace a local pin when a team already has locked CI. Engineers who need a shared box can replay the YAML there and compare deltas against the harness output.

This approach is the wrong tool for several common jobs, and those jobs should stay elsewhere. Safety certification, red-team coverage, and open-ended writing quality still need human review that a three-metric stub cannot impersonate. Teams that cannot freeze a task pack, or that change tools between the control run and the candidate run, will manufacture deltas that describe process noise. Vendor bake-offs that allow each model a private prompt and a private timeout are not using this protocol, even when the chart colors match.

The honest publication format is therefore short and slightly inconvenient. Name the protocol id, the dataset digest, the metric module, the control identity, and the paired delta with its spread. Leave the raw candidate percentage in an appendix where it cannot be screenshotted alone. If the control arm cannot run, the write-up should not run either. That rule is the entire methodology, dressed as a script that another checkout can execute.