Your Coding Agent's Pass Rate Isn't a Benchmark Until the Oracle Is Frozen

You get the slide on Thursday. Two coding agents, same internal tasks, one "wins" at 81 percent. You rerun the suite after a long weekend. The winner is now at 58 percent, and nobody can explain which file moved.

That is not a model regression. That is an unfrozen benchmark. If you cannot hash the tasks, hide the oracle, and name the controls, you do not have a number. You have a story with a percent sign.

The failure you are actually scoring

Most coding-agent demos optimize for a test file the model can see. The agent reads the tests, patches the tests, or echoes a public suite it already memorized. A single pass rate on a leaked oracle answers the wrong question. It measures exposure, not behavior.

You need three walls. The prompt the agent may read. The snapshot it may edit. The oracle it must never touch. Mix those walls and every later chart is theater.

This article is a method, not a leaderboard. You will freeze a tiny dataset, hide the tests, compute metrics that argue with each other, and refuse to publish a number you cannot replay.

1. Freeze the dataset before you touch a model

Do not start with a vendor. Start with a directory you can hash. One task is one JSON object. Keep the oracle in a second file the agent runtime cannot open.

bench/
  tasks.jsonl          # prompts + visible files only
  oracles/
    T001.json          # hidden tests, never on the agent path
  snapshots/
    T001/              # starting repo the agent may edit
  harness.py
  controls.json        # seed, timeout, network policy

A task record should be boring on purpose:

{
  "id": "T001",
  "prompt": "Add CSV export to reports.py. Do not change existing JSON output.",
  "visible_files": ["reports.py", "README.md"],
  "language": "python",
  "timeout_s": 90,
  "max_tool_calls": 12,
  "snapshot_hash": "sha256:8c1a..."
}

The matching oracle stays elsewhere:

{
  "id": "T001",
  "hidden_tests": ["test_csv_export.py"],
  "must_not_modify": ["test_csv_export.py"],
  "forbidden_substrings": ["eval(", "os.system"]
}

Hash both trees before the first run. If either hash changes, the previous score is void. You are not being pedantic. You are preventing silent dataset drift from impersonating a model win.

How to pick tasks that are not marketing bait

Use work you already own. A parser edge case. A regression you shipped last quarter. A refactor that must preserve a public function. Skip trivia that exists on every public leaderboard.

Score a task only if you can state the invariant in one sentence. "CSV export exists and JSON output is byte-identical" is an invariant. "Make the module cleaner" is a vibe. Vibes do not belong in oracles/.

Five to twenty tasks is enough to debug the method. Hundreds of tasks without a freeze are still a press release.

2. Pick metrics that disagree with each other

One number invites cheating. Four numbers that cannot all be gamed the same way are a benchmark.

Metric	What it asks	How it fails if you only publish this
hidden_pass	Did hidden tests pass after the patch?	Agents that rewrite tests, or tasks with a leaked oracle
oracle_untouched	Did the agent open or edit hidden files?	You never log file access
tool_calls	How much search/retry did the loop burn?	A pass after 80 greps is not the same product
first_diff_lines	How large was the first successful patch?	Tiny diffs can still delete a guardrail
invariant_hold	Did required old behavior survive?	New tests pass while the public API drifts

You publish a row, not a trophy. hidden_pass=1 with oracle_untouched=0 is a disqualification, not a highlight. hidden_pass=1 with tool_calls=47 is a slow wander that happened to stop on green.

Label any unpublished run as a proposal. Do not narrate a winner you did not execute under the freeze.

3. Name the controls or do not ship the chart

Write the controls in a file you commit next to the dataset. If a control is missing, you do not have a comparison. You have two anecdotes.

1. Snapshot hash. Starting files are bit-identical across agents.
2. Oracle isolation. Hidden tests are outside the sandbox filesystem, or mounted read-only after the agent exits.
3. Network policy. Default deny. If a task needs a package, vendor the wheel into the snapshot.
4. Budget. Timeout, max tokens you actually send, max tool calls. Record what you sent, not what a UI implied.
5. Decoding. Temperature, seed, and whether you allowed retries after a failed test.
6. Human edits. Zero. If you touch the patch, the run is manual and leaves the table.

{
  "dataset_hash": "sha256:…",
  "oracle_hash": "sha256:…",
  "network": "deny",
  "temperature": 0,
  "seed": 7,
  "retry_on_fail": false,
  "max_tool_calls": 12,
  "notes": "No hidden tests in the sandbox. Agent sees prompt + snapshot only."
}

Temperature 0 is not determinism. It is a control. Free-tier endpoints still jitter. That is why you store the raw patch and the tool log, not just the boolean.

4. Run a harness that cannot see the answer

The evaluator is a separate process. The agent writes into a copy of snapshots/T001. After the agent exits, you copy the tree to a clean runner that finally receives the hidden tests.

# harness.py — method sketch, not a published scoreboard
from __future__ import annotations

import hashlib, json, shutil, subprocess, sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
SANDBOX = ROOT / ".sandbox"


def sha256_tree(path: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(path.rglob("*")):
        if p.is_file():
            h.update(p.relative_to(path).as_posix().encode())
            h.update(p.read_bytes())
    return h.hexdigest()


def load_jsonl(path: Path):
    with path.open() as f:
        for line in f:
            line = line.strip()
            if line:
                yield json.loads(line)


def run_hidden_tests(work: Path, oracle: dict) -> bool:
    for test in oracle["hidden_tests"]:
        src = ROOT / "oracles" / test
        # Tests enter the runner only after the agent has exited.
        shutil.copy(src, work / test)
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-q"],
        cwd=work,
        capture_output=True,
        text=True,
    )
    return proc.returncode == 0


def evaluate(agent_fn):
    controls = json.loads((ROOT / "controls.json").read_text())
    rows = []
    for task in load_jsonl(ROOT / "tasks.jsonl"):
        snap = ROOT / "snapshots" / task["id"]
        assert sha256_tree(snap) == task["snapshot_hash"].split(":")[-1]

        work = SANDBOX / task["id"]
        if work.exists():
            shutil.rmtree(work)
        shutil.copytree(snap, work)

        log = agent_fn(prompt=task["prompt"], workdir=work, controls=controls)
        oracle = json.loads((ROOT / "oracles" / f"{task['id']}.json").read_text())

        leaked = any(
            Path(p).name in set(oracle["hidden_tests"])
            for p in log.get("opened_files", [])
        )
        passed = False if leaked else run_hidden_tests(work, oracle)
        rows.append({
            "id": task["id"],
            "hidden_pass": int(passed),
            "oracle_untouched": int(not leaked),
            "tool_calls": log.get("tool_calls", -1),
            "first_diff_lines": log.get("first_diff_lines", -1),
        })
    return rows


if __name__ == "__main__":
    # Plug in your agent. Do not paste hidden tests into the prompt.
    raise SystemExit("Wire agent_fn, then print(evaluate(agent_fn))")

Two implementation rules matter more than the code style. The agent function must not receive the oracle path. The pytest command must not run until agent_fn returns.

Log every opened path. If you cannot log it, you cannot claim oracle_untouched. A pass without that bit is incomplete, not encouraging.

5. Read the table like someone who expects to be wrong

Sort by disqualifications first. Then by invariant failures. Then by hidden_pass. Tool-call count is a cost column, not a personality trait.

A useful review looks like this:

id    hidden_pass  oracle_untouched  tool_calls  first_diff_lines
T001  1            1                 6           18
T002  0            1                 12          41
T003  1            0                 4           9     DISQUALIFIED

T003 is the row amateurs delete. Keep it. It tells you the agent found the answer file, which means your sandbox is the bug, not the model.

T002 is the honest miss. Read the patch. If the agent reimplemented JSON export while ignoring CSV, your prompt was underspecified. If it wrote CSV and broke JSON, your hidden tests are doing their job.

Do not average these into a single "quality score." A mean hides the disqualification. Report counts: n, disqualified, pass_given_clean, median tool calls on clean passes. That sentence is harder to put on a homepage. That is the point.

Where a free worker actually helps

You can run this harness on a laptop. You can also park the worker on a throwaway server so your freeze, logs, and pytest runner stay off your daily machine. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already wanted a coding-oriented environment with free model access and a free server option, MonkeyCode is one place to host the agent process while this evaluator stays local and oracle-blind.

Keep the split. Models generate patches. Your harness judges them. The product is not the metric.

Limitations

This method measures functional match to your hidden tests. It does not measure readability, licensing, or whether the patch will confuse the next human. Hidden tests can still be weak. A green suite with one happy-path file will inflate hidden_pass forever.

Free endpoints vary. Same controls, different hour, different latency, occasional truncated output. Store raw artifacts. Never treat a single pass as a seasonal ranking.

The sketch above is a procedure you can implement. It is not a claim that any named model hit a stated percent. If you publish numbers, publish the dataset hash beside them.

Who should not use this

Skip this approach if you need a legally defensible procurement bake-off with hundreds of blinded tasks and a statistics review. Skip it if the repo contains production secrets you cannot snapshot. Skip it if you will not isolate the oracle. In that last case you are running a demo. Call it a demo.

Also skip it if your real question is latency, price, or cold start. Those are different instruments. Mixing them into hidden_pass is how marketing numbers are born.

What to do on Monday

Pick three tasks from a repo you maintain. Write one invariant each. Hide the tests. Hash the snapshot. Run two agents with the same controls.json. Keep every disqualification in the table.

If the ranking flips when you freeze the oracle, trust the freeze. The earlier number was a story. This one is still small, still biased toward your taste in tests, and finally honest about that.