Pin the Oracle Bundle Before the Agent Percentage

A coding-agent percentage is not a measurement until dataset, oracle, metrics, and environment pin are frozen as one bundle. Naked success rates behave like product screenshots because later readers cannot reconstruct the track beneath the stopwatch. The method below treats that bundle as the unit of publication and withholds a percentage when any piece is missing. The harness is a labeled proposal, not a report of a completed vendor bake-off.

Marketing numbers fail the way a race fails when officials forget to paint the lanes. The athlete still runs, the clock still ticks, and a time still appears on a slide. Nobody can name the distance, and nobody can prove the finish line stayed put. Coding-agent demos repeat that failure when a chat log is graded by taste or when hidden edits move the tests after the answer lands.

The dataset is a directory of task folders rather than a prompt buried in a README file. Each task needs a stable identifier, a frozen prompt, a fixture tree, and a short license note. Tasks should be held out from any corpus the operator actually fine-tunes, and they should stay small enough for a reviewer to read. The layout that follows is clerical infrastructure, not evidence that a public leaderboard was executed here.

benchmark/
  manifest.json
  env_pin.json
  tasks/
    T001_csv_join/
      prompt.md
      fixtures/
      oracle/
        expected_files.json
        tests.py
    T002_header_normalize/
      prompt.md
      fixtures/
      oracle/
        expected_files.json
        tests.py

The manifest is the table of contents, and it should hash every file that can change a later score. Publishing a percentage without those hashes resembles publishing a checksum while withholding the bytes. Ordinary shell tools can write the manifest before any agent process is allowed to start. That ordering matters because a hash computed after a green run can hide an oracle that moved.

#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import hashlib, json, os, time
root = "benchmark"
files = []
for dirpath, _, names in os.walk(root):
    for name in sorted(names):
        path = os.path.join(dirpath, name)
        if name in {"manifest.json", "env_pin.json"}:
            continue
        with open(path, "rb") as handle:
            digest = hashlib.sha256(handle.read()).hexdigest()
        files.append({"path": os.path.relpath(path, root), "sha256": digest})
payload = {
    "schema": "oracle-bundle/v1",
    "created_unix": int(time.time()),
    "file_count": len(files),
    "files": files,
}
os.makedirs(root, exist_ok=True)
with open(os.path.join(root, "manifest.json"), "w", encoding="utf-8") as handle:
    json.dump(payload, handle, indent=2)
print("wrote benchmark/manifest.json", "files", len(files))
PY

The oracle is the benchmark because it is the only object that can disagree with the agent in binary terms. An oracle folder should list required files, forbidden files, size caps, and tests whose exit status counts as success. Taste, summary charm, and fluent self-explanation belong in a reviewer note, which is a different instrument. Mixing those notes into the same average is how a demo transcript becomes a leaderboard cell.

{
  "must_exist": ["src/join.py", "tests/test_join.py"],
  "must_not_exist": ["secrets.env"],
  "max_bytes_per_file": 65536
}

# proposed file: score_task.py
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path


def load_oracle(task_dir: Path) -> dict:
    path = task_dir / "oracle" / "expected_files.json"
    return json.loads(path.read_text(encoding="utf-8"))


def check_files(workspace: Path, oracle: dict) -> dict:
    missing = [item for item in oracle["must_exist"] if not (workspace / item).exists()]
    forbidden = [item for item in oracle["must_not_exist"] if (workspace / item).exists()]
    limit = int(oracle["max_bytes_per_file"])
    oversized = []
    for rel in oracle["must_exist"]:
        path = workspace / rel
        if path.exists() and path.stat().st_size > limit:
            oversized.append(rel)
    return {
        "missing": missing,
        "forbidden": forbidden,
        "oversized": oversized,
        "files_ok": not (missing or forbidden or oversized),
    }


def run_oracle_tests(task_dir: Path, workspace: Path) -> dict:
    test_file = task_dir / "oracle" / "tests.py"
    if not test_file.exists():
        return {"ran": False, "passed": False, "reason": "missing_tests"}
    proc = subprocess.run(
        [sys.executable, str(test_file), str(workspace)],
        capture_output=True,
        text=True,
        timeout=30,
    )
    return {
        "ran": True,
        "passed": proc.returncode == 0,
        "stdout_tail": proc.stdout[-500:],
        "stderr_tail": proc.stderr[-500:],
    }


def score_task(task_dir: Path, workspace: Path) -> dict:
    oracle = load_oracle(task_dir)
    files = check_files(workspace, oracle)
    tests = run_oracle_tests(task_dir, workspace)
    passed = bool(files["files_ok"] and tests.get("passed"))
    return {
        "task_id": task_dir.name,
        "passed": passed,
        "files": files,
        "tests": tests,
    }


if __name__ == "__main__":
    result = score_task(Path(sys.argv[1]), Path(sys.argv[2]))
    print(json.dumps(result, indent=2))
    raise SystemExit(0 if result["passed"] else 1)

Metrics are functions of oracle output, not adjectives sprayed across a transcript after the fact. A defensible starter set stays small: binary pass, count of failed checks, wall-clock seconds, and a budget declared before the run. Those fields are stored on failures as well, because a trophy cabinet cannot explain collapse later. Rolling them into one glossy figure should remain optional inside the published envelope.

# proposed file: bundle_report.py
from __future__ import annotations

import json
from pathlib import Path


def failed_check_count(task: dict) -> int:
    files = task.get("files", {})
    tests = task.get("tests", {})
    n = 0
    n += len(files.get("missing", []))
    n += len(files.get("forbidden", []))
    n += len(files.get("oversized", []))
    if tests.get("ran") and not tests.get("passed"):
        n += 1
    if not tests.get("ran"):
        n += 1
    return n


def summarize(results: list[dict]) -> dict:
    n_tasks = len(results)
    n_passed = sum(1 for item in results if item["passed"])
    return {
        "n_tasks": n_tasks,
        "n_passed": n_passed,
        "failed_checks": sum(failed_check_count(item) for item in results),
        "pass_rate": None if n_tasks == 0 else round(n_passed / n_tasks, 4),
        "incomplete": n_tasks == 0,
        "tasks": results,
    }


def reject_naked_percentage(report: dict, manifest_ok: bool, env_pin: dict) -> dict:
    pin_ready = bool(env_pin.get("python") and env_pin.get("platform"))
    if not manifest_ok or not pin_ready or report["incomplete"]:
        report["pass_rate"] = None
        report["publishable"] = False
        report["reason"] = "oracle_bundle_incomplete"
        return report
    report["publishable"] = True
    report["reason"] = "oracle_bundle_complete"
    return report


if __name__ == "__main__":
    raw = json.loads(Path("results.json").read_text(encoding="utf-8"))
    pin = json.loads(Path("benchmark/env_pin.json").read_text(encoding="utf-8"))
    report = reject_naked_percentage(summarize(raw), True, pin)
    print(json.dumps(report, indent=2))

The environment pin stops the same oracle from drifting under a new interpreter or a new testing plugin. Recording the Python version, a lockfile hash, and a platform string is usually enough for tiny coding tasks. Without that pin, a failed reproduction can be dismissed as the machine, which is how marketing numbers outlive other laptops. The snippet below writes the pin as JSON beside the manifest rather than in a slide footnote.

python3 - <<'PY'
import hashlib, json, pathlib, platform, sys
lock = pathlib.Path("requirements.lock")
digest = hashlib.sha256(lock.read_bytes()).hexdigest() if lock.exists() else None
payload = {
    "python": sys.version.split()[0],
    "platform": platform.platform(),
    "lock_sha256": digest,
}
pathlib.Path("benchmark").mkdir(exist_ok=True)
pathlib.Path("benchmark/env_pin.json").write_text(json.dumps(payload, indent=2), encoding="utf-8")
print(json.dumps(payload, indent=2))
PY

Contamination is a dataset defect, not a model personality trait, and it belongs in the methodology before any ranking is discussed. If a task prompt already lives in a public fine-tune dump, a pass may measure memorization of the oracle rather than repair skill. A local n-gram check will not prove cleanliness, yet it will catch the embarrassing copies that sit on disk. The proposed scanner below flags overlapping shingles between prompt files and a corpus the operator actually possesses.

# proposed file: shingle_scan.py
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path


def shingles(text: str, width: int = 12) -> set[str]:
    tokens = text.lower().split()
    grams = []
    for index in range(0, max(0, len(tokens) - width + 1)):
        piece = " ".join(tokens[index : index + width])
        grams.append(hashlib.sha1(piece.encode("utf-8")).hexdigest())
    return set(grams)


def scan(task_prompt: Path, corpus: Path) -> dict:
    left = shingles(task_prompt.read_text(encoding="utf-8"))
    right = shingles(corpus.read_text(encoding="utf-8"))
    overlap = left & right
    denom = max(len(left), 1)
    return {
        "prompt": str(task_prompt),
        "corpus": str(corpus),
        "overlap": len(overlap),
        "prompt_shingles": len(left),
        "overlap_ratio": round(len(overlap) / denom, 4),
        "flag": (len(overlap) / denom) >= 0.05 if left else False,
    }


if __name__ == "__main__":
    print(json.dumps(scan(Path(sys.argv[1]), Path(sys.argv[2])), indent=2))

A five percent overlap flag is a tripwire, not a scientific threshold copied from a paper. Operators should tune it against their own corpus and then freeze the constant inside the bundle so later scores stay comparable. The point is to keep dataset hygiene visible, because an invisible hygiene step is where marketing numbers usually hide. Tasks that trip the flag should be replaced or disclosed, not silently left in the average.

Controls belong in the same envelope because a pass rate without a baseline is a vibe wearing extra decimal places. The control is a weaker or cheaper agent scored on the identical oracle, not a theatrical rival review. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option that can host that control lane on the same frozen files.

export BUNDLE_HASH="$(python3 -c 'import hashlib,pathlib; print(hashlib.sha256(pathlib.Path("benchmark/manifest.json").read_bytes()).hexdigest())')"
python3 shingle_scan.py benchmark/tasks/T001_csv_join/prompt.md local_corpus.txt
python3 score_task.py benchmark/tasks/T001_csv_join /tmp/control-workspace > /tmp/control-T001.json
python3 score_task.py benchmark/tasks/T001_csv_join /tmp/candidate-workspace > /tmp/candidate-T001.json
python3 bundle_report.py

A published report should look like a sealed envelope rather than a keynote headline with a large percentage. It carries the manifest hash, the environment pin, per-task oracle outcomes, control outcomes, and only then a pass rate. If a vendor cannot attach those objects, the percentage should be read as a demo caption. The rule is unkind to slideware and kind to whoever must maintain the generated code.

The approach is the wrong instrument for several honest jobs that still matter to working teams. It will not rank chat quality, architectural taste, or the pleasure of a generated explanation, because those properties have no binary oracle here. It will not certify a model for production incident response, where the environment cannot be frozen and the task mix shifts. Teams that cannot publish even hashed fixtures should not ask strangers to trust the resulting percentage.

Readers who already keep agent evaluations in a lab notebook can adopt the oracle bundle without switching models. The extra discipline is mostly clerical, and it rewards anyone who later has to reproduce a disputed number. A cheaper control lane matters only because it makes that clerical loop repeatable on the same files. Operators who need that lane can try the free access already noted above, then publish the envelope rather than the headline.