Tuning-Set Pass Rate Is Not an Eval

Prompt iteration overfits golden cases in the same way that gradient descent overfits a frozen training set. A rising pass rate often records how closely the prompt was tailored to those known fixtures. It does not forecast how the system will behave on the next unseen production ticket. The honest metric is holdout pass rate after the prompt text is frozen and hashed.

Most prompt harnesses still treat every labeled example as both tutor and judge, which collapses two different statistical jobs. The tuning set is the surface you are allowed to read, rewrite against, and fail on while drafting. The holdout set is a locked drawer you open only after the prompt file is committed. Mixing them is the eval equivalent of reporting training accuracy as if it were a test score.

The public argument that models outgrow their benchmarks is mostly about leaderboards, yet the same failure appears in a private JSONL file. Once a fixture has been used to justify three prompt edits, it is no longer an independent measurement. It has become part of the prompt's informal training data, even though no gradient was computed. A later wording can look stronger simply because it absorbed the local folklore of those cases.

A useful analogy is a compiler suite that the author repairs while staring at each failing assertion. After enough cycles the suite describes the author's latest patch more than it describes the language. Prompt evals decay the same way, except the compiler is a stochastic completion API and the assertion is a grader function. If the grader still passes while unseen traffic would not, the only dashboard anyone watches has gone quiet.

The following harness is a worked example, not a logged production incident, and it keeps the split visible in the filesystem. Golden cases live in JSONL with a stable id, a task payload, and a structured expectation the grader can score without a second model. A seeded partition writes tuning.jsonl and holdout.jsonl so membership cannot drift between laptops. The runner hashes the prompt file, calls a chat-completions endpoint from environment variables, and writes a run artifact that CI can gate.

{"id":"inv-001","input":{"vendor":"Northwind","total":"1040.00","currency":"USD","notes":"net 30"},"expect":{"vendor":"Northwind","total_cents":104000,"currency":"USD"}}
{"id":"inv-002","input":{"vendor":"Contoso Ltd","total":"18.5","currency":"eur","notes":"duplicate of inv-991"},"expect":{"vendor":"Contoso Ltd","total_cents":1850,"currency":"EUR"}}
{"id":"inv-003","input":{"vendor":"Adventure Works","total":"0","currency":"USD","notes":"prepaid"},"expect":{"vendor":"Adventure Works","total_cents":0,"currency":"USD"}}

The splitter below sorts identifiers before shuffling so two checkouts with the same seed emit the same partition. Holdout fraction is a parameter, not a vibe, and the cut is forced to keep at least one locked case. If a later author adds fixtures, old identifiers remain in the same bucket until the seed itself changes. That stability is what makes a pass-rate delta meaningful across prompt hashes.

# split_goldens.py — proposal: seeded partition, not a production log
from __future__ import annotations

import json
import random
import sys
from pathlib import Path


def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if line:
            rows.append(json.loads(line))
    return rows


def write_jsonl(path: Path, rows: list[dict]) -> None:
    path.write_text(
        "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows),
        encoding="utf-8",
    )


def split_rows(rows: list[dict], seed: int = 20260918, holdout_frac: float = 0.3):
    by_id = {r["id"]: r for r in rows}
    ids = sorted(by_id)
    rng = random.Random(seed)
    rng.shuffle(ids)
    cut = max(1, int(len(ids) * holdout_frac))
    hold_ids = set(ids[:cut])
    holdout = [by_id[i] for i in ids if i in hold_ids]
    tuning = [by_id[i] for i in ids if i not in hold_ids]
    return tuning, holdout


def main() -> None:
    src = Path(sys.argv[1] if len(sys.argv) > 1 else "goldens.jsonl")
    rows = load_jsonl(src)
    if len(rows) < 10:
        raise SystemExit("split is noise below ten labeled cases; add fixtures first")
    tuning, holdout = split_rows(rows)
    write_jsonl(Path("tuning.jsonl"), tuning)
    write_jsonl(Path("holdout.jsonl"), holdout)
    print(f"tuning={len(tuning)} holdout={len(holdout)} seed=20260918")


if __name__ == "__main__":
    main()

Scoring stays in ordinary Python so the judge cannot be sweet-talked by the same model that produced the completion. The example extracts a vendor name, an integer cent total, and a normalized currency code, then rejects completions that wander into a long apology. Length is graded because a correct extract that now spends four hundred words is still a regression for any pipeline that bills tokens or posts into a ticket field. Exact string match on vendor is intentional here; fuzzy match would hide the drift this harness exists to catch.

# grade.py
from __future__ import annotations

import json
import re
from typing import Any

CURRENCY = {"usd": "USD", "eur": "EUR", "gbp": "GBP"}


def parse_completion(text: str) -> dict[str, Any]:
    fence = re.search(r"```

json\s*(\{.*?\})\s*

```", text, re.S)
    blob = fence.group(1) if fence else text[text.find("{") : text.rfind("}") + 1]
    return json.loads(blob)


def grade(expect: dict[str, Any], completion: str) -> dict[str, Any]:
    reasons: list[str] = []
    words = len(completion.split())
    if words > 80:
        reasons.append(f"verbose:{words}")
    try:
        got = parse_completion(completion)
    except Exception as exc:
        return {"pass": False, "reasons": [f"unparseable:{exc.__class__.__name__}"]}
    if got.get("vendor") != expect["vendor"]:
        reasons.append("vendor")
    if int(got.get("total_cents", -1)) != int(expect["total_cents"]):
        reasons.append("total_cents")
    currency = str(got.get("currency", "")).strip()
    currency = CURRENCY.get(currency.lower(), currency)
    if currency != expect["currency"]:
        reasons.append("currency")
    return {"pass": not reasons, "reasons": reasons, "words": words}

The runner records the prompt digest before it spends a single request, which stops an accidental save from being scored under the previous name. Endpoint, key, and model id come from the environment so the same files can point at a laptop proxy or a remote job without edits. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host this holdout loop when you do not want to keep a box awake for nightly runs, and the harness does not otherwise depend on that host.

# run_eval.py — proposal: OpenAI-compatible chat completions via env vars
from __future__ import annotations

import hashlib
import json
import os
import urllib.request
from pathlib import Path

from grade import grade


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def chat(prompt: str, user: str) -> str:
    body = json.dumps(
        {
            "model": os.environ["EVAL_MODEL"],
            "temperature": 0,
            "messages": [
                {"role": "system", "content": prompt},
                {"role": "user", "content": user},
            ],
        }
    ).encode("utf-8")
    req = urllib.request.Request(
        os.environ["EVAL_API_BASE"].rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ["EVAL_API_KEY"],
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    return payload["choices"][0]["message"]["content"]


def run_split(prompt: str, path: Path, reveal: bool) -> dict:
    rows = [json.loads(l) for l in path.read_text(encoding="utf-8").splitlines() if l.strip()]
    results = []
    for row in rows:
        completion = chat(prompt, json.dumps(row["input"], ensure_ascii=False))
        scored = grade(row["expect"], completion)
        item = {"id": row["id"], **scored}
        if reveal:
            item["completion"] = completion
        results.append(item)
    passed = sum(1 for r in results if r["pass"])
    return {
        "n": len(results),
        "passed": passed,
        "pass_rate": passed / max(len(results), 1),
        "failures": [r for r in results if not r["pass"]],
    }


def main() -> None:
    prompt = Path("prompts/extractor.txt").read_text(encoding="utf-8")
    reveal = os.environ.get("REVEAL_HOLDOUT") == "1"
    report = {
        "prompt_sha256": sha256_text(prompt),
        "tuning": run_split(prompt, Path("tuning.jsonl"), reveal=True),
        "holdout": run_split(prompt, Path("holdout.jsonl"), reveal=reveal),
    }
    Path("runs").mkdir(exist_ok=True)
    out = Path("runs") / (report["prompt_sha256"][:12] + ".json")
    out.write_text(json.dumps(report, indent=2), encoding="utf-8")
    print(out)
    print(
        "tuning_pass_rate={:.3f} holdout_pass_rate={:.3f} reveal={}".format(
            report["tuning"]["pass_rate"],
            report["holdout"]["pass_rate"],
            reveal,
        )
    )


if __name__ == "__main__":
    main()

When a prompt changes, tuning failures are fair game, because that split exists to teach you where the wording is still sloppy. Holdout failures stay aggregated unless REVEAL_HOLDOUT=1, which should happen after the digest is written, not while you are still editing. If holdout pass rate drops relative to the last accepted digest, the change is a regression even when tuning pass rate climbed. That comparison is the entire release rule; everything else is commentary.

# gate.py
from __future__ import annotations

import json
import sys
from pathlib import Path


def main() -> None:
    current = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    baseline = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))
    cur = current["holdout"]["pass_rate"]
    base = baseline["holdout"]["pass_rate"]
    delta = cur - base
    print(
        "holdout_now={:.3f} holdout_base={:.3f} delta={:+.3f} prompt={}".format(
            cur, base, delta, current["prompt_sha256"][:12]
        )
    )
    if current["prompt_sha256"] == baseline["prompt_sha256"]:
        raise SystemExit("gate refused: prompt hash did not change")
    if delta < 0:
        raise SystemExit("gate refused: holdout pass rate fell")
    Path("runs/accepted.json").write_text(json.dumps(current, indent=2), encoding="utf-8")


if __name__ == "__main__":
    main()

export EVAL_API_BASE='https://example.invalid/v1'
export EVAL_API_KEY='replace-me'
export EVAL_MODEL='replace-me'
python split_goldens.py goldens.jsonl
python run_eval.py
python gate.py runs/<digest>.json runs/accepted.json

The method still lies when the holdout is tiny, because a three-case drawer mostly measures luck and the last shuffle. It also lies when someone peeks at holdout completions during the same edit session, which launders those cases back into the tuning set without renaming them. Graders that accept any plausible JSON will bless format-stable nonsense, and a temperature above zero will make the gate flap unless you freeze decoding. None of those failure modes are solved by buying a larger labeled file and then iterating against every row again.

Skip this approach when the labeled set is smaller than about thirty cases, since a thirty percent holdout then behaves like a coin. Skip it for safety-critical refusal policy until a review process exists that is not a JSON equality check. Skip it if the product surface is multi-turn state, because a single-shot invoice extract will not see the regression that appears on turn three. Skip it when the team needs qualitative review more than a gate, because a pass rate cannot explain a bad tone.

Treat the tuning pass rate as a development console, and treat the holdout pass rate as the only number that can block a prompt digest. If the holdout job needs a hosted runner rather than a laptop cron, MonkeyCode's free server option is a reasonable place to park that gate. The files above remain useful when that host is absent, which is the point of keeping the endpoint behind environment variables.