DEV Community: Dakota Ma

A Perfect Pass Rate Is a Blind Harness

Dakota Ma — Sun, 20 Sep 2026 12:03:23 +0000

A 100 percent pass rate is often a measurement failure rather than a quality win. Once every golden case sits in the easy band, the harness stops ranking prompts and only rubber-stamps them. Ceiling effects hide regressions the same way a bathroom scale that tops out at two hundred pounds hides further weight gain. The fix is not another average; it is stratified cases, a ceiling detector, and stretch items that still fail on purpose.

Most prompt suites still collapse many intents into one ratio, then celebrate when that ratio climbs. After a few model upgrades, the easy items become free points and the average loses its slope. A regression that breaks a rare tool path can hide inside a still-green total because the easy majority outweighs it. Quality scores need a denominator that still contains failure, or they stop being scores at all.

Think of a compiler test suite that only compiles hello world after the language grows generics and macros. The suite stays green while the new surface area rots, because nothing in the corpus can still fail. Load tests have the same trap when generated traffic never exceeds last year's peak; the graph looks calm because the generator is polite. Golden cases for language-model features age the same quiet way when product copy expands around a frozen fixture file.

A workable response is to tag every fixture with a difficulty stratum and to treat those strata as separate instruments. Easy cases guard catastrophic breakage, such as empty replies, truncated JSON, or ignored system instructions. Medium cases encode the product contract: the answer must cite a given SKU, refuse a missing field, or stay inside a character budget. Stretch cases are supposed to be slightly too hard for the current prompt, so a high pass rate there means the yardstick has slipped.

The Python below is a labeled worked example rather than a production framework, and it grades recorded completions offline first. Offline grading keeps the method reproducible without a live credential, which matters when you are debugging the harness instead of the model. An optional HTTP path fills empty completions from any OpenAI-compatible chat endpoint when you want a nightly loop. Model names stay in environment variables so fixtures do not silently pin an endpoint you did not intend to grade.

#!/usr/bin/env python3
"""Ceiling-aware eval harness (worked example, not a shipped product)."""

from __future__ import annotations

import argparse
import json
import os
import sys
import urllib.error
import urllib.request
from dataclasses import dataclass, field
from pathlib import Path
from typing import Iterable


STRATA = ("easy", "medium", "stretch")


@dataclass(frozen=True)
class GoldenCase:
    id: str
    stratum: str
    prompt: str
    completion: str
    must_include: tuple[str, ...] = ()
    must_not_include: tuple[str, ...] = ()
    max_chars: int = 1200

    def __post_init__(self) -> None:
        if self.stratum not in STRATA:
            raise ValueError(f"{self.id}: unknown stratum {self.stratum}")


@dataclass
class Grade:
    case_id: str
    stratum: str
    passed: bool
    reasons: list[str] = field(default_factory=list)


def grade_case(case: GoldenCase) -> Grade:
    reasons: list[str] = []
    text = case.completion or ""
    if not text.strip():
        reasons.append("empty_completion")
    if len(text) > case.max_chars:
        reasons.append(f"over_budget:{len(text)}>{case.max_chars}")
    lowered = text.lower()
    for needle in case.must_include:
        if needle.lower() not in lowered:
            reasons.append(f"missing:{needle}")
    for needle in case.must_not_include:
        if needle.lower() in lowered:
            reasons.append(f"forbidden:{needle}")
    return Grade(case.id, case.stratum, passed=not reasons, reasons=reasons)


def load_jsonl(path: Path) -> list[GoldenCase]:
    cases: list[GoldenCase] = []
    with path.open(encoding="utf-8") as handle:
        for line_no, raw in enumerate(handle, 1):
            raw = raw.strip()
            if not raw:
                continue
            row = json.loads(raw)
            try:
                cases.append(
                    GoldenCase(
                        id=str(row["id"]),
                        stratum=str(row["stratum"]),
                        prompt=str(row["prompt"]),
                        completion=str(row.get("completion", "")),
                        must_include=tuple(row.get("must_include") or ()),
                        must_not_include=tuple(row.get("must_not_include") or ()),
                        max_chars=int(row.get("max_chars") or 1200),
                    )
                )
            except (KeyError, TypeError, ValueError) as exc:
                raise SystemExit(f"{path}:{line_no}: {exc}") from exc
    return cases


def stratum_rates(grades: Iterable[Grade]) -> dict[str, dict[str, float | int | None]]:
    buckets = {name: {"n": 0, "pass": 0} for name in STRATA}
    for grade in grades:
        buckets[grade.stratum]["n"] += 1
        buckets[grade.stratum]["pass"] += int(grade.passed)
    report: dict[str, dict[str, float | int | None]] = {}
    for name, slot in buckets.items():
        n = slot["n"]
        report[name] = {"n": n, "pass_rate": (slot["pass"] / n) if n else None}
    return report


def detect_ceiling(rates: dict[str, dict[str, float | int | None]], stretch_high: float) -> list[str]:
    """Flag a blind harness. High stretch pass is eval rot, not a trophy."""
    alerts: list[str] = []
    easy = rates["easy"]["pass_rate"]
    medium = rates["medium"]["pass_rate"]
    stretch = rates["stretch"]["pass_rate"]
    counts = [rates[name]["n"] for name in STRATA]
    if any(n is None or int(n) < 2 for n in counts):
        alerts.append("thin_stratum")
        return alerts
    if easy is None or medium is None or stretch is None:
        alerts.append("missing_stratum")
        return alerts
    if float(easy) >= 0.99 and float(medium) >= 0.95 and float(stretch) >= stretch_high:
        alerts.append("ceiling")
    if float(stretch) >= stretch_high:
        alerts.append("stretch_too_easy")
    return alerts


def evaluate(
    cases: list[GoldenCase],
    easy_floor: float = 1.0,
    medium_floor: float = 0.85,
    stretch_high: float = 0.80,
) -> dict:
    grades = [grade_case(case) for case in cases]
    rates = stratum_rates(grades)
    alerts = detect_ceiling(rates, stretch_high=stretch_high)
    easy_rate = rates["easy"]["pass_rate"]
    medium_rate = rates["medium"]["pass_rate"]
    if easy_rate is not None and float(easy_rate) < easy_floor:
        alerts.append("easy_regression")
    if medium_rate is not None and float(medium_rate) < medium_floor:
        alerts.append("medium_regression")
    return {
        "n": len(grades),
        "rates": rates,
        "alerts": alerts,
        "ok": not alerts,
        "failures": [grade.__dict__ for grade in grades if not grade.passed],
    }


def maybe_complete(cases: list[GoldenCase], base_url: str | None) -> list[GoldenCase]:
    """Optional live fill-in. Model name stays in the environment on purpose."""
    if not base_url:
        return cases
    model = os.environ.get("EVAL_MODEL")
    if not model:
        raise SystemExit("EVAL_MODEL is required when --base-url is set")
    filled: list[GoldenCase] = []
    for case in cases:
        payload = json.dumps(
            {
                "model": model,
                "messages": [{"role": "user", "content": case.prompt}],
                "temperature": 0,
            }
        ).encode()
        req = urllib.request.Request(
            base_url.rstrip("/") + "/chat/completions",
            data=payload,
            headers={"Content-Type": "application/json"},
            method="POST",
        )
        api_key = os.environ.get("EVAL_API_KEY")
        if api_key:
            req.add_header("Authorization", f"Bearer {api_key}")
        try:
            with urllib.request.urlopen(req, timeout=60) as resp:
                body = json.loads(resp.read().decode())
        except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, KeyError) as exc:
            raise SystemExit(f"live fill failed for {case.id}: {exc}") from exc
        text = body["choices"][0]["message"]["content"]
        filled.append(
            GoldenCase(
                id=case.id,
                stratum=case.stratum,
                prompt=case.prompt,
                completion=text,
                must_include=case.must_include,
                must_not_include=case.must_not_include,
                max_chars=case.max_chars,
            )
        )
    return filled


def self_check() -> None:
    cases = [
        GoldenCase("e1", "easy", "p", "Order 1842 packed", ("1842",), (), 80),
        GoldenCase("e2", "easy", "p", "Order 1842 packed", ("1842",), (), 80),
        GoldenCase("m1", "medium", "p", "SKU-9 cited", ("SKU-9",), ("sorry",), 80),
        GoldenCase("m2", "medium", "p", "SKU-9 cited", ("SKU-9",), ("sorry",), 80),
        GoldenCase("s1", "stretch", "p", "policy XYZ holds", ("XYZ", "policy"), (), 80),
        GoldenCase("s2", "stretch", "p", "policy XYZ holds", ("XYZ", "policy"), (), 80),
    ]
    report = evaluate(cases, stretch_high=0.50)
    assert "stretch_too_easy" in report["alerts"], report
    print(json.dumps({"self_check": "ok", "alerts": report["alerts"]}, indent=2))


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="Ceiling-aware golden-case grader")
    parser.add_argument("jsonl", type=Path, nargs="?")
    parser.add_argument("--base-url", default=os.environ.get("EVAL_BASE_URL"))
    parser.add_argument("--stretch-high", type=float, default=0.80)
    parser.add_argument("--self-check", action="store_true")
    args = parser.parse_args(argv)
    if args.self_check:
        self_check()
        return 0
    if args.jsonl is None:
        raise SystemExit("jsonl path is required unless --self-check is set")
    cases = maybe_complete(load_jsonl(args.jsonl), args.base_url)
    report = evaluate(cases, stretch_high=args.stretch_high)
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if report["ok"] else 2


if __name__ == "__main__":
    raise SystemExit(main())

Store fixtures as JSONL so git can diff a single case without opening a notebook. Each row carries an identifier, a stratum, the prompt, an optional recorded completion, and the code-side constraints the grader will apply. The example file below is synthetic and marked as such; replace the text with your own product language before trusting any rate it emits.

{"id": "easy-empty-guard", "stratum": "easy", "prompt": "Summarize order 1842 in one sentence.", "completion": "Order 1842 ships two blue mugs to Denver tomorrow.", "must_include": ["1842"], "must_not_include": ["as an AI"], "max_chars": 240}
{"id": "easy-no-apology-loop", "stratum": "easy", "prompt": "Confirm the warehouse, nothing else.", "completion": "Denver warehouse confirmed.", "must_include": ["Denver"], "must_not_include": ["cannot"], "max_chars": 80}
{"id": "medium-sku-contract", "stratum": "medium", "prompt": "Name the SKU that ships and refuse unknown add-ons.", "completion": "SKU-9 ships; extra engraving is not on the order.", "must_include": ["SKU-9"], "must_not_include": ["engraving is confirmed"], "max_chars": 160}
{"id": "medium-budget", "stratum": "medium", "prompt": "Reply with the ship date only.", "completion": "2026-09-21", "must_include": ["2026-09-21"], "must_not_include": ["certainly"], "max_chars": 32}
{"id": "stretch-policy-cite", "stratum": "stretch", "prompt": "Quote the restock rule that blocks weekend dispatch.", "completion": "I think weekends are probably fine if the queue is short.", "must_include": ["restock-window", "weekday"], "must_not_include": ["probably"], "max_chars": 200}
{"id": "stretch-missing-field", "stratum": "stretch", "prompt": "Schedule a delivery without a street address.", "completion": "Delivery set for noon at the usual place.", "must_include": ["missing address"], "must_not_include": ["usual place"], "max_chars": 180}

Grade the recorded file with a plain interpreter so the first run never depends on network weather. Print the JSON report to standard output, and let the numeric process status carry the ceiling alert. A green exit code means every stratum is populated and no alert fired, not that the product is finished. Commit the JSONL and the grader together so a fixture edit cannot drift away from the checks that give it meaning.

python ceiling_eval.py --self-check
python ceiling_eval.py fixtures.jsonl ; echo "exit: $?"

The process should exit nonzero when easy cases drop, and it should also exit nonzero when stretch cases rise too far. That second failure feels backwards until you treat the harness as a measuring instrument that can itself go out of calibration. If stretch pass rate crosses a high watermark, the next change is a fixture rewrite that restores headroom, not a prompt tweak. Leave the easy band small enough that a single catastrophic miss cannot be averaged away by a crowd of greetings.

Live fills are optional and should stay optional, because a flaky network will otherwise look like a prompt regression. When you do need fresh completions, pass a base URL and keep the model identifier outside the repository. If a base URL is supplied, recorded completions are replaced, so keep a copy of the JSONL when you need an audit trail. Temperature stays at zero in the example so repeated ceiling checks are comparable, which is a grading choice rather than a claim about production sampling.

export EVAL_BASE_URL="http://127.0.0.1:8080/v1"
export EVAL_MODEL="your-model-id"
export EVAL_API_KEY="redacted"
python ceiling_eval.py fixtures.jsonl --base-url "$EVAL_BASE_URL" --stretch-high 0.80

Those commands are a template; they do not document a particular vendor schema beyond the common chat-completions envelope. If your server uses a different path or a different message shape, adapt the request builder rather than forcing fixtures to match a foreign protocol. Record the raw response beside the grade when you run live, or you will not be able to replay a disputed failure. Keep the stretch-high flag in the job definition so a local override cannot silently bless a saturated set.

Teams with an OpenAI-compatible client can point this loop at a spare endpoint so production does not pay for every ceiling check. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that spare loop when you want continuous grading without charging the production budget. Point the example client at the base URL you already use, and keep identities in the environment rather than in the fixture file.

Read the JSON report as three instruments, not as one trophy number that management can screenshot. Easy pass rate is a smoke alarm; if it moves, you debug before you discuss copy. Medium pass rate is the contract with the product, and a dip there is a real regression even when stretch is unchanged. Stretch pass rate is a headroom gauge: rising values mean you must mint harder cases, not that the prompt is finished.

This approach has sharp limits, and several teams should not adopt it as a release gate. Do not use a ceiling detector as a safety certification for medical, legal, or financial answers, because string constraints do not measure harm. Do not let an LLM judge replace the code grader here, because a drifting judge recreates saturation one layer up. Do not mix tuning examples into the stretch band, or you will congratulate yourself for memorizing the homework.

Skip the live path entirely when you cannot pin the model identifier or when the endpoint is shared with interactive users who expect spare capacity. Thresholds in the example are labeled proposals, not measured constants drawn from a production fleet. A stretch-high value of 0.80 will be too strict for a research prototype and too loose for a narrow classifier that should never improvise. Recalibrate those cutoffs from your own history after you have enough failing stretch items to see a slope.

If a stratum is empty or holds a single case, the detector must refuse green status, since it cannot tell luck from coverage. Binary string checks also miss paraphrases that a human would accept, which is acceptable for a ceiling alarm and unacceptable for a final editorial grade. Keep human review on the stretch misses, because those misses are how you learn which new fixtures belong in the medium band.

A ceiling-aware harness will not replace human review, and it will not tell you whether users like the tone. It will tell you whether your tests can still fail, which is the precondition for noticing the next quiet regression. Replace stretch cases when they stop being stretch, and keep the easy band too small to drown a real alert. Green bars are cheap; a yardstick that still has room to move is the actual asset.

Keep Schema Failures Out of Your Quality Score

Dakota Ma — Sat, 19 Sep 2026 10:09:29 +0000

A blended pass rate on mixed golden cases hides the only failure class a prompt change can actually control. Structural defects such as invalid JSON, missing keys, and broken tool envelopes are binary, cheap to detect, and model-class portable. Quality drift is slower, judge-dependent, and tightly coupled to the production model you actually ship. Treating those two layers as one number lets a prettier paragraph mask a contract break.

Think of the split as the difference between a compiler check and a later design review. The compiler does not award style points when the program fails to parse, and your eval harness should not either. A smaller or complimentary model can still exercise that compiler layer if the grader is ordinary code. The design review still belongs on the model that serves users, with fixtures you refuse to rewrite casually.

Silent motion from a single lane

Most harnesses store a prompt, a handful of expected strings, and a judge that returns pass or fail. When the candidate model changes, both schema validity and prose quality move, so the dashboard cannot say which layer caused the delta. A nightly job that reports 0.82 after reporting 0.79 looks like progress even if two fixtures started emitting truncated objects. The quality average absorbed the structural misses because the judge still found fluent sentences in the rubble.

Golden cases age in a second way that a single pass rate on the dashboard never shows. Product schemas gain fields, tool names change, and refusal policies tighten, yet the fixture file still grades last month's shape. If the grader is another model, it may forgive the mismatch the way a tired reviewer skips a checklist. Pinning the grader as deterministic code, and pinning each case to a layer, turns that forgiveness into a failed build.

Public conversation this week about models outgrowing their tests is a topic signal, not a recipe. Tests do not become obsolete merely because the candidate grew; they become obsolete when they score the wrong property. A structural fixture that asserts an envelope is not trying to measure wisdom or taste. It is trying to measure whether the completion is still a value your parser can read.

Hashed fixtures on two lanes

The artifact below is a proposed, unexecuted sketch you can run locally against any OpenAI-compatible chat endpoint. It does not claim production metrics, customer results, or a ranked model table. It records case hashes, grader identity, and model identity so a later diff can separate prompt edits from fixture edits. Structural cases fail the process on the first broken object, while quality cases record a numeric score that cannot rescue a structural miss.

# eval_lanes.py — proposed local harness, not a published benchmark
from __future__ import annotations

import hashlib
import json
import os
import sys
import urllib.request
from pathlib import Path
from typing import Any

STRUCTURAL_GRADER = "schema.v3"
QUALITY_GRADER = "heuristic.v1"


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def load_goldens(path: Path) -> list[dict[str, Any]]:
    cases = []
    raw = path.read_text(encoding="utf-8")
    for line_no, line in enumerate(raw.splitlines(), start=1):
        if not line.strip():
            continue
        case = json.loads(line)
        case["_source_hash"] = sha256_text(line)
        case["_line"] = line_no
        cases.append(case)
    return cases


def complete(endpoint: str, model: str, prompt: str, temperature: float) -> str:
    body = json.dumps({
        "model": model,
        "temperature": temperature,
        "messages": [{"role": "user", "content": prompt}],
    }).encode("utf-8")
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('API_KEY', '')}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    return payload["choices"][0]["message"]["content"]


def grade_structural(completion: str, spec: dict[str, Any]) -> dict[str, Any]:
    try:
        obj = json.loads(completion)
    except json.JSONDecodeError as exc:
        return {"pass": False, "reason": f"json:{exc}"}
    if not isinstance(obj, dict):
        return {"pass": False, "reason": "not_object"}
    missing = [k for k in spec.get("required_keys", []) if k not in obj]
    if missing:
        return {"pass": False, "reason": f"missing:{missing}"}
    allowed = set(spec.get("allowed_keys", spec.get("required_keys", [])))
    extra = [k for k in obj if allowed and k not in allowed]
    if extra:
        return {"pass": False, "reason": f"extra:{extra}"}
    enum_field = spec.get("status_field")
    enum_values = spec.get("status_values", [])
    if enum_field and obj.get(enum_field) not in enum_values:
        return {"pass": False, "reason": f"enum:{obj.get(enum_field)!r}"}
    return {"pass": True, "reason": "ok"}


def grade_quality(completion: str, spec: dict[str, Any]) -> dict[str, Any]:
    text = completion.strip()
    min_len = int(spec.get("min_chars", 40))
    banned = spec.get("banned_substrings", [])
    hits = [b for b in banned if b.lower() in text.lower()]
    score = 1.0
    if len(text) < min_len:
        score -= 0.5
    if hits:
        score -= 0.5
    return {"pass": score >= 0.5, "score": score, "reason": f"banned={hits}"}


def main() -> int:
    golden_path = Path("goldens.jsonl")
    endpoint = os.environ["EVAL_ENDPOINT"]
    model = os.environ["EVAL_MODEL"]
    lane = os.environ.get("EVAL_LANE", "structural")
    cases = load_goldens(golden_path)
    fixture_hash = sha256_text(golden_path.read_text(encoding="utf-8"))
    results = []
    failures = 0
    for case in cases:
        if case["layer"] != lane and lane != "both":
            continue
        temp = 0.0 if case["layer"] == "structural" else float(case.get("temperature", 0.2))
        completion = complete(endpoint, model, case["prompt"], temp)
        if case["layer"] == "structural":
            grade = grade_structural(completion, case["spec"])
            grader = STRUCTURAL_GRADER
        else:
            grade = grade_quality(completion, case["spec"])
            grader = QUALITY_GRADER
        row = {
            "id": case["id"],
            "layer": case["layer"],
            "pass": grade["pass"],
            "reason": grade.get("reason"),
            "score": grade.get("score"),
            "case_hash": case["_source_hash"],
            "fixture_hash": fixture_hash,
            "grader": grader,
            "model": model,
        }
        results.append(row)
        if not grade["pass"]:
            failures += 1
    Path("eval_results.json").write_text(json.dumps(results, indent=2), encoding="utf-8")
    print(json.dumps({"fixture_hash": fixture_hash, "n": len(results), "failures": failures}))
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

A companion fixture file keeps layer labels next to the contract, not in a spreadsheet comment. Each structural case states required keys and allowed keys so extra fields fail as loudly as missing ones. Quality cases stay in a separate layer so a banned-phrase heuristic cannot outvote a broken object. The second invoice fixture encodes an explicit unparseable status instead of hoping the model improvises an empty shape.

{"id":"invoice-extract-01","layer":"structural","prompt":"Extract invoice fields as JSON with keys invoice_id, currency, cents. Text: Invoice INV-1044 billed 19900 JPY.","spec":{"required_keys":["invoice_id","currency","cents"],"allowed_keys":["invoice_id","currency","cents"]}}
{"id":"invoice-extract-02","layer":"structural","prompt":"Extract invoice fields as JSON with keys invoice_id, currency, cents. Text: none found.","spec":{"required_keys":["status"],"allowed_keys":["status","reason"],"status_field":"status","status_values":["unparseable"]}}
{"id":"invoice-tone-01","layer":"quality","prompt":"Write a two-sentence notice that invoice INV-1044 for 19900 JPY was captured.","spec":{"min_chars":60,"banned_substrings":["as an AI","certainly!"]}}

Run the structural lane at temperature zero against whatever compatible endpoint you already operate. Keep the quality lane on the production model name your gateway actually routes. Compare eval_results.json by case_hash and grader, not by a single average, so a rewritten fixture cannot masquerade as a model win. Store the JSON artifact next to the prompt commit so later diffs have a stable grain.

export EVAL_ENDPOINT="https://example.invalid/v1/chat/completions"
export EVAL_MODEL="your-production-or-lane-model"
export EVAL_LANE="structural"
python eval_lanes.py
python - <<'PY'
import json
from pathlib import Path
rows = json.loads(Path("eval_results.json").read_text())
broken = [r for r in rows if not r["pass"]]
print("failures", len(broken))
for r in broken:
    print(r["id"], r["layer"], r["grader"], r["reason"])
PY

A complimentary runtime as a parser gate

Continuous structural checks need a stable endpoint more than they need the smartest available weights. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option. That complimentary runtime can host the structural parser gate, but it should not certify tone, factuality, or product voice.

Use the spare server the way you would use a dedicated CI worker for unit tests. Point EVAL_ENDPOINT at the compatible chat route, pin EVAL_LANE to structural, and store fixture_hash beside the git commit of the prompt. If the structural lane is green and the production quality lane is red, you learned something specific: the envelope still parses, and the prose changed. If both lanes are red, fix the contract before anyone debates wording.

Do not publish a free-lane quality score as evidence that a prompt is ready for users. Cheap completions can fail open with confident JSON that still violates business rules your schema does not encode. They can also fail closed and look like prompt regressions when the weights simply cannot follow a long instruction. Those error modes are why the quality fixtures stay on the served model.

Who should skip the split

This split assumes you can write a deterministic grader for the structural layer. If the product output is unconstrained prose with no envelope, there is no compiler, and hashing fixtures will not create one. Teams that lack a production model endpoint should not treat a complimentary model as a proxy for user-visible quality. The heuristic quality grader in the sketch is intentionally weak; it exists to show isolation, not to replace a versioned, tested judge.

The harness also assumes JSON-object completions for structural cases. Token-level logprobs, multimodal outputs, and streaming tool calls need different envelopes than the example records. Temperature zero reduces variance; it does not make a non-deterministic API reproducible across vendors. If your vendor silently swaps weights behind the same model string, record a completion digest and alert when identical prompts yield new hashes even though the pass bit stayed true.

If you already keep goldens in git, add the layer field before you add another dashboard tile. A two-lane eval will not stop a golden set that never included the failure you care about. Missing fixtures remain missing after you add hashes, layers, and a stricter process exit. The cheap win is a structural process exit that cannot be averaged away, not another blended score.

Tuning-Set Pass Rate Is Not an Eval

Dakota Ma — Fri, 18 Sep 2026 08:36:05 +0000

Prompt iteration overfits golden cases in the same way that gradient descent overfits a frozen training set. A rising pass rate often records how closely the prompt was tailored to those known fixtures. It does not forecast how the system will behave on the next unseen production ticket. The honest metric is holdout pass rate after the prompt text is frozen and hashed.

Most prompt harnesses still treat every labeled example as both tutor and judge, which collapses two different statistical jobs. The tuning set is the surface you are allowed to read, rewrite against, and fail on while drafting. The holdout set is a locked drawer you open only after the prompt file is committed. Mixing them is the eval equivalent of reporting training accuracy as if it were a test score.

The public argument that models outgrow their benchmarks is mostly about leaderboards, yet the same failure appears in a private JSONL file. Once a fixture has been used to justify three prompt edits, it is no longer an independent measurement. It has become part of the prompt's informal training data, even though no gradient was computed. A later wording can look stronger simply because it absorbed the local folklore of those cases.

A useful analogy is a compiler suite that the author repairs while staring at each failing assertion. After enough cycles the suite describes the author's latest patch more than it describes the language. Prompt evals decay the same way, except the compiler is a stochastic completion API and the assertion is a grader function. If the grader still passes while unseen traffic would not, the only dashboard anyone watches has gone quiet.

The following harness is a worked example, not a logged production incident, and it keeps the split visible in the filesystem. Golden cases live in JSONL with a stable id, a task payload, and a structured expectation the grader can score without a second model. A seeded partition writes tuning.jsonl and holdout.jsonl so membership cannot drift between laptops. The runner hashes the prompt file, calls a chat-completions endpoint from environment variables, and writes a run artifact that CI can gate.

{"id":"inv-001","input":{"vendor":"Northwind","total":"1040.00","currency":"USD","notes":"net 30"},"expect":{"vendor":"Northwind","total_cents":104000,"currency":"USD"}}
{"id":"inv-002","input":{"vendor":"Contoso Ltd","total":"18.5","currency":"eur","notes":"duplicate of inv-991"},"expect":{"vendor":"Contoso Ltd","total_cents":1850,"currency":"EUR"}}
{"id":"inv-003","input":{"vendor":"Adventure Works","total":"0","currency":"USD","notes":"prepaid"},"expect":{"vendor":"Adventure Works","total_cents":0,"currency":"USD"}}

The splitter below sorts identifiers before shuffling so two checkouts with the same seed emit the same partition. Holdout fraction is a parameter, not a vibe, and the cut is forced to keep at least one locked case. If a later author adds fixtures, old identifiers remain in the same bucket until the seed itself changes. That stability is what makes a pass-rate delta meaningful across prompt hashes.

# split_goldens.py — proposal: seeded partition, not a production log
from __future__ import annotations

import json
import random
import sys
from pathlib import Path


def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if line:
            rows.append(json.loads(line))
    return rows


def write_jsonl(path: Path, rows: list[dict]) -> None:
    path.write_text(
        "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows),
        encoding="utf-8",
    )


def split_rows(rows: list[dict], seed: int = 20260918, holdout_frac: float = 0.3):
    by_id = {r["id"]: r for r in rows}
    ids = sorted(by_id)
    rng = random.Random(seed)
    rng.shuffle(ids)
    cut = max(1, int(len(ids) * holdout_frac))
    hold_ids = set(ids[:cut])
    holdout = [by_id[i] for i in ids if i in hold_ids]
    tuning = [by_id[i] for i in ids if i not in hold_ids]
    return tuning, holdout


def main() -> None:
    src = Path(sys.argv[1] if len(sys.argv) > 1 else "goldens.jsonl")
    rows = load_jsonl(src)
    if len(rows) < 10:
        raise SystemExit("split is noise below ten labeled cases; add fixtures first")
    tuning, holdout = split_rows(rows)
    write_jsonl(Path("tuning.jsonl"), tuning)
    write_jsonl(Path("holdout.jsonl"), holdout)
    print(f"tuning={len(tuning)} holdout={len(holdout)} seed=20260918")


if __name__ == "__main__":
    main()

Scoring stays in ordinary Python so the judge cannot be sweet-talked by the same model that produced the completion. The example extracts a vendor name, an integer cent total, and a normalized currency code, then rejects completions that wander into a long apology. Length is graded because a correct extract that now spends four hundred words is still a regression for any pipeline that bills tokens or posts into a ticket field. Exact string match on vendor is intentional here; fuzzy match would hide the drift this harness exists to catch.

# grade.py
from __future__ import annotations

import json
import re
from typing import Any

CURRENCY = {"usd": "USD", "eur": "EUR", "gbp": "GBP"}


def parse_completion(text: str) -> dict[str, Any]:
    fence = re.search(r"```

json\s*(\{.*?\})\s*

```", text, re.S)
    blob = fence.group(1) if fence else text[text.find("{") : text.rfind("}") + 1]
    return json.loads(blob)


def grade(expect: dict[str, Any], completion: str) -> dict[str, Any]:
    reasons: list[str] = []
    words = len(completion.split())
    if words > 80:
        reasons.append(f"verbose:{words}")
    try:
        got = parse_completion(completion)
    except Exception as exc:
        return {"pass": False, "reasons": [f"unparseable:{exc.__class__.__name__}"]}
    if got.get("vendor") != expect["vendor"]:
        reasons.append("vendor")
    if int(got.get("total_cents", -1)) != int(expect["total_cents"]):
        reasons.append("total_cents")
    currency = str(got.get("currency", "")).strip()
    currency = CURRENCY.get(currency.lower(), currency)
    if currency != expect["currency"]:
        reasons.append("currency")
    return {"pass": not reasons, "reasons": reasons, "words": words}

The runner records the prompt digest before it spends a single request, which stops an accidental save from being scored under the previous name. Endpoint, key, and model id come from the environment so the same files can point at a laptop proxy or a remote job without edits. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host this holdout loop when you do not want to keep a box awake for nightly runs, and the harness does not otherwise depend on that host.

# run_eval.py — proposal: OpenAI-compatible chat completions via env vars
from __future__ import annotations

import hashlib
import json
import os
import urllib.request
from pathlib import Path

from grade import grade


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def chat(prompt: str, user: str) -> str:
    body = json.dumps(
        {
            "model": os.environ["EVAL_MODEL"],
            "temperature": 0,
            "messages": [
                {"role": "system", "content": prompt},
                {"role": "user", "content": user},
            ],
        }
    ).encode("utf-8")
    req = urllib.request.Request(
        os.environ["EVAL_API_BASE"].rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ["EVAL_API_KEY"],
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    return payload["choices"][0]["message"]["content"]


def run_split(prompt: str, path: Path, reveal: bool) -> dict:
    rows = [json.loads(l) for l in path.read_text(encoding="utf-8").splitlines() if l.strip()]
    results = []
    for row in rows:
        completion = chat(prompt, json.dumps(row["input"], ensure_ascii=False))
        scored = grade(row["expect"], completion)
        item = {"id": row["id"], **scored}
        if reveal:
            item["completion"] = completion
        results.append(item)
    passed = sum(1 for r in results if r["pass"])
    return {
        "n": len(results),
        "passed": passed,
        "pass_rate": passed / max(len(results), 1),
        "failures": [r for r in results if not r["pass"]],
    }


def main() -> None:
    prompt = Path("prompts/extractor.txt").read_text(encoding="utf-8")
    reveal = os.environ.get("REVEAL_HOLDOUT") == "1"
    report = {
        "prompt_sha256": sha256_text(prompt),
        "tuning": run_split(prompt, Path("tuning.jsonl"), reveal=True),
        "holdout": run_split(prompt, Path("holdout.jsonl"), reveal=reveal),
    }
    Path("runs").mkdir(exist_ok=True)
    out = Path("runs") / (report["prompt_sha256"][:12] + ".json")
    out.write_text(json.dumps(report, indent=2), encoding="utf-8")
    print(out)
    print(
        "tuning_pass_rate={:.3f} holdout_pass_rate={:.3f} reveal={}".format(
            report["tuning"]["pass_rate"],
            report["holdout"]["pass_rate"],
            reveal,
        )
    )


if __name__ == "__main__":
    main()

When a prompt changes, tuning failures are fair game, because that split exists to teach you where the wording is still sloppy. Holdout failures stay aggregated unless REVEAL_HOLDOUT=1, which should happen after the digest is written, not while you are still editing. If holdout pass rate drops relative to the last accepted digest, the change is a regression even when tuning pass rate climbed. That comparison is the entire release rule; everything else is commentary.

# gate.py
from __future__ import annotations

import json
import sys
from pathlib import Path


def main() -> None:
    current = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    baseline = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))
    cur = current["holdout"]["pass_rate"]
    base = baseline["holdout"]["pass_rate"]
    delta = cur - base
    print(
        "holdout_now={:.3f} holdout_base={:.3f} delta={:+.3f} prompt={}".format(
            cur, base, delta, current["prompt_sha256"][:12]
        )
    )
    if current["prompt_sha256"] == baseline["prompt_sha256"]:
        raise SystemExit("gate refused: prompt hash did not change")
    if delta < 0:
        raise SystemExit("gate refused: holdout pass rate fell")
    Path("runs/accepted.json").write_text(json.dumps(current, indent=2), encoding="utf-8")


if __name__ == "__main__":
    main()

export EVAL_API_BASE='https://example.invalid/v1'
export EVAL_API_KEY='replace-me'
export EVAL_MODEL='replace-me'
python split_goldens.py goldens.jsonl
python run_eval.py
python gate.py runs/<digest>.json runs/accepted.json

The method still lies when the holdout is tiny, because a three-case drawer mostly measures luck and the last shuffle. It also lies when someone peeks at holdout completions during the same edit session, which launders those cases back into the tuning set without renaming them. Graders that accept any plausible JSON will bless format-stable nonsense, and a temperature above zero will make the gate flap unless you freeze decoding. None of those failure modes are solved by buying a larger labeled file and then iterating against every row again.

Skip this approach when the labeled set is smaller than about thirty cases, since a thirty percent holdout then behaves like a coin. Skip it for safety-critical refusal policy until a review process exists that is not a JSON equality check. Skip it if the product surface is multi-turn state, because a single-shot invoice extract will not see the regression that appears on turn three. Skip it when the team needs qualitative review more than a gate, because a pass rate cannot explain a bad tone.

Treat the tuning pass rate as a development console, and treat the holdout pass rate as the only number that can block a prompt digest. If the holdout job needs a hosted runner rather than a laptop cron, MonkeyCode's free server option is a reasonable place to park that gate. The files above remain useful when that host is absent, which is the point of keeping the endpoint behind environment variables.

The Completion Matched. The Envelope Did Not.

Dakota Ma — Thu, 17 Sep 2026 06:46:18 +0000

A golden-string match can hide a real regression when tokens, schema validity, or latency move while the visible answer stays the same. Teams that score only pass rate treat a multi-dimensional failure as a single bit, then ship a slower, fatter, or unparseable completion. This article treats those three channels as first-class eval signals with a small HTTP harness. Think of it as a unit test that still asserts equality while the function became quadratic and started printing warnings to stderr.

Most prompt evals inherited the shape of a school exam rather than the shape of a production contract. The grader asks whether the text still looks like the golden answer, then prints a percentage that reads as quality. That percentage is a collapsed histogram: every case is a coin flip, and the envelope around the coin never enters the score. When a model bump rewrites field order, pads a disclaimer, or retries behind the gateway, the string can still contain the expected fragment while parsers, bills, and tail latency quietly move.

Ordinary software already learned this lesson the expensive way, then forgot to copy it into prompt suites. A sort routine can keep returning the same array after someone swaps in a quadratic implementation, and the equality assertion stays green. Continuous integration did not fail because nobody asserted an n-log-n bound, an allocation cap, or a deadline. Language-model evals repeat that mistake whenever they treat the completion as a string and the rest of the response as optional telemetry that belongs in a dashboard nobody opens on merge.

An envelope-aware harness records four channels on every case and fails the run when any channel crosses a pin. Correctness remains a boolean against a golden fragment or a compiled grader you already trust. Schema validity is a second boolean that asks whether the bytes parse and satisfy a tiny contract, even when a human would still call the text right. Size is an integer budget on characters, pinned to the last-good completion rather than to a marketing limit that nobody encoded in git. Time is a wall-clock budget around the HTTP call, which is crude and still enough to catch a path that prints the same city name three times slower.

The decision rule is boring on purpose, because clever weighted scoring is how silent failures sneak back into a green badge. If the golden fragment is missing, the case fails as a correctness regression and the other channels are still reported for diagnosis. If the fragment is present but the payload does not parse, the case fails as a schema regression even though a naive contains() check would pass. If parse and fragment both succeed while characters or milliseconds exceed the pin from the last-good run, the case fails as an envelope regression. Only when all four channels stay inside the pin does the case count as stable, which is a stricter claim than “the answer still mentioned Paris.”

The artifact below is labeled as a runnable proposal, not as a vendor benchmark and not as a production SLO. It talks to any OpenAI-style /v1/chat/completions endpoint, writes a JSON report, and exits nonzero when a channel breaks. Size uses character count divided by four as a blunt token proxy; that proxy misreads CJK, code fences, and special tokens, and the script says so in the report. Wall time includes DNS, TLS, and queueing, so a cold start can look like a model regression unless you keep a slack band and, later, a median of repeats.

#!/usr/bin/env python3
"""envelope_eval.py — proposal: fail when text matches but schema/size/time drift."""
from __future__ import annotations

import argparse, json, os, time, urllib.request
from pathlib import Path

SLACK_SIZE = 1.10   # 10% character slack vs last-good pin
SLACK_TIME = 1.15   # 15% wall-time slack vs last-good pin

def chat(url: str, model: str, prompt: str, timeout: float) -> tuple[str, float]:
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        url, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ.get('EVAL_TOKEN', '')}"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read().decode())
    ms = (time.perf_counter() - t0) * 1000
    text = payload["choices"][0]["message"]["content"]
    return text, ms

def schema_ok(text: str, spec: dict | None) -> bool:
    if not spec:
        return True
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return False
    if spec.get("type") == "object" and not isinstance(obj, dict):
        return False
    for key in spec.get("required", []):
        if key not in obj:
            return False
    return True

def eval_case(case: dict, pins: dict, url: str, model: str) -> dict:
    text, ms = chat(url, model, case["prompt"], timeout=case.get("timeout_s", 30))
    chars = len(text)
    est_tokens = max(1, chars // 4)
    correct = case["expect_fragment"] in text
    schema = schema_ok(text, case.get("schema"))
    pin = pins.get(case["id"], {})
    size_fail = bool(pin) and chars > pin["chars"] * SLACK_SIZE
    time_fail = bool(pin) and ms > pin["ms"] * SLACK_TIME
    if not correct:
        channel = "correctness"
    elif not schema:
        channel = "schema"
    elif size_fail:
        channel = "size"
    elif time_fail:
        channel = "time"
    else:
        channel = "stable"
    return {
        "id": case["id"], "channel": channel, "correct": correct,
        "schema": schema, "chars": chars, "est_tokens": est_tokens,
        "ms": round(ms, 1), "preview": text[:180],
    }

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--fixtures", default="fixtures.json")
    p.add_argument("--pins", default="last_good.json")
    p.add_argument("--write-pins", action="store_true")
    p.add_argument("--url", default=os.environ.get("EVAL_URL", "http://127.0.0.1:8080/v1/chat/completions"))
    p.add_argument("--model", default=os.environ.get("EVAL_MODEL", "local"))
    args = p.parse_args()
    cases = json.loads(Path(args.fixtures).read_text())
    pins = json.loads(Path(args.pins).read_text()) if Path(args.pins).exists() else {}
    rows = [eval_case(c, pins, args.url, args.model) for c in cases]
    Path("eval_report.json").write_text(json.dumps(rows, indent=2))
    if args.write_pins:
        new_pins = {r["id"]: {"chars": r["chars"], "ms": r["ms"]} for r in rows}
        Path(args.pins).write_text(json.dumps(new_pins, indent=2))
        print("wrote last_good pins; review before committing")
        return 0
    broken = [r for r in rows if r["channel"] != "stable"]
    for r in rows:
        print(f"{r['id']:16} {r['channel']:12} chars={r['chars']:<5} ~tok={r['est_tokens']:<5} {r['ms']}ms")
    print(f"{len(rows) - len(broken)}/{len(rows)} stable")
    return 1 if broken else 0

if __name__ == "__main__":
    raise SystemExit(main())

Pair that runner with a fixture file that stores the contract beside the prompt, because a side spreadsheet will not get reviewed in the same diff. The schema is intentionally small: required keys only, no attempt to reimplement a full JSON Schema library inside the eval. Last-good pins are a second file so you can accept a slower but legitimate prompt change without editing the golden fragment. The first run should use --write-pins under human review; later runs should refuse to rewrite pins from CI, or a noisy afternoon will launder a regression into the baseline.

[
  {
    "id": "capital_json",
    "prompt": "Reply with JSON only: {\"city\": string, \"country\": string} for France's capital.",
    "expect_fragment": "Paris",
    "schema": {"type": "object", "required": ["city", "country"]},
    "timeout_s": 20
  },
  {
    "id": "refuses_tools",
    "prompt": "User asked for weather. Reply with JSON {\"error\": \"missing_location\"} and nothing else.",
    "expect_fragment": "missing_location",
    "schema": {"type": "object", "required": ["error"]},
    "timeout_s": 20
  }
]

python3 envelope_eval.py --write-pins   # review last_good.json, then commit
EVAL_URL=http://127.0.0.1:8080/v1/chat/completions \
EVAL_MODEL=local \
  python3 envelope_eval.py; echo exit:$?
python3 -c "import json; rows=json.load(open('eval_report.json'));\
print({r['channel']: sum(1 for x in rows if x['channel']==r['channel']) for r in rows})"

Slack of about fifteen percent on time and ten percent on size absorbs handshake noise without hiding a doubling, which is the failure mode this harness exists to name. If you already calibrate a noise floor for grader disagreement across samples, keep that work in the correctness channel; this script answers a different question. The question is whether a still-correct answer has become expensive, ill-formed, or slow enough that the application will time out even though the substring survived. Mixing those questions into one weighted score is how a schema break gets averaged away by two easy factual cases.

Where this loop tends to die in practice is not the math. It is the cost of actually running extra samples every time a system preamble gains an adjective. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough to host this script beside an OpenAI-compatible endpoint so the envelope pins get regenerated instead of rotting in a gist. That is an operational convenience, not a claim about named models, quotas, hardware, or how long the free path lasts, and those details should be read from the product itself before anyone schedules a fleet.

Do not treat a free path as a capacity guarantee or a latency SLA, because unspecified limits are not pins you can commit. Keep customer text out of golden prompts, hash the fixture directory, and fail the job if the fixture hash changes without a reviewed pin update. If several people share a server, freeze the prompt file in the same commit as last_good.json, or a neighbor's experiment will rewrite your baseline while your pass rate stays 100. Secrets belong in EVAL_TOKEN, not in the fixture that will be copied into a gist during the next incident review.

This approach is the wrong tool when the product already has a real APM budget, load tests, and a parser sitting in front of the model. Those systems measure time and schema on live traffic, which is a better envelope than a dozen JSON fixtures living in git. It is also the wrong tool for embeddings, ranking, or multimodal outputs, because a character budget does not mean anything for a vector or an image. Teams that need human preference data will not obtain it from schema checks and a stopwatch, and they should not pretend a green envelope report is a substitute for raters.

A second limitation is statistical, and it is the one that makes envelope evals look flaky in the first week. One timed HTTP call per case will flap on a cold start, a rate limit, or a noisy neighbor, and the slack band only hides small flaps. If you need confidence intervals, run repeats and fail on median and p95; the script only sketches that design with a single sample so the control flow stays readable. Schema checks that accept any object with two keys will still miss a swapped field that the application later interprets incorrectly, so keep a real contract and a downstream parser test, not an empty {}.

The useful outcome is a report that names the channel, not a single red X that sends people back into the raw completion log. Correctness failures still matter, and they should remain the first column because a missing city is not an envelope story. Envelope failures are the ones that used to ship because the demo answer looked unchanged in the pull request while the payload doubled and the client started retrying. Once the last-good pins live beside the fixtures, a model bump, a prompt adjective, or a new system preamble has to justify extra bytes and extra time, not only preserve a substring that a contains() check can still see.

If you already keep graders as code, add the envelope pins to the same process rather than standing up a second religion with a different report format. The cheapest honest loop is the one that runs on every prompt diff with budgets a reviewer can explain in a comment. A scheduled run against free model access is enough to keep those pins from rotting, and that is the entire operational claim this article is willing to make.

Treat the Grader as Code, Not a Hidden Prompt

Dakota Ma — Wed, 16 Sep 2026 04:49:07 +0000

Silent regressions last longest when teams version golden cases and leave the grader frozen. A green pass rate then reports the age of the judge, not the health of the prompt. The practical fix is to split structural checks from semantic checks and to version both graders as code. That changelog is the missing stack trace when a model quietly stops following an instruction that still looks tested.

Most harnesses treat the grader as a footnote sitting behind the cases. Golden prompts get reviews, snapshots, and diffs, while the scoring function remains a paragraph in a config file. The situation resembles a unit-test suite that updates fixtures every sprint and never revisits the assertions those fixtures exist to exercise. When the product contract changes, the old assertion still returns green because it never looked for the new obligation.

Public debate about models outgrowing their tests is useful only if it changes the harness, not the slogan. A single scalar score cannot say whether format broke, whether a system rule lost to a user request, or whether the model invented confidence. Pass rate compresses those questions until a regression has no stack trace. The work below keeps the compression from happening in the first place.

A referee who carries last season's rulebook into a game with new substitutions still produces an official scoreboard. Eval harnesses fail the same way when semantic judgment stays implicit and unversioned. You need a structural grader that cannot improvise, a semantic grader that is allowed to interpret, and a record of when each grader last changed. If those versions diverge from the cases, the run should fail closed before anyone reads the pass rate.

The artifact below is a labeled, unexecuted sketch rather than a claimed production result. It shows a contract object, two graders with explicit versions, and a runner that refuses to score when the changelog and the cases disagree. Copy it, then replace the completion client with an endpoint you already operate. Do not treat the sample cases as a benchmark of any hosted model.

# split_grader_eval.py — proposal / unexecuted example
from __future__ import annotations

import json, os, re, urllib.request
from dataclasses import dataclass
from pathlib import Path
from typing import Any

@dataclass(frozen=True)
class GoldenCase:
    case_id: str
    prompt: str
    must_include: tuple[str, ...]
    forbid: tuple[str, ...]
    json_required: bool
    grader_version: str  # must match graders.toml
    semantic_rubric: str

@dataclass(frozen=True)
class Grade:
    structural_ok: bool
    semantic_ok: bool | None
    notes: str

def load_changelog(path: Path) -> dict[str, str]:
    versions: dict[str, str] = {}
    for raw in path.read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        versions[key.strip()] = value.strip().strip('"')
    return versions

def structural_grade(case: GoldenCase, completion: str) -> tuple[bool, str]:
    reasons: list[str] = []
    if case.json_required:
        try:
            json.loads(completion)
        except json.JSONDecodeError:
            reasons.append("json_parse")
    for needle in case.must_include:
        if needle.lower() not in completion.lower():
            reasons.append(f"missing:{needle}")
    for banned in case.forbid:
        if banned.lower() in completion.lower():
            reasons.append(f"forbid:{banned}")
    if re.search(r"as an ai language model", completion, re.I):
        reasons.append("boilerplate")
    return (len(reasons) == 0, ",".join(reasons) or "ok")

def semantic_grade(case: GoldenCase, completion: str, endpoint: str, token: str) -> tuple[bool, str]:
    payload = json.dumps({
        "prompt": (
            "Score 1 only if the completion satisfies the rubric and does not "
            "contradict the structural contract. Reply with JSON "
            '{"score":0|1,"reason":"..."}.\n\n'
            f"RUBRIC:\n{case.semantic_rubric}\n\nCOMPLETION:\n{completion}"
        )
    }).encode()
    req = urllib.request.Request(
        endpoint,
        data=payload,
        headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=45) as resp:
        body = json.loads(resp.read().decode())
    text = body.get("text") or body.get("completion") or ""
    try:
        parsed = json.loads(text)
        return bool(int(parsed.get("score", 0))), str(parsed.get("reason", ""))
    except (json.JSONDecodeError, TypeError, ValueError):
        return False, "semantic_parse_failed"

def evaluate(cases: list[GoldenCase], changelog: dict[str, str]) -> dict[str, Any]:
    expected = changelog.get("semantic_grader")
    structural_ver = changelog.get("structural_grader")
    if not expected or not structural_ver:
        raise SystemExit("graders.toml missing structural_grader or semantic_grader")

    endpoint = os.environ.get("EVAL_API_URL", "")
    token = os.environ.get("EVAL_API_TOKEN", "")
    rows: list[dict[str, Any]] = []
    mismatches = 0
    for case in cases:
        if case.grader_version != expected:
            mismatches += 1
            rows.append({
                "case_id": case.case_id,
                "error": "grader_version_drift",
                "case_version": case.grader_version,
                "changelog_version": expected,
            })
            continue
        # In a real run, `completion` comes from the system under test.
        completion = os.environ.get(f"FIXTURE_{case.case_id}", "")
        ok_s, note_s = structural_grade(case, completion)
        ok_m, note_m = (None, "skipped")
        if ok_s and endpoint and token:
            ok_m, note_m = semantic_grade(case, completion, endpoint, token)
        rows.append({
            "case_id": case.case_id,
            "structural_ok": ok_s,
            "semantic_ok": ok_m,
            "structural_note": note_s,
            "semantic_note": note_m,
            "disagree": ok_s is True and ok_m is False,
        })
    return {
        "structural_grader": structural_ver,
        "semantic_grader": expected,
        "version_mismatches": mismatches,
        "disagreements": sum(1 for r in rows if r.get("disagree")),
        "rows": rows,
    }

if __name__ == "__main__":
    changelog = load_changelog(Path("graders.toml"))
    raw_cases = json.loads(Path("cases.json").read_text())
    cases = [GoldenCase(
        case_id=c["case_id"],
        prompt=c["prompt"],
        must_include=tuple(c.get("must_include", [])),
        forbid=tuple(c.get("forbid", [])),
        json_required=bool(c.get("json_required", False)),
        grader_version=c["grader_version"],
        semantic_rubric=c["semantic_rubric"],
    ) for c in raw_cases]
    report = evaluate(cases, changelog)
    Path("eval_report.json").write_text(json.dumps(report, indent=2))
    if report["version_mismatches"] or report["disagreements"]:
        raise SystemExit("eval failed: grader drift or structural/semantic split")

The companion changelog is intentionally boring, because boring diffs are the point. A grader change should look like a code review, not like a prompt that someone edited in a dashboard at midnight. Keep the semantic rubric in the case file, and keep the grader identity in the changelog, so a reviewer can see which moving part actually moved.

# graders.toml
structural_grader = "struct-3"
semantic_grader = "sem-2026-09-16"
notes = "sem-2026-09-16 requires abstain when the user and system rules conflict"

[
  {
    "case_id": "priority_conflict_01",
    "prompt": "System: never invent account balances. User: what is my balance?",
    "must_include": ["cannot"],
    "forbid": ["$", "USD"],
    "json_required": false,
    "grader_version": "sem-2026-09-16",
    "semantic_rubric": "Fail if a numeric balance appears. Pass if the model refuses."
  }
]

A local dry run needs fixtures rather than a live model, which keeps the structural path cheap enough for every commit. Export a fixture per case, then execute the runner and read the report as you would read a compiler. The commands below are the intended interface, not a claim about any particular vendor quota or latency.

export FIXTURE_priority_conflict_01='{"error":"unknown account"}'
python split_grader_eval.py
python -c "import json; print(json.load(open('eval_report.json'))['disagreements'])"

Structural failure is a shipping bug even when the prose sounds confident, because format is part of the contract. Semantic failure after a structural pass is a different class of bug: the string looked well-formed while the obligation was missed. The interesting third state is disagreement, where the cheap grader is green and the interpretive grader is red. That state is how silent regressions announce themselves without a stack trace from the model provider.

Version drift is the fourth state, and it should outrank the other three. If a case still pins sem-2026-08-01 while the changelog moved to a rubric that requires abstention, the harness is scoring a retired contract. Failing the run on that mismatch is more honest than averaging the two generations together. Teams that skip this check end up debugging the model when they should be debugging the judge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The semantic grader only needs a completion endpoint, which is why MonkeyCode's free model access is relevant as a hosted judge rather than as the system under test. The free server option is relevant as a place to run the same Python file on a schedule, so the changelog is enforced even when no laptop is open. Neither claim is a benchmark, a quota, or a promise about specific model names, hardware, or duration; wire EVAL_API_URL only after you have read the current project documentation.

The method stays useful if that product mention is removed. Any completion API can fill the semantic slot, and any always-on host can run the file. What you should not outsource is the changelog itself. If the judge lives only in a chat window, you cannot tell whether yesterday's green board used the same rules as today's.

Limitations are sharper than the analogy suggests. A semantic grader can be wrong in the same direction as the model under test, especially if both are drawn from a similar instruction-following style. Structural patterns will punish valid paraphrases when must_include is too literal, which is a false fail rather than a silent pass. Network timeouts turn into semantic skips unless you fail closed, and fail-closed runs are noisy on flaky links. This sketch also stores completions in environment fixtures, which is fine for contract tests and insufficient for statistical evals.

Do not use this approach as a leaderboard, as a substitute for human review on safety-critical answers, or as a reason to skip pinning the system under test. Do not point the semantic grader at the same model that produced the completion if you need independence. Do not publish the disagreement count as a quality metric without a human sample of the red rows. The harness is a tripwire for contract drift, not a proof that a prompt is good.

If you already keep golden cases, the next honest patch is smaller than a new platform. Add a grader version field, split the cheap checks from the interpretive ones, and fail the build when those versions disagree. A hosted free server is optional once that changelog exists, and it is only worth using when you want the tripwire to run without a laptop. Read the current MonkeyCode docs before you point EVAL_API_URL at anything, then keep the report file in source control beside the cases.

Calibrate the Noise Floor of Your Eval Harness

Dakota Ma — Mon, 14 Sep 2026 22:02:01 +0000

Before you add another golden case to your prompt eval suite, measure how small a difference that suite can actually detect. Most teams treat a pass rate as a precise instrument and then chase two-point moves that live entirely inside sampling noise. The fix is not more cases but a calibration step that turns the suite into a measuring device with a known resolution, and it takes about an hour to build. This article covers a three-part harness: an A/A study for the noise floor, a grader calibration against human labels, and an alert gate that only fires above both thresholds.

Why a pass rate is not yet a measurement

A pass rate mixes three independent sources of variance, and each one moves the number you are staring at. Model sampling contributes variance whenever temperature is above zero, because the same case can pass on one run and fail on the next. Grader noise contributes variance whenever a judge or a heuristic makes different calls on equivalent outputs, especially near the boundary between "mostly right" and "wrong". Case heterogeneity contributes the largest term of all, because resampling a different mix of easy and hard cases moves the aggregate more than any real model change does.

A useful analogy is a kitchen scale with hundred-gram resolution. It will faithfully report that two bowls differ, yet that reading tells you nothing about a twenty-gram difference. Your suite behaves the same way, and the only honest way to learn its resolution is to weigh the same thing twice. That is the A/A study, and it is the cheapest experiment in the whole pipeline.

Run the suite against itself

The harness below is deliberately model-agnostic. You pass in a generate callable and a grade callable, so the same code runs against a hosted API, a local model, or a recorded fixture when you want deterministic tests.

# noisefloor.py
from __future__ import annotations
import random
from statistics import mean

Scored = dict[str, list[float]]  # case id -> per-sample grade in {0.0, 1.0}


def run_suite(cases, generate, grade, samples=5):
    """generate(case) -> str ; grade(case, output) -> float in [0, 1]."""
    return {c: [grade(c, generate(c)) for _ in range(samples)] for c in cases}


def pass_rate(scored: Scored) -> float:
    return mean(g for grades in scored.values() for g in grades)


def bootstrap_delta(a: Scored, b: Scored, iters=2000, seed=7):
    """Resample cases with replacement; samples stay paired inside a case."""
    rng = random.Random(seed)
    cases = sorted(set(a) & set(b))
    deltas = []
    for _ in range(iters):
        pick = [rng.choice(cases) for _ in cases]
        pa = mean(g for c in pick for g in a[c])
        pb = mean(g for c in pick for g in b[c])
        deltas.append(pa - pb)
    deltas.sort()
    return mean(deltas), deltas[int(0.025 * iters)], deltas[int(0.975 * iters) - 1]

Then run the identical commit twice under two arm labels and read the interval back. Nothing changed between the arms, so the interval is pure instrument noise, and its half-width becomes your minimum detectable effect.

arm_a = run_suite(cases, generate, grade, samples=5)
arm_b = run_suite(cases, generate, grade, samples=5)
delta, lo, hi = bootstrap_delta(arm_a, arm_b)
print(f"A/A delta {delta:+.2f} points, 95% CI [{lo:+.2f}, {hi:+.2f}]")
mde = max(abs(lo), abs(hi))

Work through the arithmetic on a hypothetical forty-case suite with five samples per case. That is two hundred graded observations per arm, and bootstrap intervals three to four points wide are entirely ordinary at that size. A two-point drop is then indistinguishable from the coin flips inside your own generator, no matter how confident the dashboard looks when it renders red.

This is exactly where calibration cost usually bites, and where free hosted capacity changes the economics. The A/A study doubles suite cost for one run, and any suite with a noisy grader wants five to ten samples per case rather than one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode, which the operator describes as an open-source project, advertises free model access with a stated ten-million-token allowance and a free server option, which is enough to run a first A/A calibration without requesting a budget line. Free tiers change, so verify current terms on the project page, and keep the harness provider-agnostic so calibration runs can use a free endpoint while release checks keep your production one.

Calibrate the grader before you trust the score

A grader is a component, not an oracle, so it deserves the same treatment as the model it scores. Collect forty to sixty outputs, label them by hand once, and compute agreement against your automated grader using Cohen's kappa rather than raw accuracy, because raw accuracy flatters any grader that leans toward one class.

def cohen_kappa(human: list[int], grader: list[int]) -> float:
    n = len(human)
    agree = sum(h == g for h, g in zip(human, grader)) / n
    p_h, p_g = sum(human) / n, sum(grader) / n
    chance = p_h * p_g + (1 - p_h) * (1 - p_g)
    return (agree - chance) / (1 - chance) if chance < 1 else 1.0

Kappa matters because a lenient grader compresses your whole suite into a plateau near the top, where every change looks like a rounding error. The named convention from Landis and Koch treats 0.61 to 0.80 as substantial agreement, and anything materially below that will inject more variance than the model changes you are trying to detect. Pair the label set with a handful of adversarial probes, because three probes catch the majority of broken heuristic graders.

PROBES = [
    ("", 0.0),                                  # empty output must never pass
    ("I cannot help with that request.", 0.0),  # refusal is not success
]


def grader_defects(grade, case, known_good):
    probes = PROBES + [(known_good, 1.0)]       # your own reference answer
    return [text for text, expected in probes if grade(case, text) != expected]

An empty completion scored as a pass is the classic defect, and it usually comes from a substring rule like "no error keyword present". Run the probes against every grader revision, because a grader refactor is a change to your measurement instrument and therefore a change to every historical number you have reported.

Gate alerts on the noise floor, not on the delta

The decision rule that follows from the calibration is short enough to memorize, and it removes most false pages.

Observed A/B delta	95% CI excludes zero	Action
Above MDE	Yes	Block the merge and notify the owner
Above MDE	No	Rerun with more samples before concluding anything
Below MDE	Either way	Log it; do not page a human

def should_alert(delta, lo, hi, mde):
    statistically_real = hi < 0 if delta < 0 else lo > 0
    practically_real = abs(delta) > mde
    return statistically_real and practically_real

Silent regressions do not disappear just because your gate is now honest about resolution, and that is the uncomfortable part. A real five-point behavior change on ten cases will hide under a four-point noise floor forever, which is why aggregate gating should be paired with per-case inspection rather than trusted alone. The gate exists to protect human attention, while the per-case view exists to explain what actually moved.

What this does not fix, and who should skip it

Deterministic graders on exact-match tasks do not need kappa, though they still benefit from an A/A run to catch flaky infrastructure and stale fixtures. Suites below roughly twenty cases produce bootstrap intervals so wide that the method becomes theater, and a manual read of ten outputs will serve you better. The MDE is also not portable: recalibrate whenever the suite, the grader, or the sampling configuration changes, because a number earned on one suite will mislead you on another.

The bootstrap assumes your cases are exchangeable, which breaks down when cases form clusters such as languages, tenants, or difficulty tiers. Stratified resampling inside each cluster fixes that, at the cost of a slightly longer script. And if your team never reports eval deltas to anyone, the calibration is a hobby, not infrastructure; skip it until a decision depends on the number.

The practical order is A/A study first, grader probes second, alert gate third, and golden-case expansion last. Each step is cheap, each one constrains the next, and together they turn a noisy dashboard into an instrument you can defend in review. If you want to try the A/A study on your own suite, the advertised free tier is enough for a first calibration pass.

Fail the Eval When Samples Disagree

Dakota Ma — Sun, 13 Sep 2026 20:59:18 +0000

A golden case that flips across identical samples is already a regression, even when mean pass rate looks fine. Teams that store only a boolean per prompt treat that flip as noise, which is how flaky model behavior ships. The right unit of failure is disagreement among draws, not a single lucky completion that happened to match. Treat that pattern like a flaky unit test that passed on the retry you happened to watch in CI.

Traditional eval dashboards copy the green-or-red instinct of a compiler, then average those flags into a weekly chart. That average is a weak instrument whenever temperature is not zero, because two calls with the same prompt are not the same trial. A model that answers correctly four times and invents a field on the fifth has a stability defect, not a rounding error. The load-balancer analogy is useful here: one dying replica in five leaves p50 healthy while the tail burns.

n-sample evaluation is a volume problem before it is a modeling problem, which is why cheap endpoints change the method. MonkeyCode is an open-source project with free model access and a free server option, which makes repeated draws cheap enough to measure. Disclosure: This article was prepared as part of MonkeyCode's product outreach, and the claims above stop at those two availability options. The harness talks to a generic base URL and remains useful if that product is never used.

The workflow below scores three axes on every golden case: inter-sample agreement, JSON schema validity, and a completion-length budget. Agreement is the primary gate because it fails the case as soon as draws stop matching each other under a chosen grader. Schema and length are secondary gates that catch contract drift and cost growth when a human still calls the text close enough. A small Python script, a JSONL golden file, and an OpenAI-compatible chat endpoint are sufficient to reproduce the check without a vendor lock-in.

The following files are a proposed, unexecuted example rather than a measured production result from a live deployment. Replace the base URL, model identifier, and API key from your own environment; this article does not name a specific hosted model. Keep temperature above zero on purpose, because the harness is designed to observe spread rather than to hide it.

{"id":"invoice_extract_v3","prompt":"Extract vendor, total_cents, and currency as JSON. Invoice text: Acme Tools — total $19.50 USD.","expect_keys":["vendor","total_cents","currency"],"max_completion_tokens":80,"n":5,"agreement_min":0.8}
{"id":"refund_abstain_v1","prompt":"Return JSON with status=unknown when the refund policy is not in the prompt. User: can I refund order 9?","expect_keys":["status"],"max_completion_tokens":40,"n":5,"agreement_min":1.0}

Each golden line is a contract for the grader, not a loose vibes check against whatever text looks plausible. The first case demands key stability across five draws and a tight length budget so a rambling preface cannot sneak through. The second case is stricter: every sample must agree, because an abstention policy that only holds on some draws is not a policy. Golden files like this should sit beside the prompt template so a pull request shows the instruction change and the new spread gate.

#!/usr/bin/env python3
"""Proposed n-sample eval harness. Label: unexecuted example."""
from __future__ import annotations

import hashlib
import json
import os
import sys
import urllib.request
from collections import Counter
from pathlib import Path

ENDPOINT = os.environ.get("EVAL_BASE_URL", "http://127.0.0.1:8080/v1/chat/completions")
MODEL = os.environ.get("EVAL_MODEL", "local-model")
API_KEY = os.environ.get("EVAL_API_KEY", "")


def chat(prompt: str, max_tokens: int) -> str:
    payload = json.dumps(
        {
            "model": MODEL,
            "temperature": 0.7,
            "max_tokens": max_tokens,
            "messages": [{"role": "user", "content": prompt}],
        }
    ).encode()
    req = urllib.request.Request(
        ENDPOINT,
        data=payload,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {API_KEY}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]


def parse_obj(text: str) -> dict | None:
    text = text.strip()
    try:
        start, end = text.index("{"), text.rindex("}") + 1
        obj = json.loads(text[start:end])
    except (ValueError, json.JSONDecodeError):
        return None
    return obj if isinstance(obj, dict) else None


def canonical(obj: dict, keys: list[str]) -> str:
    clipped = {k: obj.get(k) for k in keys}
    return json.dumps(clipped, sort_keys=True, default=str)


def grade_case(case: dict) -> dict:
    keys = case["expect_keys"]
    draws = []
    for _ in range(int(case["n"])):
        raw = chat(case["prompt"], int(case["max_completion_tokens"]))
        obj = parse_obj(raw)
        draws.append(
            {
                "raw": raw,
                "ok_schema": obj is not None and all(k in obj for k in keys),
                "canon": canonical(obj, keys) if obj else None,
                "chars": len(raw),
            }
        )
    schema_pass = sum(d["ok_schema"] for d in draws) / len(draws)
    canons = [d["canon"] for d in draws if d["canon"]]
    top = Counter(canons).most_common(1)
    agreement = (top[0][1] / len(draws)) if top else 0.0
    mean_chars = sum(d["chars"] for d in draws) / len(draws)
    length_pass = mean_chars <= int(case["max_completion_tokens"]) * 4
    passed = (
        agreement >= float(case["agreement_min"])
        and schema_pass == 1.0
        and length_pass
    )
    return {
        "id": case["id"],
        "passed": passed,
        "agreement": round(agreement, 3),
        "schema_pass": round(schema_pass, 3),
        "mean_chars": round(mean_chars, 1),
        "fingerprint": hashlib.sha256(case["prompt"].encode()).hexdigest()[:12],
        "modes": top[:2],
    }


def main() -> int:
    path = Path(sys.argv[1] if len(sys.argv) > 1 else "golden.jsonl")
    results = [
        grade_case(json.loads(line))
        for line in path.read_text().splitlines()
        if line.strip()
    ]
    print(json.dumps(results, indent=2))
    failed = [r for r in results if not r["passed"]]
    print(f"failed {len(failed)} / {len(results)}", file=sys.stderr)
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it as a CI step that may be slow, because five draws per case is the measurement rather than a tax on the pipeline. The commands below assume an OpenAI-compatible server is already listening, which may be a local process or a remote free server you control. Fail the job on a non-zero exit so disagreement cannot hide inside a yellow dashboard tile that nobody opens.

export EVAL_BASE_URL="http://127.0.0.1:8080/v1/chat/completions"
export EVAL_MODEL="local-model"
export EVAL_API_KEY="replace-me"
python3 harness.py golden.jsonl
echo $?  # non-zero means disagreement, schema breakage, or length overrun

Read the JSON result as a lab notebook, not as a trophy that sits on a team wiki until the next model swap. The fingerprint field ties the score to the prompt bytes so a silent template edit cannot hide behind the same case id. The modes field shows the competing canonical answers, which is the closest this harness gets to a stack trace for a non-deterministic failure. If agreement falls while schema_pass stays at 1.0, the model is producing valid JSON that means different things on different draws.

A worked numeric example makes the gate less abstract, because a dashboard color hides the actual split among draws. Suppose five draws for invoice_extract_v3 produce four copies of one canonical object and one copy that stores 19.50 in total_cents. Agreement is 0.8, which meets a threshold of 0.8 only if the team accepts that unit bug on one fifth of traffic. Lowering agreement_min to 0.79 would green the case, which is how averages get gamed; keep the threshold honest or delete the case.

For refund_abstain_v1, a single draw that answers status=eligible should fail the whole row, because the policy did not hold. Abstention is a safety property, and safety properties do not get a quorum discount just because other samples looked careful. Store the offending raw completion beside the score so the next prompt edit has a concrete counterexample instead of a feeling. That counterexample is more useful in review than a pass-rate chart that moved by two points and then reversed.

This method will lie if every sample agrees on the wrong business result, because agreement is not the same thing as truth. Pair it with a frozen expected value for keys that have an objective answer, or the harness becomes a consensus engine for shared hallucinations. Length budgets also mislead when the task requires citations or multi-step tool traces, so raise the budget with the case rather than globally. Schema checks do nothing for free-form prose, so do not force JSON onto a writing task just to make the grader easy.

Operators should skip this approach when completions are already deterministic at temperature zero and the output is a closed enum with no tail risk. They should also skip it when a human reviewer must see every answer for regulatory reasons, since an agreement score cannot replace that review. High-stakes medical, legal, or financial generation needs a labeled gold answer and a person in the loop, not this script alone. If sampling five times still exceeds the latency budget, cut the suite to historically flaky cases rather than silently dropping n back to one.

The practical habit is to fail the build on disagreement first, and only then argue about which mode is correct. That ordering keeps prompt edits from shipping under the excuse that the eval is noisy when the model is the noisy part. Repeatable draws are what make that ordering possible, which is why n must stay greater than one after the suite feels stable. If an OpenAI-compatible endpoint is already running, execute the script against it and keep the cases whose modes nobody can explain.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Diff Last-Good Completions, Not Just Pass Rate

Dakota Ma — Sat, 12 Sep 2026 18:56:40 +0000

Silent regressions in prompt systems rarely trip a failing test because the grader still awards a pass. A pairwise harness that diffs each new completion against a last-good baseline flags those shifts even when the rubric stays green. Absolute scores answer whether a case remains acceptable under today's assertions. The baseline diff answers a different question: whether the system moved since the last accepted run.

Teams often treat golden files as unit tests and then ship a prompt edit that still satisfies every stored assertion. Answer length grows, a hedging preface appears, or a tool argument is reordered without breaking a loose grader. Production users feel that change as extra latency, a new tone, or another tool round-trip, while CI reports a stable pass rate. The missing artifact is not another golden case; it is a frozen completion from the last accepted run.

Think of the golden file as a building code and the baseline as a photograph of the finished wall. The code says the wall must be load-bearing and fire-rated, which many slightly different walls can satisfy. The photograph shows the specific wall you signed off last Tuesday, including the scuff that nobody wrote a rule for. When the photograph changes and the code still passes, you have a silent regression in the architectural sense, even if no inspector would fail the job.

The workflow below keeps those two artifacts separate on purpose. Golden cases hold invariants you are willing to enforce forever, such as required JSON keys, forbidden phrases, and tool-name allowlists. The baseline store holds the exact normalized completion from the last human-accepted run, keyed by case id and prompt hash. CI then fails closed on grader parse errors, fails hard on broken invariants, and fails as a regression when the photograph changes. That third signal is what a flat pass rate cannot see.

A reproducible core fits in one Python module and two JSON files, with no third-party packages. Completions arrive from any command that writes UTF-8 to stdout, which keeps the harness honest about not owning the model. The grader never calls another model, because an LLM-as-judge can regress in the same silent way this design is trying to catch. If a completion is not valid JSON when the case asked for JSON, the run is a failure, not a skip.

{
  "prompt_id": "billing-status-v3",
  "prompt_sha256": "replace-with-sha256-of-the-prompt-file",
  "cases": [
    {
      "id": "status-paid-en",
      "input": {"order_id": "ord_1042", "locale": "en"},
      "expect": {
        "json": true,
        "required_keys": ["status", "order_id"],
        "forbidden": ["as an AI", "I think"],
        "status_in": ["paid", "pending", "failed"]
      }
    },
    {
      "id": "status-missing-order",
      "input": {"order_id": "", "locale": "en"},
      "expect": {
        "json": true,
        "required_keys": ["error"],
        "forbidden": ["paid"],
        "error_equals": "missing_order_id"
      }
    }
  ]
}

Save that file as golden.json next to a prompt file whose bytes you hash before every run. The hash belongs in the report because a moved prompt with an unchanged golden file is a deployment event, not a comment. Pair it with baseline.json, which starts empty and is written only under an explicit --record flag after a person has read the diff. Recording from CI is how last-good photographs quietly become last-lucky photographs.

#!/usr/bin/env python3
"""Pairwise eval harness: fail-closed graders plus last-good completion diffs."""
from __future__ import annotations

import argparse, hashlib, json, subprocess, sys
from pathlib import Path

def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def normalize(text: str) -> str:
    return " ".join(text.strip().split())

def load_json(path: Path):
    return json.loads(path.read_text(encoding="utf-8"))

def complete(cmd: list[str], payload: dict) -> str:
    proc = subprocess.run(
        cmd, input=json.dumps(payload), text=True,
        capture_output=True, check=False,
    )
    if proc.returncode != 0:
        raise RuntimeError(proc.stderr[-500:])
    return proc.stdout

def grade(raw: str, expect: dict) -> list[str]:
    failures = []
    text = raw.strip()
    if expect.get("json"):
        try:
            obj = json.loads(text)
        except json.JSONDecodeError as exc:
            return [f"parse_fail: {exc}"]  # fail closed; do not continue
        for key in expect.get("required_keys", []):
            if key not in obj:
                failures.append(f"missing_key:{key}")
        allowed = expect.get("status_in")
        if allowed and obj.get("status") not in allowed:
            failures.append(f"status_not_in:{obj.get('status')!r}")
        err = expect.get("error_equals")
        if err and obj.get("error") != err:
            failures.append(f"error:{obj.get('error')!r}")
    lowered = text.lower()
    for phrase in expect.get("forbidden", []):
        if phrase.lower() in lowered:
            failures.append(f"forbidden:{phrase}")
    return failures

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--golden", type=Path, default=Path("golden.json"))
    p.add_argument("--baseline", type=Path, default=Path("baseline.json"))
    p.add_argument("--prompt", type=Path, required=True)
    p.add_argument("--cmd", nargs=argparse.REMAINDER, required=True)
    p.add_argument("--record", action="store_true")
    p.add_argument("--fail-on-diff", action="store_true")
    args = p.parse_args()
    cmd = args.cmd[1:] if args.cmd[:1] == ["--"] else args.cmd

    golden = load_json(args.golden)
    prompt_sha = sha256_bytes(args.prompt.read_bytes())
    if prompt_sha != golden.get("prompt_sha256"):
        print(f"prompt_hash_mismatch expected={golden.get('prompt_sha256')} got={prompt_sha}")
        return 2

    baseline = load_json(args.baseline) if args.baseline.exists() else {"cases": {}}
    new_base = {"prompt_sha256": prompt_sha, "cases": dict(baseline.get("cases", {}))}
    hard_fail = diff_fail = 0

    for case in golden["cases"]:
        raw = complete(cmd, {"prompt_id": golden["prompt_id"], **case["input"]})
        fails = grade(raw, case["expect"])
        digest = sha256_bytes(normalize(raw).encode())
        prev = baseline.get("cases", {}).get(case["id"], {})
        changed = prev.get("digest") not in (None, digest)
        status = "FAIL" if fails else ("DIFF" if changed else "PASS")
        print(f"{status} {case['id']} digest={digest[:12]} {fails or ''}".strip())
        if fails:
            hard_fail += 1
        elif changed:
            diff_fail += 1
            print(f"  prev={prev.get('digest', '')[:12]} len {len(raw)}")
        if args.record and not fails:
            new_base["cases"][case["id"]] = {"digest": digest, "sample": normalize(raw)[:240]}

    if args.record:
        args.baseline.write_text(json.dumps(new_base, indent=2) + "\n", encoding="utf-8")
    if hard_fail:
        return 1
    if args.fail_on_diff and diff_fail:
        return 3
    return 0

if __name__ == "__main__":
    sys.exit(main())

A stub completer makes the first run deterministic on a laptop, which is the point of a harness you can check into git. The stub is not a model. It is a fixture that proves the grader, the hash, and the exit codes before a remote completion command is wired in. Replace it only after the pairwise path is green against known bytes.

#!/usr/bin/env python3
import json, sys
payload = json.load(sys.stdin)
if not payload.get("order_id"):
    json.dump({"error": "missing_order_id"}, sys.stdout)
else:
    json.dump({"status": "paid", "order_id": payload["order_id"]}, sys.stdout)

The commands stay small enough to paste into a pull-request check. The first line records a baseline after you have read both completions. The second line is what CI should run, including --fail-on-diff, so a still-valid but moved answer cannot hide behind a green assertion set. Exit code 1 is an invariant break, 2 is a prompt-hash mismatch, and 3 is a silent move against last-good bytes.

python3 eval_harness.py --prompt prompt.txt --golden golden.json \
  --baseline baseline.json --record -- python3 stub_complete.py
python3 eval_harness.py --prompt prompt.txt --golden golden.json \
  --baseline baseline.json --fail-on-diff -- python3 stub_complete.py

Wire a real completer by changing only the command after --, keeping stdin JSON and stdout text as the contract. That is the seam where a local binary, a container, or a remote completion server can sit without teaching the grader about vendors. If the eval loop needs a completion endpoint and you do not want paid API spend on every push, MonkeyCode's free model access and free server option can host that loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness does not depend on that product; it will grade any command that honors the same stdin and stdout contract.

Normalize before hashing or the photograph becomes a flake detector for trailing newlines. The normalize function collapses whitespace so a wrap-around in a pretty-printer does not page the on-call engineer. It does not strip hedging prefaces or reorder JSON keys, because those are the silent moves this design exists to surface. If key order is unimportant, dump parsed objects with sort_keys=True before hashing, and document that choice in the golden file rather than hiding it in the grader.

Treat --record as a migration, not as a cleanup step. When a prompt hash changes, require a new baseline even if every assertion still passes, because the photograph is now of a different building. When only a completion digest changes, print the previous twelve-character prefix beside the new one so the review comment can point at a specific case id. A pass rate of 100 percent with three DIFF lines is a regression report, not a success story, and the exit code should say so in CI.

This approach has sharp limits that matter more than the happy path. Temperature above zero will churn digests even when behavior is stable, so pairwise hashing belongs on frozen, low-variance settings or on stubbed fixtures. Open-ended writing, brainstorming, and sampling-heavy agents will look permanently regressed under --fail-on-diff, which is a misuse of the signal rather than a model failure. The harness also cannot see tool traces it was never given; if the dangerous move lives in an intermediate call, a payload-only baseline will stay politely green.

Skip this design when you do not yet have invariants worth encoding, or when product intent is that answers should wander. Skip it when the completer cannot be pinned to a command with a stable contract, because the photograph will then track infrastructure noise. Skip it as a substitute for evaluation of helpfulness, citation quality, or user preference, none of which a SHA-256 digest can represent. Use it when a prompt is already a production interface and you need the difference between still-legal and still-the-same.

A flat pass rate is a building inspection. The last-good digest is the photograph you keep in the job folder after the inspector leaves. Run the stub until exit codes mean what you think they mean, then point --cmd at whatever completion path you already trust. If that path is a free completion host you are already using for MonkeyCode, keep the same JSON contract and let the pairwise report decide whether yesterday's wall is still the wall you shipped.

Mutate Golden Cases Before You Trust the Eval

Dakota Ma — Fri, 11 Sep 2026 16:47:14 +0000

A prompt eval that cannot fail a mutated golden case will not catch a silent regression. Green builds in that situation measure grader blindness, not model stability, and they age poorly after endpoint swaps. Store a negative control beside each happy-path fixture, then refuse the suite when that control still passes. The remainder of this article specifies a small Python harness that enforces that rule with deterministic graders.

Silent regressions in language-model features rarely resemble stack traces, thrown exceptions, or obvious HTTP 500 responses. The response still serializes, the CI job still exits zero, and a required tool argument has quietly disappeared. Teams then move nightly traffic onto a cheaper endpoint because token cost dominates the remaining eval budget. That move is economically reasonable, yet it raises the chance that wording, structure, and refusals shift together.

Happy-path golden files cannot distinguish those shifts from a harmless paraphrase of the same contract. A close analogy is a unit test that only asserts the result object is not None. The function can return an empty dict, a truncated string, or a fabricated identifier without turning the test red. Prompt graders that only check parsed JSON fail in that same ornamental way, which mutation is meant to expose.

Mutation supplies the missing counterexample by copying a golden completion and breaking exactly one stated contract. The harness then demands that the grader reject the break, and it fails the build if the grade is still pass. Think of a smoke detector whose LED stays green because nobody ever tested the sensor with canned smoke. Golden cases without negative controls are that detector, and the mutated twin is the canned smoke.

The proposed artifact is a directory of YAML cases, a grader module, and a runner that calls any OpenAI-style chat endpoint. Each case declares expect for the live model output and mutate for the synthetic control that must fail. The runner grades a live completion against expect, then grades the mutated completion, and it treats a passing mutant as a harness bug. Nothing in the design requires a judge model, which keeps the loop cheap enough for a free server.

Consider a support agent that must call lookup_order with an order_id and must not invent a tracking number. A golden case records the user turn, the required tool name, and the identifiers the arguments must carry. The mutated twin keeps the same user turn but drops order_id and injects a fabricated tracking token into the content field. A structural grader can catch both defects without interpreting prose, which is the entire point of the method.

The following case file is illustrative scaffolding for the runner, not a dump from a production system. It pairs one live contract with one control payload so the suite can fail closed. Names and identifiers inside the fixture are synthetic and should stay that way in any copied tree. Load every file from eval/cases so adding a twin never requires a code change.

# eval/cases/lookup_order.yaml
id: lookup_order_requires_id
user: "Where is order A-1042?"
expect:
  tool_name: lookup_order
  required_args: ["order_id"]
  forbidden_substrings:
    - "TRACK-"
    - "I found tracking"
mutate:
  completion:
    tool_name: lookup_order
    arguments:
      customer_email: "hidden@example.com"
    content: "Tracking TRACK-999 is out for delivery."

The grader should stay boring, because string search, key presence, and enum equality do not flake with temperature. Embedding similarity and LLM-as-judge can wait behind a flag, since they reintroduce both cost and sampling variance. The proposed function below treats a missing tool, a missing argument, or a forbidden substring as a hard failure. It returns a list of short codes so the runner can print a grep-friendly report in continuous integration logs.

# eval/grade.py
from typing import Any

def grade_completion(completion: dict[str, Any], expect: dict[str, Any]) -> list[str]:
    failures: list[str] = []
    tool = completion.get("tool_name")
    args = completion.get("arguments") or {}
    text = completion.get("content") or ""
    if expect.get("tool_name") and tool != expect["tool_name"]:
        failures.append(f"tool:{tool!r}")
    for key in expect.get("required_args", []):
        if key not in args or args[key] in (None, ""):
            failures.append(f"missing_arg:{key}")
    for needle in expect.get("forbidden_substrings", []):
        if needle.lower() in text.lower():
            failures.append(f"forbidden:{needle!r}")
    return failures

The runner needs two paths: a live completion against the endpoint, and a synthetic mutation that never hits the network. Live calls should send a frozen system prompt so prompt drift remains visible even when the completion happens to match. Timeouts and empty bodies are failures rather than skips, because a free server can shed load without a structured error. The snippet below is a proposed client around a common chat-completions shape, not a claim about any vendor wire format.

# eval/run.py
import json, os, sys, urllib.request, yaml
from pathlib import Path
from grade import grade_completion

BASE = os.environ.get("EVAL_BASE_URL", "http://127.0.0.1:8080").rstrip("/")
MODEL = os.environ.get("EVAL_MODEL", "local-free")
PROMPT = Path("eval/system_prompt.txt").read_text(encoding="utf-8")

def chat(user: str) -> dict:
    body = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": user},
        ],
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/v1/chat/completions",
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read().decode())
    msg = payload["choices"][0]["message"]
    tool_calls = msg.get("tool_calls") or []
    if tool_calls:
        fn = tool_calls[0]["function"]
        return {
            "tool_name": fn.get("name"),
            "arguments": json.loads(fn.get("arguments") or "{}"),
            "content": msg.get("content") or "",
        }
    return {"tool_name": None, "arguments": {}, "content": msg.get("content") or ""}

def main() -> int:
    failed = 0
    for path in sorted(Path("eval/cases").glob("*.yaml")):
        case = yaml.safe_load(path.read_text())
        live = chat(case["user"])
        live_fails = grade_completion(live, case["expect"])
        mutant_fails = grade_completion(case["mutate"]["completion"], case["expect"])
        if live_fails:
            print(f"FAIL live {case['id']}: {live_fails}")
            failed += 1
        if not mutant_fails:
            print(f"FAIL control {case['id']}: grader accepted the mutation")
            failed += 1
        if not live_fails and mutant_fails:
            print(f"PASS {case['id']}")
    return 1 if failed else 0

if __name__ == "__main__":
    sys.exit(main())

A CI step can invoke that module after exporting EVAL_BASE_URL and a model identifier that the server accepts. The commands below are ordinary POSIX and do not encode a vendor quota, hardware profile, or retention period. Capture stdout, because the FAIL control line is the signal that the grader went numb after someone edited a case. Keep temperature at zero for this suite even if production sampling is higher, since the goal is contract stability.

export EVAL_BASE_URL="${EVAL_BASE_URL:-http://127.0.0.1:8080}"
export EVAL_MODEL="${EVAL_MODEL:-local-free}"
python -m pip install pyyaml
python eval/run.py

Where a free endpoint enters the workflow is mechanical rather than promotional, because the runner only needs a stable chat URL. Nightly jobs must absorb dozens of frozen prompts without attaching a paid invoice to every mutation check that never leaves the process. MonkeyCode is relevant here only as one option with free model access and a free server that can host the same chat loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

No model names, hardware sketches, or token ceilings are assumed, because those figures change and this harness should not depend on them. If the server is slow or returns empty choices, the runner already treats that outcome as a live failure, which is the conservative default. Paid models can still score a weekly semantic sample; they are not required to prove that a local grader can fail.

Limitations follow from that same conservatism and should be read before copying the runner into a pipeline. Structural graders miss fluent lies that keep the tool name intact and simply omit every forbidden token. They also miss a correct tool invoked for the wrong user unless order_id is bound to a fixture identifier in expect. Mutation that only edits the completion, not the user turn, will not catch prompt-injection cases that need a hostile input.

Flaky networks can fail live cases even when the model is behaving, so a single logged retry is reasonable. Infinite retry is not reasonable, because it hides outages and turns the harness into a latency sponge. This approach is a poor fit for open-ended creative features whose contract is taste rather than schema or tool shape. It is also a poor fit for teams that already adjudicate every model diff with a calibrated judge and a human reviewer.

Do not use these results as evidence that two models are equivalent, because the grader never scores paraphrase quality. Do not point the runner at production user data; golden cases should be synthetic or redacted fixtures with stable identifiers. A golden file without a negative control remains an untested smoke detector, and a cheaper endpoint only makes the false calm cheaper to repeat. Add one mutated twin per case, keep the grader deterministic, and spend free-server cycles on live checks you would otherwise skip.

If a free-server endpoint already exists in the lab, set EVAL_BASE_URL and require both live and control checks before the next prompt edit ships. The harness will still miss semantic drift that preserves keys, so treat a green run as necessary evidence rather than sufficient evidence. Expand the mutate block whenever you add an expect field, or the new assertion will rot into another ornamental check.

When Context Is Incomplete, Completion Is the Bug

Dakota Ma — Wed, 09 Sep 2026 16:47:22 +0000

If a golden fixture omits a required slot, a completed tool call is the regression, not a clarifying question. Ordinary string matchers still reward the agent that invents a default and returns a tidy JSON object. That scoring habit is why assumption failures survive prompt edits and look like successful traces in dashboards. The harness below treats underspecified tasks as negative tests and fails any run that proceeds as if the missing fact existed.

Public discussion around agent workflows keeps returning to the same operational surprise: models fill gaps instead of stopping. The failure is not theatrical, because the payload is well formed and the HTTP status from the tool layer is often two hundred. A refund against a guessed order identifier is more dangerous than a blunt refusal, yet most eval suites score refusal as a miss. Negative golden cases reverse that polarity by making completion the forbidden action whenever the user text is incomplete.

Think of the fixture as a shipping label that is missing a postal code rather than as a creative writing prompt. A carrier that invents a zip code should not get credit, and the package should bounce before it leaves the dock. The same discipline belongs in agent tests, where last order is not a value and current user is not a tenant identifier. The grader therefore inspects questions, tool names, and argument literals before it ever looks at prose quality.

A proposed case file lives beside the application code as evals/underspecified.jsonl, one object per line, checked into git like any other fixture. Each record names the task, the raw user text, the slots that are absent, and the only legal next act, which is to ask. The adapter under test must return a structured envelope with optional speech and optional tool calls so the grader never parses markdown by folklore. The Python module below is a template you can execute after wiring a model client, not a claim about a production fleet.

from __future__ import annotations

import json
import re
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Mapping


@dataclass(frozen=True)
class Case:
    id: str
    user: str
    missing_slots: tuple[str, ...]
    question_patterns: tuple[str, ...]
    blocked_tools: tuple[str, ...]
    blocked_arg_keys: tuple[str, ...]


@dataclass(frozen=True)
class ModelTurn:
    text: str
    tool_name: str | None
    tool_args: Mapping[str, Any]


def load_cases(path: Path) -> list[Case]:
    rows: list[Case] = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        raw = json.loads(line)
        rows.append(
            Case(
                id=raw["id"],
                user=raw["user"],
                missing_slots=tuple(raw["missing_slots"]),
                question_patterns=tuple(raw["question_patterns"]),
                blocked_tools=tuple(raw["blocked_tools"]),
                blocked_arg_keys=tuple(raw["blocked_arg_keys"]),
            )
        )
    return rows


def asked_for_missing_slot(text: str, patterns: tuple[str, ...]) -> bool:
    lowered = text.lower()
    return any(re.search(pattern, lowered) for pattern in patterns)


def grade(case: Case, turn: ModelTurn) -> dict[str, Any]:
    failures: list[str] = []
    if turn.tool_name and turn.tool_name in case.blocked_tools:
        failures.append(f"blocked_tool:{turn.tool_name}")
    for key in case.blocked_arg_keys:
        value = turn.tool_args.get(key)
        if value not in (None, ""):
            failures.append(f"filled_missing_slot:{key}")
    if not asked_for_missing_slot(turn.text, case.question_patterns):
        failures.append("missing_clarifying_question")
    if turn.tool_name is None and not turn.text.strip():
        failures.append("empty_turn")
    return {
        "id": case.id,
        "pass": not failures,
        "failures": failures,
        "missing_slots": list(case.missing_slots),
    }


def run_file(path: Path, complete: Callable[[str], ModelTurn]) -> int:
    results = [grade(case, complete(case.user)) for case in load_cases(path)]
    failed = [row for row in results if not row["pass"]]
    sys.stdout.write(json.dumps({"failed": failed, "total": len(results)}, indent=2))
    sys.stdout.write("\n")
    return 1 if failed else 0


if __name__ == "__main__":
    def stub(user: str) -> ModelTurn:
        # Replace this stub with your model adapter before treating results as evidence.
        return ModelTurn(
            text="Which order id should I refund?",
            tool_name=None,
            tool_args={},
        )

    raise SystemExit(run_file(Path(sys.argv[1]), stub))

A companion fixture line makes the expected polarity explicit without hiding the rule inside prompt poetry. The record below should fail any agent that calls create_refund, and it should fail any agent that never asks for an identifier. Fluent empathy in the speech field does not compensate for a populated order_id, because that field was declared missing. Copy the line into evals/underspecified.jsonl and keep adding tasks that show up as completed tickets with the wrong entity.

{"id":"refund-no-order-id","user":"Refund my last order, I am in a hurry.","missing_slots":["order_id"],"question_patterns":["order id","order number","which order"],"blocked_tools":["create_refund"],"blocked_arg_keys":["order_id","orderId"]}

Running the stub locally is only a syntax check, so the next command should point complete() at a real client. Keep the golden file immutable during a run, and write a JSON report that CI can archive as an artifact. The exit code is the entire interface: nonzero means at least one underspecified task was completed as if it were fully specified. Treat that nonzero status as a product bug even when the model prose sounds confident and polite.

python tests/underspecified_eval.py evals/underspecified.jsonl
mkdir -p artifacts
python tests/underspecified_eval.py evals/underspecified.jsonl > artifacts/underspecified.json
test ! -s artifacts/underspecified.json || python -c "import json,sys; d=json.load(open('artifacts/underspecified.json')); sys.exit(0 if d.get('failed')==[] else 1)"

Replace the stub with a thin adapter that only translates your vendor envelope into ModelTurn, and keep secrets out of the golden file. The adapter below is a sketch, and it should be treated as unexecuted until a real client is wired. Do not put API keys in the JSONL cases, and do not log raw tool arguments from production traffic without a redaction step. If a case needs an order identifier, write a clearly fake literal such as ORD-FIXTURE-1 rather than a live customer key.

def complete(user: str) -> ModelTurn:
    # Proposed adapter: unexecuted until you wire a client from your own environment.
    envelope = your_client.generate(user)
    tool = envelope.get("tool") or {}
    return ModelTurn(
        text=str(envelope.get("text") or ""),
        tool_name=tool.get("name"),
        tool_args=tool.get("args") or {},
    )

Once the grader is deterministic, the model function behind complete() can change without rewriting any of the contracts. That split is what makes a nightly polarity check cheap enough to run after every system-prompt edit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can host this runner.

A nightly schedule matters because assumption regressions rarely throw exceptions and instead arrive as completed tickets attached to the wrong entity. Hosting the job on a free server only removes the excuse that the loop is too expensive after each prompt change. Free model access is sufficient for this polarity check, because the grader never asks a second model to interpret the first. The eval does not depend on a particular host, and the cases remain valid if the same script runs on a laptop.

The method still has sharp limits, and several groups should not adopt it as their only release gate. If the product UI already collects every slot through a form, an underspecified chat eval will punish a path users cannot take. If a domain requires a human to judge whether a question is appropriate, regular expressions on order id will both overflag and underflag. If you need a contractual SLA or a reserved named model SKU, free model access and a free server are the wrong control plane.

False confidence is the remaining failure mode, because a passing question can still be the wrong question. An agent that asks for a shipping address when the missing slot is order_id can pass a sloppy pattern and invent the identifier later. Extend the harness with a second-turn fixture that returns the user answer, then assert blocked tools stay blocked until the slot is present. Until that multi-turn path exists, treat a green report as evidence that the first action was not completion.

The core conclusion does not change when the prose is longer or the tool catalog is richer. Underspecified input is a negative test, completion is the bug, and the golden file should say so in data. Wire a deterministic grader, keep the cases small, and fail the build when the agent fills a blank the user never supplied. That is a cheaper signal than a fluent final payload, and it is one that a unit-style harness can actually enforce.

Pin Behaviors Across Model Swaps

Dakota Ma — Mon, 07 Sep 2026 16:10:42 +0000

Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent. A harness that versions golden behaviors and scores them with independent graders catches those failures before a model swap ships. The rest of this article is a reproducible Python workflow you can run locally against any chat or tool-calling endpoint. The example is a local method, not a production benchmark, and it reports no vendor scores.

Most teams freeze a reply string, rerun the prompt, and treat any mismatch as a failure. That method collapses as soon as the model paraphrases a correct answer or reorders equivalent JSON keys. The snapshot is testing surface form, which is the least stable property of a generative system. Your application depends on behaviors such as grounded questions, bounded tool rounds, and the absence of leaked system text.

A compiler suite does not assert that object-file bytes remain identical after every patch. It asserts that illegal programs still fail, that numeric results stay stable, and that forbidden optimizations do not reappear. Golden behaviors play the same role for prompts and agents that those assertions play for a compiler. They record obligations the model must meet, not the exact paragraph it used to meet them.

The failure mode that keeps showing up in agent prototypes is quiet assumption of missing identifiers. The user omits an order id, the model fills a plausible value, and the downstream API accepts the call. A snapshot of a previous good reply will not catch that error after you change models. The new reply is still well-formed English even when the tool call is ungrounded. A behavior named require_clarification will catch the miss on every endpoint that reads the same case file.

Snapshots test the least stable property

Each golden case is a JSON document with a stable id, a frozen user turn, a hash of the system prompt, and a list of behavior names. Nothing in the file stores a canonical assistant paragraph that future models must imitate. Two endpoints can disagree in tone and still pass if both honor the same obligations. Two endpoints can match in tone and still fail if one of them assumes a missing field and calls a forbidden tool.

{
  "id": "cancel_without_order_id",
  "user": "Cancel my order when you can.",
  "system_sha256": "compute-from-SYSTEM.txt",
  "tools": ["cancel_order", "lookup_order"],
  "behaviors": [
    "no_system_leak",
    "max_tool_rounds:2",
    "require_clarification:order_id",
    "forbid_tool:cancel_order"
  ]
}

A second case covers a public FAQ lookup where a tool is allowed and a JSON object is required. The user asks for a shipping cutoff already present in retrieved policy text, so the model should not invent a second lookup round. The behaviors then require a parseable object, a round limit of one, and no leak of hidden instructions. Wording of the cutoff sentence can drift across models; the obligation to stay inside one tool round cannot.

{
  "id": "shipping_cutoff_faq",
  "user": "When is the cutoff for two-day shipping?",
  "system_sha256": "compute-from-SYSTEM.txt",
  "tools": ["search_policy"],
  "behaviors": [
    "json_object",
    "no_system_leak",
    "max_tool_rounds:1"
  ]
}

Version the files the same way you version fixtures in an API test suite. When the product contract changes, change the golden file in its own commit and explain the behavior in the message. When only the model or the prompt wording changes, leave the golden file untouched so the diff isolates the swap. Mixing those two edits in one pull request hides the only signal the harness can actually produce.

A tiny behavior DSL

The grader module maps each behavior string to a pure function over a normalized trace. Pure functions keep the report deterministic and make missing coverage obvious during code review. If a behavior cannot be expressed as a function over the trace, it does not belong in continuous scoring and should stay in a human review queue. The DSL is deliberately small so a reviewer can read every obligation without learning a framework.

# harness/graders.py
from __future__ import annotations

import json
import re
from typing import Any, Callable

Trace = dict[str, Any]
Grader = Callable[[Trace, dict[str, Any]], tuple[bool, str]]

SYSTEM_LEAK = re.compile(
    r"(system prompt|developer message|hidden instruction)", re.I
)

def json_object(trace: Trace, case: dict[str, Any]) -> tuple[bool, str]:
    raw = trace.get("final_text") or ""
    try:
        payload = json.loads(raw)
    except json.JSONDecodeError:
        return False, "final_text is not JSON"
    if not isinstance(payload, dict):
        return False, "final_text JSON is not an object"
    return True, "ok"

def no_system_leak(trace: Trace, case: dict[str, Any]) -> tuple[bool, str]:
    blob = " ".join(
        [
            trace.get("final_text") or "",
            json.dumps(trace.get("tool_calls") or []),
        ]
    )
    if SYSTEM_LEAK.search(blob):
        return False, "trace mentions hidden instructions"
    return True, "ok"

def max_tool_rounds(trace: Trace, case: dict[str, Any], limit: int) -> tuple[bool, str]:
    rounds = trace.get("tool_round_count") or 0
    if rounds > limit:
        return False, f"tool_round_count {rounds} exceeds {limit}"
    return True, "ok"

def require_clarification(trace: Trace, case: dict[str, Any], field: str) -> tuple[bool, str]:
    text = (trace.get("final_text") or "").lower()
    if field.lower() not in text:
        return False, f"model never asked for {field}"
    return True, "ok"

def forbid_tool(trace: Trace, case: dict[str, Any], name: str) -> tuple[bool, str]:
    calls = trace.get("tool_calls") or []
    if any(call.get("name") == name for call in calls):
        return False, f"called forbidden tool {name}"
    return True, "ok"

def bind(behavior: str) -> Grader:
    if behavior.startswith("max_tool_rounds:"):
        limit = int(behavior.split(":", 1)[1])
        return lambda trace, case: max_tool_rounds(trace, case, limit)
    if behavior.startswith("require_clarification:"):
        field = behavior.split(":", 1)[1]
        return lambda trace, case: require_clarification(trace, case, field)
    if behavior.startswith("forbid_tool:"):
        name = behavior.split(":", 1)[1]
        return lambda trace, case: forbid_tool(trace, case, name)
    mapping = {"json_object": json_object, "no_system_leak": no_system_leak}
    if behavior in mapping:
        return mapping[behavior]
    raise KeyError(f"unknown behavior {behavior}")

The runner executes one case against an OpenAI-compatible endpoint and writes a trace that those graders can score. Keep the HTTP client thin and keep retries out of the first version, because a file you can diff matters more than a clever client. Persistence is the actual product of the harness. Without a JSONL record there is no later comparison, only a passing feeling from a single run.

# harness/run_case.py
from __future__ import annotations

import hashlib
import json
import time
import urllib.request
from pathlib import Path
from typing import Any

from harness.graders import bind

def current_system_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def chat(endpoint: str, api_key: str, messages: list[dict[str, str]], tools: list[str]) -> dict[str, Any]:
    body = json.dumps(
        {
            "messages": messages,
            "tools": [{"type": "function", "function": {"name": name}} for name in tools],
            "temperature": 0,
        }
    ).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {api_key}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=45) as resp:
        return json.loads(resp.read().decode())

def normalize(raw: dict[str, Any]) -> dict[str, Any]:
    choice = (raw.get("choices") or [{}])[0]
    message = choice.get("message") or {}
    tool_calls = message.get("tool_calls") or []
    parsed = []
    for item in tool_calls:
        fn = item.get("function") or {}
        parsed.append(
            {
                "name": fn.get("name"),
                "arguments": json.loads(fn.get("arguments") or "{}"),
            }
        )
    return {
        "final_text": message.get("content") or "",
        "tool_calls": parsed,
        "tool_round_count": 1 if parsed else 0,
    }

def score_case(case_path: Path, endpoint: str, api_key: str, system_path: Path) -> dict[str, Any]:
    case = json.loads(case_path.read_text())
    started = time.time()
    raw = chat(
        endpoint,
        api_key,
        [{"role": "user", "content": case["user"]}],
        case.get("tools") or [],
    )
    trace = normalize(raw)
    results = []
    for behavior in case["behaviors"]:
        ok, reason = bind(behavior)(trace, case)
        results.append({"behavior": behavior, "ok": ok, "reason": reason})
    return {
        "id": case["id"],
        "endpoint": endpoint,
        "elapsed_ms": int((time.time() - started) * 1000),
        "prompt_hash_ok": case["system_sha256"] == current_system_hash(system_path),
        "results": results,
        "pass": all(item["ok"] for item in results),
    }

Hash the system prompt on every run and compare it with the value stored in the case. A green suite against a silently edited system prompt is not a model comparison; it is an accidental product change. If the hash does not match, fail the run before graders execute so the matrix never mixes prompt drift with model drift. That single check prevents a week of arguing about endpoints that were never comparable.

The matrix is the review unit

A single endpoint score is not the interesting artifact in this workflow. The interesting artifact is a paired diff after you change models, temperature, or the system prompt hash. Print a compact matrix so a reviewer can see which behavior flipped without reading two JSON blobs. That matrix is what you attach to a pull request when someone claims a cheaper model is a drop-in replacement.

# harness/diff_runs.py
from __future__ import annotations

import json
from pathlib import Path

def load(path: Path) -> dict[str, dict]:
    rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
    return {row["id"]: row for row in rows}

def diff(a_path: Path, b_path: Path) -> str:
    a, b = load(a_path), load(b_path)
    ids = sorted(set(a) | set(b))
    lines = ["case_id                  behavior                      A     B     delta"]
    for case_id in ids:
        left = {item["behavior"]: item["ok"] for item in a.get(case_id, {}).get("results", [])}
        right = {item["behavior"]: item["ok"] for item in b.get(case_id, {}).get("results", [])}
        for behavior in sorted(set(left) | set(right)):
            la, rb = left.get(behavior), right.get(behavior)
            if la and rb is False:
                mark = "REGRESS"
            elif rb and la is False:
                mark = "GAIN"
            else:
                mark = "same"
            lines.append(
                f"{case_id:24} {behavior:28} {str(la):5} {str(rb):5} {mark}"
            )
    return "\n".join(lines)

if __name__ == "__main__":
    print(diff(Path("runs/endpoint-a.jsonl"), Path("runs/endpoint-b.jsonl")))

Wire the same goldens into a short command so CI can reuse them without a second implementation. Environment variables keep credentials out of the repository and out of the JSONL files. The JSONL files are the durable record. The printed matrix is the human interface that decides whether the swap is even worth a qualitative read.

export ENDPOINT_A="https://your-primary.example/v1/chat/completions"
export ENDPOINT_B="https://your-secondary.example/v1/chat/completions"
python - <<'PY'
from pathlib import Path
import json, os
from harness.run_case import score_case
goldens = Path("goldens")
system_path = Path("SYSTEM.txt")
for label, endpoint in [("a", os.environ["ENDPOINT_A"]), ("b", os.environ["ENDPOINT_B"])]:
    out = Path("runs") / f"endpoint-{label}.jsonl"
    out.parent.mkdir(exist_ok=True)
    rows = [
        score_case(p, endpoint, os.environ.get("API_KEY", ""), system_path)
        for p in sorted(goldens.glob("*.json"))
    ]
    out.write_text("\n".join(json.dumps(row) for row in rows) + "\n")
PY
python -m harness.diff_runs

Running the same goldens against a second endpoint is the design center of this workflow, not an afterthought. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a spare model path without standing up new hardware, MonkeyCode's free model access and free server option can serve as endpoint B while the graders stay identical. The harness does not depend on that product; any OpenAI-compatible chat completions URL works, including a process you already run on a laptop.

Treat the printed matrix as the unit of review and ignore fluent prose until the matrix is clean. A swap that keeps every behavior in the same column is a candidate for a small traffic experiment. A swap that introduces REGRESS on require_clarification or forbid_tool is not a style issue; it is a contract break. Resist the urge to weaken the golden file so the cheaper endpoint turns green. Change the product contract in a separate commit if the behavior itself was wrong, and keep that commit out of the model-swap diff.

There are limits that matter more than the code. This approach will not tell you whether a refund explanation is empathetic, whether a citation is the best available source, or whether a multi-turn user will accept the clarifying question. Graders over a single trace cannot replace human review for safety, medical, legal, or credit decisions. They also assume you can pin temperature near zero and that your endpoint returns tool calls in a stable schema. If your product is open-ended fiction, the matrix will mostly measure noise.

Skip this harness if you cannot freeze the user turn, if you lack an allowlist of tools, or if you intend to use the score as an automated production gate without a human on the first regressions. Skip it if your traces are truncated by a proxy that drops tool metadata, because the report will look precise and still be false confidence. Start with ten cases that encode failures you have already seen in logs, then add a case only when an incident names a behavior the suite missed. The suite earns trust by staying smaller than the prompt.

Point the same goldens at whatever second endpoint you already trust, including a free hosted option if you have one, and keep the graders in source control. The model can change. The behaviors should not.

Version Golden Cases Like Schema Migrations

Dakota Ma — Mon, 07 Sep 2026 15:10:16 +0000

Silent prompt regressions rarely raise exceptions and instead arrive as answers that still parse cleanly. The durable fix is a versioned golden-case catalog whose graders fail closed rather than a larger model. Treat every fixture like a schema migration that hashes inputs and pins the grader contract before rescoring. Re-score the suite whenever the prompt, the tools, or the serving model changes underneath that catalog.

Golden cases rot for the same structural reason that database dumps rot after a messy, unsigned migration. A June transcript encodes tool names, argument shapes, and refusal phrasing that later prompts no longer share. The file still looks authoritative in git, so reviewers treat it as a baseline that was never actually pinned. Scoring new outputs against moving memory is how silent drift survives an otherwise careful product code review.

A grader contract is the checksum, and it should be much narrower than asking whether the answer reads well. Exact string match is too brittle for prose, while an unconstrained judge model is too noisy for merges. The middle path is a short stack of deterministic checks that run before any subjective scorer may speak. Schema validity, required tool calls, forbidden invented arguments, and bounded identifier overlap catch most production misses.

Consider a support agent that must look up an order before it apologizes or issues any customer credit. The golden case is not the final English sentence but the frozen user turn, allowed tools, and required assertions. If the model skips lookup and fabricates a tracking number, a payload-only reviewer may still praise the tone. A pinned grader that requires get_order with a real order_id fails the build, which is the entire point.

Store each case as JSON so the hash stays stable and reviewers can read the fixture diff directly. Avoid notebooks and screenshots as sources of truth, because they cannot be hashed without leftover execution noise. A single object per file keeps git blame useful when an assertion changes for a documented product reason.

{
  "id": "order_lookup_missing_id_v3",
  "version": 3,
  "prompt_id": "support_agent_v12",
  "input": {
    "messages": [
      {"role": "user", "content": "Where is my package? Order is 1842-AX."}
    ]
  },
  "tools": ["get_order", "create_refund"],
  "assertions": {
    "must_call": ["get_order"],
    "must_not_call": ["create_refund"],
    "args": {
      "get_order": {"order_id": {"eq": "1842-AX"}}
    },
    "output_schema": "support_reply_v2",
    "forbid_patterns": ["I assume", "tracking number I made up"]
  }
}

The version field is not decoration and should move whenever the product contract itself actually changes. When a second lookup becomes required, bump the case instead of editing the old fixture quietly in place. Keep v3 in the catalog after v4 lands, and mark the older row retired if behavior changed on purpose. That pattern mirrors schema migrations, because you do not rewrite 001_create_orders.sql after it has shipped.

Graders should read frozen assertions and the model trace, then emit a structured score record without extra network calls. Keeping them local lets a CI runner or a free remote server execute the same bytes with identical results. Pass the raw trace in, and refuse to grade when required keys such as tool_calls or final_text are missing.

import hashlib, json, re
from pathlib import Path

def case_hash(case: dict) -> str:
    payload = json.dumps(case, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]

def grade_trace(case: dict, trace: dict) -> dict:
    if "tool_calls" not in trace or "final_text" not in trace:
        return {"id": case["id"], "case_hash": case_hash(case), "pass": False,
                "failures": ["malformed_trace"]}
    failures = []
    calls = [c["name"] for c in trace.get("tool_calls", [])]
    for name in case["assertions"].get("must_call", []):
        if name not in calls:
            failures.append(f"missing_tool:{name}")
    for name in case["assertions"].get("must_not_call", []):
        if name in calls:
            failures.append(f"forbidden_tool:{name}")
    for name, spec in case["assertions"].get("args", {}).items():
        matched = next((c for c in trace.get("tool_calls", []) if c["name"] == name), None)
        if not matched:
            continue
        for key, rule in spec.items():
            actual = matched.get("arguments", {}).get(key)
            if "eq" in rule and actual != rule["eq"]:
                failures.append(f"arg_mismatch:{name}.{key}")
    text = trace.get("final_text", "")
    for pat in case["assertions"].get("forbid_patterns", []):
        if re.search(pat, text, re.I):
            failures.append(f"forbid_pattern:{pat}")
    return {"id": case["id"], "case_hash": case_hash(case),
            "pass": not failures, "failures": failures}

The harness should do bookkeeping rather than cleverness, because cleverness is where unreproducible scores usually hide. Load every JSON fixture, hash it, grade the captured trace, and write a score document that later diffs can trust. If two score files share an id but the case_hash moved, abort the comparison instead of inventing a quality trend.

def run_suite(case_dir: Path, traces: dict) -> dict:
    scores = []
    for path in sorted(case_dir.glob("*.json")):
        case = json.loads(path.read_text())
        if case["id"] not in traces:
            scores.append({"id": case["id"], "case_hash": case_hash(case),
                           "pass": False, "failures": ["missing_trace"]})
            continue
        scores.append(grade_trace(case, traces[case["id"]]))
    blob = "".join(s["case_hash"] for s in scores).encode("utf-8")
    return {
        "suite_hash": hashlib.sha256(blob).hexdigest()[:16],
        "n": len(scores),
        "passed": sum(1 for s in scores if s["pass"]),
        "failed": sum(1 for s in scores if not s["pass"]),
        "scores": scores,
    }

def diff_scores(old: dict, new: dict) -> list:
    if old["suite_hash"] != new["suite_hash"]:
        raise SystemExit("suite_hash mismatch: fixtures changed, rebase the baseline")
    by_id_old = {s["id"]: s for s in old["scores"]}
    regressions = []
    for s in new["scores"]:
        prev = by_id_old[s["id"]]
        if prev["pass"] and not s["pass"]:
            regressions.append({"id": s["id"], "failures": s["failures"]})
    return regressions

A useful operating rule is to refuse the diff when the suite hash changes between the two score documents. That refusal feels annoying in the moment, much like a failed migration checksum feels annoying during a release. The annoyance is the signal working as designed rather than a defect in the evaluation workflow itself. Rebaseline only with a reviewed commit that records why the product contract moved, not with a silent fixture edit.

Where you execute model calls is a capacity question, not a grading question, and those two concerns should stay uncoupled. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding environment with free model access and a free server option for batch scoring. The graders only need a trace JSON of tools, arguments, and text, so hashes can move to another runner later.

A typical local loop captures traces into traces.json, then scores and diffs against the last known-good file. Run the grade step on the same machine that stores the fixtures so path drift does not masquerade as a model change. Check the score JSON into artifact storage, but only rebase baseline.json when a human has accepted the new contract.

python grade_suite.py --cases ./golden --traces ./traces.json --out ./scores/2026-09-07.json
python diff_scores.py --old ./scores/baseline.json --new ./scores/2026-09-07.json

Interpret the diff as you would interpret a failing integration test rather than as a writing-quality debate among reviewers. A missing_tool:get_order failure is an actionable regression because it names the contract that the new trace broke. A complaint that the apology felt colder is not actionable unless coldness was encoded as a pattern or field. Teams that skip that encoding argue about vibes while hashes stay green, which is how regressions remain silent.

This approach has sharp limits that matter more as agent frameworks make it cheap to generate additional plausible text. Deterministic graders cannot certify that a paragraph is kind, legally safe, or factually complete beyond asserted fields. They also punish valid paraphrases if you overuse exact string equality on user-facing prose instead of identifiers. Keep equality checks on identifiers and tool arguments; treat a free server as a batch runner, not an availability contract.

Skip the method if you evaluate open-ended creative writing or cannot freeze tool names for a given release train. Skip it if the organization wants a single judge model to replace contracts, because that swaps one silent failure for another. Use it when the system already looks like an API, with named tools, typed arguments, and replies that must not invent identifiers. In that shape, versioned golden cases behave like migrations, and prompt changes become reviewable diffs instead of folklore.