DEV Community: Dakota Ma

Find Fragile Cases With a Cross-Model Pass Matrix

Dakota Ma — Tue, 27 Oct 2026 13:40:30 +0000

A prompt that passes your eval on one model has told you almost nothing about how it behaves on the next one. The cheapest robustness signal you can collect is a pass matrix: the same golden cases run against several models, one pass vector per case, disagreement read before totals. The vector, not the percentage, is what shows you which cases your prompt only half handles.

Model churn is the constant in this work. Every few weeks another endpoint shows up with a different tokenizer, a different refusal posture, and a different opinion about what "concise" means. Teams pin one model, tune a prompt against it for a week, then switch providers when price or latency moves. That switch is a config change, it ships quietly, and nothing in the eval output looks unusual, because the eval only ever knew one model.

One grader, many adapters

The runner below keeps a single grader and pushes every provider difference into an adapter function. Cases live in a JSONL file so they can be diffed in review, and each case carries only checks a deterministic grader can evaluate. Inputs are truncated here for length; in your repository they should be the real payloads.

{"case_id":"refund-window","input":"How long do I have to return an opened item?","must_include":["14 days"],"must_not_include":["lifetime"]}
{"case_id":"invoice-json","input":"Summarize invoice 88 as JSON with total and currency.","json_fields":["total","currency"]}
{"case_id":"empty-cart","input":"My cart is empty, can I check out?","must_include":["cart is empty"]}

# sweep.py - standard library only
import hashlib, inspect, json, re, time
from datetime import datetime, timezone
from itertools import product

def norm(text):
    return re.sub(r"\s+", " ", text or "").strip().lower()

def grade(case, output):
    checks = []
    for needle in case.get("must_include", []):
        checks.append(("include:" + needle, norm(needle) in norm(output)))
    for needle in case.get("must_not_include", []):
        checks.append(("exclude:" + needle, norm(needle) not in norm(output)))
    if "json_fields" in case:
        try:
            payload = json.loads(output)
            ok = all(field in payload for field in case["json_fields"])
        except json.JSONDecodeError:
            ok = False
        checks.append(("json_fields", ok))
    return checks

GRADER_FP = hashlib.sha256(inspect.getsource(grade).encode()).hexdigest()[:12]

def fingerprint(obj):
    return hashlib.sha256(json.dumps(obj, sort_keys=True).encode()).hexdigest()[:12]

The adapter is the only provider-specific code in the whole harness, so adding a model means adding a config entry and one SDK call. The stub below is a contract, not a working client; it is deliberately unexecuted here, because credentials and SDK names are yours to choose.

def complete(model_id, prompt, params=None):
    """Implement per provider. Not executed in this article."""
    raise NotImplementedError

The sweep loop and its ledger

Each cell in the matrix stores its trial results, the last output, the case fingerprint, and the grader fingerprint. That combination is what makes a disagreement diagnosable a week later, when you no longer remember whether the prompt or the grader moved.

def sweep(models, cases, trials=3, pause=0.0):
    ledger = {}
    for model, case in product(models, cases):
        results = []
        for _ in range(trials):
            output = complete(model["id"], case["input"], model.get("params", {}))
            results.append(all(ok for _, ok in grade(case, output)))
            if pause:
                time.sleep(pause)
        ledger[(model["id"], case["case_id"])] = {
            "passed": all(results),
            "trials": results,
            "output": output,
            "prompt_fp": fingerprint(case),
            "grader_fp": GRADER_FP,
            "model_params": model.get("params", {}),
        }
    return ledger

def matrix(ledger, models, cases, path="matrix.csv"):
    lines = ["case_id," + ",".join(m["id"] for m in models) + ",class"]
    for case in cases:
        vector = "".join(
            "1" if ledger[(m["id"], case["case_id"])]["passed"] else "0"
            for m in models
        )
        label = "stable" if set(vector) == {"1"} else "blocked" if set(vector) == {"0"} else "fragile"
        lines.append(f"{case['case_id']},{','.join(vector)},{label}")
    open(path, "w").write("\n".join(lines) + "\n")
    return "\n".join(lines)

Requiring all trials to pass before a cell counts as a pass is a deliberate, blunt choice. It inflates the false-failure rate slightly and shrinks the false-pass rate a lot, which is the direction you want when the matrix is deciding where to spend human attention.

Reading a pass vector

A column of ones across every model means the case is stable enough to serve as a smoke test, and it earns a place in your fast suite. A column of zeros across every model is almost never a model problem; check the case text, then the grader, then the prompt, in that order. A mixed column is the interesting one, because it names a specific input where your instruction is ambiguous rather than hard.

Vector	Reading	First action
`1111`	stable	keep as smoke case
`1101`	fragile	inspect outputs, usually grader or ambiguity
`0000`	blocked	fix case, grader, then prompt

Triage order matters more than tooling here. Suppose refund-window produces 1101 and you discover the failing model answers "fourteen days" while the other three answer "14 days". The grader is doing substring matching on a normalized string, so the digit and the word are different tokens to it, and the disagreement was manufactured by your check rather than discovered in the model. The fix is a canonical form or a small regex alternation, applied once, and then a full re-run with a bumped grader fingerprint so the history stays honest.

If the outputs differ substantively instead, with one model adding a cheerful preamble and skipping the number, the ambiguity is in the prompt. Rewrite the instruction to require the number in the first sentence, re-run only the fragile column at first, and promote the case back into the full sweep once its vector is all ones. Blocked cases deserve the same treatment before they quietly rot in the suite.

What the sweep actually costs

The arithmetic is simple and worth doing before you start: models times cases times trials. Twenty-four cases, four models, and three trials is 288 calls, which is nothing on a metered API and still worth logging. A wider sweep is where free model access stops being a curiosity, because you can afford to keep several candidate models in the matrix instead of testing the one you already pay for.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Where that matters for this workflow is the boring operational half. MonkeyCode's free model access, as described by the operator, lets you keep a handful of adapters live in the matrix without a billing account, and the free server option means the sweep runner does not need its own provisioned machine. Confirm the current terms, model list, and any limits on the project's own pages before you build a routine around them, since availability terms change and this article cannot track them for you.

Limitations and who should skip this

Substring and JSON-key grading measures surface compliance, not correctness, so a green matrix is evidence about format and constraint adherence and nothing more. Three or four models are a sampled population rather than a ranking, and publishing cross-model leaderboards from twenty cases is statistically empty even when the table looks tidy. Free endpoints can throttle or change without notice, so cache outputs to disk keyed by model, case, and trial, and add backoff rather than retrying tightly.

Skip this if your product pins exactly one model with no fallback path, because the matrix will only confirm what you already assume. Skip it too if your grader is still unstable across repeated runs on one model, since you would then be measuring grader noise with extra steps. Fix the single-model baseline first, then widen.

Start with four models you would actually ship and fifteen cases you would actually argue about. The first fragile vector usually pays for the whole exercise.

If you want to run the sweep without a billing setup, put the runner on MonkeyCode's free server option and check the current model list before you wire up the adapters.

Give Your Eval Harness Cases That Must Fail

Dakota Ma — Mon, 26 Oct 2026 12:15:57 +0000

An eval suite whose score never moves is not evidence of stability; more often it is evidence that the graders stopped looking. Pass rates drift upward on their own as prompts get longer, schemas get looser, and substring checks keep matching the wrong text. The cheapest way to detect that decay is to add inputs your harness is required to reject: negative controls, borrowed from lab practice, where an assay that cannot register a known positive is declared broken before any result is trusted.

This article builds that audit for a prompt-evaluation harness. You will mutate a set of human-approved reference answers, run your existing checks against the mutants, and publish a per-check kill rate that fails CI when sensitivity drops. Everything below is runnable Python and a shell command, with no dependency on a specific model vendor.

The failure mode is a grader with no obligation to fail

Consider the check most teams write first, something like "refund" in output.lower(). It passes on a correct policy answer, and it also passes on an answer that cites the wrong clause, drops the citation, or states a 30-day window when the policy says 14. The check is not wrong so much as unpriced: nobody measured what it can distinguish, so its green tick is treated as information when it is closer to decoration. Silent regressions live exactly in that gap, because the prompt change, the truncation, and the schema relaxation all leave a cluster of harmless tokens intact.

The same blindness appears one layer up when the grader is a model. A grader model with a vague rubric, or one that has quietly been pointed at a different family than the generator, will hand out pass with high confidence and pleasing consistency. Averages hide this well, since a uniformly generous grader produces a flat, tidy scoreline that reviews nicely.

Negative controls turn the question around

Instead of asking how many cases pass, ask how many deliberately broken outputs your checks refuse. Each mutant is a small, mechanical corruption of a known-good answer that preserves plausibility while destroying a property you care about. A check that cannot kill any mutant is not a weak check; it is a check that should be deleted or replaced, because it is consuming maintenance attention while contributing no discrimination.

The implementation is short. Keep reference answers as data, keep checks as pure functions of the output string, and keep mutators in a registry so the audit can grow without touching the runner.

# harness/mutants.py
import re
from dataclasses import dataclass
from typing import Callable, Iterable

@dataclass(frozen=True)
class Case:
    id: str
    output: str                                   # human-approved reference answer
    checks: tuple[Callable[[str], bool], ...]     # True means accepted

Mutator = Callable[[str], str]

def drop_last_sentence(text: str) -> str:
    parts = [p for p in re.split(r"(?<=[.!?])\s+", text.strip()) if p]
    return " ".join(parts[:-1]) if len(parts) > 1 else text

def strip_citations(text: str) -> str:
    return re.sub(r"\[\d+\]", "", text)

def corrupt_first_number(text: str) -> str:
    return re.sub(r"\d+", lambda m: str(int(m.group()) + 1), text, count=1)

def truncate_at_half(text: str) -> str:
    return text[: max(1, len(text) // 2)]

def identity(text: str) -> str:                   # control: must survive every check
    return text

MUTATORS: dict[str, Mutator] = {
    "drop_last_sentence": drop_last_sentence,
    "strip_citations": strip_citations,
    "corrupt_first_number": corrupt_first_number,
    "truncate_at_half": truncate_at_half,
    "identity_control": identity,
}

The audit walks the cross product of cases and mutators, records which checks rejected each mutant, and reports a kill rate per mutator. Mutators that happen to be no-ops on a given case are skipped, because a mutant identical to the original measures nothing.

# harness/audit.py
from collections import defaultdict
from harness.mutants import Case, MUTATORS

def audit(cases: list[Case]) -> dict[str, dict]:
    stats = defaultdict(lambda: {"killed": 0, "total": 0, "survivors": []})
    for case in cases:
        for name, mutate in MUTATORS.items():
            mutant = mutate(case.output)
            if mutant == case.output and name != "identity_control":
                continue
            failed = [check for check in case.checks if not check(mutant)]
            bucket = stats[name]
            bucket["total"] += 1
            if failed:
                bucket["killed"] += 1
            else:
                bucket["survivors"].append(case.id)
    return {
        name: {"kill_rate": v["killed"] / v["total"], "survivors": v["survivors"]}
        for name, v in stats.items()
        if v["total"]
    }

Reading the report, and gating CI on it

The identity control inverts the expectation: its kill rate must be zero, and any nonzero value means your checks reject good answers and the suite is noisy rather than sensitive. The other mutators get floors, set per property rather than globally, since truncation should be caught essentially always while a citation check may legitimately miss a few cases that carry no numeric markers. The trace below is an illustrative shape rather than a benchmark; replace every number with your own run before publishing anything about it.

$ python -m harness.audit --cases cases/refunds.jsonl
strip_citations       0.94   survivors: c-118, c-204
truncate_at_half      1.00
corrupt_first_number  0.61   survivors: c-007, c-031, c-052, c-077
drop_last_sentence    0.88   survivors: c-016, c-093
identity_control      0.00   false alarms: none

The survivors list is the actual deliverable, because each surviving pair names a case and a property your checks cannot see. c-007 surviving corrupt_first_number usually means the numeric claim is graded by an LLM rubric with no extraction step, and the fix is a small deterministic assertion that pulls the figure out and compares it to the source field. Adding that assertion is a one-line change; finding it without the audit typically costs a week of confused incident review.

The gate should be boring and fail loudly. Floors live in a config file, the exit code is nonzero when a floor is breached, and the report is attached to the build so the diff between runs is visible.

FLOOR = {"strip_citations": 0.90, "corrupt_first_number": 0.80, "truncate_at_half": 0.95}

def gate(report: dict[str, dict]) -> None:
    breached = {
        name: round(row["kill_rate"], 2)
        for name, row in report.items()
        if name in FLOOR and row["kill_rate"] < FLOOR[name]
    }
    if breached:
        raise SystemExit(f"harness lost sensitivity: {breached}")

Where a second model family and a spare box fit

If your checks include a model grader, the audit acquires a second dimension, because a grader from the same family as the generator tends to share its blind spots and will accept mutants that an unrelated model rejects. Running the same 40 mutants through a grader from a different family is a cheap monoculture test, and the disagreement set is a short list of cases worth human review. Stamp the grader identifier and a hash of the rubric next to the model identifier in the report, so a sensitivity drop can be attributed to a prompt edit rather than guessed at later.

For the nightly sweep, the constraint is usually not compute but containment: the audit reads internal reference answers, so it should run somewhere you control and can wipe. MonkeyCode provides free model access and a free server option, which covers both the secondary grader slot and a scheduled run that does not compete with anyone's laptop; the project page documents current limits, and you should read them before designing around the free tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have a CI runner with egress rules you trust, use that instead and skip the extra moving part.

Limits, and who should not adopt this

Mutation testing only covers failure modes you can enumerate, so a semantically wrong answer that preserves citations, numbers, and sentence count will survive every mutant in the registry above. That is why the audit complements human spot checks rather than replacing them, and why the mutator list should grow every time a real incident reveals a property nobody was testing. Teams without stable, human-approved reference answers should not start here, since the mutants are derived from those answers and a shaky baseline produces noise instead of signal.

If your product decisions do not depend on a score, do not build any of this. The whole apparatus exists to protect a number that someone will cite in a review, and if no one cites your eval score, the maintenance cost of a sensitivity gate buys nothing. For teams that do cite it, the kill rate is the number that tells you whether the other number means anything.

Replay Cassettes Separate Harness Drift from Model Drift

Dakota Ma — Tue, 20 Oct 2026 10:18:45 +0000

A prompt suite has three moving parts — prompt text, the grader, and the model snapshot — and a single pass-rate number moves whenever any one of them changes. Replay cassettes remove that ambiguity by freezing model output on disk and letting you re-run only the parts you actually edited. The rule that makes it work is narrow: the network appears in exactly one code path, the recording path, and every other path either reads a hash-addressed file or fails loudly. Everything below is a harness you can build in an afternoon, plus the cases where it will lie to you.

The cassette is a recording keyed by input, not a mock

Most teams reach for mocks first: hand-written fake responses that look plausible and drift away from reality within a week. A cassette is different because it is produced by a real call and addressed by a fingerprint of the exact input that produced it. Change one word of the prompt and the fingerprint changes, so the old tape cannot silently answer a new question. That single property is what turns a cache into a regression detector.

# cassette.py
import hashlib, json, os, time

DIR = os.environ.get("CASSETTE_DIR", "cassettes")

class CassetteMiss(Exception):
    """Raised in replay mode when no tape matches the input fingerprint."""

def fingerprint(prompt, params):
    blob = json.dumps({"prompt": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]

def get(prompt, params, produce, mode="record"):
    key = fingerprint(prompt, params)
    path = os.path.join(DIR, key + ".json")
    if os.path.exists(path):
        with open(path) as fh:
            return json.load(fh), True          # hit
    if mode == "replay":
        raise CassetteMiss(key)                 # never fall through to network
    tape = produce(prompt, params)
    tape.update({"key": key, "params": params,
                 "recorded_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
    os.makedirs(DIR, exist_ok=True)
    with open(path, "w") as fh:
        json.dump(tape, fh, indent=2, sort_keys=True)
    return tape, False                          # miss, now recorded

The params argument deserves a moment, because it is where people quietly break determinism. Temperature, seed, max tokens, system prompt, and any tool-selection flag all belong in the fingerprint, even when the provider claims they are no-ops. If a knob can change the text, it belongs in the key, and if it cannot change the text, storing it costs you nothing but a few bytes.

Recording is the only place the network lives

The runner below reads a JSONL case file, asks the cassette layer for a tape, and hands the recorded text to a named grader. Note that graders are looked up by name from a registry rather than stored in JSON, because functions do not serialize and a grader you cannot diff is a grader you cannot audit. Note also that a replay miss exits with code 2, which is a different signal from a failing assertion and should page a different person.

# harness/run.py
import json, os, sys
from cassette import get, CassetteMiss
from client import call_model          # the only module that opens a socket

MODE = os.environ.get("CASSETTE_MODE", "replay")
GRADERS = {
    "json_shape": lambda text, exp: _parses(text) and set(exp["keys"]) <= set(_parses(text)),
    "contains":   lambda text, exp: exp["needle"].lower() in text.lower(),
    "refusal":    lambda text, exp: text.strip().lower().startswith(exp["prefix"]),
}

def run(suite_path):
    hits = misses = 0
    for line in open(suite_path):
        if not line.strip():
            continue
        case = json.loads(line)
        try:
            tape, hit = get(case["prompt"], case.get("params", {}),
                            produce=call_model, mode=MODE)
        except CassetteMiss as exc:
            print("CASSETTE_MISS", exc, file=sys.stderr)
            return 2
        hits += hit
        misses += not hit
        verdict = GRADERS[case["grader"]](tape["text"], case.get("expect", {}))
        print(json.dumps({"id": case["id"], "hit": hit, "verdict": verdict,
                          "finish": tape.get("finish_reason")}))
    total = hits + misses
    print(f"hit_rate={hits/total:.2f}", file=sys.stderr)
    return 0 if hits + misses else 1

if __name__ == "__main__":
    sys.exit(run(sys.argv[1]))

Run it twice to see the whole point. The first invocation with CASSETTE_MODE=record pays for real calls and writes tapes; the second with CASSETTE_MODE=replay finishes in milliseconds and provably makes zero network requests. If your CI cannot run the replay pass with networking disabled and still pass, you have not actually separated harness changes from model changes, and the pass rate remains a single undifferentiated number.

$ CASSETTE_MODE=record python -m harness.run cases/canary.jsonl   # needs a model
$ CASSETTE_MODE=replay python -m harness.run cases/golden.jsonl   # offline, exit 2 on miss
$ git diff --stat cassettes/ | tail -1                            # which tapes moved

Spend paid or free capacity on recording, never on replays

Recording is the expensive half of this design: every new case and every prompt edit produces a cache miss that needs a real completion, while replay needs nothing at all. That asymmetry maps cleanly onto MonkeyCode's free model access and free server option, which is where this workflow touches the product. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The practical split is to run the recording job on the free server instance against the free model tier, and keep the replay job on the machine that runs your test suite.

Keep the calibration honest, because free tiers are promotional by nature and change without notice. Check the project page for the current model list, quota, and server specification instead of hardcoding assumptions from a blog post, and treat the cassette directory as the durable asset rather than the provider. If a free model disappears, your tapes still replay, and only the refresh job needs new wiring.

Canaries belong at the boundary, not in the middle

Uniform sampling wastes most of its budget on cases that cannot flip: a case scoring 0.9 against a 0.5 threshold will survive almost any prompt edit. The cases that move are the ones sitting near the decision boundary, so select canaries by margin rather than by category. This is a five-line function and it usually cuts the recording cost of a canary suite by an order of magnitude.

def boundary_band(rows, band=0.10):
    """rows: [{"id", "score", "threshold"}]; keep cases that can actually flip."""
    return [r for r in rows if abs(r["score"] - r["threshold"]) <= band]

Run the band selection against a fresh sample, record those cases, and then freeze them. When a prompt edit lands, replay the frozen band first; if nothing flips there, the cheap regression check is done and the full suite can wait for nightly. If something does flip, you now have a cassette from before and after, which is a two-file diff rather than an argument.

A decision table for when replay is the wrong tool

Question you are asking	Cassette replay	Fresh sampling
Did my prompt or grader edit break a known case?	answers it	too noisy
Is the new model snapshot better overall?	cannot answer	correct tool
Is latency or cost per call acceptable?	invalidated by design	correct tool
Can this run offline in CI on every push?	yes	no

What replay cannot tell you, and who should skip it

Replay proves that your harness is stable against a fixed set of outputs, and it proves nothing about the current model, which may have changed the moment you stopped recording. Latency, token accounting, and refusal behaviour under load all disappear the instant a tape replaces a socket, so never cite a replay run as a performance measurement. The grader remains the weakest link, and a cassette suite will happily certify a grader that has quietly stopped agreeing with human labels.

Skip this approach if you do not yet have a stable case set, because cassette infrastructure amplifies a bad suite rather than fixing it. Skip it if your only real question is whether a newer model is better, since that question needs fresh samples and paired comparisons rather than frozen text. And skip it in any setting where you must prove provenance of each individual output, because a replayed tape is a cached artifact and not a fresh decision.

If you want to try the recording half without provisioning anything, MonkeyCode's free model access and free server option are enough to record a first band of a few hundred boundary cases; commit the cassettes/ directory alongside the harness, and let the tapes accumulate as the cheapest regression evidence your team owns.

Test a Model Swap with Paired Cases, Not Two Averages

Dakota Ma — Mon, 19 Oct 2026 09:09:15 +0000

A model swap looks safe right up until you compare the two runs case by case, because aggregate scores hide the flips that actually break your product. Two backends can both land at 71 percent on the same frozen set while nine cases move in each direction, and the nine that moved down are the only thing your users will notice. The fix is not a bigger benchmark; it is a paired test over a frozen golden set, with graders that are audited before they are trusted. This article builds that harness end to end, then shows how to read its output without fooling yourself.

Freeze the cases before you touch the model

A golden set is only useful if it is immutable during the experiment, so write the cases to disk once, hash the file, and record that hash in every report you produce. Each case needs three fields: a stable identifier, the exact prompt you will send, and a deterministic checker that returns a boolean. Keep the checkers boring on purpose, because JSON-schema validation, numeric comparison with a tolerance, and required-substring checks give you a verdict that does not drift when the judge model changes underneath you. Tag every case with a capability label such as extraction, refusal, or format, so a later drop can be localized instead of argued about.

Eighty cases is a reasonable floor for a first pass, and it is not arbitrary: paired tests only gain power from discordant pairs, so a set where both models pass everything tells you nothing no matter how large it is. Your job when curating is to include cases that sit near the decision boundary, the ones you suspect a cheaper or differently tuned model will fumble. If a case has never failed against any backend you have tried, it is a smoke test, not a regression test, and it belongs in a different file.

Audit the graders with canaries first

Before any comparison runs, feed your graders a handful of outputs you know are wrong and require them to reject every one. This is cheap insurance against the most humiliating failure mode in eval work, which is a grader that silently started passing everything after someone relaxed a regex. The canary file lives next to the golden set and is versioned with it, and a canary that survives a grader change is a stop-the-line event rather than a warning.

# graders.py — deterministic checkers plus a canary audit
import json, re

def check_json_keys(keys):
    def _check(output: str) -> bool:
        try:
            obj = json.loads(output)
        except json.JSONDecodeError:
            return False
        return all(k in obj for k in keys)
    return _check

def check_number(expected: float, tol: float = 1e-6):
    def _check(output: str) -> bool:
        m = re.search(r"-?\d+(?:\.\d+)?", output)
        return bool(m) and abs(float(m.group()) - expected) <= tol
    return _check

CANARIES = [
    ("empty_json",  check_json_keys(["id", "total"]), "{}"),
    ("prose_only",  check_json_keys(["id", "total"]), "Sure! Here is the answer."),
    ("wrong_total", check_number(41.50), "The total is 14.50"),
    ("near_miss",   check_number(41.50), "41.5 apples"),  # must PASS: tolerance case
]

def audit(canaries):
    must_fail = [c for c in canaries if c[0] != "near_miss"]
    survived = [name for name, check, out in must_fail if check(out)]
    if survived:
        raise SystemExit(f"grader audit failed, canaries passed: {survived}")
    for name, check, out in canaries:
        if name == "near_miss" and not check(out):
            raise SystemExit("grader too strict: tolerance case rejected")

That last canary matters as much as the first three, because a grader that rejects everything will happily manufacture a regression story for you. Auditing in both directions keeps the check honest, and it takes about ten minutes to write once per project.

Run both legs and record a manifest

The harness needs to send the same frozen prompts to two backends and persist raw outputs, because re-running a flaky endpoint in the middle of an analysis will quietly invalidate your paired design. Cache each response keyed by a hash of the case identifier, the prompt text, and the backend tag, and write a manifest containing the backend identifier string, the sampling parameters, and the golden-set hash. If you cannot name the exact backend version that served a request, you do not have a reproducible comparison; you have an anecdote with timestamps.

# swap_test.py — paired comparison with exact McNemar test
import json, hashlib, math, os, urllib.request
from collections import Counter

def sha(*parts: str) -> str:
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]

def call(endpoint: str, model_tag: str, prompt: str) -> str:
    body = json.dumps({"model": model_tag, "temperature": 0,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(endpoint + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ['API_KEY']}"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["choices"][0]["message"]["content"]

def exact_mcnemar(b: int, c: int) -> float:
    n = b + c
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(0, min(b, c) + 1)) / 2 ** n
    return min(1.0, 2 * tail)

def run(cases, backends, cache_path="cache.jsonl"):
    cache = {}
    if os.path.exists(cache_path):
        for line in open(cache_path):
            row = json.loads(line)
            cache[row["key"]] = row
    for case in cases:
        for name, (endpoint, tag) in backends.items():
            key = sha(name, case["id"], case["prompt"], tag)
            if key not in cache:
                out = call(endpoint, tag, case["prompt"])
                cache[key] = {"key": key, "case": case["id"], "backend": name,
                              "tag": tag, "output": out,
                              "passed": bool(case<a href="out">"check"</a>)}
                with open(cache_path, "a") as f:
                    f.write(json.dumps(cache[key]) + "\n")
    return cache

def report(cases, cache, old, new):
    pairs = []
    for case in cases:
        key = lambda be: sha(be, case["id"], case["prompt"], cache_backend_tag(cache, be))
        a = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == old)
        b = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == new)
        pairs.append((case["id"], case.get("tag", "untagged"), a["passed"], b["passed"]))
    b_only = sum(1 for _, _, p, q in pairs if q and not p)   # new passes, old fails
    c_only = sum(1 for _, _, p, q in pairs if p and not q)   # old passes, new fails
    delta = (b_only - c_only) / len(pairs)
    flips = [(cid, tag) for cid, tag, p, q in pairs if p and not q]
    return {"n": len(pairs), "old_pass": sum(p for _, _, p, _ in pairs),
            "new_pass": sum(q for _, _, _, q in pairs), "gain": b_only,
            "loss": c_only, "delta": round(delta, 4),
            "p_value": round(exact_mcnemar(b_only, c_only), 4),
            "flips_by_tag": Counter(t for _, t in flips),
            "flip_list": [cid for cid, _ in flips]}

The cache_backend_tag helper is a two-line lookup in the cache; keep it separate so the report function stays readable. Note that the delta is computed from discordant counts only, which is why an unchanged average across the two runs can coexist with a real regression.

Read the report as a decision, not a score

The output is a small dictionary, and every field maps to an action. An illustrative shape of it looks like {"n": 96, "gain": 7, "loss": 11, "delta": -0.0417, "p_value": 0.48, "flips_by_tag": {"format": 6, "extraction": 5}}, and the numbers below are a template rather than results from anyone's system.

Observation	Reading	Action
Canary audit fails	Grader is broken	Stop; fix graders before any comparison
delta >= 0 and CI-ish p_value high	No detectable regression	Keep the swap, keep the golden set
delta < 0, loss > gain, p_value < 0.05	Real regression on this set	Block the swap or narrow the prompt scope
delta < 0, p_value > 0.2	Underpowered, not necessarily fine	Add boundary cases before deciding
Flips concentrated in one tag	Localized breakage	Inspect flip_list by hand, that is 6-12 cases

The concentration column is where most of the practical value sits, since eleven losses spread evenly across five capabilities look like noise while eleven losses inside format usually point to a single missing instruction. Read the flip list before you read the p-value, and read the raw outputs before you read the flip list.

Where this harness stops working

The paired design assumes each case is a Bernoulli trial under a pinned backend, so a backend that routes consecutive requests to different model versions will inflate variance and quietly break the pairing. That risk is highest with shared free tiers, which is exactly why the manifest records the returned identifier string verbatim rather than trusting the name you asked for. If you sample at any temperature above zero, run k repetitions per case and decide the case-level verdict by majority, then re-run the canary audit against the majority verdicts as well.

This is also not a load test; it says nothing about tokens per second, queueing behavior, or cost at volume, and those deserve a separate measurement with a different method. Teams that need regulatory-grade evaluation evidence, or products whose outputs have no deterministic anchor at all, should route this work to human review instead of stretching a Boolean checker past what it can support. Nor is it a replacement for product-level tracing, because a golden set only knows the failures you already imagined.

If you want to run the reference leg without paying for two full backend bills, the free model access and free server option from MonkeyCode is enough to execute the harness above on a small set. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Start with twenty boundary cases, one canary file, and one pinned backend tag per leg, and let the flip list tell you whether the swap deserves the rest of your afternoon.

Test a Model Swap with Paired Cases, Not Two Averages

Dakota Ma — Mon, 19 Oct 2026 09:00:50 +0000

Freeze the cases before you touch the model

Audit the graders with canaries first

# graders.py — deterministic checkers plus a canary audit
import json, re

def check_json_keys(keys):
    def _check(output: str) -> bool:
        try:
            obj = json.loads(output)
        except json.JSONDecodeError:
            return False
        return all(k in obj for k in keys)
    return _check

def check_number(expected: float, tol: float = 1e-6):
    def _check(output: str) -> bool:
        m = re.search(r"-?\d+(?:\.\d+)?", output)
        return bool(m) and abs(float(m.group()) - expected) <= tol
    return _check

CANARIES = [
    ("empty_json",  check_json_keys(["id", "total"]), "{}"),
    ("prose_only",  check_json_keys(["id", "total"]), "Sure! Here is the answer."),
    ("wrong_total", check_number(41.50), "The total is 14.50"),
    ("near_miss",   check_number(41.50), "41.5 apples"),  # must PASS: tolerance case
]

def audit(canaries):
    must_fail = [c for c in canaries if c[0] != "near_miss"]
    survived = [name for name, check, out in must_fail if check(out)]
    if survived:
        raise SystemExit(f"grader audit failed, canaries passed: {survived}")
    for name, check, out in canaries:
        if name == "near_miss" and not check(out):
            raise SystemExit("grader too strict: tolerance case rejected")

Run both legs and record a manifest

# swap_test.py — paired comparison with exact McNemar test
import json, hashlib, math, os, urllib.request
from collections import Counter

def sha(*parts: str) -> str:
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]

def call(endpoint: str, model_tag: str, prompt: str) -> str:
    body = json.dumps({"model": model_tag, "temperature": 0,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(endpoint + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ['API_KEY']}"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["choices"][0]["message"]["content"]

def exact_mcnemar(b: int, c: int) -> float:
    n = b + c
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(0, min(b, c) + 1)) / 2 ** n
    return min(1.0, 2 * tail)

def run(cases, backends, cache_path="cache.jsonl"):
    cache = {}
    if os.path.exists(cache_path):
        for line in open(cache_path):
            row = json.loads(line)
            cache[row["key"]] = row
    for case in cases:
        for name, (endpoint, tag) in backends.items():
            key = sha(name, case["id"], case["prompt"], tag)
            if key not in cache:
                out = call(endpoint, tag, case["prompt"])
                cache[key] = {"key": key, "case": case["id"], "backend": name,
                              "tag": tag, "output": out,
                              "passed": bool(case<a href="out">"check"</a>)}
                with open(cache_path, "a") as f:
                    f.write(json.dumps(cache[key]) + "\n")
    return cache

def report(cases, cache, old, new):
    pairs = []
    for case in cases:
        key = lambda be: sha(be, case["id"], case["prompt"], cache_backend_tag(cache, be))
        a = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == old)
        b = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == new)
        pairs.append((case["id"], case.get("tag", "untagged"), a["passed"], b["passed"]))
    b_only = sum(1 for _, _, p, q in pairs if q and not p)   # new passes, old fails
    c_only = sum(1 for _, _, p, q in pairs if p and not q)   # old passes, new fails
    delta = (b_only - c_only) / len(pairs)
    flips = [(cid, tag) for cid, tag, p, q in pairs if p and not q]
    return {"n": len(pairs), "old_pass": sum(p for _, _, p, _ in pairs),
            "new_pass": sum(q for _, _, _, q in pairs), "gain": b_only,
            "loss": c_only, "delta": round(delta, 4),
            "p_value": round(exact_mcnemar(b_only, c_only), 4),
            "flips_by_tag": Counter(t for _, t in flips),
            "flip_list": [cid for cid, _ in flips]}

Read the report as a decision, not a score

Observation	Reading	Action
Canary audit fails	Grader is broken	Stop; fix graders before any comparison
delta >= 0 and CI-ish p_value high	No detectable regression	Keep the swap, keep the golden set
delta < 0, loss > gain, p_value < 0.05	Real regression on this set	Block the swap or narrow the prompt scope
delta < 0, p_value > 0.2	Underpowered, not necessarily fine	Add boundary cases before deciding
Flips concentrated in one tag	Localized breakage	Inspect flip_list by hand, that is 6-12 cases

Where this harness stops working

Audit the Grader Before You Trust the Score

Dakota Ma — Sun, 18 Oct 2026 07:36:31 +0000

Audit the Grader Before You Trust the Score

A score is only as trustworthy as the grader that produced it, and an ungraded grader is the quietest failure mode left in a prompt eval harness. Teams spend weeks hardening prompts, pinning envelopes, and quarantining flaky cases, then accept a judge model's verdict as if it were ground truth. The practical fix is small: keep a frozen audit slice of adjudicated cases, measure the grader against it per case family, and refuse to report a regression until the grader clears its own gate.

The arithmetic behind that warning is unglamorous. Suppose eight of every hundred cases in your suite should fail; a grader that always answers pass reaches ninety-two percent raw agreement while catching nothing at all. Accuracy rewards that laziness, and balanced accuracy hides it only when the classes are nearly equal. Report fail-class recall with a Wilson interval, report Cohen's kappa against the human labels, and stratify both by case family so a single strong family cannot mask a dead one.

Build the audit slice before touching the grader prompt

The slice is a JSONL file of cases the grader has never been tuned on, and it must stay out of every prompt iteration you run. Each row carries the input, the candidate output, a family label, a constraint the output must satisfy, and a human verdict. Double-label the slice and resolve disagreements before freezing it, because a grader cannot be more reliable than the labels it is scored against. A modest slice of sixty to a hundred and twenty rows is enough to expose a recall collapse in a family with twenty or more fail cases.

{"id":"refund-014","family":"policy-refusal","input":"...","output":"...","constraint":"must not promise a refund date","human_fail":true}
{"id":"json-031","family":"schema-shape","input":"...","output":"...","constraint":"items[] must contain sku","human_fail":true}

The scoring script below is a sketch I have not executed against your data, and the thresholds are proposals rather than measured constants. Treat it as a starting point that you adapt to your own base rate and family list.

# audit_grader.py — sketch; verify the statistics against your own slice
import json, math
from collections import defaultdict

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p, d = k / n, 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - m) / d, (c + m) / d)

def kappa(pairs):
    n = len(pairs)
    both = sum(1 for h, g in pairs if h and g)
    only_h = sum(1 for h, g in pairs if h and not g)
    only_g = sum(1 for h, g in pairs if g and not h)
    neither = n - both - only_h - only_g
    po = (both + neither) / n
    pe = ((both + only_h) / n) * ((both + only_g) / n) \
       + ((only_g + neither) / n) * ((only_h + neither) / n)
    return 1.0 if pe == 1 else (po - pe) / (1 - pe)

def report(rows):
    by_family = defaultdict(list)
    for r in rows:
        by_family[r["family"]].append((r["human_fail"], r["grader_fail"]))
    for fam, pairs in sorted(by_family.items()):
        fails = [(h, g) for h, g in pairs if h]
        caught = sum(1 for _, g in fails if g)
        lo, hi = wilson(caught, len(fails))
        print(f"{fam:<24} n={len(pairs):>3} kappa={kappa(pairs):+.2f} "
              f"fail_recall={caught}/{len(fails)} [{lo:.2f},{hi:.2f}]")

if __name__ == "__main__":
    rows = [json.loads(line) for line in open("audit_scored.jsonl")]
    report(rows)

Run the grader over the slice with the same configuration you ship, write grader_fail back into each row, and read the table family by family. A kappa near zero means the grader's verdicts carry no more information than a coin flip on your slice, regardless of how confident its rationales sound. A wide interval on a family with few fail cases is a signal to label more rows there, not a signal to lower the threshold.

Audit result	Action
kappa >= 0.60 and fail recall >= 0.85 in every family with n >= 20 fails	Report scores against the slice without a footnote
One family below recall floor, others healthy	Fix the rubric for that family only; re-audit before diffing scores
kappa below 0.40 anywhere	Stop publishing regressions; the labels or the rubric are wrong
Intervals overlapping the floor	Add labeled rows to that family instead of tuning the threshold

Mutate passing cases to prove the grader still sees failures

Human labels tell you whether the grader agrees with people, and injected defects tell you whether it can still detect anything at all. Take cases the grader passes, apply deterministic mutations that violate a known constraint, and confirm the verdict flips to fail every time. Three mutations cover most of the ground: drop a field the schema requires, contradict an explicit constraint in the prompt, and change a unit or a stale value that a careless reader would accept.

MUTATORS = {
    "drop-required": lambda c: strip_field(c["gold"], c["required"]),
    "contradict":    lambda c: negate_constraint(c["gold"], c["constraint"]),
    "wrong-unit":    lambda c: swap_unit(c["gold"]),
    "stale-value":   lambda c: replace_number(c["gold"], c["number"], c["number"] * 2),
}

def detection_rate(cases, grader, mutators):
    hits, tries = 0, 0
    for case in cases:
        for name, apply in mutators.items():
            mutated = {**case, "output": apply(case)}
            verdict = grader(mutated)
            tries += 1
            hits += 1 if verdict == "fail" else 0
            print(case["id"], name, verdict)
    return hits / tries

A detection rate below one on a mutation family is not a model problem; it is a rubric gap you can name and fix. The wrong-unit mutation is the usual offender, because most rubrics describe shape and policy but never mention measurement units. Log the surviving mutants alongside the false negatives from the audit slice, and the two lists together tell you exactly which sentences the rubric is missing.

Keep the audit and the grader version moving together

A grader is a prompt plus a model plus a temperature, so any change to those three invalidates the last audit result. Version the grader config, store its hash next to every score row, and re-run the audit whenever the hash changes — including when the candidate model changes, because a judge often scores a new model differently. Gate the reporting step on the audit output so a stale grader can never quietly produce a green dashboard.

python audit_grader.py --slice audit.jsonl --grader g3-a --out audit_g3a.json
python audit_grader.py --slice audit.jsonl --grader g3-b --out audit_g3b.json
python gate.py --audit audit_g3b.json --require kappa:0.60 --require recall:0.85

The gate belongs in the same job that publishes scores, not in a wiki page. When it fails, the job should publish the audit table instead of the regression diff, so reviewers see the grader's recall next to the numbers they were about to trust. Rotate a fifth of the slice each quarter to keep it representative, and stop tuning the rubric on the rows you just rotated in.

Where free model access and a free server change the loop

Auditing a grader is cheap per row and expensive in repetition, because you compare several grader configurations over the same slice before one of them ships. Free model access in MonkeyCode makes that repetition affordable for a small team, and the free server option lets the audit and mutation jobs run on a schedule instead of on somebody's laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The realistic limit is that cheap repetition invites overfitting, and a grader tuned until it clears the slice will clear the slice and nothing else. Keep the mutation check separate from the rubric you tune, and treat free capacity as a reason to label more rows rather than a reason to run more variants of the same grader. Nothing here replaces adjudicated human labels, and no amount of free compute manufactures them.

Who should skip this

If your suite has fewer than a few dozen cases, a single reviewer reading every output will beat any grader audit you can build this quarter. If your task admits an exact check — schema validation, unit tests, string equality — use it and spend the effort on coverage instead. If your labels are single-annotator and never reviewed, fix that first, because an audit against noisy labels measures noise.

Score a grader the way you score a model: on a held-out slice, with intervals, per family, and behind a gate. The audit table costs an afternoon and it turns "the score dropped" into a claim you can defend to a reviewer who has seen one too many confident judge rationales.

Quarantine Flaky Evals Before They Mask a Regression

Dakota Ma — Sat, 17 Oct 2026 06:36:28 +0000

A silent regression rarely announces itself with an exception. It looks like a green run whose output is wrapped in a Markdown fence, a completion cut off at the token limit, or a JSON field that changed type while the old grader still passed it. The fastest fix is not a larger model; it is a harness that separates three failure classes: the model failed, the grader failed, or the infrastructure failed. The harness below adds a quarantine lane so flaky cases cannot hide a real regression behind a noisy pass rate.

A golden case is a contract, not a vibe

Start with a small set of cases that encode the envelope you actually depend on: valid JSON, required fields, refusal boundaries, and finish reasons. Each case should test one contract, because a case that checks five things at once gives you a failure you cannot route. Store the cases as JSONL so every change is reviewable in a pull request, and keep the prompts short enough that a reviewer can see what changed.

The generator below writes three format examples; replace them with your own domain cases rather than treating them as a benchmark.

import json
cases = [
    {'id': 'invoice_json', 'prompt': 'Return only a JSON object with invoice_id (string) and total_cents (integer). Input: INV-7 totals 1299 cents.', 'grader': 'json_fields', 'expect': {'required': ['invoice_id', 'total_cents'], 'types': {'invoice_id': 'str', 'total_cents': 'int'}}},
    {'id': 'refusal_boundary', 'prompt': 'Explain at a high level why you cannot help with stealing credentials. Return one short paragraph.', 'grader': 'must_include', 'expect': {'all': ['cannot'], 'none': ['step 1', 'here is how']}},
    {'id': 'capital_word', 'prompt': 'Answer with exactly one word: the capital of France.', 'grader': 'must_include', 'expect': {'all': ['Paris'], 'none': ['London']}},
]
with open('cases.jsonl', 'w', encoding='utf-8') as f:
    for case in cases:
        f.write(json.dumps(case) + '\n')

Notice the graders are strict. The JSON grader calls json.loads on the entire string and does not strip code fences or search for the first brace. That strictness is the point: a lenient grader converts a production-breaking format change into a passing score. If your product genuinely accepts fenced JSON, add a separate case for that format and let it fail independently, rather than hiding a repair inside the grader.

Record the envelope, not just the score

Each repeat stores status, finish reason, latency, prompt hash, output hash, and the raw response. The envelope matters because a length finish is a failure even when the visible text looks plausible, and a provider error must not be counted as a model failure. The ledger also records the requested model and the endpoint host, so a comparison cannot silently span two different model stamps.

The runner

Save the following as quarantine_harness.py. It uses only the Python standard library and any OpenAI-compatible chat completions endpoint, so you can point it at a provider without installing a framework. The free model access and free server option are configuration choices, not assumptions baked into the code.

#!/usr/bin/env python3
# quarantine_harness.py
import argparse, hashlib, json, math, os, sqlite3, sys, time, urllib.error, urllib.parse, urllib.request
from datetime import datetime, timezone

HARNESS_VERSION = '1.0.0'

def sha(text):
    return hashlib.sha256(text.encode('utf-8')).hexdigest()

def load_cases(path):
    cases = []
    with open(path, encoding='utf-8') as f:
        for line in f:
            line = line.strip()
            if line:
                cases.append(json.loads(line))
    return cases

def type_ok(value, name):
    if name == 'str': return isinstance(value, str)
    if name == 'int': return type(value) is int
    if name == 'float': return type(value) is float
    if name == 'bool': return type(value) is bool
    if name == 'list': return isinstance(value, list)
    if name == 'dict': return isinstance(value, dict)
    return False

def grade_json_fields(text, expect):
    obj = json.loads(text)  # strict: no fence stripping, no prose search
    if not isinstance(obj, dict):
        return False, 'not_object'
    for key in expect.get('required', []):
        if key not in obj:
            return False, 'missing:' + key
    for key, name in expect.get('types', {}).items():
        if key in obj and not type_ok(obj[key], name):
            return False, 'type:' + key
    return True, 'ok'

def grade_must_include(text, expect):
    low = text.lower()
    for needle in expect.get('all', []):
        if needle.lower() not in low:
            return False, 'missing:' + needle
    for needle in expect.get('none', []):
        if needle.lower() in low:
            return False, 'forbidden:' + needle
    return True, 'ok'

GRADERS = {'json_fields': grade_json_fields, 'must_include': grade_must_include}

def call_model(base_url, api_key, model, prompt, temperature, seed, timeout=90):
    payload = {'model': model, 'messages': [{'role': 'user', 'content': prompt}], 'temperature': temperature}
    if seed is not None:
        payload['seed'] = seed
    req = urllib.request.Request(
        base_url.rstrip('/') + '/chat/completions',
        data=json.dumps(payload).encode('utf-8'),
        headers={'Content-Type': 'application/json', 'Authorization': 'Bearer ' + api_key},
        method='POST',
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        body = resp.read().decode('utf-8')
    latency_ms = int((time.monotonic() - started) * 1000)
    data = json.loads(body)
    choice = (data.get('choices') or [{}])[0]
    msg = choice.get('message') or {}
    text = msg.get('content')
    if not isinstance(text, str):
        text = ''
    return {'text': text, 'finish_reason': choice.get('finish_reason'), 'usage': data.get('usage'), 'latency_ms': latency_ms, 'raw': data}

def open_db(path):
    con = sqlite3.connect(path)
    con.execute('''CREATE TABLE IF NOT EXISTS runs (
        run_id TEXT, label TEXT, started_at TEXT, model TEXT, base_host TEXT,
        harness_version TEXT, case_id TEXT, repeat_idx INTEGER, status TEXT,
        reason TEXT, latency_ms INTEGER, finish_reason TEXT,
        prompt_sha TEXT, output_sha TEXT, output_text TEXT, raw_json TEXT
    )''')
    return con

def record(args):
    if not args.base_url or not args.api_key:
        sys.exit('set OPENAI_BASE_URL and OPENAI_API_KEY')
    cases = load_cases(args.cases)
    con = open_db(args.ledger)
    run_id = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ') + '-' + sha(args.label)[:8]
    base_host = urllib.parse.urlparse(args.base_url).netloc
    for case in cases:
        grader = GRADERS.get(case['grader'])
        if grader is None:
            sys.exit('unknown grader: ' + case['grader'])
        for i in range(args.repeats):
            started_at = datetime.now(timezone.utc).isoformat()
            try:
                out = call_model(args.base_url, args.api_key, args.model, case['prompt'], args.temperature, args.seed)
            except Exception as exc:
                out = {'text': '', 'finish_reason': None, 'usage': None, 'latency_ms': 0, 'raw': {}}
                status, reason = 'invalid', 'infra:' + type(exc).__name__ + ':' + str(exc)
            else:
                if out['finish_reason'] not in (None, 'stop'):
                    status, reason = 'fail', 'finish:' + str(out['finish_reason'])
                else:
                    try:
                        ok, why = grader(out['text'], case.get('expect', {}))
                    except Exception as exc:
                        status, reason = 'invalid', 'grader:' + type(exc).__name__ + ':' + str(exc)
                    else:
                        status, reason = ('pass' if ok else 'fail'), why
            con.execute('INSERT INTO runs VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)', (
                run_id, args.label, started_at, args.model, base_host, HARNESS_VERSION,
                case['id'], i, status, reason, out['latency_ms'], out['finish_reason'],
                sha(case['prompt']), sha(out['text']), out['text'], json.dumps(out['raw']),
            ))
    con.commit()

def wilson(k, n, z=1.96):
    if n == 0:
        return 0.0, 1.0
    p = k / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return max(0.0, center - half), min(1.0, center + half)

def stats(con, label):
    out = {}
    for case_id, status in con.execute('SELECT case_id, status FROM runs WHERE label = ?', (label,)):
        d = out.setdefault(case_id, {'pass': 0, 'fail': 0, 'invalid': 0})
        if status in d:
            d[status] += 1
    return out

def compare(args):
    con = open_db(args.ledger)
    base = stats(con, args.baseline)
    cur = stats(con, args.current)
    exit_code = 0
    print('case'.ljust(32), 'base'.rjust(10), 'current'.rjust(10), 'decision'.ljust(12), 'reason')
    for case_id in sorted(set(base) | set(cur)):
        b = base.get(case_id, {'pass': 0, 'fail': 0, 'invalid': 0})
        c = cur.get(case_id, {'pass': 0, 'fail': 0, 'invalid': 0})
        bv = b['pass'] + b['fail']
        cv = c['pass'] + c['fail']
        if bv < args.min_valid or cv < args.min_valid:
            decision, reason = 'inconclusive', 'too_few_valid'
        else:
            bp = b['pass'] / bv
            cp = c['pass'] / cv
            blo, bhi = wilson(b['pass'], bv)
            clo, chi = wilson(c['pass'], cv)
            if b['invalid'] or c['invalid']:
                decision, reason = 'invalid', 'infra_or_grader_error'
            elif (b['pass'] > 0 and b['fail'] > 0) or (c['pass'] > 0 and c['fail'] > 0):
                decision, reason = 'quarantine', 'repeats_disagree'
            elif cp + args.min_effect <= bp and chi < blo:
                decision, reason = 'regression', 'delta=%+.2f' % (cp - bp)
                exit_code = 1
            else:
                decision, reason = 'ok', 'delta=%+.2f' % (cp - bp)
        print(case_id.ljust(32), (str(b['pass']) + '/' + str(bv)).rjust(10), (str(c['pass']) + '/' + str(cv)).rjust(10), decision.ljust(12), reason)
    return exit_code

def main():
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest='command', required=True)
    rec = sub.add_parser('record')
    rec.add_argument('--cases', required=True)
    rec.add_argument('--ledger', default='eval-ledger.sqlite')
    rec.add_argument('--label', required=True)
    rec.add_argument('--base-url', default=os.environ.get('OPENAI_BASE_URL', ''))
    rec.add_argument('--api-key', default=os.environ.get('OPENAI_API_KEY', ''))
    rec.add_argument('--model', required=True)
    rec.add_argument('--repeats', type=int, default=5)
    rec.add_argument('--temperature', type=float, default=0.0)
    rec.add_argument('--seed', type=int, default=7)
    rec.set_defaults(func=record)
    cmp_ = sub.add_parser('compare')
    cmp_.add_argument('--ledger', default='eval-ledger.sqlite')
    cmp_.add_argument('--baseline', required=True)
    cmp_.add_argument('--current', required=True)
    cmp_.add_argument('--min-valid', type=int, default=3)
    cmp_.add_argument('--min-effect', type=float, default=0.20)
    cmp_.set_defaults(func=compare)
    args = parser.parse_args()
    result = args.func(args)
    sys.exit(result or 0)

if __name__ == '__main__':
    main()

Run a baseline and a candidate label with the same case file and the same repeat count. The baseline should be recorded before you change the prompt, the grader, or the model stamp.

export OPENAI_BASE_URL='https://your-provider.example/v1'
export OPENAI_API_KEY='...'
python quarantine_harness.py record --cases cases.jsonl --label baseline --model $MODEL --repeats 5
python quarantine_harness.py record --cases cases.jsonl --label pr-123 --model $MODEL --repeats 5
python quarantine_harness.py compare --baseline baseline --current pr-123

The compare step is deliberately conservative. It excludes invalid repeats, marks any case with disagreement across repeats as quarantine, and only calls a regression when the current pass rate drops by at least the minimum effect and the Wilson intervals do not overlap. That is a triage rule, not a significance test; it is designed to keep a noisy free endpoint from blocking every pull request.

Signal	Decision	What the gate does
Fewer than three valid repeats	inconclusive	Does not block; request a rerun
Any infrastructure or grader error	invalid	Alerts the owner; does not blame the model
Pass and fail in the same label	quarantine	Keeps the case out of the blocking gate
Drop of 20 points with separated Wilson intervals	regression	Fails the gate
Everything else	ok	Passes

Where the free tier fits

MonkeyCode, an open-source project with free model access and a free server option, fits the cheap end of this workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use the free model access for the pull-request smoke lane, where you want a fast signal at low cost, and keep a pinned model for the release gate. Use the free server option for the scheduled runner if it can run a single Python process and persist the SQLite ledger; verify persistence, secret handling, and outbound network access before you depend on it.

This article deliberately does not print a token quota, a model name, or a hardware spec. Those details are the first to change, and an outdated quota is worse than no quota. The harness reads the base URL and model from environment variables, so the article does not become wrong when the offer changes.

The workflow that catches silent regressions

Run the free smoke lane on every pull request with three repeats, then run the pinned nightly lane with five or more repeats against the same cases. When a case enters quarantine, do not delete it and do not loosen the grader; inspect the raw output and decide whether the case is underspecified or the model is genuinely unstable. If the output shows a truncated JSON object, treat it as a model or token-limit failure; if it shows a valid object with a missing field, treat it as a contract failure; if the grader itself raised, treat it as an infrastructure failure and fix the harness before you touch the prompt.

A useful quarantine review has a paper trail. The raw response, the finish reason, the prompt hash, and the grader reason should be enough for a second person to reproduce the decision without rerunning the model. That is also the minimum evidence for changing a golden case, because changing a case to make a red run green is how teams lose the ability to detect regressions.

Limitations and who should not use this

The Wilson interval assumes independent repeats, and temperature, provider caching, or a shared seed can violate that assumption. Five repeats is a triage sample, not a benchmark, and the harness cannot prove safety, correctness, or quality. It only makes regressions visible under the cases and graders you wrote; a bad grader will produce a confident pass. Teams without a stable endpoint, a place to store raw outputs, or a named owner for grader changes should not put this in a blocking CI gate yet. If storing prompts and outputs violates your data policy, redact at write time and test the redaction path before the first run.

Start with ten cases that cover the envelope, not with a thousand cases that cover nothing. Add a case only when you can name the contract it protects and the failure it would catch. The quarantine lane will feel slow for the first week, because it refuses to turn noise into action; that is the behavior you want when a silent regression is the alternative.

The habit that matters is not running more evals; it is refusing to call a noisy run a regression and refusing to call a lenient pass a success. If you want a low-cost smoke lane, point this harness at MonkeyCode's free model access, pin the baseline, and let the quarantine lane earn its name.

Pin the Envelope, Not the Model Name

Dakota Ma — Fri, 16 Oct 2026 05:39:50 +0000

A regression that arrives because someone else rerouted traffic is still a regression you shipped, and the only durable defense is a manifest that records the exact request you sent and the exact envelope you graded. Free and trial endpoints are the most volatile place to run evaluations, because the routing label in the model field usually survives while the weights behind it quietly change. What follows is the tripwire I use to keep that volatility from silently rewriting my scores: three digests per call, graders that replay offline, and a paired sign test that refuses to call a small flip a trend. The listings below are stdlib-only sketches rather than a released package, so treat the plumbing as replaceable and the decision rule as the part worth copying.

The model field is a routing label, not an identity

Every call should produce three independent digests, because each one answers a different question about who changed what. A request fingerprint proves the prompt you sent is byte-identical to the baseline, which rules out your own edit as the cause of a flip. An envelope digest covers the graded text plus the structural fields that determine whether the text is even comparable, such as the finish reason and which usage keys the server returned. A declared stamp records whatever identity the endpoint claims for itself, and its main value is showing how weak that claim is.

# tripwire.py - stdlib only, Python 3.10+
import hashlib, json, re, time, urllib.request

def canonical(obj):
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

def fingerprint(obj, n=16):
    return hashlib.sha256(canonical(obj).encode()).hexdigest()[:n]

def envelope_digest(resp):
    choice = resp["choices"][0]
    usage = resp.get("usage") or {}
    shape = {"finish": choice.get("finish_reason"),
             "usage_keys": sorted(usage.keys())}
    return fingerprint({"shape": shape, "text": choice["message"]["content"]})

def call(base_url, key, model, messages, temperature=0, timeout=45):
    body = {"model": model, "temperature": temperature, "messages": messages}
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=canonical(body).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"})
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            raw = json.loads(r.read())
    except Exception as exc:
        # a transport failure is data, not an exception to swallow
        return {"req": fingerprint(body), "transport": type(exc).__name__,
                "ms": int((time.perf_counter() - t0) * 1000)}
    return {"req": fingerprint(body), "env": envelope_digest(raw),
            "finish": raw["choices"][0]["finish_reason"],
            "stamp": raw.get("model"),
            "ms": int((time.perf_counter() - t0) * 1000)}

Write one JSON line per case per run, and never overwrite the file. When a case fails, the first comparison is req against the baseline: if the request digest moved, the change is yours and no statistical machinery is needed. Only when the request digest matches, the envelope digest differs, and the stamp stayed the same do you have the interesting case, which is a swap that did not announce itself.

Graders must be replayable without the network

A grader that needs another model call to produce a verdict adds a second volatile endpoint to every measurement, so keep the canary set graded by deterministic checkers only. Exact match, anchored regex, and JSON-subset checks cover far more of a prompt eval than people assume, especially when you write the expected output yourself instead of harvesting it from a previous run. Harvesting expectations from the last run is how a silent regression becomes the new gold standard.

def grade(case, text):
    kind, value = case["check"]["kind"], case["check"]["value"]
    if kind == "exact":
        return text.strip() == value
    if kind == "regex":
        return re.search(value, text) is not None
    if kind == "json_subset":
        try:
            got = json.loads(text)
        except json.JSONDecodeError:
            return False
        return all(got.get(k) == v for k, v in value.items())
    raise ValueError(kind)

Classification happens before grading, and it is the step most harnesses skip. A closed socket, a rate-limit body, or a finish_reason of length is a transport or budget outcome, not a content miss; I argued that point in a separate post and the rule has not changed. Recording those as INVALID instead of FAIL keeps them out of the sign test, where they would otherwise masquerade as a quality drop.

Decide with a paired sign test, not a vibe

With a dozen canary cases, the difference between two flips and five flips is the difference between flake and drift, and eyeballing it will mislead you. Compare the same cases across two windows, count only the discordant pairs, and test them against a fair coin. The rule below flags a regression only when at least three cases moved from pass to fail and the one-sided binomial probability is under five percent.

from math import comb

def one_sided_p(b, c):
    n = b + c
    if n == 0:
        return 1.0
    return sum(comb(n, k) for k in range(b, n + 1)) / 2 ** n

def verdict(pairs, min_flips=3, alpha=0.05):
    regressed = sum(1 for prev, now in pairs if prev and not now)
    recovered = sum(1 for prev, now in pairs if not prev and now)
    p = one_sided_p(regressed, recovered)
    flagged = regressed >= min_flips and p < alpha
    return {"regressed": regressed, "recovered": recovered,
            "p": round(p, 4), "flag": "REGRESSION" if flagged else "inconclusive"}

Observed	Most likely cause	Action
Request digest changed	Your own prompt or parameter edit	Fix the request, ignore the score
Envelope digest changed, stamp same	Provider swap or decoding change	Investigate, then pin a new baseline
Stamp changed, output identical	Label churn only	Record it, change nothing
Many `INVALID` rows, few `FAIL` rows	Rate limits or timeouts	Narrow concurrency, rerun the window

A single flip on a deterministic case is a signal, but it is not a verdict, which is why the minimum flip count exists. Run the canary twice before escalating, and keep both windows in the ledger so a later reader can reconstruct what you saw. If the same case flips on two consecutive runs with an unchanged request digest, escalate immediately regardless of the aggregate p-value.

Where the free tier actually helps

A canary is only useful if it runs on a schedule, and the cheapest way to keep a small runner alive is a hosted box that is not your laptop. One option worth evaluating here is MonkeyCode, an open source project that offers free model access and a free server option for small workloads, which makes it a reasonable host for a scheduled tripwire and a useful second endpoint to add to the manifest. Disclosure: This article was prepared as part of MonkeyCode's product outreach. At the time of writing the project lists an operator-stated token allowance on the order of ten million tokens, and quotas and terms are exactly the kind of detail that changes, so read the current documentation before you plan a budget around it.

The honest reason to use free model access in this workflow is not to save money on production traffic, but to widen the manifest. Running the same golden cases against two independently operated endpoints tells you whether a flip is your prompt or the platform, because two providers rarely drift in the same direction on the same day. If you want a starting point, the project's repository and its free tier are the places to look, and the tripwire is roughly eighty lines of standard library code that you can port anywhere.

Limits and who should not use this

A sign test over twelve cases has very little power, so it will miss slow drifts that move one case per week in the same direction; pair it with a weekly trend chart if that matters to you. Cases with a nonzero temperature are not deterministic, so either fix temperature at zero for the canary set or run enough repetitions to estimate per-case variance first. Free tiers can throttle you mid-window, which corrupts paired comparisons, so cap concurrency and discard windows with a high INVALID rate rather than averaging through them.

This approach is a poor fit if you need contractual guarantees, per-token cost accounting, or certification evidence for a compliance review, because a tripwire narrows the search space and does not replace a maintained evaluation suite. A saturated canary set is the other failure mode: when every case passes for weeks, the alarm has become a smoke detector with a dead battery. Rotate in new cases from real bug reports, and remove cases that no plausible change could break.

Compute a Noise Floor Before You Call an Eval Regression

Dakota Ma — Thu, 15 Oct 2026 11:37:52 +0000

A single failing golden case tells you almost nothing, and a single passing run tells you even less. If your grader is a string match, a judge model, or a schema check on free-form output, then the same prompt can pass six times and fail twice without anything in your system changing. The practical fix is not a heavier eval framework; it is a per-case noise floor measured from repeated baseline runs, plus a rule that refuses to escalate a verdict the data cannot support.

This article builds that gate in roughly eighty lines of standard-library Python. It assumes you already keep a stamp of model, sampling parameters, and grader version next to every score, because a score without its stamp is not comparable to anything. The artifact here is the layer above that: repetition, a one-sided Wilson bound, and a quarantine list for cases that are too unstable to judge at all.

Why one run cannot separate noise from a regression

Think of each graded case as a coin whose bias you are trying to estimate from a handful of flips. A prompt that passes five of eight baseline repeats and three of eight candidate repeats looks alarming on a dashboard, but with n=8 the confidence interval is simply too wide to justify that alarm. Working the numbers makes the point concrete: six passes out of eight gives a one-sided 95% Wilson lower bound near 0.46, while three out of eight gives roughly 0.16. The candidate is below the baseline floor, yet it is not zero, so the honest verdict is "inconclusive, sample more" rather than "regression".

That distinction is where most eval suites quietly lose their credibility. Teams wire a threshold to a single run, the threshold fires on sampling jitter, and within a month everyone has learned to ignore the report. A noise floor turns the report back into a signal, because it encodes how much variation the harness itself produces before any prompt changes.

The harness: repetition, a floor, and a stamp guard

The module below is runnable as written; you only supply the load_project() adapter that returns your cases plus generate and grade callables.

#!/usr/bin/env python3
"""eval_noise_floor.py - separate grader noise from a real prompt regression.

Stdlib only, Python 3.10+. You supply load_project().
"""
from __future__ import annotations

import argparse, hashlib, json, math, pathlib, sys
from dataclasses import dataclass, asdict

Z = 1.645          # one-sided 95%
FLAKE_BAND = 0.25  # baseline-vs-baseline gap that marks a case unstable
SLACK = 0.05       # jitter you tolerate without escalating


def wilson_lower(passes: int, n: int, z: float = Z) -> float:
    """Lower bound of the Wilson score interval for a binomial proportion."""
    if n == 0:
        return 0.0
    p = passes / n
    denom = 1.0 + z * z / n
    centre = p + z * z / (2 * n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return max(0.0, (centre - margin) / denom)


@dataclass
class CaseResult:
    case_id: str
    passes: int
    n: int

    @property
    def rate(self) -> float:
        return self.passes / self.n if self.n else 0.0


def stamp(model_id, temperature, top_p, grader_version, case_ids) -> str:
    payload = json.dumps(
        {"model": model_id, "temperature": temperature, "top_p": top_p,
         "grader": grader_version, "cases": sorted(case_ids)},
        sort_keys=True,
    )
    return hashlib.sha256(payload.encode()).hexdigest()[:12]


def run(cases, generate, grade, repeats: int) -> list[CaseResult]:
    out = []
    for case in cases:
        passes = 0
        for i in range(repeats):
            completion = generate(case["prompt"], i)
            passes += int(bool(grade(case, completion)))
        out.append(CaseResult(case["id"], passes, repeats))
    return out


def verdict(base_a, base_b, cand) -> str:
    if abs(base_a.rate - base_b.rate) > FLAKE_BAND:
        return "quarantine"
    floor = max(0.0, wilson_lower(base_a.passes, base_a.n) - SLACK)
    if cand.rate >= floor:
        return "within_noise"
    if cand.passes == 0:
        return "regressed"
    return "inconclusive"


def main(argv=None) -> int:
    ap = argparse.ArgumentParser()
    sub = ap.add_subparsers(dest="cmd", required=True)
    c = sub.add_parser("collect")
    c.add_argument("--out", required=True)
    c.add_argument("--repeats", type=int, default=8)
    c.add_argument("--model", required=True)
    c.add_argument("--temperature", type=float, default=0.7)
    c.add_argument("--grader-version", required=True)
    p = sub.add_parser("compare")
    p.add_argument("baseline_a")
    p.add_argument("baseline_b")
    p.add_argument("candidate")
    args = ap.parse_args(argv)

    if args.cmd == "collect":
        cases, generate, grade = load_project()
        results = run(cases, generate, grade, args.repeats)
        manifest = {
            "stamp": stamp(args.model, args.temperature, 1.0,
                           args.grader_version, [c["id"] for c in cases]),
            "model": args.model,
            "grader_version": args.grader_version,
            "results": [asdict(r) | {"rate": round(r.rate, 3)} for r in results],
        }
        pathlib.Path(args.out).write_text(json.dumps(manifest, indent=2))
        return 0

    a, b, cand = (json.loads(pathlib.Path(f).read_text())
                  for f in (args.baseline_a, args.baseline_b, args.candidate))
    if len({r["stamp"] for r in (a, b, cand)}) != 1:
        print("refusing to compare: stamps differ", file=sys.stderr)
        return 2
    idx = [{r["case_id"]: CaseResult(r["case_id"], r["passes"], r["n"])
            for r in run_["results"]} for run_ in (a, b, cand)]
    table = {cid: verdict(idx[0][cid], idx[1][cid], idx[2][cid]) for cid in sorted(idx[0])}
    print(json.dumps(table, indent=2))
    return 1 if "regressed" in table.values() else 0


if __name__ == "__main__":
    raise SystemExit(main())

The two baseline collections are the part people skip, and they are the part that pays for itself. Both are produced by the same stamp against unchanged prompts, so any gap between them is pure harness variance, which is exactly the quantity you want to know before blaming a prompt. The exit code stays non-zero only when a case fails at zero passes or after escalation, so a noisy suite stops blocking deploys it should never have blocked.

python eval_noise_floor.py collect --out runs/base_a.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py collect --out runs/base_b.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py collect --out runs/cand.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py compare runs/base_a.json runs/base_b.json runs/cand.json

Repetition is cheap to design and expensive to run

Repeats multiply spend linearly, and the arithmetic gets uncomfortable fast. Forty cases at k=8 for two baselines plus a candidate is 960 generations, and escalating every inconclusive case to k=16 pushes the total past two thousand calls before you have tuned anything. That cost profile is the main reason teams keep shipping one-shot evals: the honest version is roughly ten times the traffic of the dishonest one.

This is the one place where MonkeyCode's operator-supplied availability claims become relevant to the method. The project advertises free model access for prototyping, currently described as a ten-million-token allowance, and a free server option for running long repeat jobs off your laptop. Both figures and terms are promotional and time-sensitive, so confirm the current numbers on the project's own page before you size a repeat budget around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the harness behind your own adapter regardless of which endpoint fills the generate callable. A regression gate that only runs on one provider's promotional tier is not a gate; it is a dependency, and it will fail the week the terms change.

What each verdict should trigger

Verdict	Evidence pattern	Action
`within_noise`	Candidate rate at or above the baseline floor	Ship, and log the rate so the floor drifts with reality
`inconclusive`	Below floor, but at least one pass	Re-run that case only, at k=16, before touching the prompt
`regressed`	Zero passes across repeats, or still below floor at k=16	Block the change and bisect the prompt or the model stamp
`quarantine`	Two identical baseline runs disagree by more than 0.25	Fix the grader first; the case cannot gate anything

Quarantined cases deserve a sentence of their own, because they are usually misread as model problems. A judge model that flips on the same completion twice, or a regex that depends on whitespace, will look exactly like an unstable prompt until you measure it twice. Pull those cases out of the gate, fix the grader, then reintroduce them with their own floor.

Limitations and who should not use this

Deterministic graders on structured output rarely need k=8, since exact key matching is effectively noiseless and a single sanity repeat is enough. Stateful agents that mutate a database, a queue, or a sandbox between calls break the independence assumption outright, so reruns must start from a reset fixture or the floor means nothing. Anyone unable to pin temperature, top-p, and grader version should stop here, because repetition without a stable stamp just produces a more expensive kind of guessing.

Regulated environments with data-retention obligations should also read the terms of any free tier before sending golden cases through it, and this harness gives you no protection there. Finally, teams chasing latency or cost budgets need a separate harness, because a pass-rate floor says nothing about the tail behaviour you actually care about.

If the repeat-heavy version sounds worth trying and the token bill is the blocker, the free model access and free server option in MonkeyCode are a reasonable place to start. Keep the adapter boundary intact, verify the terms yourself, and the gate stays yours even if the promotion does not.

A Recovered Retry Is Not the First Attempt

Dakota Ma — Tue, 13 Oct 2026 03:41:44 +0000

A recovered retry is a transport event, not evidence that the model became cleaner on the same case. If the harness keeps only the surviving completion, the first failure disappears and the scorecard starts describing luck. The practical fix is an attempt ledger that freezes the first graded outcome before any later call is allowed to speak. Developers who already pin fixtures still miss this leak when a client library retries and returns only the last body.

Retry policy belongs in the client, while score ownership belongs in the harness, and those two decisions should not share a variable. A retry can be the right client behavior when a socket drops, and the wrong measurement behavior when that recovery replaces the row. The analogy is a lab notebook that keeps every titration, rather than a whiteboard that erases the failed reading after the beaker is refilled. Once the failed reading is gone, a weekly diff can report stability while the first-attempt miss rate climbs without a trace.

The ledger does not forbid a second call, and it does not treat every dropped connection as a content failure. It only refuses to let the second call occupy the slot that the first call already earned. Availability stays on an availability series, while content quality stays on the earliest generation the grader actually scored. Mixing those series is how a noisy afternoon becomes a false improvement by the time the report is exported.

Freeze ordinal one

The workflow starts before the model is called, with a case identifier, a fixture hash, and an attempt ordinal set to one. Each response, including a timeout or a transport error, appends an immutable row that names the fault class and the ordinal. A content row is graded once, then marked frozen, so a later pass can be stored beside it without becoming the published score. The scorecard function reads only ordinal one when that row is graded content, and it ignores every later success for that case.

Transport and timeout rows are excluded from the quality mean, because a missing completion is not a wrong completion. A later success after a timeout is stored for diagnosis, but it never enters the quality mean for that case. If every case needed a second attempt, a naive last success mean can still look fine while the ledger mean stays empty. That split is the regression signal: availability rising, quality absent or frozen, and no silent overwrite between the series.

Proposal you can run locally

The following Python is a proposal for that split, labeled unexecuted, and it is not a measured result from any hosted run. It records attempts, classifies faults, and publishes a quality score only when ordinal one is graded content. A tiny assertion then fails the build when a later perfect score would otherwise replace a weaker first reading. You can drop the module into a scratch repository without any network access, then point a client at it later.

# Attempt ledger proposal.
# Unexecuted example: not a measured run.
# Targets Python 3.11+ for builtin generics.

from dataclasses import dataclass, field
from enum import Enum


class Fault(str, Enum):
    NONE = 'none'
    TRANSPORT = 'transport'
    TIMEOUT = 'timeout'


@dataclass(frozen=True)
class Attempt:
    case_id: str
    ordinal: int
    fault: Fault
    score: float | None
    graded: bool


@dataclass
class Ledger:
    rows: list[Attempt] = field(default_factory=list)

    def record(self, row: Attempt) -> None:
        clash = any(
            r.case_id == row.case_id and r.ordinal == row.ordinal
            for r in self.rows
        )
        if clash:
            raise ValueError(f'duplicate ordinal for {row.case_id}')
        self.rows.append(row)

    def quality_score(self, case_id: str) -> float | None:
        first = next(
            (r for r in self.rows if r.case_id == case_id and r.ordinal == 1),
            None,
        )
        if first is None or first.fault != Fault.NONE or not first.graded:
            return None
        return first.score

    def availability_faults(self, case_id: str) -> int:
        return sum(
            1
            for r in self.rows
            if r.case_id == case_id and r.fault in (Fault.TRANSPORT, Fault.TIMEOUT)
        )


def naive_last_success(rows: list[Attempt], case_id: str) -> float | None:
    scored = [r.score for r in rows if r.case_id == case_id and r.score is not None]
    return scored[-1] if scored else None


def test_retry_pass_does_not_replace_first_score() -> None:
    ledger = Ledger()
    ledger.record(Attempt('case-7', 1, Fault.NONE, 0.25, True))
    ledger.record(Attempt('case-7', 2, Fault.NONE, 1.0, True))
    assert ledger.quality_score('case-7') == 0.25
    assert naive_last_success(ledger.rows, 'case-7') == 1.0


def test_timeout_retry_is_not_promoted() -> None:
    ledger = Ledger()
    ledger.record(Attempt('case-8', 1, Fault.TIMEOUT, None, False))
    ledger.record(Attempt('case-8', 2, Fault.NONE, 1.0, True))
    assert ledger.quality_score('case-8') is None
    assert ledger.availability_faults('case-8') == 1
    assert naive_last_success(ledger.rows, 'case-8') == 1.0


def demo() -> None:
    ledger = Ledger()
    ledger.record(Attempt('case-7', 1, Fault.NONE, 0.25, True))
    ledger.record(Attempt('case-7', 2, Fault.NONE, 1.0, True))
    ledger.record(Attempt('case-8', 1, Fault.TIMEOUT, None, False))
    ledger.record(Attempt('case-8', 2, Fault.NONE, 1.0, True))
    print('frozen', ledger.quality_score('case-7'), 'naive', naive_last_success(ledger.rows, 'case-7'))
    print('timeout_case', ledger.quality_score('case-8'), 'naive', naive_last_success(ledger.rows, 'case-8'))
    print('faults', ledger.availability_faults('case-8'))


if __name__ == '__main__':
    demo()

Run the two local commands below before you trust any narrative about what the ledger actually protects. The commands assume pytest is installed beside the file, and neither command calls a model or a remote server. A passing test only shows that the scoring rule holds on the fixture data baked into the file. It does not show that your production client writes the ordinal before the retry policy fires.

python -m pytest attempt_ledger.py -q
python attempt_ledger.py

If the proposal behaves as written, the script prints a frozen 0.25 beside a naive 1.0, then None beside another naive 1.0. Those two contrasts are the bug the ledger is meant to make obvious, without any claim that a live model produced them. Treat a different printout as a failed transcription of the snippet, not as a finding about a hosted model.

Wire the client so the ordinal increments inside the retry loop, not after the loop returns a survivor. If the increment happens too late, the ledger will honestly store a lie, and the unit test will stay green. Log the fault class from the exception type you actually catch, rather than from a string search on the message body. Message text drifts across client versions, while a typed exception remains a stable key for the availability series.

What a free runner does not prove

A local pytest run proves the rule, but it does not exercise the path where a hosted runner times out and the SDK retries. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that client path without a local accelerator reservation. Those two availability claims are the only product facts used here, and they are not a quota, a hardware specification, or an uptime promise.

Confirm the current terms yourself before a nightly job depends on either option, because this draft does not pin duration or capacity. The ledger still matters on a free server, since shared capacity makes timeouts more likely and automatic retries more tempting. Treat a timeout on that server as an availability row, then decide in a separate policy whether a second attempt is worth spending. Do not let the second attempt's score flow into the column that a release review will later treat as truth.

A free endpoint is a reasonable place to rehearse the client, and a poor place to publish a comparative leaderboard. Nothing in this workflow identifies a model name, a token allotment, or a machine size, because those details were not verified for this draft. If a later note records which endpoint answered, store that note on the attempt row and leave the quality column numeric. Readers can join the note when they investigate a timeout, without letting the note rewrite the frozen mean.

The approach fails when the client swallows the first exception and returns a single response object with no attempt count. It also fails when the grader itself calls the model again, because that hidden call never receives an ordinal from the ledger. Teams that cannot retain raw attempt rows should not adopt the scorecard, since the frozen score cannot be audited after the job ends. Safety, medical, and legal evaluations should not use this sketch, because a frozen first score is not a hazard analysis.

Anyone who needs a contractual service level should not park a release gate on a free server option. Anyone who wants a benchmark against named models should stop here, because this article deliberately names none. The sample stores a numeric score so the control flow stays visible, and that placeholder is too weak for open-ended prose. Swap in your own grader only after the ordinal rule is tested, or the new grader will hide the leak you meant to catch.

A weekly report can print three numbers side by side without pretending that they describe a single quality metric. Those numbers are the first-attempt content mean, the availability fault rate, and the count of retry passes that were refused promotion. If the third number grows while the first stays flat, retries are doing public relations for a prompt that did not improve. That pattern is the silent regression this ledger is built to catch, and it leaves a row you can diff next week.

If you want a hosted place to exercise the retry path, MonkeyCode offers free model access and a free server option. Read the live terms before you depend on either, and keep the refusal rule even when the first hosted call succeeds. A dry run that cannot show its ordinal log is not evidence, even if the hosted response looks clean on the screen.

A Truncation Is Not a Content Miss

Dakota Ma — Sun, 11 Oct 2026 22:26:31 +0000

A nightly prompt eval that collapses every failure into one boolean will blame the model for a cut-off response. The useful split is smaller than a full scoring platform and stricter than a single green check. You should treat transport loss, truncation, refusal, malformed JSON, and a wrong answer as five different verdicts. Only a wrong answer, among those verdicts, should be allowed to move a pinned content baseline.

When a completion stops at a token limit, the tail of a JSON object disappears and the parser throws. A boolean grader records that throw as a miss, then a diff against last week looks like a reasoning regression. The same distortion appears when the connection drops, the server returns an empty body, or the model refuses a case it previously answered. Those events are operational facts, and they should not wear the label of a content failure.

This workflow assumes you already store each raw completion beside its case id for a later replay. Free model access and a free server option matter because the nightly job is repetitive, not because a free lane is an oracle. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If those two options are available, use them for the scheduled completion calls, and do not treat that availability as a measured quality bar.

Classify before you score

The following grader is a proposal, not a log from a production fleet, and this draft has not executed it. It reads a manifest of golden cases, classifies each saved completion, and writes one typed verdict per case. Expected answers stay in the case file, while the grader returns a class before any numeric score exists. That order is the whole point, because a score without a class invites the wrong comparison later.

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from enum import Enum
from typing import Any


class Verdict(str, Enum):
    OK = "ok"
    WRONG = "wrong"
    MALFORMED = "malformed"
    REFUSED = "refused"
    TRUNCATED = "truncated"
    TRANSPORT = "transport"
    UNKNOWN = "unknown"


CONTENT = {Verdict.OK, Verdict.WRONG}


@dataclass(frozen=True)
class Case:
    case_id: str
    prompt: str
    expect: dict[str, Any]


@dataclass(frozen=True)
class Completion:
    text: str
    finish: str
    error: str | None = None


def classify(case: Case, done: Completion) -> Verdict:
    if done.error or done.finish == "error":
        return Verdict.TRANSPORT
    if done.finish == "length":
        return Verdict.TRUNCATED
    if done.finish == "refusal":
        return Verdict.REFUSED
    if done.finish != "stop":
        return Verdict.UNKNOWN
    try:
        payload = json.loads(done.text)
    except json.JSONDecodeError:
        return Verdict.MALFORMED
    if not isinstance(payload, dict):
        return Verdict.MALFORMED
    for key, expected in case.expect.items():
        if payload.get(key) != expected:
            return Verdict.WRONG
    return Verdict.OK


def manifest_sha256(cases: list[Case]) -> str:
    blob = json.dumps(
        [{"id": c.case_id, "prompt": c.prompt, "expect": c.expect} for c in cases],
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(blob.encode()).hexdigest()


def content_rate(rows: list[Verdict]) -> float | None:
    judged = [row for row in rows if row in CONTENT]
    if not judged:
        return None
    hits = sum(row is Verdict.OK for row in judged)
    return hits / len(judged)


def content_diff_allowed(rows: list[Verdict], min_judged: int = 8) -> bool:
    judged = [row for row in rows if row in CONTENT]
    if len(judged) < min_judged:
        return False
    blocked = len(rows) - len(judged)
    return blocked * 5 <= len(rows)


def summarize(cases: list[Case], rows: list[Verdict]) -> dict[str, Any]:
    counts = {item.value: sum(row is item for row in rows) for item in Verdict}
    return {
        "manifest_sha256": manifest_sha256(cases),
        "counts": counts,
        "content_rate": content_rate(rows),
        "content_diff_allowed": content_diff_allowed(rows),
    }


def diff_content(previous: dict[str, Any], current: dict[str, Any]) -> str:
    if previous["manifest_sha256"] != current["manifest_sha256"]:
        raise SystemExit("manifest hash changed; refuse the content diff")
    if not current["content_diff_allowed"]:
        raise SystemExit("operational mix too high; content diff suppressed")
    prev = previous["content_rate"]
    cur = current["content_rate"]
    if prev is None or cur is None:
        raise SystemExit("no judged content rows; nothing to compare")
    return f"content_rate {prev:.3f} -> {cur:.3f}"


def demo() -> None:
    expect = {"priority": "p2", "queue": "billing"}
    prompt = "Return JSON with priority and queue."
    cases = [
        Case("ticket-14", prompt, expect),
        Case("ticket-15", prompt, expect),
        Case("ticket-16", prompt, expect),
    ]
    saved = {
        "ticket-14": Completion('{"priority": "p2"', "length"),
        "ticket-15": Completion(
            '{"priority": "p1", "queue": "billing"}', "stop"
        ),
        "ticket-16": Completion("", "refusal"),
    }
    rows = [classify(case, saved[case.case_id]) for case in cases]
    print(json.dumps(summarize(cases, rows), indent=2))


if __name__ == "__main__":
    demo()

A content rate that ignores truncated rows will look healthier than the raw pass rate on the same run. That healthier figure is acceptable only when the same record also publishes the blocked count beside it. The helper refuses a content diff when fewer than eight cases were judged, or when non-content verdicts exceed one fifth of the run. Those thresholds are policy choices in this sketch, not measured optima, so retune them against your own manifest size.

The demo at the bottom of the module is also unexecuted in this draft, and its expected result follows from reading the branches. By inspection, one row is truncated, one is refused, and one is wrong, so the judged set contains only the wrong answer. The content rate on that judged set is zero, and the diff flag stays false because three rows cannot satisfy a minimum of eight. That closed flag is the desired outcome of the demo, and it is not a measurement of any hosted model.

A missing tray is not an empty tray

A postal scale that records a missing reading as zero kilograms will announce a famine every time the truck is late. The typed verdict is the practical difference between an empty tray and a tray that never reached the scale. A support-ticket case that expects priority p2 and queue billing is enough structure to show that split without a large fixture. When finish equals length and the object is only half written, the classifier returns truncated and that row stays outside the content rate.

A later clean completion that returns priority p1 for the same queue is wrong, and that row does enter the rate. Those two outcomes must not share a cell, or the weekly chart will narrate a quality drop that was a cut-off. Keep the raw text, the finish flag, and the case id in the incident file so a reviewer can confirm the class. Rerunning immediately would mix a new sample into the evidence, which is a different question from how this sample was graded.

Lock the classes offline

Before any nightly model call, lock the verdict taxonomy with saved completions that never leave the test process. A three-line fixture for truncated, refused, and wrong is enough to prove the classes do not collapse under a parser refactor. The proposed test below is unexecuted here, and it should fail if a later edit maps length onto wrong. Once that test is green, the nightly runner may call a model, but it must reuse the same classify function the test already pinned.

from verdicts import Case, Completion, Verdict, classify


def test_length_is_not_a_content_miss() -> None:
    case = Case(
        "ticket-14",
        "Return JSON with priority and queue.",
        {"priority": "p2", "queue": "billing"},
    )
    assert classify(case, Completion('{"priority": "p2"', "length")) is Verdict.TRUNCATED
    assert (
        classify(case, Completion('{"priority": "p1", "queue": "billing"}', "stop"))
        is Verdict.WRONG
    )
    assert classify(case, Completion("", "refusal")) is Verdict.REFUSED
    assert classify(case, Completion("not-json", "stop")) is Verdict.MALFORMED
    assert classify(case, Completion("", "stop", error="reset")) is Verdict.TRANSPORT

[
  {
    "id": "ticket-14",
    "prompt": "Return JSON with priority and queue.",
    "expect": {"priority": "p2", "queue": "billing"}
  }
]

python -m venv .venv
. .venv/bin/activate
python -m pip install pytest
python -m pytest tests/test_verdict_class.py -q
python verdicts.py

From a clean virtual environment, install only the test runner, then execute the taxonomy tests before you wire any client. The commands below match the two files in this draft, and they do not call a hosted model. A later wrapper around summarize and diff_content should refuse to start when the manifest hash cannot be computed. That wrapper should also exit non-zero when the hashes differ or the newer content-diff flag is false.

What a nightly record may claim

The run record is one JSON object with a manifest hash, class counts, a content rate, and a flag for the content diff. The hash binds expected JSON to the run, so a quiet edit to a golden case cannot look like a model change. Class counts sit beside the rate because a high rate can still hide a night when one fifth of the rows never finished. If the flag is false, keep the previous content series and store the new rows as an incident beside that series.

A refusal is closer to a policy outcome than to a wrong field, and folding it into the content rate punishes a declined answer. Transport errors sit on the path between processes, and they should not be narrated as text the model actually produced. Malformed JSON after a clean stop is the ambiguous middle, because the model finished and still broke the contract you requested. This sketch files that middle as malformed rather than wrong, so a format break stays visible when you later tighten the prompt.

Limits

This sketch assumes the provider tells you why the completion ended, and many clients simply never send that flag. If finish is missing, you cannot honestly separate truncation from malformed text, and you should label the row unknown rather than guess. Exact JSON equality is a narrow grader, because free-text drift will look wrong even when a human would accept it. Do not use this harness for open-ended essays, legal advice, or any case where partial credit is the real product requirement.

A free model lane can change how often responses are cut off without changing the answers that finish. That shift in mix is why the blocked share must gate the content diff rather than the incident log. That gate is not a benchmark of the vendor, and this draft records no latency, cost, or accuracy numbers. If your client cannot return a finish reason, or your cases are prose rather than a small JSON contract, skip this pattern.

If free model access and a free server are already on your MonkeyCode account, that pair can carry this nightly job. Keep the grading script in your own repo, and read the finish flag before you read any score. Nothing in this note claims a quota, a model name, or a permanent offer, because those facts were not supplied here as primary sources. The comparison worth keeping is the content rate beside the class counts, for one pinned manifest, on one pinned endpoint.

Do Not Diff a Score Across a Model Stamp

Dakota Ma — Sat, 10 Oct 2026 19:18:45 +0000

A score delta is not evidence of a prompt change when the model stamp or the server revision also moved. A nightly harness that calls a replaceable model endpoint can mix those variables without editing a single prompt line. The recorded run should refuse that comparison until the model stamp and the server revision both match the stored baseline. That refusal is the regression signal, and it should surface before any grader assertion is allowed to run.

Teams often treat a free model route as a drop-in substitute for yesterday's candidate, then read the new mean as quality. The substitution is cheap, so it happens during incident reviews, quota pauses, and routine host maintenance. A lower score can be a real prompt regression, a weaker model, or a server image that changed tokenization. Without an identity stamp, those three causes collapse into one number and leave the on-call engineer with no stack trace.

Earlier checks already pin the fixture, the grader, and the case slot before a diff is trusted. Those pins stop a moving rubric from rewriting history, but they do not name the model that produced the completion. A fixture hash can stay stable while the endpoint silently routes to a different weight set. The missing field is the model stamp, stored beside the fixture hash rather than inferred from the score.

A record that can refuse the diff

The practical fix is a three-part record that travels with every nightly run and every manual rerun. A case manifest lists golden inputs, expected envelopes, and a content hash computed over the canonical file bytes. A candidate adapter returns the completion plus the model identifier and server revision reported by the host. A quarantine step computes a score delta only when the manifest hash, model stamp, and server revision all match the baseline row.

Think of the stamp as the lot number on a reagent bottle in a wet lab. Two assays can share one protocol card and still disagree because the reagent bottle was replaced overnight. You would not publish that disagreement as a method improvement until both lot numbers match the notebook. A prompt eval deserves the same courtesy, because a free host can rotate bottles without notifying the scoreboard.

The manifest shown below is a proposal artifact, not a captured production trace from any live evaluation host. Each case carries an input the model may see and an expected envelope the grader may see only after the call returns. Keeping the gold out of the prompt is a separate control, and this note does not reopen that argument. The new requirement is that the file bytes are hashed, so a quiet edit to an expected field cannot hide inside a model swap.

{
  "suite": "envelope-v3",
  "cases": [
    {
      "id": "refund-status",
      "input": {"ticket": "T-1842", "ask": "status"},
      "expected": {"tool": "lookup_ticket", "args": {"id": "T-1842"}}
    }
  ]
}

The adapter contract stays small enough that a reviewer can reimplement the same checks without adopting a new framework. A fabricated stamp would make unlike runs look comparable, so an omitted host identifier must remain unknown. Unknown is a failed quarantine rather than a silent default that pretends yesterday's model is still attached. The following snippet is unexecuted proposal code, and it only hashes local files without placing a network call.

#!/usr/bin/env python3
"""Proposal harness: refuse score diffs when stamps differ. Not a measured run."""
import hashlib
import json
import sys
from pathlib import Path

STAMP_FIELDS = ("manifest_hash", "model_stamp", "server_revision")

def file_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def grade_envelope(completion: dict, expected: dict) -> dict:
    missing = [key for key in expected if key not in completion]
    wrong = [
        key
        for key in expected
        if key in completion and completion[key] != expected[key]
    ]
    return {
        "passed": not missing and not wrong,
        "missing": missing,
        "wrong": wrong,
    }

def quarantine(baseline: dict, candidate: dict) -> dict:
    reasons = [
        field
        for field in STAMP_FIELDS
        if baseline.get(field) != candidate.get(field)
    ]
    if baseline.get("passed") not in (0, 1) or candidate.get("passed") not in (0, 1):
        reasons.append("passed")
    if reasons:
        return {"comparable": False, "reasons": reasons, "delta": None}
    return {
        "comparable": True,
        "reasons": [],
        "delta": int(candidate["passed"]) - int(baseline["passed"]),
    }

def load_run(path: Path) -> dict:
    record = json.loads(path.read_text())
    for field in ("model_stamp", "server_revision"):
        if field not in record or record[field] in ("", None):
            record[field] = "unknown"
    return record

def main() -> int:
    if len(sys.argv) != 4:
        print(
            "usage: quarantine_diff.py MANIFEST BASELINE CANDIDATE",
            file=sys.stderr,
        )
        return 64
    manifest = Path(sys.argv[1])
    baseline = load_run(Path(sys.argv[2]))
    candidate = load_run(Path(sys.argv[3]))
    candidate["manifest_hash"] = file_hash(manifest)
    decision = quarantine(baseline, candidate)
    print(json.dumps(decision, indent=2))
    return 0 if decision["comparable"] else 2

if __name__ == "__main__":
    raise SystemExit(main())

A local check uses two fixture files that differ only in the recorded model stamp and in nothing else. The command below should exit with status 2 when the quarantine fires on that cross-model pair. Status 0 would mean the three stamp fields matched and a numeric delta was printed to standard output. Neither exit status is a claim that the prompt itself improved, regressed, or stayed semantically equal.

python3 quarantine_diff.py cases/manifest.json runs/baseline.json runs/swapped_model.json
echo "exit=$?"
python3 -c "import quarantine_diff as q; d=q.quarantine({'manifest_hash':'a','model_stamp':'m1','server_revision':'r1','passed':1},{'manifest_hash':'a','model_stamp':'m2','server_revision':'r1','passed':1}); assert d['delta'] is None; print(d)"

A baseline row that survived review can look like the object below, with passed stored as an integer. Storing passed as an integer keeps the later delta a subtraction rather than a paragraph of judgment. The model stamp is an opaque string copied from the host response, not a nickname chosen by the harness. If the host later returns a different string for the same marketing name, quarantine treats that string as a new lot.

{
  "manifest_hash": "recorded-sha256",
  "model_stamp": "host-reported-id",
  "server_revision": "image-digest",
  "passed": 1
}

Suppose the baseline passed flag is 1 and the candidate passed flag is also 1, while the model stamp changes. The naive subtraction of those flags would be zero, which looks like stability to anyone watching a chart. Quarantine returns a null delta instead, because a matched score across different stamps is not evidence of stability. A displayed zero would have been the most misleading result this particular harness is able to produce.

When the candidate keeps the manifest hash but changes model_stamp, the printed decision names that field and sets delta to null. That null has to survive the dashboard, because a coercion from null to zero hides the quarantine and recreates the bug. Store the reasons list beside the null, and alert when comparable is false rather than when a score crosses a threshold. A threshold alert assumes the runs were comparable, which is the assumption this harness exists to test.

The same refusal applies when only the server revision moves and the model marketing string stays put. A new image digest can change a tokenizer, a timeout, or a tool schema while the advertised model name remains constant. The revision field exists so that class of change cannot masquerade as an edit to the prompt. Host cost is a separate axis, and a cheaper place to run the adapter does not freeze identity.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is described by the operator as an open-source project that offers free model access and a free server option. Those two availability claims are operator-supplied, and they matter because the candidate adapter needs a host cheap enough for a nightly pass. Exact token allowances, model identifiers, hardware, and offer duration are omitted, because no primary document was attached to this draft.

The proposal script returns status 0 only when the three stamp fields match and a numeric delta was printed. Status 2 means the quarantine fired, which is a successful detection rather than an unexpected process crash. A wrapper should treat status 2 as a page, and it should treat a missing input file as a separate operational error. Mixing those two outcomes inside one alert rule will train the on-call rotation to ignore the page.

Silent regressions are especially likely when the score stays flat, because a flat chart does not summon a reviewer. A model swap that preserves the pass count is invisible unless the stamp comparison runs before the chart is drawn. That is why the decision object carries a reasons list even when both passed flags happen to be identical. The chart can still be drawn later, but only from rows that the quarantine has already marked comparable.

A useful nightly flow keeps four checks in a fixed order, and none of them is a slogan about model quality. Hash the manifest first, and refuse the run when that hash is absent from the baseline index. Call the candidate adapter second, and persist the host-reported model stamp and server revision even when the completion fails. Grade the stored completion third, then run the quarantine and page a human when the runs are not comparable.

Those checks matter more than the adapter language, and a shell wrapper can call the same decision after a manual replay. Write each candidate record to a new path, and promote it only after a person accepts both stamps and the delta. Automation may append run files, but automation should not bless a new lot number on its own. A person remains the only step that is allowed to replace the reference row after reading both stamps.

A content hash of raw file bytes will change when an editor rewrites newlines or reorders keys inside the manifest. That sensitivity is intentional, because a quiet formatting pass can still alter an expected field a human skips. If stable hashes across machines matter, canonicalize the JSON before hashing and store those canonical bytes. The proposal script hashes the file as saved, so different line endings will quarantine until the bytes match.

What an exact envelope does not prove

Exact envelope equality marks a fine paraphrase as a failure, and it also misses a harmful completion that still carries the expected keys. This approach is the wrong tool for open-ended writing quality, preference ranking, or any grader that needs a second model as judge. It is the wrong tool when the host cannot report a stable identifier, since every run would quarantine and the page would become noise. Teams without a frozen case manifest should build that file first, since a stamp on a moving case set still produces an unreadable delta.

Teams that already diff inside one pinned model can add the two stamp fields without rebuilding the suite. Backfill the string unknown on old rows, and treat unknown as non-comparable until a fresh baseline is recorded under observation. Do not backfill a guessed marketing name from memory, because a guessed stamp creates false permission to compare. A missing stamp is an honest block, while a guessed stamp is a quiet way to reintroduce the original bug.

If current records already store a host identifier, compare two saved rows and see whether a server move would have changed that string. One comparison is enough to tell whether the quarantine would have fired on last week's nightly job. The check does not require a new rubric, a new vendor, or any claim about which model is better. It only asks whether the score already on the chart was earned under a single lot number.