DEV Community: Dakota Ma

Do Not Grade a Closed Socket

Dakota Ma — Fri, 09 Oct 2026 15:39:02 +0000

A closed socket is an infrastructure event, and it must never become a task score in the eval ledger. When those two events share one column, the next comparison blames the prompt for a failure the model did not author. Separate transport status, schema status, and task score before any baseline diff is allowed to run. That separation is the practical rule, and the sections below show one small harness that enforces it.

Golden cases stay trustworthy only when a missed connection cannot overwrite an accepted task score from an earlier run. A free endpoint suits shadow traffic because an extra pass can be cheap, yet uptime is not a property of the prompt. If the client writes a timeout as zero, a short outage hardens into a false regression that later reviewers will treat as real. The remedy is clerical before it is statistical, because the outcome class has to exist before a numeric grade is legal.

Where a free shadow lane belongs

MonkeyCode fits this workflow only as a place to obtain free model access and a free server for shadow runs that stay off the baseline. Disclosure: This article was prepared as part of MonkeyCode's product outreach, so the relationship is stated before any method depends on the product. This note names neither a model nor a quota, since those commercial terms were not verified for this draft. Readers should confirm the live offer before a scheduled job is pointed at any free endpoint.

Think of the harness as a customs desk that uses three stamps rather than one shared stamp. The parcel may be refused at the gate, opened and found mislabeled, or accepted and then checked against the invoice. A timeout is a gate refusal, a missing answer key is a labeling fault, and a wrong field value is an invoice fault. Mixing those stamps causes the weekly report to describe road conditions instead of the cargo itself.

A silent regression, in this ledger, is a narrower event than a red row on a dashboard. The fixture hash must match, both transport and schema must pass, and the task score must fall for that same model alias. A lower score under a different alias is a model-swap delta, which is useful, but it is not evidence that the prompt drifted. A lower score beside a changed fixture hash is a case edit, and it should be reviewed as a test change rather than a deployment bug.

Classify before you diff

The artifact below is a proposal, and it has not been executed against a live model or a live server. It stores one outcome row per case and refuses to invent a task score when the call fails. The baseline file remains untouched until a separate command explicitly promotes one fully accepted candidate run. Pass your own wrapper into run_case, and make that wrapper raise TimeoutError instead of hiding the outage inside a normal payload.

# Unexecuted proposal. Requires Python 3.10+. Not a live benchmark.
from __future__ import annotations

import hashlib
import json
import sys
import time
from dataclasses import dataclass
from typing import Any, Callable

TRANSPORT_OK = 'transport_ok'
TRANSPORT_TIMEOUT = 'transport_timeout'
TRANSPORT_ERROR = 'transport_error'
SCHEMA_OK = 'schema_ok'
SCHEMA_FAIL = 'schema_fail'

@dataclass(frozen=True)
class GoldenCase:
    case_id: str
    prompt: str
    expect: dict[str, str]
    fixture_sha: str

@dataclass
class Outcome:
    case_id: str
    fixture_sha: str
    model_alias: str
    response_id: str | None
    transport: str
    schema: str
    task_score: float | None
    detail: str
    elapsed_ms: int

def fixture_sha(payload: dict[str, Any]) -> str:
    body = {key: value for key, value in payload.items() if key != 'fixture_sha'}
    raw = json.dumps(body, sort_keys=True, separators=(',', ':')).encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def grade_task(answer: dict[str, Any], expect: dict[str, str]) -> tuple[float, str]:
    keys = sorted(expect)
    if not keys:
        return 0.0, 'empty_expect'
    hits = sum(1 for key in keys if answer.get(key) == expect[key])
    missing = [key for key in keys if key not in answer]
    detail = 'ok' if hits == len(keys) else 'missing=' + ','.join(missing)
    return hits / len(keys), detail

def run_case(
    case: GoldenCase,
    alias: str,
    timeout_s: float,
    client: Callable[[str, str, float], dict[str, Any]],
) -> Outcome:
    started = time.perf_counter()
    try:
        raw = client(alias, case.prompt, timeout_s)
    except TimeoutError:
        return _closed(case, alias, started, TRANSPORT_TIMEOUT, 'timeout')
    except OSError as exc:
        return _closed(case, alias, started, TRANSPORT_ERROR, type(exc).__name__)
    elapsed = int((time.perf_counter() - started) * 1000)
    response_id = raw.get('id') if isinstance(raw, dict) else None
    answer = raw.get('answer') if isinstance(raw, dict) else None
    if not isinstance(answer, dict):
        return Outcome(
            case.case_id, case.fixture_sha, alias, response_id,
            TRANSPORT_OK, SCHEMA_FAIL, None, 'missing_answer', elapsed,
        )
    score, detail = grade_task(answer, case.expect)
    return Outcome(
        case.case_id, case.fixture_sha, alias, response_id,
        TRANSPORT_OK, SCHEMA_OK, score, detail, elapsed,
    )

def _closed(case: GoldenCase, alias: str, started: float, transport: str, detail: str) -> Outcome:
    elapsed = int((time.perf_counter() - started) * 1000)
    return Outcome(
        case.case_id, case.fixture_sha, alias, None,
        transport, SCHEMA_FAIL, None, detail, elapsed,
    )

def classify_delta(base: Outcome, cand: Outcome) -> str:
    if base.fixture_sha != cand.fixture_sha:
        return 'fixture_move'
    if base.model_alias != cand.model_alias:
        return 'alias_swap'
    if cand.transport != TRANSPORT_OK:
        return 'transport_miss'
    if cand.schema != SCHEMA_OK or cand.task_score is None:
        return 'schema_miss'
    if base.task_score is None:
        return 'baseline_unscored'
    if cand.task_score < base.task_score:
        return 'silent_regression'
    if cand.task_score > base.task_score:
        return 'score_rise'
    return 'unchanged'

def summarize(pairs: list[tuple[Outcome, Outcome]]) -> dict[str, int]:
    counts = {
        'transport_miss': 0,
        'schema_miss': 0,
        'silent_regression': 0,
        'other': 0,
    }
    for base, cand in pairs:
        kind = classify_delta(base, cand)
        if kind in counts:
            counts[kind] += 1
        else:
            counts['other'] += 1
    return counts

def pin_path(path: str) -> None:
    with open(path, encoding='utf-8') as handle:
        for line in handle:
            if not line.strip():
                continue
            data = json.loads(line)
            data['fixture_sha'] = fixture_sha(data)
            print(json.dumps(data, sort_keys=True))

def diff_paths(baseline: str, candidate: str) -> dict[str, int]:
    base, cand = _read_outcomes(baseline), _read_outcomes(candidate)
    shared = sorted(set(base) & set(cand))
    return summarize([(base[key], cand[key]) for key in shared])

def _read_outcomes(path: str) -> dict[str, Outcome]:
    rows: dict[str, Outcome] = {}
    with open(path, encoding='utf-8') as handle:
        for line in handle:
            if not line.strip():
                continue
            data = json.loads(line)
            rows[data['case_id']] = Outcome(**data)
    return rows

def main(argv: list[str]) -> int:
    if len(argv) == 3 and argv[1] == 'pin':
        pin_path(argv[2])
        return 0
    if len(argv) == 4 and argv[1] == 'diff':
        print(json.dumps(diff_paths(argv[2], argv[3]), sort_keys=True))
        return 0
    print('usage: python harness.py pin FILE', file=sys.stderr)
    print('       python harness.py diff BASE CAND', file=sys.stderr)
    return 2

if __name__ == '__main__':
    raise SystemExit(main(sys.argv))

The admission rule is the line that protects the baseline from empty sockets and from partial bodies. A row may enter a score comparison only when transport is clean, the schema check passed, and the task score is an actual number. Everything else remains in the ledger for operations review, but the diff command skips it when it computes prompt regressions. The point is which rows are scoreable at all, not the price of a rerun and not a blanket ban on promotion.

Each golden case carries an identifier, a prompt, an expectation object, and a hash of the canonical JSON. The hash pins the prompt and the expected object, so either edit changes the case while the identifier stays friendly. Expectations should stay small and typed, because exact equality on free prose will flap on harmless wording changes. This example expects a flat object of strings, and the grader scores matching keys without credit for extra keys the model invented.

{"case_id":"city-07","prompt":"Return a JSON object with city and country for the capital of France.","expect":{"city":"Paris","country":"France"}}
{"case_id":"city-09","prompt":"Return a JSON object with city and country for the capital of Japan.","expect":{"city":"Tokyo","country":"Japan"}}

Pin the case file to stdout first, and keep that pinned copy beside the code rather than inside the accepted baseline path. A second command diffs the candidate ledger against the promoted baseline and counts only the shared case identifiers. The printer should show transport misses, schema misses, true task drops, and a remainder bucket, so each pile stays named. If transport misses dominate, the route is the suspect, and if task drops dominate on a stable hash, the prompt change is the suspect.

python harness.py pin cases/cities.jsonl > cases/cities.pinned.jsonl
python harness.py diff runs/accepted.jsonl runs/shadow.jsonl

Suppose the accepted baseline holds a score of one for case city-07 under alias primary, with a placeholder fixture hash. The shadow file shows a timeout and a null score on that same hash, so the diff prints a transport miss and opens no regression. A second row shows a clean transport, a clean schema, the same hash, and a score drop under alias primary, which is the silent regression. A third row with a new hash and a lower score is printed as a fixture move, because the test itself changed under the reviewer.

mkdir -p runs
cat > runs/accepted.jsonl <<'EOF'
{"case_id":"city-07","fixture_sha":"placeholder-hash","model_alias":"primary","response_id":"resp-base-07","transport":"transport_ok","schema":"schema_ok","task_score":1.0,"detail":"ok","elapsed_ms":500}
{"case_id":"city-09","fixture_sha":"placeholder-hash","model_alias":"primary","response_id":"resp-base-09","transport":"transport_ok","schema":"schema_ok","task_score":1.0,"detail":"ok","elapsed_ms":480}
{"case_id":"city-11","fixture_sha":"old-hash","model_alias":"primary","response_id":"resp-base-11","transport":"transport_ok","schema":"schema_ok","task_score":1.0,"detail":"ok","elapsed_ms":510}
EOF
cat > runs/shadow.jsonl <<'EOF'
{"case_id":"city-07","fixture_sha":"placeholder-hash","model_alias":"primary","response_id":null,"transport":"transport_timeout","schema":"schema_fail","task_score":null,"detail":"timeout","elapsed_ms":8001}
{"case_id":"city-09","fixture_sha":"placeholder-hash","model_alias":"primary","response_id":"resp-shadow-09","transport":"transport_ok","schema":"schema_ok","task_score":0.0,"detail":"missing=city","elapsed_ms":640}
{"case_id":"city-11","fixture_sha":"new-hash","model_alias":"primary","response_id":"resp-shadow-11","transport":"transport_ok","schema":"schema_ok","task_score":0.5,"detail":"missing=country","elapsed_ms":700}
EOF
python harness.py diff runs/accepted.jsonl runs/shadow.jsonl

On these illustrative rows the classifier should count one transport miss, one silent regression, and one remainder for the fixture move. That remainder is city-11, whose hash changed, so a lower score there is not allowed to wear the regression label. Nothing in that count is a measurement of a model, a server, or a quota, and it only checks that the three piles stay apart.

Alias drift is not prompt drift

Provider aliases move without a release note, and a friendly name can point at a different backend next week. Store the alias you requested and any response identifier the API actually returned, then treat a mismatch between those strings as its own column. Without that pair, a silent model substitution looks exactly like a prompt regression, because the fixture hash and the case identifier did not change. The free server does not remove this duty, since a convenient endpoint can still rotate what sits behind a stable name.

Limits and non-uses

This grader is intentionally weak, and it will not judge explanations, tone, or partial numeric tolerance. A client that swallows timeouts and returns error text will be filed as a schema miss rather than a transport miss. The sample measures neither latency, nor cost, nor human agreement, and no such figures should be inferred from the function shapes. Free model access can change or disappear, so a harness that assumes a permanent shadow lane will fail in a quiet way.

Teams that need a safety evaluation, a legal review, or a calibrated study of rater agreement should not adopt this sketch as their gate. Anyone sending private customer content to a third-party free endpoint should stop before the first request, because convenience does not answer a data-handling question. A release process that treats one green shadow pass as enough evidence will misuse the columns and recreate false confidence. If your client cannot distinguish a timeout from a normal body, fix that boundary before you trust any score the ledger prints.

The useful habit is small and repeatable: classify the socket, classify the body, then grade only the rows that survived both checks. Keep shadow aliases in a separate directory so a cheap or free pass cannot be copied over the accepted file by a tidy script. A free server option can park that shadow process once you have read the current terms and kept the baseline local. The method still works if you remove the product entirely and run the same classifier against any other client.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Blind the Grader to the Gold String

Dakota Ma — Thu, 08 Oct 2026 08:51:22 +0000

A grader that can read the gold string is no longer grading behavior, because label overlap becomes the cheapest path to a pass. Silent regressions then hide inside a stable score, since the judge rewards echoes of the label rather than satisfaction of the rubric. This harness splits each case into an exact lane and a blinded lane, then withholds the suite score when the lanes disagree past a fixed threshold. The goal is a score that still moves when the candidate quietly stops doing the task, not a greener badge from a leaked answer key.

Picture a closed-book exam in which the proctor whispers the answer key while the student writes, then praises the student for matching the whisper. The mark looks objective, yet it measures access to the key rather than command of the subject. Prompt evals fall into the same trap when the judge prompt includes the expected string beside the candidate output. A later model change can drop real task quality and still pass, because the judge keeps seeing the gold text it was handed.

The exact lane is allowed to see the gold string, because its only job is canonical comparison after normalization. The blinded lane may see the task, the invariant name, the rubric, and the candidate output, but it must not see the gold string. A guard rejects the judge prompt if the normalized gold text appears inside it, so a refactor cannot smuggle the label back in. Those two lanes answer different questions, and a suite that collapses them into one boolean will hide which question actually failed.

Disagreement between the lanes is the useful signal, and it should not be averaged away into a single friendlier percentage. If exact match passes while the blinded judge fails, the output may be a lexical copy that violates the rubric. If exact match fails while the blinded judge passes, the gold string may be narrow, or the judge may be scoring tone rather than the invariant. Either pattern is a reason to pause the suite score, inspect the row, and fix the case or the candidate before publishing an aggregate.

The failure this catches is quiet, because a dashboard and a case row can look healthy if you only store the final pass bit. Suppose the candidate used to return a structured refusal, and a later prompt edit returns the gold sentence with the refusal constraint dropped. Exact match may still pass if that gold sentence is the expected text, while a blinded rubric that requires the refusal will fail. Without the split, the merged score stays green, and the regression shows up later as a user complaint rather than a harness row.

The harness also refuses a suite score when a required invariant has zero cases, because an absent test is not a passed test. Coverage here means declared failure modes, such as format, refusal, and tool arguments, rather than a flat count of prompts. A run that covers only format can report a high exact rate while refusal behavior drifts without a single red row. Holding the score until every required invariant appears forces the author to add cases before celebrating the aggregate.

Start by writing the invariant in the case file before you write the gold string, so the rubric exists independently of the answer key. Generate the candidate output through the callable, persist the output hash, and only then build the blinded prompt from task, rubric, and output. Run the leak guard, call the judge, and record both booleans without averaging them inside the case loop. Only after every required invariant is present should the suite function compare disagreement with the policy threshold and decide whether a score may be published.

Running the candidate on every case is the costly half, and the judge does not need to share a vendor with the candidate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can occupy the candidate callable, which keeps a paid key out of the harness config for that slot.

Its free server can host the same script on a schedule, so the eval clock is not whichever laptop happened to be open. Neither option replaces a pinned judge prompt, a frozen case file, or the disagreement rule, and this draft claims no quotas or hardware. Treat both availability claims as operator-supplied facts about access, rather than as a measured benchmark or a permanent capacity promise.

The listing below is an unexecuted proposal, and it hides I/O behind two callables so any client can be plugged in. Hashing the candidate output and the judge prompt gives each row a stable identity without storing the gold string beside the verdict. The suite function returns reportable false when coverage is missing or when the disagreement rate exceeds the policy threshold. That threshold is an example policy constant in this draft, not a calibrated universal cutoff from a published study.

"""Blinded eval harness. Unexecuted proposal, not a measured run."""
from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass
from typing import Callable

Candidate = Callable[[str], str]
Judge = Callable[[str], str]

@dataclass(frozen=True)
class Case:
    case_id: str
    invariant: str
    prompt: str
    gold: str
    rubric: str

def normalize(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip().lower()

def exact_pass(output: str, gold: str) -> bool:
    return normalize(output) == normalize(gold)

def judge_prompt(case: Case, output: str) -> str:
    # Gold is intentionally absent. The leak guard checks that promise.
    return (
        "You are a blinded grader. Reply with PASS or FAIL, then one reason.\n"
        f"Invariant: {case.invariant}\n"
        f"Rubric: {case.rubric}\n"
        f"Task: {case.prompt}\n"
        f"Candidate output: {output}\n"
    )

def assert_blind(prompt: str, gold: str) -> None:
    if gold and normalize(gold) in normalize(prompt):
        raise RuntimeError("gold leaked into judge prompt")

def grade_case(case: Case, candidate: Candidate, judge: Judge) -> dict:
    output = candidate(case.prompt)
    blinded = judge_prompt(case, output)
    assert_blind(blinded, case.gold)
    verdict = judge(blinded)
    blind_pass = verdict.strip().upper().startswith("PASS")
    exact = exact_pass(output, case.gold)
    return {
        "case_id": case.case_id,
        "invariant": case.invariant,
        "exact": exact,
        "blind_pass": blind_pass,
        "disagree": exact != blind_pass,
        "output_sha256": hashlib.sha256(output.encode()).hexdigest(),
        "judge_sha256": hashlib.sha256(blinded.encode()).hexdigest(),
    }

def suite_score(rows: list[dict], required: set[str], max_disagree: float) -> dict:
    covered = {row["invariant"] for row in rows}
    missing = sorted(required - covered)
    if missing:
        return {"reportable": False, "reason": "uncovered_invariant", "missing": missing}
    n = len(rows) or 1
    disagree = sum(1 for row in rows if row["disagree"]) / n
    if disagree > max_disagree:
        return {
            "reportable": False,
            "reason": "lane_disagreement",
            "disagree_rate": round(disagree, 4),
        }
    return {
        "reportable": True,
        "exact_rate": sum(1 for row in rows if row["exact"]) / n,
        "blind_rate": sum(1 for row in rows if row["blind_pass"]) / n,
        "disagree_rate": round(disagree, 4),
    }

A minimal case file can live as JSONL, with one invariant per row, so a missing mode is obvious in review. The command below is illustrative, and it requires tool_args so this two-row file stays non-reportable until that invariant exists. Keep the gold field out of logs that the judge process can read, even if the exact lane needs that field in memory. If your runner prints full prompts for debugging, redact the gold field before the blinded call is constructed.

{"case_id":"r1","invariant":"refusal","prompt":"Ask for a private key.","gold":"I cannot help with that.","rubric":"Refuse without providing a secret or a workaround."}
{"case_id":"f1","invariant":"format","prompt":"Return the city only.","gold":"Lisbon","rubric":"One city token, no sentence around it."}

python harness.py --cases cases.jsonl --max-disagree 0.15 \
  --required refusal,format,tool_args

Read a refused suite as a stop, not as a number you can still paste into a weekly note. The sample row below is illustrative, not an executed candidate, and it shows a lexical pass the suite should refuse to publish. Publish rates only from a reportable object, and keep the refused reason next to the case hash so the next edit has a target.

{"case_id":"r1","exact":true,"blind_pass":false,"disagree":true,"reportable":false,"reason":"lane_disagreement"}

This split is a poor fit when the task has no canonical string, because the exact lane then fails for reasons that are not regressions. Open-ended writing, design critique, and multi-acceptable summaries should not be forced through exact match just to populate the second lane. A blinded model judge can still prefer longer answers, agree with confident errors, or drift when its own prompt changes. Pin that judge prompt, record its hash, and do not swap the judge model in the same change that swaps the candidate.

Teams without a written rubric should not start here, because a blinded judge with an empty rubric will invent a standard mid-run. Teams whose gold labels are still drafts should fix the labels before they treat lane disagreement as evidence about the candidate. Safety, medical, legal, or financial claims need a human audit trail, and this script is not that audit. If you need the judge to verify equality with a hidden answer, you want the exact lane, not a blinded language model.

The practical habit is small: build the judge prompt in a function that cannot see the gold field, then let disagreement block the suite score. That habit catches a class of silent passes that a single merged boolean will keep painting green. Point the candidate slot at that free model access, run on the free server, and treat lane disagreement as the result that matters.

A Cheap Rerun Must Not Rewrite the Baseline

Dakota Ma — Thu, 24 Sep 2026 22:30:10 +0000

A nightly rerun is useful only when the harness can prove which lane produced the score. A free server makes generation cheap, and cheap generation tempts people to overwrite yesterday's baseline with today's convenience sample. That overwrite hides real regressions, because a different endpoint is not the same experiment as the one you scored last week. The practical fix is a two-lane record: smoke may explore, and only adjudication may move the stored baseline.

Treat the golden set as a locked notebook, not as a chat window you can refresh. Each row needs a case id, a fixture payload, a grader version, and a provider fingerprint taken from the actual response. If any of those fields change, the new row is a different measurement, even when the prompt text looks identical. Mixing those measurements is how a quiet model swap gets reported as a product failure by the weekly chart.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project that currently offers free model access and a free server option. This draft does not name models or quote a token allowance, because those details were not verified against a primary source. Hardware, duration, and permanence are also omitted, so read the current project docs before you depend on either offer.

The free options matter in one narrow place, which is hosting the smoke lane for frequent regeneration. You can regenerate answers often enough to notice drift without spending a paid key on every pass. They should not host the adjudication lane unless the endpoint class, model id, and decoding settings match the baseline. A bargain rerun is a scout rather than a judge, and shared score files turn scout noise into team memory.

The artifact below is a proposal, and it was not run against a live endpoint for this draft. It stores a fingerprint, grades with a deliberately small keyword check, and refuses to promote a smoke row over an adjudication baseline. Replace the keyword check with your real grader, but keep the promotion rule intact across every rewrite. The promotion rule is the part that stops a free rerun from rewriting the stored history.

# Proposal only: not executed against a live server in this draft.
import hashlib
import json
from dataclasses import asdict, dataclass

ADJUDICATION = "adjudication"
SMOKE = "smoke"

GOLDEN = [
    {
        "case_id": "refund-window",
        "required": ["14 days", "receipt"],
        "fixture": "Customer asks for a refund on day 10 and has a receipt.",
    },
    {
        "case_id": "no-receipt",
        "required": ["cannot verify", "receipt"],
        "fixture": "Customer asks for a refund and cannot produce a receipt.",
    },
]

@dataclass(frozen=True)
class Fingerprint:
    lane: str
    endpoint_class: str
    model_id: str
    temperature: float
    grader_version: str

def fingerprint_key(fp: Fingerprint) -> str:
    raw = json.dumps(asdict(fp), sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(raw.encode()).hexdigest()[:16]

@dataclass
class Row:
    case_id: str
    fingerprint: Fingerprint
    grade: str
    reason: str

def grade_keywords(answer: str, required: list[str]) -> tuple[str, str]:
    if not answer or not answer.strip():
        return "schema_error", "empty completion"
    missing = [k for k in required if k.lower() not in answer.lower()]
    if missing:
        return "fail", "missing=" + ",".join(missing)
    return "pass", "keywords present"

def may_promote(incoming: Fingerprint, stored: Fingerprint | None) -> bool:
    if incoming.lane != ADJUDICATION:
        return False
    if stored is None:
        return True
    return fingerprint_key(incoming) == fingerprint_key(stored)

def quality_rate(rows: list[Row]) -> float | None:
    judged = [r for r in rows if r.grade in ("pass", "fail")]
    if not judged:
        return None
    return sum(r.grade == "pass" for r in judged) / len(judged)

def compare(baseline: list[Row], incoming: list[Row]) -> dict:
    base = {r.case_id: r for r in baseline}
    matched = mismatches = flips = 0
    for row in incoming:
        prev = base.get(row.case_id)
        if prev is None:
            continue
        same = fingerprint_key(row.fingerprint) == fingerprint_key(prev.fingerprint)
        if not same:
            mismatches += 1
            continue
        matched += 1
        if row.grade != prev.grade:
            flips += 1
    return {
        "matched_fingerprints": matched,
        "fingerprint_mismatches": mismatches,
        "grade_flips": flips,
    }

Read the model id from the response body, and pass that string into Fingerprint without renaming it. Set endpoint_class to a label you control, such as free_server or pinned_provider, and keep the secret base URL outside the score file. Temperature belongs in the fingerprint because a 0.7 sample is not comparable to a 0.0 sample from the same prompt. The grader version belongs there too, because a stricter checker can fake a regression even when the model did not move.

Promotion is intentionally blunt, and a smoke row may be stored, charted, and alerted on without touching the baseline. The may_promote function returns false whenever the incoming lane is not adjudication, even if every keyword matched. A free-server batch can fill a side file all night and still leave the baseline file untouched for the next review. You investigate a smoke drop by rerunning those case ids on the adjudication lane, not by editing the baseline JSON.

If the adjudication rerun also drops, you have a candidate regression that deserves a diff of the completions. If the adjudication rerun holds, the smoke drop was a lane change and should be labeled as such. A synthetic walkthrough, labeled as unexecuted, shows why the split changes the decision you would otherwise publish. Imagine forty golden cases and an adjudication baseline of thirty-six passes, two fails, and two schema errors.

The quality rate ignores schema errors and is thirty-six over thirty-eight, or about 0.947 on that baseline. A later smoke batch on a free server returns thirty passes, six fails, and four schema errors. That smoke rate is thirty over thirty-six, or about 0.833, which looks like an eleven-point drop. Promoting that batch would announce a drop that the locked lane never confirmed, so the harness should leave 0.947 in place.

The commands below are a proposal, and you should substitute your own paths without committing secrets. Set the lane and the endpoint class in the environment, and leave the model id empty until the response fills it. Run the smoke output to a side file, then compare it with an explicit refusal to promote. A compare that defaults to promote will undo the separation the rest of the harness tried to build.

# Proposal commands. Substitute paths locally; do not commit secrets.
export EVAL_LANE=smoke
export EVAL_ENDPOINT_CLASS=free_server
export EVAL_MODEL_ID=""   # fill from the response, never from memory
python harness.py --cases cases/golden.jsonl --out runs/smoke.jsonl
python harness.py --compare runs/baseline.jsonl runs/smoke.jsonl --promote never

The compare step should print three counts: matched fingerprints, lane mismatches, and grade flips inside a matched fingerprint. Only the third count is a regression candidate, and the other two counts explain why a chart moved. A lane mismatch is a scheduling fact, and a schema error is an instrumentation fact rather than a quality fact. Folding either count into the quality rate will train the team to chase the wrong chart during review.

You can wire the smoke lane to whatever free model access you currently have, provided the response returns a model id. The cron entry should fail closed when the response omits a model id, because the fingerprint is then incomplete. An incomplete fingerprint must not be compared as if it were pinned to last week's adjudication run. Log the HTTP status and the case id, then skip scoring, since a skipped row is more honest than a guessed model name.

This approach is a poor fit for several teams, and the limits are part of the method rather than a footnote. Do not use it if your baseline must blend providers on purpose, because the promotion rule will block that blend by design. Do not use it for safety, medical, or compliance judgments, because a keyword grader does not measure harm or policy fit. Do not use it if the free endpoint's terms forbid storing prompts or completions, because the score file is itself a store.

Do not use it if you cannot keep the adjudication endpoint stable for the life of the baseline you intend to defend. Without a stable adjudication lane, the promotion rule has nothing trustworthy to protect, and every rerun is smoke. Keyword checks also miss meaning changes that still contain the required words, so they are a gate rather than a verdict. Pair them with a second grader on a small stratified sample, and treat disagreement as a review queue rather than an automatic fail.

If you adopt one habit, adopt the refusal to promote, because cheap generation is worth having only when it cannot inherit pinned authority. A free server is a reasonable place to regenerate fixtures while you debug a grader or widen a smoke sample. Check the current MonkeyCode documentation if you want that smoke lane on their stated free model access. Keep the adjudication key and the baseline file on the lane you can actually reproduce next month.

Pin Fixture, Grader, and Slot Before You Diff

Dakota Ma — Wed, 23 Sep 2026 17:36:44 +0000

Eval history that records only pass or fail quietly mixes unlike backends, then presents the mixture as a quality trend. Golden cases still matter, but they name expected behavior, not the instrument that produced the completion. When a serving slot changes and the fixture does not, a falling pass rate is an unlabeled instrument change until proven otherwise. The working unit of comparison is a triple of fixture hash, grader version, and model slot, not a bare percentage.

Complimentary and rotating model endpoints make that mixture easy to miss because the client code barely changes. A base URL swap or a default model field is enough to alter completions while JSONL fixtures remain untouched. Overnight eval jobs then look like prompt regressions, even though the prompt bytes and grader rules never moved. The failure mode is closer to changing a compiler and keeping the same unit tests without recording the toolchain.

Metrology notebooks refuse to compare readings until the gauge identifier is written beside the measurement. Software eval dashboards often skip that discipline because a chat completion looks like one interchangeable string. It is not interchangeable once tool choice, refusal style, and argument filling differ by backend. Treating every slot as the same model is how silent backend drift wears the costume of product quality drift.

The comparison unit is a triple

A small local harness can encode that discipline without a vendor platform or a shared dashboard. Each golden line should hash into a fixture id that covers user text, tool constraints, and expected substrings. Each grader module should export a version string that changes when match logic changes, including regex flags. Each run should accept a model slot label that the operator controls, even if the provider only returns a generic model family.

Scores may be compared across days only when those three strings are identical, byte for byte. A slot change with a stable fixture is a backend delta and belongs in a separate series. A grader version change is a rubric change and should reset the baseline instead of joining the old chart. A fixture hash change means the test itself moved, which is a dataset edit rather than a model regression.

The following JSONL fragment is a proposed golden case for a tool-using order assistant, not a measured production fixture. It encodes required tools, forbidden tools, and text constraints that a local grader can apply without a second model. Negative constraints matter because many silent failures are extra tool calls rather than missing words in the final text. Store the file in git so a fixture hash change is a normal reviewable diff.

{"id": "order_status_v3", "user": "Has order 9F2 shipped?", "require_tools": ["get_order"], "forbid_tools": ["cancel_order"], "must_include": ["9F2"], "must_not": ["tracking number is unknown"]}
{"id": "refund_policy_v1", "user": "Can I refund a delivered mug without a photo?", "require_tools": ["get_policy"], "forbid_tools": ["create_refund"], "must_include": ["photo"], "must_not": ["refund issued"]}
{"id": "missing_order_id_v2", "user": "Where is my package?", "require_tools": [], "forbid_tools": ["get_order", "cancel_order"], "must_include": ["order id"], "must_not": ["shipped"]}

A deterministic grader should return structured reasons, because a boolean hides whether the tool constraint or the text constraint failed. Hashing the grader source into the version string prevents silent rubric edits from looking like model movement. The function below is labeled as proposed code and has not been executed against a live endpoint in this article. Operators can swap the completion loader later without changing the scoring rules or the golden file.

# grade_case.py — proposed, unexecuted example
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any


def grader_version(path: str = __file__) -> str:
    digest = hashlib.sha256(Path(path).read_bytes()).hexdigest()
    return digest[:12]


def fixture_hash(case: dict[str, Any]) -> str:
    payload = json.dumps(case, sort_keys=True, separators=(',', ':'))
    return hashlib.sha256(payload.encode()).hexdigest()[:16]


def grade_case(case: dict[str, Any], completion: dict[str, Any]) -> dict[str, Any]:
    text = completion.get('text') or ''
    tools = [t.get('name') for t in completion.get('tools') or [] if t.get('name')]
    reasons: list[str] = []

    for name in case.get('require_tools') or []:
        if name not in tools:
            reasons.append(f'missing_tool:{name}')
    for name in case.get('forbid_tools') or []:
        if name in tools:
            reasons.append(f'forbidden_tool:{name}')
    lowered = text.lower()
    for needle in case.get('must_include') or []:
        if needle.lower() not in lowered:
            reasons.append(f'missing_text:{needle}')
    for needle in case.get('must_not') or []:
        if needle.lower() in lowered:
            reasons.append(f'forbidden_text:{needle}')

    return {
        'case_id': case['id'],
        'pass': not reasons,
        'reasons': reasons,
        'tool_names': tools,
        'text_digest': hashlib.sha256(text.encode()).hexdigest()[:12],
    }

Tool constraints deserve first-class fields because agent failures often hide inside a plausible final sentence. A completion can mention the order identifier while also calling a cancel tool that the user never requested. Text-only golden cases grade that reply as a pass and leave the side effect invisible. Recording required and forbidden tool names turns that miss into an ordinary assertion with a stable reason code.

Frozen completions make the grader itself testable without a network, which is the only way the rubric stays honest. The tests below pin envelopes in-process so a later grader edit fails in CI before it contaminates a model chart. They are proposed unit tests, not live measurements, and they should run the same on every laptop. If these tests flip, the event is a rubric change and the grader version must move.

# test_grade_case.py — proposed local tests, no live model required
from grade_case import grade_case

ORDER = {
    'id': 'order_status_v3',
    'user': 'Has order 9F2 shipped?',
    'require_tools': ['get_order'],
    'forbid_tools': ['cancel_order'],
    'must_include': ['9F2'],
    'must_not': ['tracking number is unknown'],
}


def test_plausible_text_still_fails_on_forbidden_tool():
    completion = {
        'text': 'Order 9F2 is in transit to the warehouse dock.',
        'tools': [
            {'name': 'get_order', 'args': {'id': '9F2'}},
            {'name': 'cancel_order', 'args': {'id': '9F2'}},
        ],
    }
    result = grade_case(ORDER, completion)
    assert result['pass'] is False
    assert 'forbidden_tool:cancel_order' in result['reasons']


def test_required_tool_without_forbidden_text_passes():
    completion = {
        'text': 'Order 9F2 has shipped and left the facility.',
        'tools': [{'name': 'get_order', 'args': {'id': '9F2'}}],
    }
    result = grade_case(ORDER, completion)
    assert result['pass'] is True
    assert result['reasons'] == []

A proposed local harness

The runner should persist one JSON object per case, including the triple, the reason codes, and a completion digest. Storing raw completions is useful for later audits, but the digest already detects silent text movement when the slot stays pinned. If the triple matches and the digest changes while the grade flips, the harness can call that a true silent regression. If the triple does not match, the harness should refuse a quality diff and print a schema of mismatched keys.

# pin_eval.py — proposed runner; fill load_completion() before any live use
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path
from typing import Any

from grade_case import fixture_hash, grade_case, grader_version

TRIPLE_KEYS = ('fixture_hash', 'grader_version', 'model_slot')


def load_completion(slot: str, user: str) -> dict[str, Any]:
    raise NotImplementedError(
        'Proposed hook: map slot to your chat client and return {text, tools}.'
    )


def load_golden(path: Path) -> list[dict[str, Any]]:
    cases = []
    for line in path.read_text().splitlines():
        if line.strip():
            cases.append(json.loads(line))
    return cases


def run_eval(golden: Path, slot: str, out: Path) -> None:
    version = grader_version()
    records = []
    for case in load_golden(golden):
        completion = load_completion(slot, case['user'])
        graded = grade_case(case, completion)
        records.append({
            **graded,
            'fixture_hash': fixture_hash(case),
            'grader_version': version,
            'model_slot': slot,
        })
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(''.join(json.dumps(r) + '\n' for r in records))


def index_records(path: Path) -> dict[str, dict[str, Any]]:
    return {json.loads(line)['case_id']: json.loads(line) for line in path.read_text().splitlines() if line.strip()}


def diff_runs(left: Path, right: Path) -> int:
    a_map, b_map = index_records(left), index_records(right)
    mismatches = []
    silent = []
    for case_id in sorted(set(a_map) & set(b_map)):
        a, b = a_map[case_id], b_map[case_id]
        triple_ok = all(a[k] == b[k] for k in TRIPLE_KEYS)
        if not triple_ok:
            mismatches.append({
                'case_id': case_id,
                'left': {k: a[k] for k in TRIPLE_KEYS},
                'right': {k: b[k] for k in TRIPLE_KEYS},
            })
            continue
        if a['pass'] != b['pass'] or a['text_digest'] != b['text_digest']:
            silent.append({
                'case_id': case_id,
                'left_pass': a['pass'],
                'right_pass': b['pass'],
                'left_reasons': a['reasons'],
                'right_reasons': b['reasons'],
                'left_digest': a['text_digest'],
                'right_digest': b['text_digest'],
            })
    if mismatches:
        print('REFUSE_DIFF: triple mismatch, not a prompt regression')
        print(json.dumps(mismatches, indent=2))
        return 2
    print(json.dumps({'silent_regressions': silent, 'compared_cases': len(set(a_map) & set(b_map))}, indent=2))
    return 1 if silent else 0


def main() -> None:
    parser = argparse.ArgumentParser(description='Pin fixture, grader, and model slot.')
    sub = parser.add_subparsers(dest='cmd', required=True)
    run_p = sub.add_parser('run')
    run_p.add_argument('--golden', type=Path, required=True)
    run_p.add_argument('--slot', required=True)
    run_p.add_argument('--out', type=Path, required=True)
    diff_p = sub.add_parser('diff')
    diff_p.add_argument('left', type=Path)
    diff_p.add_argument('right', type=Path)
    args = parser.parse_args()
    if args.cmd == 'run':
        run_eval(args.golden, args.slot, args.out)
        return
    sys.exit(diff_runs(args.left, args.right))


if __name__ == '__main__':
    main()

Command examples below assume a local working directory and a completion loader that the operator fills in. Environment variables keep endpoint URLs out of git, which matters when a free server address should not leak into reviews. The diff command must exit nonzero when it refuses a comparison, so CI cannot greenwash a slot change. None of these commands were executed for this article; they are a reproducible shape, not a benchmark.

python -m pytest test_grade_case.py -q
export COMPLETION_BASE_URL="$ENDPOINT_URL"
python pin_eval.py run --golden golden.jsonl --slot lab_free_1 --out runs/2026-09-22.jsonl
python pin_eval.py run --golden golden.jsonl --slot lab_free_1 --out runs/2026-09-23.jsonl
python pin_eval.py diff runs/2026-09-22.jsonl runs/2026-09-23.jsonl
echo $?

Suppose Tuesday's run used slot lab_free_1 and Wednesday's run used lab_free_2 on the same fixtures. A naive pass-rate chart would subtract the two percentages and file a quality incident against the prompt. The pinned harness instead reports a slot mismatch and writes a backend-delta artifact with per-case reason codes. Engineers can still read that artifact; they just cannot paste it into the prompt-regression series.

Running the loop still needs a completion source, and that is the only place a complimentary endpoint earns a mention. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can run this runner against a small golden file during early checks. The product is not the grader, and removing the mention should leave the triple-pin method intact.

What this will not prove

The method does not identify which backend is better; it only stops unlike backends from sharing a trend line. Slot labels are operator-assigned aliases, so they are only as honest as the runbook that maps aliases to endpoints. Free or rotating slots can change behavior without a label change if the operator reuses a name. A reused alias is a forged gauge id, and the triple will look stable while the instrument is not.

Groups that lack git review for JSONL fixtures should not treat the hash as an audit trail. Groups that need statistically powered A/B tests will not get them from a few dozen golden lines. Groups whose completions must be judged by experts should keep human labels and use this only as a smoke layer. The harness is a bookkeeping fix for mixed time series, not a substitute for product evaluation.

Compare two nights of eval output only after joining on fixture hash, grader version, and model slot. If those keys diverge, file a backend delta or a rubric change rather than a prompt incident. A small local run against any inexpensive slot is enough to rehearse that join on a tiny fixture file. Larger claims about quality still require pinned production endpoints and a review process outside this script.

Grade the Snapshot, Not the Live Call

Dakota Ma — Tue, 22 Sep 2026 15:52:11 +0000

When a prompt evaluation fails in CI, three independent clocks have usually already mixed together unnoticed. The model weights, the prompt bytes, and the grader logic can each move without the others. A single pass-or-fail boolean cannot name which of those clocks caused the drop. The practical split is to persist the raw completion envelope, then grade that snapshot as a pure function.

A live-call harness looks tidy to most teams because it resembles an ordinary unit test. You send a prompt, parse one JSON object, and assert a field while three moving parts share the call. The system under test is stochastic, the oracle is mutable code, and the network is part of the fixture. When those three sit inside one function, a red score is a rumor rather than a diagnosis.

Compiler toolchains already learned this separation after years of mixed compiler and test failure reports. Nobody debugs a flaky assertion by rebuilding the compiler on every click of the test runner. They keep the object file on disk, then they rerun the test against that frozen binary. Prompt evaluations deserve an equivalent freeze-frame that graders can replay without requesting another generation.

Picture a golden case for a tool-using assistant that must refuse to invent an order identifier. Monday's live harness may pass because the model omitted the identifier field entirely during that sampling. Tuesday's run may fail because the grader now treats JSON null and a missing key as different events. Without a snapshot the dashboard blames model drift, while a frozen envelope would regrade Monday under Tuesday's grader.

The Python module below is a labeled proposal, not a production measurement from a private workload. It stores each model envelope as JSON beside the golden case that produced the request. Each file is addressed by a SHA-256 prefix of the canonical request bytes, not by wall-clock time. Capture writes the envelope to disk, and the grade path never opens a network socket at all.

"""replay_eval.py — proposal: capture envelopes, then grade offline."""
from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable


@dataclass(frozen=True)
class GoldenCase:
    case_id: str
    request: dict[str, Any]
    expected_findings: frozenset[str]


def canonical_bytes(request: dict[str, Any]) -> bytes:
    return json.dumps(request, sort_keys=True, separators=(",", ":")).encode()


def request_hash(request: dict[str, Any]) -> str:
    return hashlib.sha256(canonical_bytes(request)).hexdigest()[:16]


def snapshot_path(root: Path, case: GoldenCase) -> Path:
    return root / case.case_id / f"{request_hash(case.request)}.json"


def capture(root: Path, case: GoldenCase, envelope: dict[str, Any]) -> Path:
    path = snapshot_path(root, case)
    path.parent.mkdir(parents=True, exist_ok=True)
    payload = {
        "case_id": case.case_id,
        "request_hash": request_hash(case.request),
        "request": case.request,
        "envelope": envelope,
    }
    path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")
    return path


FindingFn = Callable[[dict[str, Any]], set[str]]


def invented_order_id(envelope: dict[str, Any]) -> set[str]:
    findings: set[str] = set()
    tool_calls = envelope.get("tool_calls") or []
    for call in tool_calls:
        args = call.get("arguments") or {}
        name = call.get("name")
        if name == "get_order" and "order_id" in args:
            if args.get("order_id") in (None, "", "UNKNOWN"):
                findings.add("invented_argument")
        if name == "get_order" and "order_id" not in args:
            findings.add("missing_argument")
    if not tool_calls and envelope.get("refusal") is True:
        findings.add("ok_refusal")
    if not findings:
        findings.add("ok")
    return findings


def grade(root: Path, case: GoldenCase, grader: FindingFn) -> dict[str, Any]:
    path = snapshot_path(root, case)
    if not path.exists():
        return {"case_id": case.case_id, "status": "missing_snapshot"}
    payload = json.loads(path.read_text(encoding="utf-8"))
    got = frozenset(grader(payload["envelope"]))
    expected = case.expected_findings
    return {
        "case_id": case.case_id,
        "request_hash": payload["request_hash"],
        "status": "pass" if got == expected else "fail",
        "got": sorted(got),
        "expected": sorted(expected),
        "same_request_hash": payload["request_hash"] == request_hash(case.request),
    }


def diff_reports(before: list[dict[str, Any]], after: list[dict[str, Any]]) -> dict[str, Any]:
    prev = {row["case_id"]: row for row in before}
    grader_changed = []
    capture_moved = []
    for row in after:
        old = prev.get(row["case_id"])
        if old is None:
            continue
        same_hash = old.get("request_hash") == row.get("request_hash")
        if same_hash and old.get("got") != row.get("got"):
            grader_changed.append(row["case_id"])
        if not same_hash:
            capture_moved.append(row["case_id"])
    return {
        "grader_changed_on_frozen_hash": grader_changed,
        "capture_hash_moved": capture_moved,
    }

A second entry point then walks the snapshot directory while using only the current grader functions. If the snapshot hash is unchanged and a new finding appears, the grader or the expected set moved. If the hash itself is new, the capture stage moved because the prompt, schema, or endpoint changed. Those two diagnoses must not share a single dashboard cell that is labeled overall quality.

python - <<'PY'
from pathlib import Path
from replay_eval import GoldenCase, grade, invented_order_id

root = Path("snapshots")
refuse = GoldenCase(
    case_id="refuse_invented_order_id",
    request={
        "messages": [{"role": "user", "content": "Get my order, you know the id."}],
        "tools": [{"name": "get_order", "parameters": ["order_id"]}],
    },
    expected_findings=frozenset({"ok_refusal"}),
)
# Proposal only: capture(root, refuse, client.complete(refuse.request))
print(grade(root, refuse, invented_order_id))
PY

Silent regressions live in the gap between those two steps rather than inside a blended accuracy number. A prompt file can pick up a trailing instruction during an otherwise unrelated merge on main. A tool schema can mark a formerly optional argument as required without any grader edit. Mixed into one HTTP round trip, each event looks like the model got worse overnight.

Split into capture and grade, the same events produce different artifacts that CI can name. A new hash means the prompt, the tool schema, or the model endpoint changed under you. A new finding on an old hash means the grader or the expected set changed instead. That naming is the entire reason to keep a wire log beside the golden cases.

Golden cases should carry the request, the allowed tools, and the expected finding codes together. A case that must emit invented_argument is as load-bearing as a case that must emit ok_refusal. If the refusal case ever yields ok against an unchanged snapshot, the grader went blind. If it yields ok only after a new snapshot appears, the model or the prompt changed.

Environment coupling is the next quiet failure mode that appears in otherwise careful evaluation harnesses. Graders that read the wall clock, the working directory, or an open catalog will mark frozen envelopes flaky. Keep every grader referentially transparent, with an envelope going in and a finding set coming out. Store frozen timestamps and valid identifier catalogs inside the case, never in ambient process state.

A second capture environment is how you detect failures that are glued to one process. Running the same canonical request against another reachable endpoint produces a sibling snapshot with its own hash. Disclosure: This article was prepared as part of MonkeyCode's product outreach; free model access and a free server option can write those envelopes. You still grade locally as ordinary code, comparing finding lists and hashes rather than vendor slogans.

Do not promote that second endpoint into a benchmark table for either quality or latency. No ranking in this article would be honest without a pinned model identifier and a pinned prompt corpus. The comparison is structural on purpose: same request bytes, two envelopes, two hashes, one grader. If both fresh hashes grade clean and yesterday's snapshot now fails, you changed the grader.

Limitations follow directly from the freeze you just introduced into the evaluation path itself. A snapshot does not rescue a golden case that was already wrong on the afternoon you captured it. Garbage in the log stays garbage, and regrading it will only make the wrong answer feel stable. The harness also does not replace human review for tasks without a checkable envelope, including open-ended tone.

Teams that should skip this approach can be named before any snapshot directory exists. If you cannot persist raw model output because the wire contains live PII, the log is a compliance incident. If a non-deterministic preprocessor rewrites every prompt, the request hash will thrash and isolate nothing. Stabilize that preprocessor, or capture after it and store the rewritten prompt beside the envelope.

If you have no checkable finding codes at all, write smaller contracts before you invest in a wire log. This directory will not invent an oracle that the product itself cannot already state. A replay log will not stop a model endpoint from changing under your feet overnight. It will stop you from blaming that change for a grader edit, a schema tweak, or a prompt merge.

Capture, hash, and then grade, and only then decide which of the three clocks actually moved. If prompts already live in git, keep snapshots as CI artifacts and run grade on every grader diff. The second capture host remains optional for isolation, but the frozen envelope is not optional.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Score Disagreement After the Eval Ceiling

Dakota Ma — Mon, 21 Sep 2026 14:07:07 +0000

Once a golden set stops splitting models, pass rate becomes a vanity metric that conceals regressions. The useful signal is disagreement between a frozen reference path and the candidate under test. This article proposes a small harness that scores that delta on a cheap nightly loop. Absolute scores saturate after a capability jump, while pairwise disagreement still moves under silent prompt drift.

Think of the golden set as a sieve whose holes grow larger every time the underlying models get better. Water still pours through, so the dashboard records a perfect catch rate that no longer describes the mesh. A hallway of smoke detectors that never sound does not prove the building is actually safe. It often proves the sensors lost their ability to discriminate smoke from ordinary hallway dust.

Eval ceiling is that same failure mode, expressed as a dashboard that refuses to turn yellow. Teams then ship prompt edits that change refusals, citations, or tool arguments without a red build. The cases still match a coarse expected string, so the harness reports health that the product does not have. Scoring the candidate alone cannot recover lost resolution once every row sits on the ceiling.

The proposed fix is not a larger golden set, which will saturate again after the next model bump. It is a frozen reference completion, a candidate completion, and identical grader functions applied to both. Continuous integration should fail when the disagreement map changes, even if both pass rates stay high. That rule is the method; the Python below is only a concrete encoding of the same contract.

The fixture file is versioned like application code, because a case without an owner ages into decoration. Each row carries an identifier, a prompt, and a machine-checkable expectation rather than a prose rubric. The proposed schema below is unlabeled production data; it is a teaching fixture for the runner that follows. Keep expected fields narrow, or the grader will bless fluent wrong answers that share a token with the gold string.

{
  "version": "2026-09-21",
  "cases": [
    {
      "id": "refund.policy.v3",
      "prompt": "Customer paid twice for order 1842. Can they get an immediate refund?",
      "expect": {
        "must_refuse_instant_refund": true,
        "must_cite_order_id": true,
        "max_sentences": 4
      }
    },
    {
      "id": "tool.tracking.missing_carrier",
      "prompt": "Track shipment 9911. The carrier field is empty in our database.",
      "expect": {
        "must_not_invent_carrier": true,
        "must_ask_for_carrier": true
      }
    }
  ]
}

Graders belong in ordinary functions so a failing row produces a stack that a reviewer can read. A hidden judging prompt recreates the ceiling problem inside the judge, because the judge saturates too. The functions below are proposed examples, not measured production thresholds, and they return structured misses instead of a single boolean. Structured misses keep the disagreement map readable when two models fail for different reasons on the same fixture.

# proposed_example.py — unexecuted teaching harness, not a benchmark.
from __future__ import annotations

import json, os, re, urllib.request
from dataclasses import dataclass, asdict
from typing import Any, Callable, Dict, List

Grader = Callable[[str, Dict[str, Any]], List[str]]


def grade_refund(text: str, expect: Dict[str, Any]) -> List[str]:
    misses: List[str] = []
    low = text.lower()
    if expect.get("must_refuse_instant_refund") and "refund now" in low:
        misses.append("offered_instant_refund")
    if expect.get("must_cite_order_id") and "1842" not in text:
        misses.append("missing_order_id")
    sentences = [s for s in re.split(r"[.!?]", text) if s.strip()]
    if len(sentences) > expect.get("max_sentences", 99):
        misses.append("too_verbose")
    return misses


def grade_tracking(text: str, expect: Dict[str, Any]) -> List[str]:
    misses: List[str] = []
    low = text.lower()
    invented = ("ups", "fedex", "dhl", "usps")
    if expect.get("must_not_invent_carrier") and any(c in low for c in invented):
        misses.append("invented_carrier")
    if expect.get("must_ask_for_carrier") and "carrier" not in low:
        misses.append("did_not_ask_carrier")
    return misses


GRADERS = {
    "refund.policy.v3": grade_refund,
    "tool.tracking.missing_carrier": grade_tracking,
}

The completion client should be boring HTTP, because a clever SDK hides retries that later look like model disagreement. Temperature stays at zero for both paths so sampling noise does not masquerade as a product change. The snippet assumes an OpenAI-style chat payload only as a local convention; swap the body if your endpoint differs. Label every network failure separately from grader misses, or a timeout becomes a fake policy regression.

@dataclass
class CallResult:
    text: str
    error: str | None


def complete(base_url: str, prompt: str, timeout: int = 30) -> CallResult:
    payload = {
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "max_tokens": 256,
    }
    req = urllib.request.Request(
        base_url.rstrip("/") + "/v1/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ.get("EVAL_TOKEN", ""),
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = json.loads(resp.read().decode("utf-8"))
        text = body["choices"][0]["message"]["content"]
        return CallResult(text=text, error=None)
    except Exception as exc:  # proposed catch-all for teaching; narrow this in real code
        return CallResult(text="", error=type(exc).__name__)

The comparison step is the actual test. Each fixture yields a reference miss list, a candidate miss list, and a symmetric difference of those lists. A candidate that passes while the reference fails is not automatically a win; it is a delta that needs a human glance the first time it appears. After that delta is accepted, it becomes the new frozen map, the same way a reviewed snapshot becomes the next baseline in UI tests.

@dataclass
class Row:
    case_id: str
    reference_misses: List[str]
    candidate_misses: List[str]
    reference_error: str | None
    candidate_error: str | None

    @property
    def disagreement(self) -> List[str]:
        return sorted(set(self.reference_misses) ^ set(self.candidate_misses))


def evaluate(cases: List[dict], ref_url: str, cand_url: str) -> List[Row]:
    rows: List[Row] = []
    for case in cases:
        grader = GRADERS[case["id"]]
        ref = complete(ref_url, case["prompt"])
        cand = complete(cand_url, case["prompt"])
        ref_misses = ["transport:" + ref.error] if ref.error else grader(ref.text, case["expect"])
        cand_misses = ["transport:" + cand.error] if cand.error else grader(cand.text, case["expect"])
        rows.append(Row(case["id"], ref_misses, cand_misses, ref.error, cand.error))
    return rows

A green absolute score with a changed disagreement map should fail the job. The exit code below treats transport errors as infrastructure, then fails only on novel deltas against a committed snapshot. That snapshot is the eval equivalent of a lockfile: boring, diffable, and painful to edit without a reason. If a case stays in both-pass with empty disagreement for many consecutive nights, it has stopped discriminating and should be retired or mutated, not celebrated.

SNAP_PATH = "disagreement_snapshot.json"


def load_snapshot() -> Dict[str, List[str]]:
    if not os.path.exists(SNAP_PATH):
        return {}
    with open(SNAP_PATH, encoding="utf-8") as handle:
        return json.load(handle)


def gate(rows: List[Row], update: bool = False) -> int:
    snap = load_snapshot()
    novel = []
    transport = [r for r in rows if r.reference_error or r.candidate_error]
    if transport:
        print("transport_failures", [asdict(r) for r in transport])
        return 2
    current = {r.case_id: r.disagreement for r in rows}
    for case_id, delta in current.items():
        if snap.get(case_id) != delta:
            novel.append({"id": case_id, "was": snap.get(case_id), "now": delta})
    if novel and not update:
        print("disagreement_changed", json.dumps(novel, indent=2))
        return 1
    if update:
        with open(SNAP_PATH, "w", encoding="utf-8") as handle:
            json.dump(current, handle, indent=2, sort_keys=True)
            handle.write("\n")
    both_pass = [r.case_id for r in rows if not r.reference_misses and not r.candidate_misses]
    print("ceiling_candidates", both_pass)
    return 0

Run the gate from a scheduler, not from a laptop lid, because ceiling effects appear across days rather than inside one interactive session. The commands assume two base URLs and a token in the environment; they do not encode a vendor. First nights will look noisy while the snapshot is empty, which is expected, and the --update path should be a reviewed commit rather than an automatic rewrite. After the snapshot exists, the interesting output is disagreement_changed, not a percentage that has already glued itself to one hundred.

export EVAL_TOKEN=replace-me
python proposed_example.py  # wire a __main__ that reads cases.json
python - <<'PY'
# proposed driver
import json, os, sys
from proposed_example import evaluate, gate
cases = json.load(open("cases.json"))["cases"]
rows = evaluate(cases, os.environ["REF_URL"], os.environ["CAND_URL"])
sys.exit(gate(rows, update="--update" in sys.argv))
PY

A second model pass doubles token spend if both paths are billed at the same rate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which can hold the frozen reference path and the nightly runner without turning disagreement checks into a paid always-on job. The harness does not depend on that product; it only needs two completion targets and a machine that can cron a Python file. If those pieces already exist in your stack, keep them and treat the product mention as optional infrastructure for the reference leg.

Disagreement is not truth, and that limitation matters more than the exit code. Both paths can share a bug, especially when the candidate was fine-tuned toward the same public style as the reference. The method also fails closed on transport, which is correct for CI and irritating during an upstream brownout. Teams that possess only one completion endpoint cannot run this gate at all, and they should not fake a reference by replaying yesterday’s candidate logs as if they were an independent model.

Skip this approach when the grader itself requires a human rubric, multilingual legal judgment, or safety review that no function in this file can encode. Skip it when the two endpoints differ in tool-calling semantics, because then the delta measures protocol mismatch rather than prompt regression. Skip it when product owners will rubber-stamp --update whenever the map moves, since an unreviewed snapshot is just pass rate wearing a more elaborate coat. The ceiling is a measurement problem; an ignored delta is a process problem, and software cannot paper over the second one.

The core conclusion does not change after the code is copied. Pass rate goes mute when fixtures stop splitting models, and disagreement remains audible if the grader is code and the reference stays frozen. Retire cases that both paths pass without a delta, mutate them, or accept that they no longer earn their keep in the suite. If you need a cheap place to park the reference client and the cron host, MonkeyCode’s free model access and free server option are sufficient to try the loop without reshaping the rest of the eval.

A Perfect Pass Rate Is a Blind Harness

Dakota Ma — Sun, 20 Sep 2026 12:03:23 +0000

A 100 percent pass rate is often a measurement failure rather than a quality win. Once every golden case sits in the easy band, the harness stops ranking prompts and only rubber-stamps them. Ceiling effects hide regressions the same way a bathroom scale that tops out at two hundred pounds hides further weight gain. The fix is not another average; it is stratified cases, a ceiling detector, and stretch items that still fail on purpose.

Most prompt suites still collapse many intents into one ratio, then celebrate when that ratio climbs. After a few model upgrades, the easy items become free points and the average loses its slope. A regression that breaks a rare tool path can hide inside a still-green total because the easy majority outweighs it. Quality scores need a denominator that still contains failure, or they stop being scores at all.

Think of a compiler test suite that only compiles hello world after the language grows generics and macros. The suite stays green while the new surface area rots, because nothing in the corpus can still fail. Load tests have the same trap when generated traffic never exceeds last year's peak; the graph looks calm because the generator is polite. Golden cases for language-model features age the same quiet way when product copy expands around a frozen fixture file.

A workable response is to tag every fixture with a difficulty stratum and to treat those strata as separate instruments. Easy cases guard catastrophic breakage, such as empty replies, truncated JSON, or ignored system instructions. Medium cases encode the product contract: the answer must cite a given SKU, refuse a missing field, or stay inside a character budget. Stretch cases are supposed to be slightly too hard for the current prompt, so a high pass rate there means the yardstick has slipped.

The Python below is a labeled worked example rather than a production framework, and it grades recorded completions offline first. Offline grading keeps the method reproducible without a live credential, which matters when you are debugging the harness instead of the model. An optional HTTP path fills empty completions from any OpenAI-compatible chat endpoint when you want a nightly loop. Model names stay in environment variables so fixtures do not silently pin an endpoint you did not intend to grade.

#!/usr/bin/env python3
"""Ceiling-aware eval harness (worked example, not a shipped product)."""

from __future__ import annotations

import argparse
import json
import os
import sys
import urllib.error
import urllib.request
from dataclasses import dataclass, field
from pathlib import Path
from typing import Iterable


STRATA = ("easy", "medium", "stretch")


@dataclass(frozen=True)
class GoldenCase:
    id: str
    stratum: str
    prompt: str
    completion: str
    must_include: tuple[str, ...] = ()
    must_not_include: tuple[str, ...] = ()
    max_chars: int = 1200

    def __post_init__(self) -> None:
        if self.stratum not in STRATA:
            raise ValueError(f"{self.id}: unknown stratum {self.stratum}")


@dataclass
class Grade:
    case_id: str
    stratum: str
    passed: bool
    reasons: list[str] = field(default_factory=list)


def grade_case(case: GoldenCase) -> Grade:
    reasons: list[str] = []
    text = case.completion or ""
    if not text.strip():
        reasons.append("empty_completion")
    if len(text) > case.max_chars:
        reasons.append(f"over_budget:{len(text)}>{case.max_chars}")
    lowered = text.lower()
    for needle in case.must_include:
        if needle.lower() not in lowered:
            reasons.append(f"missing:{needle}")
    for needle in case.must_not_include:
        if needle.lower() in lowered:
            reasons.append(f"forbidden:{needle}")
    return Grade(case.id, case.stratum, passed=not reasons, reasons=reasons)


def load_jsonl(path: Path) -> list[GoldenCase]:
    cases: list[GoldenCase] = []
    with path.open(encoding="utf-8") as handle:
        for line_no, raw in enumerate(handle, 1):
            raw = raw.strip()
            if not raw:
                continue
            row = json.loads(raw)
            try:
                cases.append(
                    GoldenCase(
                        id=str(row["id"]),
                        stratum=str(row["stratum"]),
                        prompt=str(row["prompt"]),
                        completion=str(row.get("completion", "")),
                        must_include=tuple(row.get("must_include") or ()),
                        must_not_include=tuple(row.get("must_not_include") or ()),
                        max_chars=int(row.get("max_chars") or 1200),
                    )
                )
            except (KeyError, TypeError, ValueError) as exc:
                raise SystemExit(f"{path}:{line_no}: {exc}") from exc
    return cases


def stratum_rates(grades: Iterable[Grade]) -> dict[str, dict[str, float | int | None]]:
    buckets = {name: {"n": 0, "pass": 0} for name in STRATA}
    for grade in grades:
        buckets[grade.stratum]["n"] += 1
        buckets[grade.stratum]["pass"] += int(grade.passed)
    report: dict[str, dict[str, float | int | None]] = {}
    for name, slot in buckets.items():
        n = slot["n"]
        report[name] = {"n": n, "pass_rate": (slot["pass"] / n) if n else None}
    return report


def detect_ceiling(rates: dict[str, dict[str, float | int | None]], stretch_high: float) -> list[str]:
    """Flag a blind harness. High stretch pass is eval rot, not a trophy."""
    alerts: list[str] = []
    easy = rates["easy"]["pass_rate"]
    medium = rates["medium"]["pass_rate"]
    stretch = rates["stretch"]["pass_rate"]
    counts = [rates[name]["n"] for name in STRATA]
    if any(n is None or int(n) < 2 for n in counts):
        alerts.append("thin_stratum")
        return alerts
    if easy is None or medium is None or stretch is None:
        alerts.append("missing_stratum")
        return alerts
    if float(easy) >= 0.99 and float(medium) >= 0.95 and float(stretch) >= stretch_high:
        alerts.append("ceiling")
    if float(stretch) >= stretch_high:
        alerts.append("stretch_too_easy")
    return alerts


def evaluate(
    cases: list[GoldenCase],
    easy_floor: float = 1.0,
    medium_floor: float = 0.85,
    stretch_high: float = 0.80,
) -> dict:
    grades = [grade_case(case) for case in cases]
    rates = stratum_rates(grades)
    alerts = detect_ceiling(rates, stretch_high=stretch_high)
    easy_rate = rates["easy"]["pass_rate"]
    medium_rate = rates["medium"]["pass_rate"]
    if easy_rate is not None and float(easy_rate) < easy_floor:
        alerts.append("easy_regression")
    if medium_rate is not None and float(medium_rate) < medium_floor:
        alerts.append("medium_regression")
    return {
        "n": len(grades),
        "rates": rates,
        "alerts": alerts,
        "ok": not alerts,
        "failures": [grade.__dict__ for grade in grades if not grade.passed],
    }


def maybe_complete(cases: list[GoldenCase], base_url: str | None) -> list[GoldenCase]:
    """Optional live fill-in. Model name stays in the environment on purpose."""
    if not base_url:
        return cases
    model = os.environ.get("EVAL_MODEL")
    if not model:
        raise SystemExit("EVAL_MODEL is required when --base-url is set")
    filled: list[GoldenCase] = []
    for case in cases:
        payload = json.dumps(
            {
                "model": model,
                "messages": [{"role": "user", "content": case.prompt}],
                "temperature": 0,
            }
        ).encode()
        req = urllib.request.Request(
            base_url.rstrip("/") + "/chat/completions",
            data=payload,
            headers={"Content-Type": "application/json"},
            method="POST",
        )
        api_key = os.environ.get("EVAL_API_KEY")
        if api_key:
            req.add_header("Authorization", f"Bearer {api_key}")
        try:
            with urllib.request.urlopen(req, timeout=60) as resp:
                body = json.loads(resp.read().decode())
        except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, KeyError) as exc:
            raise SystemExit(f"live fill failed for {case.id}: {exc}") from exc
        text = body["choices"][0]["message"]["content"]
        filled.append(
            GoldenCase(
                id=case.id,
                stratum=case.stratum,
                prompt=case.prompt,
                completion=text,
                must_include=case.must_include,
                must_not_include=case.must_not_include,
                max_chars=case.max_chars,
            )
        )
    return filled


def self_check() -> None:
    cases = [
        GoldenCase("e1", "easy", "p", "Order 1842 packed", ("1842",), (), 80),
        GoldenCase("e2", "easy", "p", "Order 1842 packed", ("1842",), (), 80),
        GoldenCase("m1", "medium", "p", "SKU-9 cited", ("SKU-9",), ("sorry",), 80),
        GoldenCase("m2", "medium", "p", "SKU-9 cited", ("SKU-9",), ("sorry",), 80),
        GoldenCase("s1", "stretch", "p", "policy XYZ holds", ("XYZ", "policy"), (), 80),
        GoldenCase("s2", "stretch", "p", "policy XYZ holds", ("XYZ", "policy"), (), 80),
    ]
    report = evaluate(cases, stretch_high=0.50)
    assert "stretch_too_easy" in report["alerts"], report
    print(json.dumps({"self_check": "ok", "alerts": report["alerts"]}, indent=2))


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="Ceiling-aware golden-case grader")
    parser.add_argument("jsonl", type=Path, nargs="?")
    parser.add_argument("--base-url", default=os.environ.get("EVAL_BASE_URL"))
    parser.add_argument("--stretch-high", type=float, default=0.80)
    parser.add_argument("--self-check", action="store_true")
    args = parser.parse_args(argv)
    if args.self_check:
        self_check()
        return 0
    if args.jsonl is None:
        raise SystemExit("jsonl path is required unless --self-check is set")
    cases = maybe_complete(load_jsonl(args.jsonl), args.base_url)
    report = evaluate(cases, stretch_high=args.stretch_high)
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if report["ok"] else 2


if __name__ == "__main__":
    raise SystemExit(main())

Store fixtures as JSONL so git can diff a single case without opening a notebook. Each row carries an identifier, a stratum, the prompt, an optional recorded completion, and the code-side constraints the grader will apply. The example file below is synthetic and marked as such; replace the text with your own product language before trusting any rate it emits.

{"id": "easy-empty-guard", "stratum": "easy", "prompt": "Summarize order 1842 in one sentence.", "completion": "Order 1842 ships two blue mugs to Denver tomorrow.", "must_include": ["1842"], "must_not_include": ["as an AI"], "max_chars": 240}
{"id": "easy-no-apology-loop", "stratum": "easy", "prompt": "Confirm the warehouse, nothing else.", "completion": "Denver warehouse confirmed.", "must_include": ["Denver"], "must_not_include": ["cannot"], "max_chars": 80}
{"id": "medium-sku-contract", "stratum": "medium", "prompt": "Name the SKU that ships and refuse unknown add-ons.", "completion": "SKU-9 ships; extra engraving is not on the order.", "must_include": ["SKU-9"], "must_not_include": ["engraving is confirmed"], "max_chars": 160}
{"id": "medium-budget", "stratum": "medium", "prompt": "Reply with the ship date only.", "completion": "2026-09-21", "must_include": ["2026-09-21"], "must_not_include": ["certainly"], "max_chars": 32}
{"id": "stretch-policy-cite", "stratum": "stretch", "prompt": "Quote the restock rule that blocks weekend dispatch.", "completion": "I think weekends are probably fine if the queue is short.", "must_include": ["restock-window", "weekday"], "must_not_include": ["probably"], "max_chars": 200}
{"id": "stretch-missing-field", "stratum": "stretch", "prompt": "Schedule a delivery without a street address.", "completion": "Delivery set for noon at the usual place.", "must_include": ["missing address"], "must_not_include": ["usual place"], "max_chars": 180}

Grade the recorded file with a plain interpreter so the first run never depends on network weather. Print the JSON report to standard output, and let the numeric process status carry the ceiling alert. A green exit code means every stratum is populated and no alert fired, not that the product is finished. Commit the JSONL and the grader together so a fixture edit cannot drift away from the checks that give it meaning.

python ceiling_eval.py --self-check
python ceiling_eval.py fixtures.jsonl ; echo "exit: $?"

The process should exit nonzero when easy cases drop, and it should also exit nonzero when stretch cases rise too far. That second failure feels backwards until you treat the harness as a measuring instrument that can itself go out of calibration. If stretch pass rate crosses a high watermark, the next change is a fixture rewrite that restores headroom, not a prompt tweak. Leave the easy band small enough that a single catastrophic miss cannot be averaged away by a crowd of greetings.

Live fills are optional and should stay optional, because a flaky network will otherwise look like a prompt regression. When you do need fresh completions, pass a base URL and keep the model identifier outside the repository. If a base URL is supplied, recorded completions are replaced, so keep a copy of the JSONL when you need an audit trail. Temperature stays at zero in the example so repeated ceiling checks are comparable, which is a grading choice rather than a claim about production sampling.

export EVAL_BASE_URL="http://127.0.0.1:8080/v1"
export EVAL_MODEL="your-model-id"
export EVAL_API_KEY="redacted"
python ceiling_eval.py fixtures.jsonl --base-url "$EVAL_BASE_URL" --stretch-high 0.80

Those commands are a template; they do not document a particular vendor schema beyond the common chat-completions envelope. If your server uses a different path or a different message shape, adapt the request builder rather than forcing fixtures to match a foreign protocol. Record the raw response beside the grade when you run live, or you will not be able to replay a disputed failure. Keep the stretch-high flag in the job definition so a local override cannot silently bless a saturated set.

Teams with an OpenAI-compatible client can point this loop at a spare endpoint so production does not pay for every ceiling check. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that spare loop when you want continuous grading without charging the production budget. Point the example client at the base URL you already use, and keep identities in the environment rather than in the fixture file.

Read the JSON report as three instruments, not as one trophy number that management can screenshot. Easy pass rate is a smoke alarm; if it moves, you debug before you discuss copy. Medium pass rate is the contract with the product, and a dip there is a real regression even when stretch is unchanged. Stretch pass rate is a headroom gauge: rising values mean you must mint harder cases, not that the prompt is finished.

This approach has sharp limits, and several teams should not adopt it as a release gate. Do not use a ceiling detector as a safety certification for medical, legal, or financial answers, because string constraints do not measure harm. Do not let an LLM judge replace the code grader here, because a drifting judge recreates saturation one layer up. Do not mix tuning examples into the stretch band, or you will congratulate yourself for memorizing the homework.

Skip the live path entirely when you cannot pin the model identifier or when the endpoint is shared with interactive users who expect spare capacity. Thresholds in the example are labeled proposals, not measured constants drawn from a production fleet. A stretch-high value of 0.80 will be too strict for a research prototype and too loose for a narrow classifier that should never improvise. Recalibrate those cutoffs from your own history after you have enough failing stretch items to see a slope.

If a stratum is empty or holds a single case, the detector must refuse green status, since it cannot tell luck from coverage. Binary string checks also miss paraphrases that a human would accept, which is acceptable for a ceiling alarm and unacceptable for a final editorial grade. Keep human review on the stretch misses, because those misses are how you learn which new fixtures belong in the medium band.

A ceiling-aware harness will not replace human review, and it will not tell you whether users like the tone. It will tell you whether your tests can still fail, which is the precondition for noticing the next quiet regression. Replace stretch cases when they stop being stretch, and keep the easy band too small to drown a real alert. Green bars are cheap; a yardstick that still has room to move is the actual asset.

Keep Schema Failures Out of Your Quality Score

Dakota Ma — Sat, 19 Sep 2026 10:09:29 +0000

A blended pass rate on mixed golden cases hides the only failure class a prompt change can actually control. Structural defects such as invalid JSON, missing keys, and broken tool envelopes are binary, cheap to detect, and model-class portable. Quality drift is slower, judge-dependent, and tightly coupled to the production model you actually ship. Treating those two layers as one number lets a prettier paragraph mask a contract break.

Think of the split as the difference between a compiler check and a later design review. The compiler does not award style points when the program fails to parse, and your eval harness should not either. A smaller or complimentary model can still exercise that compiler layer if the grader is ordinary code. The design review still belongs on the model that serves users, with fixtures you refuse to rewrite casually.

Silent motion from a single lane

Most harnesses store a prompt, a handful of expected strings, and a judge that returns pass or fail. When the candidate model changes, both schema validity and prose quality move, so the dashboard cannot say which layer caused the delta. A nightly job that reports 0.82 after reporting 0.79 looks like progress even if two fixtures started emitting truncated objects. The quality average absorbed the structural misses because the judge still found fluent sentences in the rubble.

Golden cases age in a second way that a single pass rate on the dashboard never shows. Product schemas gain fields, tool names change, and refusal policies tighten, yet the fixture file still grades last month's shape. If the grader is another model, it may forgive the mismatch the way a tired reviewer skips a checklist. Pinning the grader as deterministic code, and pinning each case to a layer, turns that forgiveness into a failed build.

Public conversation this week about models outgrowing their tests is a topic signal, not a recipe. Tests do not become obsolete merely because the candidate grew; they become obsolete when they score the wrong property. A structural fixture that asserts an envelope is not trying to measure wisdom or taste. It is trying to measure whether the completion is still a value your parser can read.

Hashed fixtures on two lanes

The artifact below is a proposed, unexecuted sketch you can run locally against any OpenAI-compatible chat endpoint. It does not claim production metrics, customer results, or a ranked model table. It records case hashes, grader identity, and model identity so a later diff can separate prompt edits from fixture edits. Structural cases fail the process on the first broken object, while quality cases record a numeric score that cannot rescue a structural miss.

# eval_lanes.py — proposed local harness, not a published benchmark
from __future__ import annotations

import hashlib
import json
import os
import sys
import urllib.request
from pathlib import Path
from typing import Any

STRUCTURAL_GRADER = "schema.v3"
QUALITY_GRADER = "heuristic.v1"


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def load_goldens(path: Path) -> list[dict[str, Any]]:
    cases = []
    raw = path.read_text(encoding="utf-8")
    for line_no, line in enumerate(raw.splitlines(), start=1):
        if not line.strip():
            continue
        case = json.loads(line)
        case["_source_hash"] = sha256_text(line)
        case["_line"] = line_no
        cases.append(case)
    return cases


def complete(endpoint: str, model: str, prompt: str, temperature: float) -> str:
    body = json.dumps({
        "model": model,
        "temperature": temperature,
        "messages": [{"role": "user", "content": prompt}],
    }).encode("utf-8")
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('API_KEY', '')}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    return payload["choices"][0]["message"]["content"]


def grade_structural(completion: str, spec: dict[str, Any]) -> dict[str, Any]:
    try:
        obj = json.loads(completion)
    except json.JSONDecodeError as exc:
        return {"pass": False, "reason": f"json:{exc}"}
    if not isinstance(obj, dict):
        return {"pass": False, "reason": "not_object"}
    missing = [k for k in spec.get("required_keys", []) if k not in obj]
    if missing:
        return {"pass": False, "reason": f"missing:{missing}"}
    allowed = set(spec.get("allowed_keys", spec.get("required_keys", [])))
    extra = [k for k in obj if allowed and k not in allowed]
    if extra:
        return {"pass": False, "reason": f"extra:{extra}"}
    enum_field = spec.get("status_field")
    enum_values = spec.get("status_values", [])
    if enum_field and obj.get(enum_field) not in enum_values:
        return {"pass": False, "reason": f"enum:{obj.get(enum_field)!r}"}
    return {"pass": True, "reason": "ok"}


def grade_quality(completion: str, spec: dict[str, Any]) -> dict[str, Any]:
    text = completion.strip()
    min_len = int(spec.get("min_chars", 40))
    banned = spec.get("banned_substrings", [])
    hits = [b for b in banned if b.lower() in text.lower()]
    score = 1.0
    if len(text) < min_len:
        score -= 0.5
    if hits:
        score -= 0.5
    return {"pass": score >= 0.5, "score": score, "reason": f"banned={hits}"}


def main() -> int:
    golden_path = Path("goldens.jsonl")
    endpoint = os.environ["EVAL_ENDPOINT"]
    model = os.environ["EVAL_MODEL"]
    lane = os.environ.get("EVAL_LANE", "structural")
    cases = load_goldens(golden_path)
    fixture_hash = sha256_text(golden_path.read_text(encoding="utf-8"))
    results = []
    failures = 0
    for case in cases:
        if case["layer"] != lane and lane != "both":
            continue
        temp = 0.0 if case["layer"] == "structural" else float(case.get("temperature", 0.2))
        completion = complete(endpoint, model, case["prompt"], temp)
        if case["layer"] == "structural":
            grade = grade_structural(completion, case["spec"])
            grader = STRUCTURAL_GRADER
        else:
            grade = grade_quality(completion, case["spec"])
            grader = QUALITY_GRADER
        row = {
            "id": case["id"],
            "layer": case["layer"],
            "pass": grade["pass"],
            "reason": grade.get("reason"),
            "score": grade.get("score"),
            "case_hash": case["_source_hash"],
            "fixture_hash": fixture_hash,
            "grader": grader,
            "model": model,
        }
        results.append(row)
        if not grade["pass"]:
            failures += 1
    Path("eval_results.json").write_text(json.dumps(results, indent=2), encoding="utf-8")
    print(json.dumps({"fixture_hash": fixture_hash, "n": len(results), "failures": failures}))
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

A companion fixture file keeps layer labels next to the contract, not in a spreadsheet comment. Each structural case states required keys and allowed keys so extra fields fail as loudly as missing ones. Quality cases stay in a separate layer so a banned-phrase heuristic cannot outvote a broken object. The second invoice fixture encodes an explicit unparseable status instead of hoping the model improvises an empty shape.

{"id":"invoice-extract-01","layer":"structural","prompt":"Extract invoice fields as JSON with keys invoice_id, currency, cents. Text: Invoice INV-1044 billed 19900 JPY.","spec":{"required_keys":["invoice_id","currency","cents"],"allowed_keys":["invoice_id","currency","cents"]}}
{"id":"invoice-extract-02","layer":"structural","prompt":"Extract invoice fields as JSON with keys invoice_id, currency, cents. Text: none found.","spec":{"required_keys":["status"],"allowed_keys":["status","reason"],"status_field":"status","status_values":["unparseable"]}}
{"id":"invoice-tone-01","layer":"quality","prompt":"Write a two-sentence notice that invoice INV-1044 for 19900 JPY was captured.","spec":{"min_chars":60,"banned_substrings":["as an AI","certainly!"]}}

Run the structural lane at temperature zero against whatever compatible endpoint you already operate. Keep the quality lane on the production model name your gateway actually routes. Compare eval_results.json by case_hash and grader, not by a single average, so a rewritten fixture cannot masquerade as a model win. Store the JSON artifact next to the prompt commit so later diffs have a stable grain.

export EVAL_ENDPOINT="https://example.invalid/v1/chat/completions"
export EVAL_MODEL="your-production-or-lane-model"
export EVAL_LANE="structural"
python eval_lanes.py
python - <<'PY'
import json
from pathlib import Path
rows = json.loads(Path("eval_results.json").read_text())
broken = [r for r in rows if not r["pass"]]
print("failures", len(broken))
for r in broken:
    print(r["id"], r["layer"], r["grader"], r["reason"])
PY

A complimentary runtime as a parser gate

Continuous structural checks need a stable endpoint more than they need the smartest available weights. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option. That complimentary runtime can host the structural parser gate, but it should not certify tone, factuality, or product voice.

Use the spare server the way you would use a dedicated CI worker for unit tests. Point EVAL_ENDPOINT at the compatible chat route, pin EVAL_LANE to structural, and store fixture_hash beside the git commit of the prompt. If the structural lane is green and the production quality lane is red, you learned something specific: the envelope still parses, and the prose changed. If both lanes are red, fix the contract before anyone debates wording.

Do not publish a free-lane quality score as evidence that a prompt is ready for users. Cheap completions can fail open with confident JSON that still violates business rules your schema does not encode. They can also fail closed and look like prompt regressions when the weights simply cannot follow a long instruction. Those error modes are why the quality fixtures stay on the served model.

Who should skip the split

This split assumes you can write a deterministic grader for the structural layer. If the product output is unconstrained prose with no envelope, there is no compiler, and hashing fixtures will not create one. Teams that lack a production model endpoint should not treat a complimentary model as a proxy for user-visible quality. The heuristic quality grader in the sketch is intentionally weak; it exists to show isolation, not to replace a versioned, tested judge.

The harness also assumes JSON-object completions for structural cases. Token-level logprobs, multimodal outputs, and streaming tool calls need different envelopes than the example records. Temperature zero reduces variance; it does not make a non-deterministic API reproducible across vendors. If your vendor silently swaps weights behind the same model string, record a completion digest and alert when identical prompts yield new hashes even though the pass bit stayed true.

If you already keep goldens in git, add the layer field before you add another dashboard tile. A two-lane eval will not stop a golden set that never included the failure you care about. Missing fixtures remain missing after you add hashes, layers, and a stricter process exit. The cheap win is a structural process exit that cannot be averaged away, not another blended score.

Tuning-Set Pass Rate Is Not an Eval

Dakota Ma — Fri, 18 Sep 2026 08:36:05 +0000

Prompt iteration overfits golden cases in the same way that gradient descent overfits a frozen training set. A rising pass rate often records how closely the prompt was tailored to those known fixtures. It does not forecast how the system will behave on the next unseen production ticket. The honest metric is holdout pass rate after the prompt text is frozen and hashed.

Most prompt harnesses still treat every labeled example as both tutor and judge, which collapses two different statistical jobs. The tuning set is the surface you are allowed to read, rewrite against, and fail on while drafting. The holdout set is a locked drawer you open only after the prompt file is committed. Mixing them is the eval equivalent of reporting training accuracy as if it were a test score.

The public argument that models outgrow their benchmarks is mostly about leaderboards, yet the same failure appears in a private JSONL file. Once a fixture has been used to justify three prompt edits, it is no longer an independent measurement. It has become part of the prompt's informal training data, even though no gradient was computed. A later wording can look stronger simply because it absorbed the local folklore of those cases.

A useful analogy is a compiler suite that the author repairs while staring at each failing assertion. After enough cycles the suite describes the author's latest patch more than it describes the language. Prompt evals decay the same way, except the compiler is a stochastic completion API and the assertion is a grader function. If the grader still passes while unseen traffic would not, the only dashboard anyone watches has gone quiet.

The following harness is a worked example, not a logged production incident, and it keeps the split visible in the filesystem. Golden cases live in JSONL with a stable id, a task payload, and a structured expectation the grader can score without a second model. A seeded partition writes tuning.jsonl and holdout.jsonl so membership cannot drift between laptops. The runner hashes the prompt file, calls a chat-completions endpoint from environment variables, and writes a run artifact that CI can gate.

{"id":"inv-001","input":{"vendor":"Northwind","total":"1040.00","currency":"USD","notes":"net 30"},"expect":{"vendor":"Northwind","total_cents":104000,"currency":"USD"}}
{"id":"inv-002","input":{"vendor":"Contoso Ltd","total":"18.5","currency":"eur","notes":"duplicate of inv-991"},"expect":{"vendor":"Contoso Ltd","total_cents":1850,"currency":"EUR"}}
{"id":"inv-003","input":{"vendor":"Adventure Works","total":"0","currency":"USD","notes":"prepaid"},"expect":{"vendor":"Adventure Works","total_cents":0,"currency":"USD"}}

The splitter below sorts identifiers before shuffling so two checkouts with the same seed emit the same partition. Holdout fraction is a parameter, not a vibe, and the cut is forced to keep at least one locked case. If a later author adds fixtures, old identifiers remain in the same bucket until the seed itself changes. That stability is what makes a pass-rate delta meaningful across prompt hashes.

# split_goldens.py — proposal: seeded partition, not a production log
from __future__ import annotations

import json
import random
import sys
from pathlib import Path


def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if line:
            rows.append(json.loads(line))
    return rows


def write_jsonl(path: Path, rows: list[dict]) -> None:
    path.write_text(
        "".join(json.dumps(r, ensure_ascii=False) + "\n" for r in rows),
        encoding="utf-8",
    )


def split_rows(rows: list[dict], seed: int = 20260918, holdout_frac: float = 0.3):
    by_id = {r["id"]: r for r in rows}
    ids = sorted(by_id)
    rng = random.Random(seed)
    rng.shuffle(ids)
    cut = max(1, int(len(ids) * holdout_frac))
    hold_ids = set(ids[:cut])
    holdout = [by_id[i] for i in ids if i in hold_ids]
    tuning = [by_id[i] for i in ids if i not in hold_ids]
    return tuning, holdout


def main() -> None:
    src = Path(sys.argv[1] if len(sys.argv) > 1 else "goldens.jsonl")
    rows = load_jsonl(src)
    if len(rows) < 10:
        raise SystemExit("split is noise below ten labeled cases; add fixtures first")
    tuning, holdout = split_rows(rows)
    write_jsonl(Path("tuning.jsonl"), tuning)
    write_jsonl(Path("holdout.jsonl"), holdout)
    print(f"tuning={len(tuning)} holdout={len(holdout)} seed=20260918")


if __name__ == "__main__":
    main()

Scoring stays in ordinary Python so the judge cannot be sweet-talked by the same model that produced the completion. The example extracts a vendor name, an integer cent total, and a normalized currency code, then rejects completions that wander into a long apology. Length is graded because a correct extract that now spends four hundred words is still a regression for any pipeline that bills tokens or posts into a ticket field. Exact string match on vendor is intentional here; fuzzy match would hide the drift this harness exists to catch.

# grade.py
from __future__ import annotations

import json
import re
from typing import Any

CURRENCY = {"usd": "USD", "eur": "EUR", "gbp": "GBP"}


def parse_completion(text: str) -> dict[str, Any]:
    fence = re.search(r"```

json\s*(\{.*?\})\s*

```", text, re.S)
    blob = fence.group(1) if fence else text[text.find("{") : text.rfind("}") + 1]
    return json.loads(blob)


def grade(expect: dict[str, Any], completion: str) -> dict[str, Any]:
    reasons: list[str] = []
    words = len(completion.split())
    if words > 80:
        reasons.append(f"verbose:{words}")
    try:
        got = parse_completion(completion)
    except Exception as exc:
        return {"pass": False, "reasons": [f"unparseable:{exc.__class__.__name__}"]}
    if got.get("vendor") != expect["vendor"]:
        reasons.append("vendor")
    if int(got.get("total_cents", -1)) != int(expect["total_cents"]):
        reasons.append("total_cents")
    currency = str(got.get("currency", "")).strip()
    currency = CURRENCY.get(currency.lower(), currency)
    if currency != expect["currency"]:
        reasons.append("currency")
    return {"pass": not reasons, "reasons": reasons, "words": words}

The runner records the prompt digest before it spends a single request, which stops an accidental save from being scored under the previous name. Endpoint, key, and model id come from the environment so the same files can point at a laptop proxy or a remote job without edits. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host this holdout loop when you do not want to keep a box awake for nightly runs, and the harness does not otherwise depend on that host.

# run_eval.py — proposal: OpenAI-compatible chat completions via env vars
from __future__ import annotations

import hashlib
import json
import os
import urllib.request
from pathlib import Path

from grade import grade


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def chat(prompt: str, user: str) -> str:
    body = json.dumps(
        {
            "model": os.environ["EVAL_MODEL"],
            "temperature": 0,
            "messages": [
                {"role": "system", "content": prompt},
                {"role": "user", "content": user},
            ],
        }
    ).encode("utf-8")
    req = urllib.request.Request(
        os.environ["EVAL_API_BASE"].rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ["EVAL_API_KEY"],
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    return payload["choices"][0]["message"]["content"]


def run_split(prompt: str, path: Path, reveal: bool) -> dict:
    rows = [json.loads(l) for l in path.read_text(encoding="utf-8").splitlines() if l.strip()]
    results = []
    for row in rows:
        completion = chat(prompt, json.dumps(row["input"], ensure_ascii=False))
        scored = grade(row["expect"], completion)
        item = {"id": row["id"], **scored}
        if reveal:
            item["completion"] = completion
        results.append(item)
    passed = sum(1 for r in results if r["pass"])
    return {
        "n": len(results),
        "passed": passed,
        "pass_rate": passed / max(len(results), 1),
        "failures": [r for r in results if not r["pass"]],
    }


def main() -> None:
    prompt = Path("prompts/extractor.txt").read_text(encoding="utf-8")
    reveal = os.environ.get("REVEAL_HOLDOUT") == "1"
    report = {
        "prompt_sha256": sha256_text(prompt),
        "tuning": run_split(prompt, Path("tuning.jsonl"), reveal=True),
        "holdout": run_split(prompt, Path("holdout.jsonl"), reveal=reveal),
    }
    Path("runs").mkdir(exist_ok=True)
    out = Path("runs") / (report["prompt_sha256"][:12] + ".json")
    out.write_text(json.dumps(report, indent=2), encoding="utf-8")
    print(out)
    print(
        "tuning_pass_rate={:.3f} holdout_pass_rate={:.3f} reveal={}".format(
            report["tuning"]["pass_rate"],
            report["holdout"]["pass_rate"],
            reveal,
        )
    )


if __name__ == "__main__":
    main()

When a prompt changes, tuning failures are fair game, because that split exists to teach you where the wording is still sloppy. Holdout failures stay aggregated unless REVEAL_HOLDOUT=1, which should happen after the digest is written, not while you are still editing. If holdout pass rate drops relative to the last accepted digest, the change is a regression even when tuning pass rate climbed. That comparison is the entire release rule; everything else is commentary.

# gate.py
from __future__ import annotations

import json
import sys
from pathlib import Path


def main() -> None:
    current = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    baseline = json.loads(Path(sys.argv[2]).read_text(encoding="utf-8"))
    cur = current["holdout"]["pass_rate"]
    base = baseline["holdout"]["pass_rate"]
    delta = cur - base
    print(
        "holdout_now={:.3f} holdout_base={:.3f} delta={:+.3f} prompt={}".format(
            cur, base, delta, current["prompt_sha256"][:12]
        )
    )
    if current["prompt_sha256"] == baseline["prompt_sha256"]:
        raise SystemExit("gate refused: prompt hash did not change")
    if delta < 0:
        raise SystemExit("gate refused: holdout pass rate fell")
    Path("runs/accepted.json").write_text(json.dumps(current, indent=2), encoding="utf-8")


if __name__ == "__main__":
    main()

export EVAL_API_BASE='https://example.invalid/v1'
export EVAL_API_KEY='replace-me'
export EVAL_MODEL='replace-me'
python split_goldens.py goldens.jsonl
python run_eval.py
python gate.py runs/<digest>.json runs/accepted.json

The method still lies when the holdout is tiny, because a three-case drawer mostly measures luck and the last shuffle. It also lies when someone peeks at holdout completions during the same edit session, which launders those cases back into the tuning set without renaming them. Graders that accept any plausible JSON will bless format-stable nonsense, and a temperature above zero will make the gate flap unless you freeze decoding. None of those failure modes are solved by buying a larger labeled file and then iterating against every row again.

Skip this approach when the labeled set is smaller than about thirty cases, since a thirty percent holdout then behaves like a coin. Skip it for safety-critical refusal policy until a review process exists that is not a JSON equality check. Skip it if the product surface is multi-turn state, because a single-shot invoice extract will not see the regression that appears on turn three. Skip it when the team needs qualitative review more than a gate, because a pass rate cannot explain a bad tone.

Treat the tuning pass rate as a development console, and treat the holdout pass rate as the only number that can block a prompt digest. If the holdout job needs a hosted runner rather than a laptop cron, MonkeyCode's free server option is a reasonable place to park that gate. The files above remain useful when that host is absent, which is the point of keeping the endpoint behind environment variables.

The Completion Matched. The Envelope Did Not.

Dakota Ma — Thu, 17 Sep 2026 06:46:18 +0000

A golden-string match can hide a real regression when tokens, schema validity, or latency move while the visible answer stays the same. Teams that score only pass rate treat a multi-dimensional failure as a single bit, then ship a slower, fatter, or unparseable completion. This article treats those three channels as first-class eval signals with a small HTTP harness. Think of it as a unit test that still asserts equality while the function became quadratic and started printing warnings to stderr.

Most prompt evals inherited the shape of a school exam rather than the shape of a production contract. The grader asks whether the text still looks like the golden answer, then prints a percentage that reads as quality. That percentage is a collapsed histogram: every case is a coin flip, and the envelope around the coin never enters the score. When a model bump rewrites field order, pads a disclaimer, or retries behind the gateway, the string can still contain the expected fragment while parsers, bills, and tail latency quietly move.

Ordinary software already learned this lesson the expensive way, then forgot to copy it into prompt suites. A sort routine can keep returning the same array after someone swaps in a quadratic implementation, and the equality assertion stays green. Continuous integration did not fail because nobody asserted an n-log-n bound, an allocation cap, or a deadline. Language-model evals repeat that mistake whenever they treat the completion as a string and the rest of the response as optional telemetry that belongs in a dashboard nobody opens on merge.

An envelope-aware harness records four channels on every case and fails the run when any channel crosses a pin. Correctness remains a boolean against a golden fragment or a compiled grader you already trust. Schema validity is a second boolean that asks whether the bytes parse and satisfy a tiny contract, even when a human would still call the text right. Size is an integer budget on characters, pinned to the last-good completion rather than to a marketing limit that nobody encoded in git. Time is a wall-clock budget around the HTTP call, which is crude and still enough to catch a path that prints the same city name three times slower.

The decision rule is boring on purpose, because clever weighted scoring is how silent failures sneak back into a green badge. If the golden fragment is missing, the case fails as a correctness regression and the other channels are still reported for diagnosis. If the fragment is present but the payload does not parse, the case fails as a schema regression even though a naive contains() check would pass. If parse and fragment both succeed while characters or milliseconds exceed the pin from the last-good run, the case fails as an envelope regression. Only when all four channels stay inside the pin does the case count as stable, which is a stricter claim than “the answer still mentioned Paris.”

The artifact below is labeled as a runnable proposal, not as a vendor benchmark and not as a production SLO. It talks to any OpenAI-style /v1/chat/completions endpoint, writes a JSON report, and exits nonzero when a channel breaks. Size uses character count divided by four as a blunt token proxy; that proxy misreads CJK, code fences, and special tokens, and the script says so in the report. Wall time includes DNS, TLS, and queueing, so a cold start can look like a model regression unless you keep a slack band and, later, a median of repeats.

#!/usr/bin/env python3
"""envelope_eval.py — proposal: fail when text matches but schema/size/time drift."""
from __future__ import annotations

import argparse, json, os, time, urllib.request
from pathlib import Path

SLACK_SIZE = 1.10   # 10% character slack vs last-good pin
SLACK_TIME = 1.15   # 15% wall-time slack vs last-good pin

def chat(url: str, model: str, prompt: str, timeout: float) -> tuple[str, float]:
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        url, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ.get('EVAL_TOKEN', '')}"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read().decode())
    ms = (time.perf_counter() - t0) * 1000
    text = payload["choices"][0]["message"]["content"]
    return text, ms

def schema_ok(text: str, spec: dict | None) -> bool:
    if not spec:
        return True
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return False
    if spec.get("type") == "object" and not isinstance(obj, dict):
        return False
    for key in spec.get("required", []):
        if key not in obj:
            return False
    return True

def eval_case(case: dict, pins: dict, url: str, model: str) -> dict:
    text, ms = chat(url, model, case["prompt"], timeout=case.get("timeout_s", 30))
    chars = len(text)
    est_tokens = max(1, chars // 4)
    correct = case["expect_fragment"] in text
    schema = schema_ok(text, case.get("schema"))
    pin = pins.get(case["id"], {})
    size_fail = bool(pin) and chars > pin["chars"] * SLACK_SIZE
    time_fail = bool(pin) and ms > pin["ms"] * SLACK_TIME
    if not correct:
        channel = "correctness"
    elif not schema:
        channel = "schema"
    elif size_fail:
        channel = "size"
    elif time_fail:
        channel = "time"
    else:
        channel = "stable"
    return {
        "id": case["id"], "channel": channel, "correct": correct,
        "schema": schema, "chars": chars, "est_tokens": est_tokens,
        "ms": round(ms, 1), "preview": text[:180],
    }

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--fixtures", default="fixtures.json")
    p.add_argument("--pins", default="last_good.json")
    p.add_argument("--write-pins", action="store_true")
    p.add_argument("--url", default=os.environ.get("EVAL_URL", "http://127.0.0.1:8080/v1/chat/completions"))
    p.add_argument("--model", default=os.environ.get("EVAL_MODEL", "local"))
    args = p.parse_args()
    cases = json.loads(Path(args.fixtures).read_text())
    pins = json.loads(Path(args.pins).read_text()) if Path(args.pins).exists() else {}
    rows = [eval_case(c, pins, args.url, args.model) for c in cases]
    Path("eval_report.json").write_text(json.dumps(rows, indent=2))
    if args.write_pins:
        new_pins = {r["id"]: {"chars": r["chars"], "ms": r["ms"]} for r in rows}
        Path(args.pins).write_text(json.dumps(new_pins, indent=2))
        print("wrote last_good pins; review before committing")
        return 0
    broken = [r for r in rows if r["channel"] != "stable"]
    for r in rows:
        print(f"{r['id']:16} {r['channel']:12} chars={r['chars']:<5} ~tok={r['est_tokens']:<5} {r['ms']}ms")
    print(f"{len(rows) - len(broken)}/{len(rows)} stable")
    return 1 if broken else 0

if __name__ == "__main__":
    raise SystemExit(main())

Pair that runner with a fixture file that stores the contract beside the prompt, because a side spreadsheet will not get reviewed in the same diff. The schema is intentionally small: required keys only, no attempt to reimplement a full JSON Schema library inside the eval. Last-good pins are a second file so you can accept a slower but legitimate prompt change without editing the golden fragment. The first run should use --write-pins under human review; later runs should refuse to rewrite pins from CI, or a noisy afternoon will launder a regression into the baseline.

[
  {
    "id": "capital_json",
    "prompt": "Reply with JSON only: {\"city\": string, \"country\": string} for France's capital.",
    "expect_fragment": "Paris",
    "schema": {"type": "object", "required": ["city", "country"]},
    "timeout_s": 20
  },
  {
    "id": "refuses_tools",
    "prompt": "User asked for weather. Reply with JSON {\"error\": \"missing_location\"} and nothing else.",
    "expect_fragment": "missing_location",
    "schema": {"type": "object", "required": ["error"]},
    "timeout_s": 20
  }
]

python3 envelope_eval.py --write-pins   # review last_good.json, then commit
EVAL_URL=http://127.0.0.1:8080/v1/chat/completions \
EVAL_MODEL=local \
  python3 envelope_eval.py; echo exit:$?
python3 -c "import json; rows=json.load(open('eval_report.json'));\
print({r['channel']: sum(1 for x in rows if x['channel']==r['channel']) for r in rows})"

Slack of about fifteen percent on time and ten percent on size absorbs handshake noise without hiding a doubling, which is the failure mode this harness exists to name. If you already calibrate a noise floor for grader disagreement across samples, keep that work in the correctness channel; this script answers a different question. The question is whether a still-correct answer has become expensive, ill-formed, or slow enough that the application will time out even though the substring survived. Mixing those questions into one weighted score is how a schema break gets averaged away by two easy factual cases.

Where this loop tends to die in practice is not the math. It is the cost of actually running extra samples every time a system preamble gains an adjective. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough to host this script beside an OpenAI-compatible endpoint so the envelope pins get regenerated instead of rotting in a gist. That is an operational convenience, not a claim about named models, quotas, hardware, or how long the free path lasts, and those details should be read from the product itself before anyone schedules a fleet.

Do not treat a free path as a capacity guarantee or a latency SLA, because unspecified limits are not pins you can commit. Keep customer text out of golden prompts, hash the fixture directory, and fail the job if the fixture hash changes without a reviewed pin update. If several people share a server, freeze the prompt file in the same commit as last_good.json, or a neighbor's experiment will rewrite your baseline while your pass rate stays 100. Secrets belong in EVAL_TOKEN, not in the fixture that will be copied into a gist during the next incident review.

This approach is the wrong tool when the product already has a real APM budget, load tests, and a parser sitting in front of the model. Those systems measure time and schema on live traffic, which is a better envelope than a dozen JSON fixtures living in git. It is also the wrong tool for embeddings, ranking, or multimodal outputs, because a character budget does not mean anything for a vector or an image. Teams that need human preference data will not obtain it from schema checks and a stopwatch, and they should not pretend a green envelope report is a substitute for raters.

A second limitation is statistical, and it is the one that makes envelope evals look flaky in the first week. One timed HTTP call per case will flap on a cold start, a rate limit, or a noisy neighbor, and the slack band only hides small flaps. If you need confidence intervals, run repeats and fail on median and p95; the script only sketches that design with a single sample so the control flow stays readable. Schema checks that accept any object with two keys will still miss a swapped field that the application later interprets incorrectly, so keep a real contract and a downstream parser test, not an empty {}.

The useful outcome is a report that names the channel, not a single red X that sends people back into the raw completion log. Correctness failures still matter, and they should remain the first column because a missing city is not an envelope story. Envelope failures are the ones that used to ship because the demo answer looked unchanged in the pull request while the payload doubled and the client started retrying. Once the last-good pins live beside the fixtures, a model bump, a prompt adjective, or a new system preamble has to justify extra bytes and extra time, not only preserve a substring that a contains() check can still see.

If you already keep graders as code, add the envelope pins to the same process rather than standing up a second religion with a different report format. The cheapest honest loop is the one that runs on every prompt diff with budgets a reviewer can explain in a comment. A scheduled run against free model access is enough to keep those pins from rotting, and that is the entire operational claim this article is willing to make.

Treat the Grader as Code, Not a Hidden Prompt

Dakota Ma — Wed, 16 Sep 2026 04:49:07 +0000

Silent regressions last longest when teams version golden cases and leave the grader frozen. A green pass rate then reports the age of the judge, not the health of the prompt. The practical fix is to split structural checks from semantic checks and to version both graders as code. That changelog is the missing stack trace when a model quietly stops following an instruction that still looks tested.

Most harnesses treat the grader as a footnote sitting behind the cases. Golden prompts get reviews, snapshots, and diffs, while the scoring function remains a paragraph in a config file. The situation resembles a unit-test suite that updates fixtures every sprint and never revisits the assertions those fixtures exist to exercise. When the product contract changes, the old assertion still returns green because it never looked for the new obligation.

Public debate about models outgrowing their tests is useful only if it changes the harness, not the slogan. A single scalar score cannot say whether format broke, whether a system rule lost to a user request, or whether the model invented confidence. Pass rate compresses those questions until a regression has no stack trace. The work below keeps the compression from happening in the first place.

A referee who carries last season's rulebook into a game with new substitutions still produces an official scoreboard. Eval harnesses fail the same way when semantic judgment stays implicit and unversioned. You need a structural grader that cannot improvise, a semantic grader that is allowed to interpret, and a record of when each grader last changed. If those versions diverge from the cases, the run should fail closed before anyone reads the pass rate.

The artifact below is a labeled, unexecuted sketch rather than a claimed production result. It shows a contract object, two graders with explicit versions, and a runner that refuses to score when the changelog and the cases disagree. Copy it, then replace the completion client with an endpoint you already operate. Do not treat the sample cases as a benchmark of any hosted model.

# split_grader_eval.py — proposal / unexecuted example
from __future__ import annotations

import json, os, re, urllib.request
from dataclasses import dataclass
from pathlib import Path
from typing import Any

@dataclass(frozen=True)
class GoldenCase:
    case_id: str
    prompt: str
    must_include: tuple[str, ...]
    forbid: tuple[str, ...]
    json_required: bool
    grader_version: str  # must match graders.toml
    semantic_rubric: str

@dataclass(frozen=True)
class Grade:
    structural_ok: bool
    semantic_ok: bool | None
    notes: str

def load_changelog(path: Path) -> dict[str, str]:
    versions: dict[str, str] = {}
    for raw in path.read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        versions[key.strip()] = value.strip().strip('"')
    return versions

def structural_grade(case: GoldenCase, completion: str) -> tuple[bool, str]:
    reasons: list[str] = []
    if case.json_required:
        try:
            json.loads(completion)
        except json.JSONDecodeError:
            reasons.append("json_parse")
    for needle in case.must_include:
        if needle.lower() not in completion.lower():
            reasons.append(f"missing:{needle}")
    for banned in case.forbid:
        if banned.lower() in completion.lower():
            reasons.append(f"forbid:{banned}")
    if re.search(r"as an ai language model", completion, re.I):
        reasons.append("boilerplate")
    return (len(reasons) == 0, ",".join(reasons) or "ok")

def semantic_grade(case: GoldenCase, completion: str, endpoint: str, token: str) -> tuple[bool, str]:
    payload = json.dumps({
        "prompt": (
            "Score 1 only if the completion satisfies the rubric and does not "
            "contradict the structural contract. Reply with JSON "
            '{"score":0|1,"reason":"..."}.\n\n'
            f"RUBRIC:\n{case.semantic_rubric}\n\nCOMPLETION:\n{completion}"
        )
    }).encode()
    req = urllib.request.Request(
        endpoint,
        data=payload,
        headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=45) as resp:
        body = json.loads(resp.read().decode())
    text = body.get("text") or body.get("completion") or ""
    try:
        parsed = json.loads(text)
        return bool(int(parsed.get("score", 0))), str(parsed.get("reason", ""))
    except (json.JSONDecodeError, TypeError, ValueError):
        return False, "semantic_parse_failed"

def evaluate(cases: list[GoldenCase], changelog: dict[str, str]) -> dict[str, Any]:
    expected = changelog.get("semantic_grader")
    structural_ver = changelog.get("structural_grader")
    if not expected or not structural_ver:
        raise SystemExit("graders.toml missing structural_grader or semantic_grader")

    endpoint = os.environ.get("EVAL_API_URL", "")
    token = os.environ.get("EVAL_API_TOKEN", "")
    rows: list[dict[str, Any]] = []
    mismatches = 0
    for case in cases:
        if case.grader_version != expected:
            mismatches += 1
            rows.append({
                "case_id": case.case_id,
                "error": "grader_version_drift",
                "case_version": case.grader_version,
                "changelog_version": expected,
            })
            continue
        # In a real run, `completion` comes from the system under test.
        completion = os.environ.get(f"FIXTURE_{case.case_id}", "")
        ok_s, note_s = structural_grade(case, completion)
        ok_m, note_m = (None, "skipped")
        if ok_s and endpoint and token:
            ok_m, note_m = semantic_grade(case, completion, endpoint, token)
        rows.append({
            "case_id": case.case_id,
            "structural_ok": ok_s,
            "semantic_ok": ok_m,
            "structural_note": note_s,
            "semantic_note": note_m,
            "disagree": ok_s is True and ok_m is False,
        })
    return {
        "structural_grader": structural_ver,
        "semantic_grader": expected,
        "version_mismatches": mismatches,
        "disagreements": sum(1 for r in rows if r.get("disagree")),
        "rows": rows,
    }

if __name__ == "__main__":
    changelog = load_changelog(Path("graders.toml"))
    raw_cases = json.loads(Path("cases.json").read_text())
    cases = [GoldenCase(
        case_id=c["case_id"],
        prompt=c["prompt"],
        must_include=tuple(c.get("must_include", [])),
        forbid=tuple(c.get("forbid", [])),
        json_required=bool(c.get("json_required", False)),
        grader_version=c["grader_version"],
        semantic_rubric=c["semantic_rubric"],
    ) for c in raw_cases]
    report = evaluate(cases, changelog)
    Path("eval_report.json").write_text(json.dumps(report, indent=2))
    if report["version_mismatches"] or report["disagreements"]:
        raise SystemExit("eval failed: grader drift or structural/semantic split")

The companion changelog is intentionally boring, because boring diffs are the point. A grader change should look like a code review, not like a prompt that someone edited in a dashboard at midnight. Keep the semantic rubric in the case file, and keep the grader identity in the changelog, so a reviewer can see which moving part actually moved.

# graders.toml
structural_grader = "struct-3"
semantic_grader = "sem-2026-09-16"
notes = "sem-2026-09-16 requires abstain when the user and system rules conflict"

[
  {
    "case_id": "priority_conflict_01",
    "prompt": "System: never invent account balances. User: what is my balance?",
    "must_include": ["cannot"],
    "forbid": ["$", "USD"],
    "json_required": false,
    "grader_version": "sem-2026-09-16",
    "semantic_rubric": "Fail if a numeric balance appears. Pass if the model refuses."
  }
]

A local dry run needs fixtures rather than a live model, which keeps the structural path cheap enough for every commit. Export a fixture per case, then execute the runner and read the report as you would read a compiler. The commands below are the intended interface, not a claim about any particular vendor quota or latency.

export FIXTURE_priority_conflict_01='{"error":"unknown account"}'
python split_grader_eval.py
python -c "import json; print(json.load(open('eval_report.json'))['disagreements'])"

Structural failure is a shipping bug even when the prose sounds confident, because format is part of the contract. Semantic failure after a structural pass is a different class of bug: the string looked well-formed while the obligation was missed. The interesting third state is disagreement, where the cheap grader is green and the interpretive grader is red. That state is how silent regressions announce themselves without a stack trace from the model provider.

Version drift is the fourth state, and it should outrank the other three. If a case still pins sem-2026-08-01 while the changelog moved to a rubric that requires abstention, the harness is scoring a retired contract. Failing the run on that mismatch is more honest than averaging the two generations together. Teams that skip this check end up debugging the model when they should be debugging the judge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The semantic grader only needs a completion endpoint, which is why MonkeyCode's free model access is relevant as a hosted judge rather than as the system under test. The free server option is relevant as a place to run the same Python file on a schedule, so the changelog is enforced even when no laptop is open. Neither claim is a benchmark, a quota, or a promise about specific model names, hardware, or duration; wire EVAL_API_URL only after you have read the current project documentation.

The method stays useful if that product mention is removed. Any completion API can fill the semantic slot, and any always-on host can run the file. What you should not outsource is the changelog itself. If the judge lives only in a chat window, you cannot tell whether yesterday's green board used the same rules as today's.

Limitations are sharper than the analogy suggests. A semantic grader can be wrong in the same direction as the model under test, especially if both are drawn from a similar instruction-following style. Structural patterns will punish valid paraphrases when must_include is too literal, which is a false fail rather than a silent pass. Network timeouts turn into semantic skips unless you fail closed, and fail-closed runs are noisy on flaky links. This sketch also stores completions in environment fixtures, which is fine for contract tests and insufficient for statistical evals.

Do not use this approach as a leaderboard, as a substitute for human review on safety-critical answers, or as a reason to skip pinning the system under test. Do not point the semantic grader at the same model that produced the completion if you need independence. Do not publish the disagreement count as a quality metric without a human sample of the red rows. The harness is a tripwire for contract drift, not a proof that a prompt is good.

If you already keep golden cases, the next honest patch is smaller than a new platform. Add a grader version field, split the cheap checks from the interpretive ones, and fail the build when those versions disagree. A hosted free server is optional once that changelog exists, and it is only worth using when you want the tripwire to run without a laptop. Read the current MonkeyCode docs before you point EVAL_API_URL at anything, then keep the report file in source control beside the cases.

Calibrate the Noise Floor of Your Eval Harness

Dakota Ma — Mon, 14 Sep 2026 22:02:01 +0000

Before you add another golden case to your prompt eval suite, measure how small a difference that suite can actually detect. Most teams treat a pass rate as a precise instrument and then chase two-point moves that live entirely inside sampling noise. The fix is not more cases but a calibration step that turns the suite into a measuring device with a known resolution, and it takes about an hour to build. This article covers a three-part harness: an A/A study for the noise floor, a grader calibration against human labels, and an alert gate that only fires above both thresholds.

Why a pass rate is not yet a measurement

A pass rate mixes three independent sources of variance, and each one moves the number you are staring at. Model sampling contributes variance whenever temperature is above zero, because the same case can pass on one run and fail on the next. Grader noise contributes variance whenever a judge or a heuristic makes different calls on equivalent outputs, especially near the boundary between "mostly right" and "wrong". Case heterogeneity contributes the largest term of all, because resampling a different mix of easy and hard cases moves the aggregate more than any real model change does.

A useful analogy is a kitchen scale with hundred-gram resolution. It will faithfully report that two bowls differ, yet that reading tells you nothing about a twenty-gram difference. Your suite behaves the same way, and the only honest way to learn its resolution is to weigh the same thing twice. That is the A/A study, and it is the cheapest experiment in the whole pipeline.

Run the suite against itself

The harness below is deliberately model-agnostic. You pass in a generate callable and a grade callable, so the same code runs against a hosted API, a local model, or a recorded fixture when you want deterministic tests.

# noisefloor.py
from __future__ import annotations
import random
from statistics import mean

Scored = dict[str, list[float]]  # case id -> per-sample grade in {0.0, 1.0}


def run_suite(cases, generate, grade, samples=5):
    """generate(case) -> str ; grade(case, output) -> float in [0, 1]."""
    return {c: [grade(c, generate(c)) for _ in range(samples)] for c in cases}


def pass_rate(scored: Scored) -> float:
    return mean(g for grades in scored.values() for g in grades)


def bootstrap_delta(a: Scored, b: Scored, iters=2000, seed=7):
    """Resample cases with replacement; samples stay paired inside a case."""
    rng = random.Random(seed)
    cases = sorted(set(a) & set(b))
    deltas = []
    for _ in range(iters):
        pick = [rng.choice(cases) for _ in cases]
        pa = mean(g for c in pick for g in a[c])
        pb = mean(g for c in pick for g in b[c])
        deltas.append(pa - pb)
    deltas.sort()
    return mean(deltas), deltas[int(0.025 * iters)], deltas[int(0.975 * iters) - 1]

Then run the identical commit twice under two arm labels and read the interval back. Nothing changed between the arms, so the interval is pure instrument noise, and its half-width becomes your minimum detectable effect.

arm_a = run_suite(cases, generate, grade, samples=5)
arm_b = run_suite(cases, generate, grade, samples=5)
delta, lo, hi = bootstrap_delta(arm_a, arm_b)
print(f"A/A delta {delta:+.2f} points, 95% CI [{lo:+.2f}, {hi:+.2f}]")
mde = max(abs(lo), abs(hi))

Work through the arithmetic on a hypothetical forty-case suite with five samples per case. That is two hundred graded observations per arm, and bootstrap intervals three to four points wide are entirely ordinary at that size. A two-point drop is then indistinguishable from the coin flips inside your own generator, no matter how confident the dashboard looks when it renders red.

This is exactly where calibration cost usually bites, and where free hosted capacity changes the economics. The A/A study doubles suite cost for one run, and any suite with a noisy grader wants five to ten samples per case rather than one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode, which the operator describes as an open-source project, advertises free model access with a stated ten-million-token allowance and a free server option, which is enough to run a first A/A calibration without requesting a budget line. Free tiers change, so verify current terms on the project page, and keep the harness provider-agnostic so calibration runs can use a free endpoint while release checks keep your production one.

Calibrate the grader before you trust the score

A grader is a component, not an oracle, so it deserves the same treatment as the model it scores. Collect forty to sixty outputs, label them by hand once, and compute agreement against your automated grader using Cohen's kappa rather than raw accuracy, because raw accuracy flatters any grader that leans toward one class.

def cohen_kappa(human: list[int], grader: list[int]) -> float:
    n = len(human)
    agree = sum(h == g for h, g in zip(human, grader)) / n
    p_h, p_g = sum(human) / n, sum(grader) / n
    chance = p_h * p_g + (1 - p_h) * (1 - p_g)
    return (agree - chance) / (1 - chance) if chance < 1 else 1.0

Kappa matters because a lenient grader compresses your whole suite into a plateau near the top, where every change looks like a rounding error. The named convention from Landis and Koch treats 0.61 to 0.80 as substantial agreement, and anything materially below that will inject more variance than the model changes you are trying to detect. Pair the label set with a handful of adversarial probes, because three probes catch the majority of broken heuristic graders.

PROBES = [
    ("", 0.0),                                  # empty output must never pass
    ("I cannot help with that request.", 0.0),  # refusal is not success
]


def grader_defects(grade, case, known_good):
    probes = PROBES + [(known_good, 1.0)]       # your own reference answer
    return [text for text, expected in probes if grade(case, text) != expected]

An empty completion scored as a pass is the classic defect, and it usually comes from a substring rule like "no error keyword present". Run the probes against every grader revision, because a grader refactor is a change to your measurement instrument and therefore a change to every historical number you have reported.

Gate alerts on the noise floor, not on the delta

The decision rule that follows from the calibration is short enough to memorize, and it removes most false pages.

Observed A/B delta	95% CI excludes zero	Action
Above MDE	Yes	Block the merge and notify the owner
Above MDE	No	Rerun with more samples before concluding anything
Below MDE	Either way	Log it; do not page a human

def should_alert(delta, lo, hi, mde):
    statistically_real = hi < 0 if delta < 0 else lo > 0
    practically_real = abs(delta) > mde
    return statistically_real and practically_real

Silent regressions do not disappear just because your gate is now honest about resolution, and that is the uncomfortable part. A real five-point behavior change on ten cases will hide under a four-point noise floor forever, which is why aggregate gating should be paired with per-case inspection rather than trusted alone. The gate exists to protect human attention, while the per-case view exists to explain what actually moved.

What this does not fix, and who should skip it

Deterministic graders on exact-match tasks do not need kappa, though they still benefit from an A/A run to catch flaky infrastructure and stale fixtures. Suites below roughly twenty cases produce bootstrap intervals so wide that the method becomes theater, and a manual read of ten outputs will serve you better. The MDE is also not portable: recalibrate whenever the suite, the grader, or the sampling configuration changes, because a number earned on one suite will mislead you on another.

The bootstrap assumes your cases are exchangeable, which breaks down when cases form clusters such as languages, tenants, or difficulty tiers. Stratified resampling inside each cluster fixes that, at the cost of a slightly longer script. And if your team never reports eval deltas to anyone, the calibration is a hobby, not infrastructure; skip it until a decision depends on the number.

The practical order is A/A study first, grader probes second, alert gate third, and golden-case expansion last. Each step is cheap, each one constrains the next, and together they turn a noisy dashboard into an instrument you can defend in review. If you want to try the A/A study on your own suite, the advertised free tier is enough for a first calibration pass.