Pin Fixture, Grader, and Slot Before You Diff

Eval history that records only pass or fail quietly mixes unlike backends, then presents the mixture as a quality trend. Golden cases still matter, but they name expected behavior, not the instrument that produced the completion. When a serving slot changes and the fixture does not, a falling pass rate is an unlabeled instrument change until proven otherwise. The working unit of comparison is a triple of fixture hash, grader version, and model slot, not a bare percentage.

Complimentary and rotating model endpoints make that mixture easy to miss because the client code barely changes. A base URL swap or a default model field is enough to alter completions while JSONL fixtures remain untouched. Overnight eval jobs then look like prompt regressions, even though the prompt bytes and grader rules never moved. The failure mode is closer to changing a compiler and keeping the same unit tests without recording the toolchain.

Metrology notebooks refuse to compare readings until the gauge identifier is written beside the measurement. Software eval dashboards often skip that discipline because a chat completion looks like one interchangeable string. It is not interchangeable once tool choice, refusal style, and argument filling differ by backend. Treating every slot as the same model is how silent backend drift wears the costume of product quality drift.

The comparison unit is a triple

A small local harness can encode that discipline without a vendor platform or a shared dashboard. Each golden line should hash into a fixture id that covers user text, tool constraints, and expected substrings. Each grader module should export a version string that changes when match logic changes, including regex flags. Each run should accept a model slot label that the operator controls, even if the provider only returns a generic model family.

Scores may be compared across days only when those three strings are identical, byte for byte. A slot change with a stable fixture is a backend delta and belongs in a separate series. A grader version change is a rubric change and should reset the baseline instead of joining the old chart. A fixture hash change means the test itself moved, which is a dataset edit rather than a model regression.

The following JSONL fragment is a proposed golden case for a tool-using order assistant, not a measured production fixture. It encodes required tools, forbidden tools, and text constraints that a local grader can apply without a second model. Negative constraints matter because many silent failures are extra tool calls rather than missing words in the final text. Store the file in git so a fixture hash change is a normal reviewable diff.

{"id": "order_status_v3", "user": "Has order 9F2 shipped?", "require_tools": ["get_order"], "forbid_tools": ["cancel_order"], "must_include": ["9F2"], "must_not": ["tracking number is unknown"]}
{"id": "refund_policy_v1", "user": "Can I refund a delivered mug without a photo?", "require_tools": ["get_policy"], "forbid_tools": ["create_refund"], "must_include": ["photo"], "must_not": ["refund issued"]}
{"id": "missing_order_id_v2", "user": "Where is my package?", "require_tools": [], "forbid_tools": ["get_order", "cancel_order"], "must_include": ["order id"], "must_not": ["shipped"]}

A deterministic grader should return structured reasons, because a boolean hides whether the tool constraint or the text constraint failed. Hashing the grader source into the version string prevents silent rubric edits from looking like model movement. The function below is labeled as proposed code and has not been executed against a live endpoint in this article. Operators can swap the completion loader later without changing the scoring rules or the golden file.

# grade_case.py — proposed, unexecuted example
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any


def grader_version(path: str = __file__) -> str:
    digest = hashlib.sha256(Path(path).read_bytes()).hexdigest()
    return digest[:12]


def fixture_hash(case: dict[str, Any]) -> str:
    payload = json.dumps(case, sort_keys=True, separators=(',', ':'))
    return hashlib.sha256(payload.encode()).hexdigest()[:16]


def grade_case(case: dict[str, Any], completion: dict[str, Any]) -> dict[str, Any]:
    text = completion.get('text') or ''
    tools = [t.get('name') for t in completion.get('tools') or [] if t.get('name')]
    reasons: list[str] = []

    for name in case.get('require_tools') or []:
        if name not in tools:
            reasons.append(f'missing_tool:{name}')
    for name in case.get('forbid_tools') or []:
        if name in tools:
            reasons.append(f'forbidden_tool:{name}')
    lowered = text.lower()
    for needle in case.get('must_include') or []:
        if needle.lower() not in lowered:
            reasons.append(f'missing_text:{needle}')
    for needle in case.get('must_not') or []:
        if needle.lower() in lowered:
            reasons.append(f'forbidden_text:{needle}')

    return {
        'case_id': case['id'],
        'pass': not reasons,
        'reasons': reasons,
        'tool_names': tools,
        'text_digest': hashlib.sha256(text.encode()).hexdigest()[:12],
    }

Tool constraints deserve first-class fields because agent failures often hide inside a plausible final sentence. A completion can mention the order identifier while also calling a cancel tool that the user never requested. Text-only golden cases grade that reply as a pass and leave the side effect invisible. Recording required and forbidden tool names turns that miss into an ordinary assertion with a stable reason code.

Frozen completions make the grader itself testable without a network, which is the only way the rubric stays honest. The tests below pin envelopes in-process so a later grader edit fails in CI before it contaminates a model chart. They are proposed unit tests, not live measurements, and they should run the same on every laptop. If these tests flip, the event is a rubric change and the grader version must move.

# test_grade_case.py — proposed local tests, no live model required
from grade_case import grade_case

ORDER = {
    'id': 'order_status_v3',
    'user': 'Has order 9F2 shipped?',
    'require_tools': ['get_order'],
    'forbid_tools': ['cancel_order'],
    'must_include': ['9F2'],
    'must_not': ['tracking number is unknown'],
}


def test_plausible_text_still_fails_on_forbidden_tool():
    completion = {
        'text': 'Order 9F2 is in transit to the warehouse dock.',
        'tools': [
            {'name': 'get_order', 'args': {'id': '9F2'}},
            {'name': 'cancel_order', 'args': {'id': '9F2'}},
        ],
    }
    result = grade_case(ORDER, completion)
    assert result['pass'] is False
    assert 'forbidden_tool:cancel_order' in result['reasons']


def test_required_tool_without_forbidden_text_passes():
    completion = {
        'text': 'Order 9F2 has shipped and left the facility.',
        'tools': [{'name': 'get_order', 'args': {'id': '9F2'}}],
    }
    result = grade_case(ORDER, completion)
    assert result['pass'] is True
    assert result['reasons'] == []

A proposed local harness

The runner should persist one JSON object per case, including the triple, the reason codes, and a completion digest. Storing raw completions is useful for later audits, but the digest already detects silent text movement when the slot stays pinned. If the triple matches and the digest changes while the grade flips, the harness can call that a true silent regression. If the triple does not match, the harness should refuse a quality diff and print a schema of mismatched keys.

# pin_eval.py — proposed runner; fill load_completion() before any live use
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path
from typing import Any

from grade_case import fixture_hash, grade_case, grader_version

TRIPLE_KEYS = ('fixture_hash', 'grader_version', 'model_slot')


def load_completion(slot: str, user: str) -> dict[str, Any]:
    raise NotImplementedError(
        'Proposed hook: map slot to your chat client and return {text, tools}.'
    )


def load_golden(path: Path) -> list[dict[str, Any]]:
    cases = []
    for line in path.read_text().splitlines():
        if line.strip():
            cases.append(json.loads(line))
    return cases


def run_eval(golden: Path, slot: str, out: Path) -> None:
    version = grader_version()
    records = []
    for case in load_golden(golden):
        completion = load_completion(slot, case['user'])
        graded = grade_case(case, completion)
        records.append({
            **graded,
            'fixture_hash': fixture_hash(case),
            'grader_version': version,
            'model_slot': slot,
        })
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(''.join(json.dumps(r) + '\n' for r in records))


def index_records(path: Path) -> dict[str, dict[str, Any]]:
    return {json.loads(line)['case_id']: json.loads(line) for line in path.read_text().splitlines() if line.strip()}


def diff_runs(left: Path, right: Path) -> int:
    a_map, b_map = index_records(left), index_records(right)
    mismatches = []
    silent = []
    for case_id in sorted(set(a_map) & set(b_map)):
        a, b = a_map[case_id], b_map[case_id]
        triple_ok = all(a[k] == b[k] for k in TRIPLE_KEYS)
        if not triple_ok:
            mismatches.append({
                'case_id': case_id,
                'left': {k: a[k] for k in TRIPLE_KEYS},
                'right': {k: b[k] for k in TRIPLE_KEYS},
            })
            continue
        if a['pass'] != b['pass'] or a['text_digest'] != b['text_digest']:
            silent.append({
                'case_id': case_id,
                'left_pass': a['pass'],
                'right_pass': b['pass'],
                'left_reasons': a['reasons'],
                'right_reasons': b['reasons'],
                'left_digest': a['text_digest'],
                'right_digest': b['text_digest'],
            })
    if mismatches:
        print('REFUSE_DIFF: triple mismatch, not a prompt regression')
        print(json.dumps(mismatches, indent=2))
        return 2
    print(json.dumps({'silent_regressions': silent, 'compared_cases': len(set(a_map) & set(b_map))}, indent=2))
    return 1 if silent else 0


def main() -> None:
    parser = argparse.ArgumentParser(description='Pin fixture, grader, and model slot.')
    sub = parser.add_subparsers(dest='cmd', required=True)
    run_p = sub.add_parser('run')
    run_p.add_argument('--golden', type=Path, required=True)
    run_p.add_argument('--slot', required=True)
    run_p.add_argument('--out', type=Path, required=True)
    diff_p = sub.add_parser('diff')
    diff_p.add_argument('left', type=Path)
    diff_p.add_argument('right', type=Path)
    args = parser.parse_args()
    if args.cmd == 'run':
        run_eval(args.golden, args.slot, args.out)
        return
    sys.exit(diff_runs(args.left, args.right))


if __name__ == '__main__':
    main()

Command examples below assume a local working directory and a completion loader that the operator fills in. Environment variables keep endpoint URLs out of git, which matters when a free server address should not leak into reviews. The diff command must exit nonzero when it refuses a comparison, so CI cannot greenwash a slot change. None of these commands were executed for this article; they are a reproducible shape, not a benchmark.

python -m pytest test_grade_case.py -q
export COMPLETION_BASE_URL="$ENDPOINT_URL"
python pin_eval.py run --golden golden.jsonl --slot lab_free_1 --out runs/2026-09-22.jsonl
python pin_eval.py run --golden golden.jsonl --slot lab_free_1 --out runs/2026-09-23.jsonl
python pin_eval.py diff runs/2026-09-22.jsonl runs/2026-09-23.jsonl
echo $?

Suppose Tuesday's run used slot lab_free_1 and Wednesday's run used lab_free_2 on the same fixtures. A naive pass-rate chart would subtract the two percentages and file a quality incident against the prompt. The pinned harness instead reports a slot mismatch and writes a backend-delta artifact with per-case reason codes. Engineers can still read that artifact; they just cannot paste it into the prompt-regression series.

Running the loop still needs a completion source, and that is the only place a complimentary endpoint earns a mention. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can run this runner against a small golden file during early checks. The product is not the grader, and removing the mention should leave the triple-pin method intact.

What this will not prove

The method does not identify which backend is better; it only stops unlike backends from sharing a trend line. Slot labels are operator-assigned aliases, so they are only as honest as the runbook that maps aliases to endpoints. Free or rotating slots can change behavior without a label change if the operator reuses a name. A reused alias is a forged gauge id, and the triple will look stable while the instrument is not.

Groups that lack git review for JSONL fixtures should not treat the hash as an audit trail. Groups that need statistically powered A/B tests will not get them from a few dozen golden lines. Groups whose completions must be judged by experts should keep human labels and use this only as a smoke layer. The harness is a bookkeeping fix for mixed time series, not a substitute for product evaluation.

Compare two nights of eval output only after joining on fixture hash, grader version, and model slot. If those keys diverge, file a backend delta or a rubric change rather than a prompt incident. A small local run against any inexpensive slot is enough to rehearse that join on a tiny fixture file. Larger claims about quality still require pinned production endpoints and a review process outside this script.