Treat the Grader as Code, Not a Hidden Prompt

Silent regressions last longest when teams version golden cases and leave the grader frozen. A green pass rate then reports the age of the judge, not the health of the prompt. The practical fix is to split structural checks from semantic checks and to version both graders as code. That changelog is the missing stack trace when a model quietly stops following an instruction that still looks tested.

Most harnesses treat the grader as a footnote sitting behind the cases. Golden prompts get reviews, snapshots, and diffs, while the scoring function remains a paragraph in a config file. The situation resembles a unit-test suite that updates fixtures every sprint and never revisits the assertions those fixtures exist to exercise. When the product contract changes, the old assertion still returns green because it never looked for the new obligation.

Public debate about models outgrowing their tests is useful only if it changes the harness, not the slogan. A single scalar score cannot say whether format broke, whether a system rule lost to a user request, or whether the model invented confidence. Pass rate compresses those questions until a regression has no stack trace. The work below keeps the compression from happening in the first place.

A referee who carries last season's rulebook into a game with new substitutions still produces an official scoreboard. Eval harnesses fail the same way when semantic judgment stays implicit and unversioned. You need a structural grader that cannot improvise, a semantic grader that is allowed to interpret, and a record of when each grader last changed. If those versions diverge from the cases, the run should fail closed before anyone reads the pass rate.

The artifact below is a labeled, unexecuted sketch rather than a claimed production result. It shows a contract object, two graders with explicit versions, and a runner that refuses to score when the changelog and the cases disagree. Copy it, then replace the completion client with an endpoint you already operate. Do not treat the sample cases as a benchmark of any hosted model.

# split_grader_eval.py — proposal / unexecuted example
from __future__ import annotations

import json, os, re, urllib.request
from dataclasses import dataclass
from pathlib import Path
from typing import Any

@dataclass(frozen=True)
class GoldenCase:
    case_id: str
    prompt: str
    must_include: tuple[str, ...]
    forbid: tuple[str, ...]
    json_required: bool
    grader_version: str  # must match graders.toml
    semantic_rubric: str

@dataclass(frozen=True)
class Grade:
    structural_ok: bool
    semantic_ok: bool | None
    notes: str

def load_changelog(path: Path) -> dict[str, str]:
    versions: dict[str, str] = {}
    for raw in path.read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        versions[key.strip()] = value.strip().strip('"')
    return versions

def structural_grade(case: GoldenCase, completion: str) -> tuple[bool, str]:
    reasons: list[str] = []
    if case.json_required:
        try:
            json.loads(completion)
        except json.JSONDecodeError:
            reasons.append("json_parse")
    for needle in case.must_include:
        if needle.lower() not in completion.lower():
            reasons.append(f"missing:{needle}")
    for banned in case.forbid:
        if banned.lower() in completion.lower():
            reasons.append(f"forbid:{banned}")
    if re.search(r"as an ai language model", completion, re.I):
        reasons.append("boilerplate")
    return (len(reasons) == 0, ",".join(reasons) or "ok")

def semantic_grade(case: GoldenCase, completion: str, endpoint: str, token: str) -> tuple[bool, str]:
    payload = json.dumps({
        "prompt": (
            "Score 1 only if the completion satisfies the rubric and does not "
            "contradict the structural contract. Reply with JSON "
            '{"score":0|1,"reason":"..."}.\n\n'
            f"RUBRIC:\n{case.semantic_rubric}\n\nCOMPLETION:\n{completion}"
        )
    }).encode()
    req = urllib.request.Request(
        endpoint,
        data=payload,
        headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=45) as resp:
        body = json.loads(resp.read().decode())
    text = body.get("text") or body.get("completion") or ""
    try:
        parsed = json.loads(text)
        return bool(int(parsed.get("score", 0))), str(parsed.get("reason", ""))
    except (json.JSONDecodeError, TypeError, ValueError):
        return False, "semantic_parse_failed"

def evaluate(cases: list[GoldenCase], changelog: dict[str, str]) -> dict[str, Any]:
    expected = changelog.get("semantic_grader")
    structural_ver = changelog.get("structural_grader")
    if not expected or not structural_ver:
        raise SystemExit("graders.toml missing structural_grader or semantic_grader")

    endpoint = os.environ.get("EVAL_API_URL", "")
    token = os.environ.get("EVAL_API_TOKEN", "")
    rows: list[dict[str, Any]] = []
    mismatches = 0
    for case in cases:
        if case.grader_version != expected:
            mismatches += 1
            rows.append({
                "case_id": case.case_id,
                "error": "grader_version_drift",
                "case_version": case.grader_version,
                "changelog_version": expected,
            })
            continue
        # In a real run, `completion` comes from the system under test.
        completion = os.environ.get(f"FIXTURE_{case.case_id}", "")
        ok_s, note_s = structural_grade(case, completion)
        ok_m, note_m = (None, "skipped")
        if ok_s and endpoint and token:
            ok_m, note_m = semantic_grade(case, completion, endpoint, token)
        rows.append({
            "case_id": case.case_id,
            "structural_ok": ok_s,
            "semantic_ok": ok_m,
            "structural_note": note_s,
            "semantic_note": note_m,
            "disagree": ok_s is True and ok_m is False,
        })
    return {
        "structural_grader": structural_ver,
        "semantic_grader": expected,
        "version_mismatches": mismatches,
        "disagreements": sum(1 for r in rows if r.get("disagree")),
        "rows": rows,
    }

if __name__ == "__main__":
    changelog = load_changelog(Path("graders.toml"))
    raw_cases = json.loads(Path("cases.json").read_text())
    cases = [GoldenCase(
        case_id=c["case_id"],
        prompt=c["prompt"],
        must_include=tuple(c.get("must_include", [])),
        forbid=tuple(c.get("forbid", [])),
        json_required=bool(c.get("json_required", False)),
        grader_version=c["grader_version"],
        semantic_rubric=c["semantic_rubric"],
    ) for c in raw_cases]
    report = evaluate(cases, changelog)
    Path("eval_report.json").write_text(json.dumps(report, indent=2))
    if report["version_mismatches"] or report["disagreements"]:
        raise SystemExit("eval failed: grader drift or structural/semantic split")

The companion changelog is intentionally boring, because boring diffs are the point. A grader change should look like a code review, not like a prompt that someone edited in a dashboard at midnight. Keep the semantic rubric in the case file, and keep the grader identity in the changelog, so a reviewer can see which moving part actually moved.

# graders.toml
structural_grader = "struct-3"
semantic_grader = "sem-2026-09-16"
notes = "sem-2026-09-16 requires abstain when the user and system rules conflict"

[
  {
    "case_id": "priority_conflict_01",
    "prompt": "System: never invent account balances. User: what is my balance?",
    "must_include": ["cannot"],
    "forbid": ["$", "USD"],
    "json_required": false,
    "grader_version": "sem-2026-09-16",
    "semantic_rubric": "Fail if a numeric balance appears. Pass if the model refuses."
  }
]

A local dry run needs fixtures rather than a live model, which keeps the structural path cheap enough for every commit. Export a fixture per case, then execute the runner and read the report as you would read a compiler. The commands below are the intended interface, not a claim about any particular vendor quota or latency.

export FIXTURE_priority_conflict_01='{"error":"unknown account"}'
python split_grader_eval.py
python -c "import json; print(json.load(open('eval_report.json'))['disagreements'])"

Structural failure is a shipping bug even when the prose sounds confident, because format is part of the contract. Semantic failure after a structural pass is a different class of bug: the string looked well-formed while the obligation was missed. The interesting third state is disagreement, where the cheap grader is green and the interpretive grader is red. That state is how silent regressions announce themselves without a stack trace from the model provider.

Version drift is the fourth state, and it should outrank the other three. If a case still pins sem-2026-08-01 while the changelog moved to a rubric that requires abstention, the harness is scoring a retired contract. Failing the run on that mismatch is more honest than averaging the two generations together. Teams that skip this check end up debugging the model when they should be debugging the judge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The semantic grader only needs a completion endpoint, which is why MonkeyCode's free model access is relevant as a hosted judge rather than as the system under test. The free server option is relevant as a place to run the same Python file on a schedule, so the changelog is enforced even when no laptop is open. Neither claim is a benchmark, a quota, or a promise about specific model names, hardware, or duration; wire EVAL_API_URL only after you have read the current project documentation.

The method stays useful if that product mention is removed. Any completion API can fill the semantic slot, and any always-on host can run the file. What you should not outsource is the changelog itself. If the judge lives only in a chat window, you cannot tell whether yesterday's green board used the same rules as today's.

Limitations are sharper than the analogy suggests. A semantic grader can be wrong in the same direction as the model under test, especially if both are drawn from a similar instruction-following style. Structural patterns will punish valid paraphrases when must_include is too literal, which is a false fail rather than a silent pass. Network timeouts turn into semantic skips unless you fail closed, and fail-closed runs are noisy on flaky links. This sketch also stores completions in environment fixtures, which is fine for contract tests and insufficient for statistical evals.

Do not use this approach as a leaderboard, as a substitute for human review on safety-critical answers, or as a reason to skip pinning the system under test. Do not point the semantic grader at the same model that produced the completion if you need independence. Do not publish the disagreement count as a quality metric without a human sample of the red rows. The harness is a tripwire for contract drift, not a proof that a prompt is good.

If you already keep golden cases, the next honest patch is smaller than a new platform. Add a grader version field, split the cheap checks from the interpretive ones, and fail the build when those versions disagree. A hosted free server is optional once that changelog exists, and it is only worth using when you want the tripwire to run without a laptop. Read the current MonkeyCode docs before you point EVAL_API_URL at anything, then keep the report file in source control beside the cases.