Score Disagreement After the Eval Ceiling

Once a golden set stops splitting models, pass rate becomes a vanity metric that conceals regressions. The useful signal is disagreement between a frozen reference path and the candidate under test. This article proposes a small harness that scores that delta on a cheap nightly loop. Absolute scores saturate after a capability jump, while pairwise disagreement still moves under silent prompt drift.

Think of the golden set as a sieve whose holes grow larger every time the underlying models get better. Water still pours through, so the dashboard records a perfect catch rate that no longer describes the mesh. A hallway of smoke detectors that never sound does not prove the building is actually safe. It often proves the sensors lost their ability to discriminate smoke from ordinary hallway dust.

Eval ceiling is that same failure mode, expressed as a dashboard that refuses to turn yellow. Teams then ship prompt edits that change refusals, citations, or tool arguments without a red build. The cases still match a coarse expected string, so the harness reports health that the product does not have. Scoring the candidate alone cannot recover lost resolution once every row sits on the ceiling.

The proposed fix is not a larger golden set, which will saturate again after the next model bump. It is a frozen reference completion, a candidate completion, and identical grader functions applied to both. Continuous integration should fail when the disagreement map changes, even if both pass rates stay high. That rule is the method; the Python below is only a concrete encoding of the same contract.

The fixture file is versioned like application code, because a case without an owner ages into decoration. Each row carries an identifier, a prompt, and a machine-checkable expectation rather than a prose rubric. The proposed schema below is unlabeled production data; it is a teaching fixture for the runner that follows. Keep expected fields narrow, or the grader will bless fluent wrong answers that share a token with the gold string.

{
  "version": "2026-09-21",
  "cases": [
    {
      "id": "refund.policy.v3",
      "prompt": "Customer paid twice for order 1842. Can they get an immediate refund?",
      "expect": {
        "must_refuse_instant_refund": true,
        "must_cite_order_id": true,
        "max_sentences": 4
      }
    },
    {
      "id": "tool.tracking.missing_carrier",
      "prompt": "Track shipment 9911. The carrier field is empty in our database.",
      "expect": {
        "must_not_invent_carrier": true,
        "must_ask_for_carrier": true
      }
    }
  ]
}

Graders belong in ordinary functions so a failing row produces a stack that a reviewer can read. A hidden judging prompt recreates the ceiling problem inside the judge, because the judge saturates too. The functions below are proposed examples, not measured production thresholds, and they return structured misses instead of a single boolean. Structured misses keep the disagreement map readable when two models fail for different reasons on the same fixture.

# proposed_example.py — unexecuted teaching harness, not a benchmark.
from __future__ import annotations

import json, os, re, urllib.request
from dataclasses import dataclass, asdict
from typing import Any, Callable, Dict, List

Grader = Callable[[str, Dict[str, Any]], List[str]]


def grade_refund(text: str, expect: Dict[str, Any]) -> List[str]:
    misses: List[str] = []
    low = text.lower()
    if expect.get("must_refuse_instant_refund") and "refund now" in low:
        misses.append("offered_instant_refund")
    if expect.get("must_cite_order_id") and "1842" not in text:
        misses.append("missing_order_id")
    sentences = [s for s in re.split(r"[.!?]", text) if s.strip()]
    if len(sentences) > expect.get("max_sentences", 99):
        misses.append("too_verbose")
    return misses


def grade_tracking(text: str, expect: Dict[str, Any]) -> List[str]:
    misses: List[str] = []
    low = text.lower()
    invented = ("ups", "fedex", "dhl", "usps")
    if expect.get("must_not_invent_carrier") and any(c in low for c in invented):
        misses.append("invented_carrier")
    if expect.get("must_ask_for_carrier") and "carrier" not in low:
        misses.append("did_not_ask_carrier")
    return misses


GRADERS = {
    "refund.policy.v3": grade_refund,
    "tool.tracking.missing_carrier": grade_tracking,
}

The completion client should be boring HTTP, because a clever SDK hides retries that later look like model disagreement. Temperature stays at zero for both paths so sampling noise does not masquerade as a product change. The snippet assumes an OpenAI-style chat payload only as a local convention; swap the body if your endpoint differs. Label every network failure separately from grader misses, or a timeout becomes a fake policy regression.

@dataclass
class CallResult:
    text: str
    error: str | None


def complete(base_url: str, prompt: str, timeout: int = 30) -> CallResult:
    payload = {
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "max_tokens": 256,
    }
    req = urllib.request.Request(
        base_url.rstrip("/") + "/v1/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ.get("EVAL_TOKEN", ""),
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = json.loads(resp.read().decode("utf-8"))
        text = body["choices"][0]["message"]["content"]
        return CallResult(text=text, error=None)
    except Exception as exc:  # proposed catch-all for teaching; narrow this in real code
        return CallResult(text="", error=type(exc).__name__)

The comparison step is the actual test. Each fixture yields a reference miss list, a candidate miss list, and a symmetric difference of those lists. A candidate that passes while the reference fails is not automatically a win; it is a delta that needs a human glance the first time it appears. After that delta is accepted, it becomes the new frozen map, the same way a reviewed snapshot becomes the next baseline in UI tests.

@dataclass
class Row:
    case_id: str
    reference_misses: List[str]
    candidate_misses: List[str]
    reference_error: str | None
    candidate_error: str | None

    @property
    def disagreement(self) -> List[str]:
        return sorted(set(self.reference_misses) ^ set(self.candidate_misses))


def evaluate(cases: List[dict], ref_url: str, cand_url: str) -> List[Row]:
    rows: List[Row] = []
    for case in cases:
        grader = GRADERS[case["id"]]
        ref = complete(ref_url, case["prompt"])
        cand = complete(cand_url, case["prompt"])
        ref_misses = ["transport:" + ref.error] if ref.error else grader(ref.text, case["expect"])
        cand_misses = ["transport:" + cand.error] if cand.error else grader(cand.text, case["expect"])
        rows.append(Row(case["id"], ref_misses, cand_misses, ref.error, cand.error))
    return rows

A green absolute score with a changed disagreement map should fail the job. The exit code below treats transport errors as infrastructure, then fails only on novel deltas against a committed snapshot. That snapshot is the eval equivalent of a lockfile: boring, diffable, and painful to edit without a reason. If a case stays in both-pass with empty disagreement for many consecutive nights, it has stopped discriminating and should be retired or mutated, not celebrated.

SNAP_PATH = "disagreement_snapshot.json"


def load_snapshot() -> Dict[str, List[str]]:
    if not os.path.exists(SNAP_PATH):
        return {}
    with open(SNAP_PATH, encoding="utf-8") as handle:
        return json.load(handle)


def gate(rows: List[Row], update: bool = False) -> int:
    snap = load_snapshot()
    novel = []
    transport = [r for r in rows if r.reference_error or r.candidate_error]
    if transport:
        print("transport_failures", [asdict(r) for r in transport])
        return 2
    current = {r.case_id: r.disagreement for r in rows}
    for case_id, delta in current.items():
        if snap.get(case_id) != delta:
            novel.append({"id": case_id, "was": snap.get(case_id), "now": delta})
    if novel and not update:
        print("disagreement_changed", json.dumps(novel, indent=2))
        return 1
    if update:
        with open(SNAP_PATH, "w", encoding="utf-8") as handle:
            json.dump(current, handle, indent=2, sort_keys=True)
            handle.write("\n")
    both_pass = [r.case_id for r in rows if not r.reference_misses and not r.candidate_misses]
    print("ceiling_candidates", both_pass)
    return 0

Run the gate from a scheduler, not from a laptop lid, because ceiling effects appear across days rather than inside one interactive session. The commands assume two base URLs and a token in the environment; they do not encode a vendor. First nights will look noisy while the snapshot is empty, which is expected, and the --update path should be a reviewed commit rather than an automatic rewrite. After the snapshot exists, the interesting output is disagreement_changed, not a percentage that has already glued itself to one hundred.

export EVAL_TOKEN=replace-me
python proposed_example.py  # wire a __main__ that reads cases.json
python - <<'PY'
# proposed driver
import json, os, sys
from proposed_example import evaluate, gate
cases = json.load(open("cases.json"))["cases"]
rows = evaluate(cases, os.environ["REF_URL"], os.environ["CAND_URL"])
sys.exit(gate(rows, update="--update" in sys.argv))
PY

A second model pass doubles token spend if both paths are billed at the same rate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which can hold the frozen reference path and the nightly runner without turning disagreement checks into a paid always-on job. The harness does not depend on that product; it only needs two completion targets and a machine that can cron a Python file. If those pieces already exist in your stack, keep them and treat the product mention as optional infrastructure for the reference leg.

Disagreement is not truth, and that limitation matters more than the exit code. Both paths can share a bug, especially when the candidate was fine-tuned toward the same public style as the reference. The method also fails closed on transport, which is correct for CI and irritating during an upstream brownout. Teams that possess only one completion endpoint cannot run this gate at all, and they should not fake a reference by replaying yesterday’s candidate logs as if they were an independent model.

Skip this approach when the grader itself requires a human rubric, multilingual legal judgment, or safety review that no function in this file can encode. Skip it when the two endpoints differ in tool-calling semantics, because then the delta measures protocol mismatch rather than prompt regression. Skip it when product owners will rubber-stamp --update whenever the map moves, since an unreviewed snapshot is just pass rate wearing a more elaborate coat. The ceiling is a measurement problem; an ignored delta is a process problem, and software cannot paper over the second one.

The core conclusion does not change after the code is copied. Pass rate goes mute when fixtures stop splitting models, and disagreement remains audible if the grader is code and the reference stays frozen. Retire cases that both paths pass without a delta, mutate them, or accept that they no longer earn their keep in the suite. If you need a cheap place to park the reference client and the cron host, MonkeyCode’s free model access and free server option are sufficient to try the loop without reshaping the rest of the eval.