Keep Schema Failures Out of Your Quality Score

A blended pass rate on mixed golden cases hides the only failure class a prompt change can actually control. Structural defects such as invalid JSON, missing keys, and broken tool envelopes are binary, cheap to detect, and model-class portable. Quality drift is slower, judge-dependent, and tightly coupled to the production model you actually ship. Treating those two layers as one number lets a prettier paragraph mask a contract break.

Think of the split as the difference between a compiler check and a later design review. The compiler does not award style points when the program fails to parse, and your eval harness should not either. A smaller or complimentary model can still exercise that compiler layer if the grader is ordinary code. The design review still belongs on the model that serves users, with fixtures you refuse to rewrite casually.

Silent motion from a single lane

Most harnesses store a prompt, a handful of expected strings, and a judge that returns pass or fail. When the candidate model changes, both schema validity and prose quality move, so the dashboard cannot say which layer caused the delta. A nightly job that reports 0.82 after reporting 0.79 looks like progress even if two fixtures started emitting truncated objects. The quality average absorbed the structural misses because the judge still found fluent sentences in the rubble.

Golden cases age in a second way that a single pass rate on the dashboard never shows. Product schemas gain fields, tool names change, and refusal policies tighten, yet the fixture file still grades last month's shape. If the grader is another model, it may forgive the mismatch the way a tired reviewer skips a checklist. Pinning the grader as deterministic code, and pinning each case to a layer, turns that forgiveness into a failed build.

Public conversation this week about models outgrowing their tests is a topic signal, not a recipe. Tests do not become obsolete merely because the candidate grew; they become obsolete when they score the wrong property. A structural fixture that asserts an envelope is not trying to measure wisdom or taste. It is trying to measure whether the completion is still a value your parser can read.

Hashed fixtures on two lanes

The artifact below is a proposed, unexecuted sketch you can run locally against any OpenAI-compatible chat endpoint. It does not claim production metrics, customer results, or a ranked model table. It records case hashes, grader identity, and model identity so a later diff can separate prompt edits from fixture edits. Structural cases fail the process on the first broken object, while quality cases record a numeric score that cannot rescue a structural miss.

# eval_lanes.py — proposed local harness, not a published benchmark
from __future__ import annotations

import hashlib
import json
import os
import sys
import urllib.request
from pathlib import Path
from typing import Any

STRUCTURAL_GRADER = "schema.v3"
QUALITY_GRADER = "heuristic.v1"


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def load_goldens(path: Path) -> list[dict[str, Any]]:
    cases = []
    raw = path.read_text(encoding="utf-8")
    for line_no, line in enumerate(raw.splitlines(), start=1):
        if not line.strip():
            continue
        case = json.loads(line)
        case["_source_hash"] = sha256_text(line)
        case["_line"] = line_no
        cases.append(case)
    return cases


def complete(endpoint: str, model: str, prompt: str, temperature: float) -> str:
    body = json.dumps({
        "model": model,
        "temperature": temperature,
        "messages": [{"role": "user", "content": prompt}],
    }).encode("utf-8")
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('API_KEY', '')}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    return payload["choices"][0]["message"]["content"]


def grade_structural(completion: str, spec: dict[str, Any]) -> dict[str, Any]:
    try:
        obj = json.loads(completion)
    except json.JSONDecodeError as exc:
        return {"pass": False, "reason": f"json:{exc}"}
    if not isinstance(obj, dict):
        return {"pass": False, "reason": "not_object"}
    missing = [k for k in spec.get("required_keys", []) if k not in obj]
    if missing:
        return {"pass": False, "reason": f"missing:{missing}"}
    allowed = set(spec.get("allowed_keys", spec.get("required_keys", [])))
    extra = [k for k in obj if allowed and k not in allowed]
    if extra:
        return {"pass": False, "reason": f"extra:{extra}"}
    enum_field = spec.get("status_field")
    enum_values = spec.get("status_values", [])
    if enum_field and obj.get(enum_field) not in enum_values:
        return {"pass": False, "reason": f"enum:{obj.get(enum_field)!r}"}
    return {"pass": True, "reason": "ok"}


def grade_quality(completion: str, spec: dict[str, Any]) -> dict[str, Any]:
    text = completion.strip()
    min_len = int(spec.get("min_chars", 40))
    banned = spec.get("banned_substrings", [])
    hits = [b for b in banned if b.lower() in text.lower()]
    score = 1.0
    if len(text) < min_len:
        score -= 0.5
    if hits:
        score -= 0.5
    return {"pass": score >= 0.5, "score": score, "reason": f"banned={hits}"}


def main() -> int:
    golden_path = Path("goldens.jsonl")
    endpoint = os.environ["EVAL_ENDPOINT"]
    model = os.environ["EVAL_MODEL"]
    lane = os.environ.get("EVAL_LANE", "structural")
    cases = load_goldens(golden_path)
    fixture_hash = sha256_text(golden_path.read_text(encoding="utf-8"))
    results = []
    failures = 0
    for case in cases:
        if case["layer"] != lane and lane != "both":
            continue
        temp = 0.0 if case["layer"] == "structural" else float(case.get("temperature", 0.2))
        completion = complete(endpoint, model, case["prompt"], temp)
        if case["layer"] == "structural":
            grade = grade_structural(completion, case["spec"])
            grader = STRUCTURAL_GRADER
        else:
            grade = grade_quality(completion, case["spec"])
            grader = QUALITY_GRADER
        row = {
            "id": case["id"],
            "layer": case["layer"],
            "pass": grade["pass"],
            "reason": grade.get("reason"),
            "score": grade.get("score"),
            "case_hash": case["_source_hash"],
            "fixture_hash": fixture_hash,
            "grader": grader,
            "model": model,
        }
        results.append(row)
        if not grade["pass"]:
            failures += 1
    Path("eval_results.json").write_text(json.dumps(results, indent=2), encoding="utf-8")
    print(json.dumps({"fixture_hash": fixture_hash, "n": len(results), "failures": failures}))
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

A companion fixture file keeps layer labels next to the contract, not in a spreadsheet comment. Each structural case states required keys and allowed keys so extra fields fail as loudly as missing ones. Quality cases stay in a separate layer so a banned-phrase heuristic cannot outvote a broken object. The second invoice fixture encodes an explicit unparseable status instead of hoping the model improvises an empty shape.

{"id":"invoice-extract-01","layer":"structural","prompt":"Extract invoice fields as JSON with keys invoice_id, currency, cents. Text: Invoice INV-1044 billed 19900 JPY.","spec":{"required_keys":["invoice_id","currency","cents"],"allowed_keys":["invoice_id","currency","cents"]}}
{"id":"invoice-extract-02","layer":"structural","prompt":"Extract invoice fields as JSON with keys invoice_id, currency, cents. Text: none found.","spec":{"required_keys":["status"],"allowed_keys":["status","reason"],"status_field":"status","status_values":["unparseable"]}}
{"id":"invoice-tone-01","layer":"quality","prompt":"Write a two-sentence notice that invoice INV-1044 for 19900 JPY was captured.","spec":{"min_chars":60,"banned_substrings":["as an AI","certainly!"]}}

Run the structural lane at temperature zero against whatever compatible endpoint you already operate. Keep the quality lane on the production model name your gateway actually routes. Compare eval_results.json by case_hash and grader, not by a single average, so a rewritten fixture cannot masquerade as a model win. Store the JSON artifact next to the prompt commit so later diffs have a stable grain.

export EVAL_ENDPOINT="https://example.invalid/v1/chat/completions"
export EVAL_MODEL="your-production-or-lane-model"
export EVAL_LANE="structural"
python eval_lanes.py
python - <<'PY'
import json
from pathlib import Path
rows = json.loads(Path("eval_results.json").read_text())
broken = [r for r in rows if not r["pass"]]
print("failures", len(broken))
for r in broken:
    print(r["id"], r["layer"], r["grader"], r["reason"])
PY

A complimentary runtime as a parser gate

Continuous structural checks need a stable endpoint more than they need the smartest available weights. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option. That complimentary runtime can host the structural parser gate, but it should not certify tone, factuality, or product voice.

Use the spare server the way you would use a dedicated CI worker for unit tests. Point EVAL_ENDPOINT at the compatible chat route, pin EVAL_LANE to structural, and store fixture_hash beside the git commit of the prompt. If the structural lane is green and the production quality lane is red, you learned something specific: the envelope still parses, and the prose changed. If both lanes are red, fix the contract before anyone debates wording.

Do not publish a free-lane quality score as evidence that a prompt is ready for users. Cheap completions can fail open with confident JSON that still violates business rules your schema does not encode. They can also fail closed and look like prompt regressions when the weights simply cannot follow a long instruction. Those error modes are why the quality fixtures stay on the served model.

Who should skip the split

This split assumes you can write a deterministic grader for the structural layer. If the product output is unconstrained prose with no envelope, there is no compiler, and hashing fixtures will not create one. Teams that lack a production model endpoint should not treat a complimentary model as a proxy for user-visible quality. The heuristic quality grader in the sketch is intentionally weak; it exists to show isolation, not to replace a versioned, tested judge.

The harness also assumes JSON-object completions for structural cases. Token-level logprobs, multimodal outputs, and streaming tool calls need different envelopes than the example records. Temperature zero reduces variance; it does not make a non-deterministic API reproducible across vendors. If your vendor silently swaps weights behind the same model string, record a completion digest and alert when identical prompts yield new hashes even though the pass bit stayed true.

If you already keep goldens in git, add the layer field before you add another dashboard tile. A two-lane eval will not stop a golden set that never included the failure you care about. Missing fixtures remain missing after you add hashes, layers, and a stricter process exit. The cheap win is a structural process exit that cannot be averaged away, not another blended score.