A 100 percent pass rate is often a measurement failure rather than a quality win. Once every golden case sits in the easy band, the harness stops ranking prompts and only rubber-stamps them. Ceiling effects hide regressions the same way a bathroom scale that tops out at two hundred pounds hides further weight gain. The fix is not another average; it is stratified cases, a ceiling detector, and stretch items that still fail on purpose.
Most prompt suites still collapse many intents into one ratio, then celebrate when that ratio climbs. After a few model upgrades, the easy items become free points and the average loses its slope. A regression that breaks a rare tool path can hide inside a still-green total because the easy majority outweighs it. Quality scores need a denominator that still contains failure, or they stop being scores at all.
Think of a compiler test suite that only compiles hello world after the language grows generics and macros. The suite stays green while the new surface area rots, because nothing in the corpus can still fail. Load tests have the same trap when generated traffic never exceeds last year's peak; the graph looks calm because the generator is polite. Golden cases for language-model features age the same quiet way when product copy expands around a frozen fixture file.
A workable response is to tag every fixture with a difficulty stratum and to treat those strata as separate instruments. Easy cases guard catastrophic breakage, such as empty replies, truncated JSON, or ignored system instructions. Medium cases encode the product contract: the answer must cite a given SKU, refuse a missing field, or stay inside a character budget. Stretch cases are supposed to be slightly too hard for the current prompt, so a high pass rate there means the yardstick has slipped.
The Python below is a labeled worked example rather than a production framework, and it grades recorded completions offline first. Offline grading keeps the method reproducible without a live credential, which matters when you are debugging the harness instead of the model. An optional HTTP path fills empty completions from any OpenAI-compatible chat endpoint when you want a nightly loop. Model names stay in environment variables so fixtures do not silently pin an endpoint you did not intend to grade.
#!/usr/bin/env python3
"""Ceiling-aware eval harness (worked example, not a shipped product)."""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.request
from dataclasses import dataclass, field
from pathlib import Path
from typing import Iterable
STRATA = ("easy", "medium", "stretch")
@dataclass(frozen=True)
class GoldenCase:
id: str
stratum: str
prompt: str
completion: str
must_include: tuple[str, ...] = ()
must_not_include: tuple[str, ...] = ()
max_chars: int = 1200
def __post_init__(self) -> None:
if self.stratum not in STRATA:
raise ValueError(f"{self.id}: unknown stratum {self.stratum}")
@dataclass
class Grade:
case_id: str
stratum: str
passed: bool
reasons: list[str] = field(default_factory=list)
def grade_case(case: GoldenCase) -> Grade:
reasons: list[str] = []
text = case.completion or ""
if not text.strip():
reasons.append("empty_completion")
if len(text) > case.max_chars:
reasons.append(f"over_budget:{len(text)}>{case.max_chars}")
lowered = text.lower()
for needle in case.must_include:
if needle.lower() not in lowered:
reasons.append(f"missing:{needle}")
for needle in case.must_not_include:
if needle.lower() in lowered:
reasons.append(f"forbidden:{needle}")
return Grade(case.id, case.stratum, passed=not reasons, reasons=reasons)
def load_jsonl(path: Path) -> list[GoldenCase]:
cases: list[GoldenCase] = []
with path.open(encoding="utf-8") as handle:
for line_no, raw in enumerate(handle, 1):
raw = raw.strip()
if not raw:
continue
row = json.loads(raw)
try:
cases.append(
GoldenCase(
id=str(row["id"]),
stratum=str(row["stratum"]),
prompt=str(row["prompt"]),
completion=str(row.get("completion", "")),
must_include=tuple(row.get("must_include") or ()),
must_not_include=tuple(row.get("must_not_include") or ()),
max_chars=int(row.get("max_chars") or 1200),
)
)
except (KeyError, TypeError, ValueError) as exc:
raise SystemExit(f"{path}:{line_no}: {exc}") from exc
return cases
def stratum_rates(grades: Iterable[Grade]) -> dict[str, dict[str, float | int | None]]:
buckets = {name: {"n": 0, "pass": 0} for name in STRATA}
for grade in grades:
buckets[grade.stratum]["n"] += 1
buckets[grade.stratum]["pass"] += int(grade.passed)
report: dict[str, dict[str, float | int | None]] = {}
for name, slot in buckets.items():
n = slot["n"]
report[name] = {"n": n, "pass_rate": (slot["pass"] / n) if n else None}
return report
def detect_ceiling(rates: dict[str, dict[str, float | int | None]], stretch_high: float) -> list[str]:
"""Flag a blind harness. High stretch pass is eval rot, not a trophy."""
alerts: list[str] = []
easy = rates["easy"]["pass_rate"]
medium = rates["medium"]["pass_rate"]
stretch = rates["stretch"]["pass_rate"]
counts = [rates[name]["n"] for name in STRATA]
if any(n is None or int(n) < 2 for n in counts):
alerts.append("thin_stratum")
return alerts
if easy is None or medium is None or stretch is None:
alerts.append("missing_stratum")
return alerts
if float(easy) >= 0.99 and float(medium) >= 0.95 and float(stretch) >= stretch_high:
alerts.append("ceiling")
if float(stretch) >= stretch_high:
alerts.append("stretch_too_easy")
return alerts
def evaluate(
cases: list[GoldenCase],
easy_floor: float = 1.0,
medium_floor: float = 0.85,
stretch_high: float = 0.80,
) -> dict:
grades = [grade_case(case) for case in cases]
rates = stratum_rates(grades)
alerts = detect_ceiling(rates, stretch_high=stretch_high)
easy_rate = rates["easy"]["pass_rate"]
medium_rate = rates["medium"]["pass_rate"]
if easy_rate is not None and float(easy_rate) < easy_floor:
alerts.append("easy_regression")
if medium_rate is not None and float(medium_rate) < medium_floor:
alerts.append("medium_regression")
return {
"n": len(grades),
"rates": rates,
"alerts": alerts,
"ok": not alerts,
"failures": [grade.__dict__ for grade in grades if not grade.passed],
}
def maybe_complete(cases: list[GoldenCase], base_url: str | None) -> list[GoldenCase]:
"""Optional live fill-in. Model name stays in the environment on purpose."""
if not base_url:
return cases
model = os.environ.get("EVAL_MODEL")
if not model:
raise SystemExit("EVAL_MODEL is required when --base-url is set")
filled: list[GoldenCase] = []
for case in cases:
payload = json.dumps(
{
"model": model,
"messages": [{"role": "user", "content": case.prompt}],
"temperature": 0,
}
).encode()
req = urllib.request.Request(
base_url.rstrip("/") + "/chat/completions",
data=payload,
headers={"Content-Type": "application/json"},
method="POST",
)
api_key = os.environ.get("EVAL_API_KEY")
if api_key:
req.add_header("Authorization", f"Bearer {api_key}")
try:
with urllib.request.urlopen(req, timeout=60) as resp:
body = json.loads(resp.read().decode())
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, KeyError) as exc:
raise SystemExit(f"live fill failed for {case.id}: {exc}") from exc
text = body["choices"][0]["message"]["content"]
filled.append(
GoldenCase(
id=case.id,
stratum=case.stratum,
prompt=case.prompt,
completion=text,
must_include=case.must_include,
must_not_include=case.must_not_include,
max_chars=case.max_chars,
)
)
return filled
def self_check() -> None:
cases = [
GoldenCase("e1", "easy", "p", "Order 1842 packed", ("1842",), (), 80),
GoldenCase("e2", "easy", "p", "Order 1842 packed", ("1842",), (), 80),
GoldenCase("m1", "medium", "p", "SKU-9 cited", ("SKU-9",), ("sorry",), 80),
GoldenCase("m2", "medium", "p", "SKU-9 cited", ("SKU-9",), ("sorry",), 80),
GoldenCase("s1", "stretch", "p", "policy XYZ holds", ("XYZ", "policy"), (), 80),
GoldenCase("s2", "stretch", "p", "policy XYZ holds", ("XYZ", "policy"), (), 80),
]
report = evaluate(cases, stretch_high=0.50)
assert "stretch_too_easy" in report["alerts"], report
print(json.dumps({"self_check": "ok", "alerts": report["alerts"]}, indent=2))
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Ceiling-aware golden-case grader")
parser.add_argument("jsonl", type=Path, nargs="?")
parser.add_argument("--base-url", default=os.environ.get("EVAL_BASE_URL"))
parser.add_argument("--stretch-high", type=float, default=0.80)
parser.add_argument("--self-check", action="store_true")
args = parser.parse_args(argv)
if args.self_check:
self_check()
return 0
if args.jsonl is None:
raise SystemExit("jsonl path is required unless --self-check is set")
cases = maybe_complete(load_jsonl(args.jsonl), args.base_url)
report = evaluate(cases, stretch_high=args.stretch_high)
json.dump(report, sys.stdout, indent=2)
sys.stdout.write("\n")
return 0 if report["ok"] else 2
if __name__ == "__main__":
raise SystemExit(main())
Store fixtures as JSONL so git can diff a single case without opening a notebook. Each row carries an identifier, a stratum, the prompt, an optional recorded completion, and the code-side constraints the grader will apply. The example file below is synthetic and marked as such; replace the text with your own product language before trusting any rate it emits.
{"id": "easy-empty-guard", "stratum": "easy", "prompt": "Summarize order 1842 in one sentence.", "completion": "Order 1842 ships two blue mugs to Denver tomorrow.", "must_include": ["1842"], "must_not_include": ["as an AI"], "max_chars": 240}
{"id": "easy-no-apology-loop", "stratum": "easy", "prompt": "Confirm the warehouse, nothing else.", "completion": "Denver warehouse confirmed.", "must_include": ["Denver"], "must_not_include": ["cannot"], "max_chars": 80}
{"id": "medium-sku-contract", "stratum": "medium", "prompt": "Name the SKU that ships and refuse unknown add-ons.", "completion": "SKU-9 ships; extra engraving is not on the order.", "must_include": ["SKU-9"], "must_not_include": ["engraving is confirmed"], "max_chars": 160}
{"id": "medium-budget", "stratum": "medium", "prompt": "Reply with the ship date only.", "completion": "2026-09-21", "must_include": ["2026-09-21"], "must_not_include": ["certainly"], "max_chars": 32}
{"id": "stretch-policy-cite", "stratum": "stretch", "prompt": "Quote the restock rule that blocks weekend dispatch.", "completion": "I think weekends are probably fine if the queue is short.", "must_include": ["restock-window", "weekday"], "must_not_include": ["probably"], "max_chars": 200}
{"id": "stretch-missing-field", "stratum": "stretch", "prompt": "Schedule a delivery without a street address.", "completion": "Delivery set for noon at the usual place.", "must_include": ["missing address"], "must_not_include": ["usual place"], "max_chars": 180}
Grade the recorded file with a plain interpreter so the first run never depends on network weather. Print the JSON report to standard output, and let the numeric process status carry the ceiling alert. A green exit code means every stratum is populated and no alert fired, not that the product is finished. Commit the JSONL and the grader together so a fixture edit cannot drift away from the checks that give it meaning.
python ceiling_eval.py --self-check
python ceiling_eval.py fixtures.jsonl ; echo "exit: $?"
The process should exit nonzero when easy cases drop, and it should also exit nonzero when stretch cases rise too far. That second failure feels backwards until you treat the harness as a measuring instrument that can itself go out of calibration. If stretch pass rate crosses a high watermark, the next change is a fixture rewrite that restores headroom, not a prompt tweak. Leave the easy band small enough that a single catastrophic miss cannot be averaged away by a crowd of greetings.
Live fills are optional and should stay optional, because a flaky network will otherwise look like a prompt regression. When you do need fresh completions, pass a base URL and keep the model identifier outside the repository. If a base URL is supplied, recorded completions are replaced, so keep a copy of the JSONL when you need an audit trail. Temperature stays at zero in the example so repeated ceiling checks are comparable, which is a grading choice rather than a claim about production sampling.
export EVAL_BASE_URL="http://127.0.0.1:8080/v1"
export EVAL_MODEL="your-model-id"
export EVAL_API_KEY="redacted"
python ceiling_eval.py fixtures.jsonl --base-url "$EVAL_BASE_URL" --stretch-high 0.80
Those commands are a template; they do not document a particular vendor schema beyond the common chat-completions envelope. If your server uses a different path or a different message shape, adapt the request builder rather than forcing fixtures to match a foreign protocol. Record the raw response beside the grade when you run live, or you will not be able to replay a disputed failure. Keep the stretch-high flag in the job definition so a local override cannot silently bless a saturated set.
Teams with an OpenAI-compatible client can point this loop at a spare endpoint so production does not pay for every ceiling check. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that spare loop when you want continuous grading without charging the production budget. Point the example client at the base URL you already use, and keep identities in the environment rather than in the fixture file.
Read the JSON report as three instruments, not as one trophy number that management can screenshot. Easy pass rate is a smoke alarm; if it moves, you debug before you discuss copy. Medium pass rate is the contract with the product, and a dip there is a real regression even when stretch is unchanged. Stretch pass rate is a headroom gauge: rising values mean you must mint harder cases, not that the prompt is finished.
This approach has sharp limits, and several teams should not adopt it as a release gate. Do not use a ceiling detector as a safety certification for medical, legal, or financial answers, because string constraints do not measure harm. Do not let an LLM judge replace the code grader here, because a drifting judge recreates saturation one layer up. Do not mix tuning examples into the stretch band, or you will congratulate yourself for memorizing the homework.
Skip the live path entirely when you cannot pin the model identifier or when the endpoint is shared with interactive users who expect spare capacity. Thresholds in the example are labeled proposals, not measured constants drawn from a production fleet. A stretch-high value of 0.80 will be too strict for a research prototype and too loose for a narrow classifier that should never improvise. Recalibrate those cutoffs from your own history after you have enough failing stretch items to see a slope.
If a stratum is empty or holds a single case, the detector must refuse green status, since it cannot tell luck from coverage. Binary string checks also miss paraphrases that a human would accept, which is acceptable for a ceiling alarm and unacceptable for a final editorial grade. Keep human review on the stretch misses, because those misses are how you learn which new fixtures belong in the medium band.
A ceiling-aware harness will not replace human review, and it will not tell you whether users like the tone. It will tell you whether your tests can still fail, which is the precondition for noticing the next quiet regression. Replace stretch cases when they stop being stretch, and keep the easy band too small to drown a real alert. Green bars are cheap; a yardstick that still has room to move is the actual asset.
Top comments (0)