Silent LLM regressions usually violate output contracts long before they fail a subjective quality grade in review. A deterministic harness can assert keys, types, numeric bounds, and forbidden phrases without introducing another model. That split lets you spend free generation capacity on repeated samples while the grader itself stays boring and stable. The rest of this article specifies a proposed runner, a small contract file, and the failure modes it will not catch.
When a classifier drops confidence or a summarizer omits citations, dashboards often stay green because nobody asserted shape. Golden expected strings still help for brittle phrases, yet they punish valid paraphrases that would satisfy the same interface. An LLM-as-judge can score fluency, but the judge spends tokens, drifts across versions, and rarely fails closed on schema. Treating the model like an unreliable backend restores an old service habit: contract tests before taste tests.
A payments API that returns HTTP 200 with a missing amount would not ship because the error prose sounded confident. Prompted features deserve the same suspicion, even when the surrounding copy reads polished. The contract is the unit of failure, and leftover prose is a residual that a human or a later grader can inspect. If the JSON already broke, scoring style is mostly theater.
The proposed contract is intentionally dull. Each case names an id, a prompt fixture, and a list of assertions that Python can evaluate without calling a second model. Assertions cover required keys, JSON Schema types, numeric ranges, substring bans, and simple cardinality checks on arrays. Nothing in this layer tries to decide whether the answer is wise, only whether it is still a well-formed reply your code can parse.
# Proposed harness (not executed against a live vendor account).
# Save as contracts_eval.py and keep fixtures in git beside the prompt.
from __future__ import annotations
import json, os, re, time, urllib.request
from dataclasses import dataclass
from typing import Any, Callable
@dataclass(frozen=True)
class Case:
id: str
prompt: str
repeats: int = 3
require_keys: tuple[str, ...] = ()
json_mode: bool = True
max_chars: int = 4000
forbidden: tuple[str, ...] = ()
number_bounds: tuple[tuple[str, float, float], ...] = ()
min_list_len: tuple[tuple[str, int], ...] = ()
CASES = [
Case(
id="support_triage_v3",
prompt=open("fixtures/support_triage_v3.txt", encoding="utf-8").read(),
require_keys=("label", "confidence", "rationale"),
number_bounds=(("confidence", 0.0, 1.0),),
forbidden=("as an AI language model", "I cannot browse"),
),
Case(
id="release_notes_v2",
prompt=open("fixtures/release_notes_v2.txt", encoding="utf-8").read(),
require_keys=("title", "bullets", "risk"),
min_list_len=("bullets", 3),
forbidden=("lorem ipsum",),
),
]
def chat(prompt: str) -> str:
body = json.dumps({
"model": os.environ["MODEL_NAME"],
"messages": [{"role": "user", "content": prompt}],
"temperature": 0,
}).encode()
req = urllib.request.Request(
os.environ["MODEL_BASE_URL"].rstrip("/") + "/chat/completions",
data=body,
headers={
"Content-Type": "application/json",
"Authorization": "Bearer " + os.environ.get("MODEL_API_KEY", ""),
},
method="POST",
)
with urllib.request.urlopen(req, timeout=60) as resp:
payload = json.loads(resp.read().decode())
return payload["choices"][0]["message"]["content"]
def parse_json(text: str) -> Any:
fence = re.search(r"```
(?:json)?\s*(\{.*?\}|\[.*?\])\s*
```", text, re.S)
raw = fence.group(1) if fence else text
return json.loads(raw)
def check(case: Case, text: str) -> list[str]:
failures: list[str] = []
if len(text) > case.max_chars:
failures.append(f"length {len(text)} > {case.max_chars}")
lowered = text.lower()
for phrase in case.forbidden:
if phrase.lower() in lowered:
failures.append(f"forbidden phrase: {phrase}")
if not case.json_mode:
return failures
try:
data = parse_json(text)
except Exception as exc:
return failures + [f"json parse: {exc}"]
if not isinstance(data, dict):
return failures + ["root is not an object"]
for key in case.require_keys:
if key not in data:
failures.append(f"missing key: {key}")
for key, lo, hi in case.number_bounds:
value = data.get(key)
if not isinstance(value, (int, float)) or not (lo <= float(value) <= hi):
failures.append(f"{key}={value!r} outside [{lo}, {hi}]")
for key, n in case.min_list_len:
value = data.get(key)
if not isinstance(value, list) or len(value) < n:
failures.append(f"{key} length {value!r} < {n}")
return failures
def run() -> int:
rows = []
for case in CASES:
sample_fails = 0
for i in range(case.repeats):
text = chat(case.prompt)
fails = check(case, text)
sample_fails += int(bool(fails))
rows.append({"id": case.id, "i": i, "fails": fails, "n": len(text)})
time.sleep(0.2)
# Fail the case if a majority of repeats violate the contract.
print(json.dumps({"id": case.id, "broken_repeats": sample_fails,
"repeats": case.repeats}, ensure_ascii=False))
open("contract_report.jsonl", "w", encoding="utf-8").write(
"\n".join(json.dumps(r, ensure_ascii=False) for r in rows) + "\n"
)
broken = [r for r in rows if r["fails"]]
return 1 if broken else 0
if __name__ == "__main__":
raise SystemExit(run())
The majority-of-repeats rule is the whole statistical idea, and it stays small on purpose. A single malformed sample is a flake when temperature cannot be forced to zero, or when a free endpoint retries internally. Three identical contract breaks in a row is a regression you can attach to a prompt diff. The report file is JSONL so a later job can plot break rates without teaching the grader new tastes.
You can wire the same exit code into CI with a one-line command after exporting a base URL and a model name. The command below is a labeled example, not a claim about any particular vendor quota or uptime target. Keep secrets in the environment, never in the fixture files that you commit next to prompts.
export MODEL_BASE_URL="http://127.0.0.1:8080/v1"
export MODEL_NAME="local-or-free-chat"
python contracts_eval.py
echo $? # 0 means every case kept its contract on a majority of repeats
Generation still needs a model, which is where a free endpoint earns its keep without becoming the story. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which is enough to generate candidates and run this runner on a box you do not have to size yourself. The harness does not depend on that product; any OpenAI-compatible chat completions URL will exercise the same contracts.
A useful operating loop is prompt change, then three contract repeats, then a human glance only if the JSON still stands. That order reverses the common habit of reading a fluent paragraph and assuming the parser will be fine. It also keeps spend near zero on the grading side, because Python is the judge and the model is only the system under test. If you later add a semantic grader, feed it only the samples that already passed the contract, not the wreckage.
Limitations are sharp, and they matter more than the happy path. Contracts cannot tell you that a rationale is wrong, that a citation is fabricated, or that a label is unfair to a user. Repeats estimate flake rate; they do not produce a confidence interval you should quote in a paper. Free generation endpoints can change behavior without a version bump, which is exactly why the contract file belongs in git beside the prompt rather than in a vendor console. Latency, context windows, and rate limits are not characterized here because those numbers go stale and were not measured for this draft.
Teams that should not start here include safety reviewers who need groundedness, medical or legal products that require cited evidence, and ranking problems where the only bug is a subtle preference shift. If your output is free-form narrative with no parseable envelope, wrap it first or skip this method. If your failure mode is “the answer is fluent and well-typed but strategically wrong,” you still need golden judgments or human review after the contract passes.
The practical conclusion is narrow. Score the envelope before you score the prose, fail closed on shape, and spend scarce attention on the cases that survived parsing. A free generation path and a free server make that loop cheap enough to run after ordinary prompt edits instead of once a quarter. If you want a place to host the runner and the candidate model together, MonkeyCode’s free model access and free server option are sufficient to try the script on your own fixtures.
Top comments (0)