Most LLM regressions do not throw exceptions; they quietly change the shape of a JSON response or move a confidence score by four percent. A passing test suite that only asserts a 200 response will never see it. This article presents a golden-set harness that treats model output as a contract, with graders that measure structure and values, and a schedule that runs it overnight. The setup is provider-agnostic, but MonkeyCode's free model access and free server option make it cheap to run continuously. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
A golden set is a fixed collection of inputs paired with expected outputs and a rule for scoring each response. You build it once for the decisions your prompt actually makes, then run it after every prompt edit, model swap, or system change. The key is that the expected output is not a full reference answer; it is a structural contract: a JSON schema, a required field value, or a numeric tolerance. This is the same spirit as defensive unit testing, applied to a function whose output is non-deterministic.
The harness below uses three graders: exact equality for reproducible extractors, contains for classifiers where extra fields are acceptable, and numeric tolerance for scores or probabilities. It reads the endpoint and key from environment variables, so it works with any endpoint that accepts a chat-completions-style payload, including a local server or a free remote one. The threshold at the end turns a regression into a non-zero exit code, which is what lets cron treat it like a failed build.
import json
import os
import sys
import requests
ENDPOINT = os.environ["LLM_ENDPOINT"]
API_KEY = os.environ.get("LLM_API_KEY", "")
def call_model(prompt, system=None, timeout=30):
headers = {"Authorization": f"Bearer {API_KEY}"} if API_KEY else {}
payload = {"model": os.environ.get("LLM_MODEL", "default"), "messages": []}
if system:
payload["messages"].append({"role": "system", "content": system})
payload["messages"].append({"role": "user", "content": prompt})
response = requests.post(ENDPOINT, headers=headers, json=payload, timeout=timeout)
response.raise_for_status()
return response.json()["choices"][0]["message"]["content"]
GOLDEN = [
{
"name": "extract_quantity",
"system": "Extract quantity and unit as JSON.",
"input": "Add 2 kg of flour.",
"expected": {"quantity": 2, "unit": "kg"},
"grader": "exact",
},
{
"name": "urgency_classifier",
"system": "Return JSON with tone: professional, friendly, or urgent.",
"input": "We need this report now.",
"expected": {"tone": "urgent"},
"grader": "contains",
},
]
def grade_exact(output, expected):
try:
return 1.0 if json.loads(output) == expected else 0.0
except Exception:
return 0.0
def grade_contains(output, expected):
try:
data = json.loads(output)
return 1.0 if all(data.get(k) == v for k, v in expected.items()) else 0.0
except Exception:
return 0.0
def grade_numeric(output, field, target, tolerance=0.05):
try:
value = float(json.loads(output)[field])
if abs(value - target) <= tolerance:
return 1.0
return max(0.0, 1.0 - (abs(value - target) - tolerance) / (tolerance * 4))
except Exception:
return 0.0
def run_case(case):
output = call_model(case["input"], system=case.get("system"))
if case["grader"] == "exact":
score = grade_exact(output, case["expected"])
elif case["grader"] == "contains":
score = grade_contains(output, case["expected"])
else:
raise ValueError(f"Unknown grader: {case['grader']}")
return score, output
def main(threshold=0.9):
scores = []
for case in GOLDEN:
score, output = run_case(case)
scores.append(score)
print(f"{case['name']}: {score:.2f}\n---\n{output}\n")
average = sum(scores) / len(scores)
print(f"AVERAGE: {average:.2f}")
if average < threshold:
sys.exit(1)
if __name__ == "__main__":
main(threshold=float(os.environ.get("THRESHOLD", "0.9")))
The graders are deliberately simple because they are easier to audit than a second LLM. An LLM-as-judge can evaluate nuance, but it can also introduce its own drift and cost; a dictionary comparison is deterministic, transparent, and cheap. Once every golden case is mapped to a grader, you effectively have a schema test for natural language. When the average score drops below your threshold, the script prints each failing response so you can see exactly which instruction caused the break.
Scheduling is the second half of the harness. Locally, you would run it after editing a prompt and forget it after two days; on a schedule, it becomes a continuous safety net. A cron entry such as 0 3 * * * cd /path/to/llm-eval && THRESHOLD=0.9 python eval.py >> eval.log 2>&1 runs every night and sends a non-zero exit code to your monitoring. If you do not want to maintain a VM for this, MonkeyCode's free server option gives you a place to host the same script, while the free model access removes the per-call cost concern.
The biggest catch is that golden sets encode yesterday's correctness. When a prompt change intentionally alters behavior, you must re-baseline only the affected cases; otherwise the harness fails for the right reason but stalls your work. Exact JSON matching is also brittle about key ordering and whitespace, so prefer contains or custom numeric graders for anything that touches a real product. Finally, a free server can have cold start latency or job limits, so keep the evaluation short and monitor the log.
Do not use this approach for subjective tasks like brand voice, creative writing, or open-ended Q&A, where a single expected answer does not exist. It shines only when the model's output has a bounded shape: extraction, classification, routing, or scoring. If you are still exploring what a model can do, run manual trials first; a golden set will only lock in assumptions you have not validated. When the contract is clear, however, a thirty-line script gives you a nightly alarm that wakes you up the moment a prompt drifts.
Top comments (0)