A prompt eval that cannot fail a mutated golden case will not catch a silent regression. Green builds in that situation measure grader blindness, not model stability, and they age poorly after endpoint swaps. Store a negative control beside each happy-path fixture, then refuse the suite when that control still passes. The remainder of this article specifies a small Python harness that enforces that rule with deterministic graders.
Silent regressions in language-model features rarely resemble stack traces, thrown exceptions, or obvious HTTP 500 responses. The response still serializes, the CI job still exits zero, and a required tool argument has quietly disappeared. Teams then move nightly traffic onto a cheaper endpoint because token cost dominates the remaining eval budget. That move is economically reasonable, yet it raises the chance that wording, structure, and refusals shift together.
Happy-path golden files cannot distinguish those shifts from a harmless paraphrase of the same contract. A close analogy is a unit test that only asserts the result object is not None. The function can return an empty dict, a truncated string, or a fabricated identifier without turning the test red. Prompt graders that only check parsed JSON fail in that same ornamental way, which mutation is meant to expose.
Mutation supplies the missing counterexample by copying a golden completion and breaking exactly one stated contract. The harness then demands that the grader reject the break, and it fails the build if the grade is still pass. Think of a smoke detector whose LED stays green because nobody ever tested the sensor with canned smoke. Golden cases without negative controls are that detector, and the mutated twin is the canned smoke.
The proposed artifact is a directory of YAML cases, a grader module, and a runner that calls any OpenAI-style chat endpoint. Each case declares expect for the live model output and mutate for the synthetic control that must fail. The runner grades a live completion against expect, then grades the mutated completion, and it treats a passing mutant as a harness bug. Nothing in the design requires a judge model, which keeps the loop cheap enough for a free server.
Consider a support agent that must call lookup_order with an order_id and must not invent a tracking number. A golden case records the user turn, the required tool name, and the identifiers the arguments must carry. The mutated twin keeps the same user turn but drops order_id and injects a fabricated tracking token into the content field. A structural grader can catch both defects without interpreting prose, which is the entire point of the method.
The following case file is illustrative scaffolding for the runner, not a dump from a production system. It pairs one live contract with one control payload so the suite can fail closed. Names and identifiers inside the fixture are synthetic and should stay that way in any copied tree. Load every file from eval/cases so adding a twin never requires a code change.
# eval/cases/lookup_order.yaml
id: lookup_order_requires_id
user: "Where is order A-1042?"
expect:
tool_name: lookup_order
required_args: ["order_id"]
forbidden_substrings:
- "TRACK-"
- "I found tracking"
mutate:
completion:
tool_name: lookup_order
arguments:
customer_email: "hidden@example.com"
content: "Tracking TRACK-999 is out for delivery."
The grader should stay boring, because string search, key presence, and enum equality do not flake with temperature. Embedding similarity and LLM-as-judge can wait behind a flag, since they reintroduce both cost and sampling variance. The proposed function below treats a missing tool, a missing argument, or a forbidden substring as a hard failure. It returns a list of short codes so the runner can print a grep-friendly report in continuous integration logs.
# eval/grade.py
from typing import Any
def grade_completion(completion: dict[str, Any], expect: dict[str, Any]) -> list[str]:
failures: list[str] = []
tool = completion.get("tool_name")
args = completion.get("arguments") or {}
text = completion.get("content") or ""
if expect.get("tool_name") and tool != expect["tool_name"]:
failures.append(f"tool:{tool!r}")
for key in expect.get("required_args", []):
if key not in args or args[key] in (None, ""):
failures.append(f"missing_arg:{key}")
for needle in expect.get("forbidden_substrings", []):
if needle.lower() in text.lower():
failures.append(f"forbidden:{needle!r}")
return failures
The runner needs two paths: a live completion against the endpoint, and a synthetic mutation that never hits the network. Live calls should send a frozen system prompt so prompt drift remains visible even when the completion happens to match. Timeouts and empty bodies are failures rather than skips, because a free server can shed load without a structured error. The snippet below is a proposed client around a common chat-completions shape, not a claim about any vendor wire format.
# eval/run.py
import json, os, sys, urllib.request, yaml
from pathlib import Path
from grade import grade_completion
BASE = os.environ.get("EVAL_BASE_URL", "http://127.0.0.1:8080").rstrip("/")
MODEL = os.environ.get("EVAL_MODEL", "local-free")
PROMPT = Path("eval/system_prompt.txt").read_text(encoding="utf-8")
def chat(user: str) -> dict:
body = json.dumps({
"model": MODEL,
"temperature": 0,
"messages": [
{"role": "system", "content": PROMPT},
{"role": "user", "content": user},
],
}).encode()
req = urllib.request.Request(
f"{BASE}/v1/chat/completions",
data=body,
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req, timeout=30) as resp:
payload = json.loads(resp.read().decode())
msg = payload["choices"][0]["message"]
tool_calls = msg.get("tool_calls") or []
if tool_calls:
fn = tool_calls[0]["function"]
return {
"tool_name": fn.get("name"),
"arguments": json.loads(fn.get("arguments") or "{}"),
"content": msg.get("content") or "",
}
return {"tool_name": None, "arguments": {}, "content": msg.get("content") or ""}
def main() -> int:
failed = 0
for path in sorted(Path("eval/cases").glob("*.yaml")):
case = yaml.safe_load(path.read_text())
live = chat(case["user"])
live_fails = grade_completion(live, case["expect"])
mutant_fails = grade_completion(case["mutate"]["completion"], case["expect"])
if live_fails:
print(f"FAIL live {case['id']}: {live_fails}")
failed += 1
if not mutant_fails:
print(f"FAIL control {case['id']}: grader accepted the mutation")
failed += 1
if not live_fails and mutant_fails:
print(f"PASS {case['id']}")
return 1 if failed else 0
if __name__ == "__main__":
sys.exit(main())
A CI step can invoke that module after exporting EVAL_BASE_URL and a model identifier that the server accepts. The commands below are ordinary POSIX and do not encode a vendor quota, hardware profile, or retention period. Capture stdout, because the FAIL control line is the signal that the grader went numb after someone edited a case. Keep temperature at zero for this suite even if production sampling is higher, since the goal is contract stability.
export EVAL_BASE_URL="${EVAL_BASE_URL:-http://127.0.0.1:8080}"
export EVAL_MODEL="${EVAL_MODEL:-local-free}"
python -m pip install pyyaml
python eval/run.py
Where a free endpoint enters the workflow is mechanical rather than promotional, because the runner only needs a stable chat URL. Nightly jobs must absorb dozens of frozen prompts without attaching a paid invoice to every mutation check that never leaves the process. MonkeyCode is relevant here only as one option with free model access and a free server that can host the same chat loop.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
No model names, hardware sketches, or token ceilings are assumed, because those figures change and this harness should not depend on them. If the server is slow or returns empty choices, the runner already treats that outcome as a live failure, which is the conservative default. Paid models can still score a weekly semantic sample; they are not required to prove that a local grader can fail.
Limitations follow from that same conservatism and should be read before copying the runner into a pipeline. Structural graders miss fluent lies that keep the tool name intact and simply omit every forbidden token. They also miss a correct tool invoked for the wrong user unless order_id is bound to a fixture identifier in expect. Mutation that only edits the completion, not the user turn, will not catch prompt-injection cases that need a hostile input.
Flaky networks can fail live cases even when the model is behaving, so a single logged retry is reasonable. Infinite retry is not reasonable, because it hides outages and turns the harness into a latency sponge. This approach is a poor fit for open-ended creative features whose contract is taste rather than schema or tool shape. It is also a poor fit for teams that already adjudicate every model diff with a calibrated judge and a human reviewer.
Do not use these results as evidence that two models are equivalent, because the grader never scores paraphrase quality. Do not point the runner at production user data; golden cases should be synthetic or redacted fixtures with stable identifiers. A golden file without a negative control remains an untested smoke detector, and a cheaper endpoint only makes the false calm cheaper to repeat. Add one mutated twin per case, keep the grader deterministic, and spend free-server cycles on live checks you would otherwise skip.
If a free-server endpoint already exists in the lab, set EVAL_BASE_URL and require both live and control checks before the next prompt edit ships. The harness will still miss semantic drift that preserves keys, so treat a green run as necessary evidence rather than sufficient evidence. Expand the mutate block whenever you add an expect field, or the new assertion will rot into another ornamental check.
Top comments (0)