A prompt regression is the only production bug that never throws an exception and never writes to your error log. When you change a prompt, the API still returns 200, the JSON still parses, and the latency chart still looks healthy, but the model quietly stops honoring a constraint it used to follow. The reliable fix is not more careful diff reading; it is a small eval harness with golden cases, deterministic graders, and a baseline comparison that fails your pipeline when the score drops.
Manual spot-checking fails for a structural reason: humans test happy paths, and happy paths are exactly where language models stay stable. The regressions that matter live in edge cases, in formatting constraints, in refusal behavior, and in instructions that conflict with the model's prior. I have caught more of these by accident in production than by intention at the keyboard. That is why I now treat an eval suite as a required artifact of any prompt change.
The harness I use is deliberately small, because a complicated eval framework is just another thing that can break silently. It has three pieces: a JSON file of golden cases, a registry of grader functions, and a comparison step that measures the new score against a stored baseline. The whole thing fits in one Python file and one workflow file, and it runs anywhere a cron job can run.
# eval_harness.py
import json
import re
import sys
from pathlib import Path
GRADERS = {}
def grader(name):
def wrap(fn):
GRADERS[name] = fn
return fn
return wrap
@grader("regex")
def grade_regex(case, output):
return bool(re.search(case["pattern"], output))
@grader("keyword")
def grade_keyword(case, output):
lowered = output.lower()
return any(k.lower() in lowered for k in case["keywords"])
@grader("word_count_max")
def grade_word_count(case, output):
return len(output.split()) <= case["max_words"]
def make_model_fn(base_url, api_key, model):
from openai import OpenAI
client = OpenAI(base_url=base_url, api_key=api_key)
def fn(prompt):
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0.2,
)
return response.choices[0].message.content
return fn
def run_case(case, model_fn):
output = model_fn(case["prompt"])
passed = GRADERS[case["grader"]](case, output)
return {"id": case["id"], "passed": passed, "output": output}
def run_suite(cases, model_fn):
results = [run_case(c, model_fn) for c in cases]
score = sum(r["passed"] for r in results) / len(results)
return score, results
def main():
cases = json.loads(Path("golden_cases.json").read_text())
base_url, api_key, model = sys.argv[1], sys.argv[2], sys.argv[3]
model_fn = make_model_fn(base_url, api_key, model)
score, results = run_suite(cases, model_fn)
if "--save-baseline" in sys.argv:
Path("baseline.json").write_text(json.dumps({"score": score}))
print(f"baseline saved: {score:.2f}")
return
baseline = json.loads(Path("baseline.json").read_text())
delta = score - baseline["score"]
print(f"score={score:.2f} baseline={baseline['score']:.2f} delta={delta:+.2f}")
for r in results:
if not r["passed"]:
print(f"FAIL {r['id']}: {r['output'][:160]}")
if delta < -0.05:
sys.exit(1)
if __name__ == "__main__":
main()
The golden cases file is where the real thinking happens, because each case pairs a prompt with a grader and an explicit constraint. A good case reads like a contract: it states what the output must contain, how long it may be, or which category it must fall into. Vague cases produce vague signals, so I write them as if a future developer will have to debug them at 2 a.m.
[
{
"id": "date-format-001",
"prompt": "Extract the ISO date from: 'Deploy shipped 2026-08-27 at 14:00 UTC.'",
"grader": "regex",
"pattern": "2026-08-27"
},
{
"id": "summary-length-001",
"prompt": "Summarize in at most 30 words: 'The batch job failed at 03:12 because the queue backlog exceeded the visibility timeout, and the retry policy was exhausted after three attempts.'",
"grader": "word_count_max",
"max_words": 30
},
{
"id": "sentiment-001",
"prompt": "Classify the sentiment as positive, negative, or neutral: 'The API returned 500s for an hour during peak traffic.'",
"grader": "keyword",
"keywords": ["negative"]
}
]
The first run establishes the baseline, and every subsequent run compares against it. Saving the baseline is a one-liner, and so is the comparison that follows.
python eval_harness.py "$BASE_URL" "$API_KEY" "model-x" --save-baseline
python eval_harness.py "$BASE_URL" "$API_KEY" "model-x"
The exit code matters more than the printed score, because it lets the harness act as a gate in a pipeline. A delta below minus five points fails the run, and a failed run is what forces a human to look at the per-case output instead of trusting the average.
The second half of the workflow is scheduling, because an eval you run once is a snapshot and an eval you run on every prompt change is a tripwire. Continuous runs cost tokens and need a host, and that friction is what kills most eval setups before they catch anything. In my case the harness runs on a free server option from MonkeyCode, and the model calls go through MonkeyCode's free model access, so the tripwire stays armed at zero cost under the current terms, which include a 10 million token allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Token allowances and server terms change over time, so check the project page before you rely on them; the harness itself is endpoint-agnostic and works with any OpenAI-compatible API.
A scheduled workflow keeps the tripwire armed without anyone remembering to run it. The GitHub Actions version below runs the suite weekly, fails loudly on regression, and leaves the results in the workflow log.
name: prompt-eval
on:
schedule:
- cron: "0 6 * * 1"
workflow_dispatch:
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install openai
- run: python eval_harness.py "$BASE_URL" "$API_KEY" "$MODEL"
env:
BASE_URL: ${{ secrets.BASE_URL }}
API_KEY: ${{ secrets.API_KEY }}
MODEL: ${{ vars.MODEL }}
The aggregate score hides the most interesting failure mode, which is one case flipping while another improves and the average stays flat. That is why the harness prints failing case IDs and why I keep the raw outputs in the log: the per-case diff is the real signal, and the score is only a summary of it. When a case starts failing, the output snippet usually reveals whether the prompt lost a constraint or the model changed its behavior.
The limitations are real and worth stating plainly. Golden cases encode today's assumptions, so they go stale as your product changes and need the same maintenance as tests. A regex grader cannot catch semantic drift, and a keyword grader rewards guessing, so the graders only enforce the constraints you know how to express. Twenty cases will not catch everything; this is a tripwire, not a benchmark, and its job is to make silent regressions loud, not to prove quality.
This approach is also not for everyone. If you are prototyping and the prompt changes hourly, a harness will slow you down more than it helps, because the baseline churns faster than you can read it. If you have no labeled cases yet, spend a week logging real outputs and turn the painful failures into golden cases before you build anything. Teams that already ship prompt changes without review are the ones that benefit most, because the harness gives them a reviewable artifact instead of a vibes-based approval.
The model will change under you whether you watch it or not, and the only question is whether the change arrives as a loud failure or a quiet degradation. A minimal harness with golden cases, deterministic graders, and a baseline gate turns the second category into the first. If you want a free place to arm this tripwire, MonkeyCode's free model access and free server option are a reasonable starting point, but the harness itself is the part you should keep.
Top comments (1)
I like keeping the harness boring. The piece I would add is one deliberately ugly golden case per prompt, something that used to pass by accident. My prompt tests got much more useful once I stopped only saving the clean examples and started saving the weird failures with a short note on why they matter.