A small change to a system prompt quietly broke one of my test cases last month. The model still sounded right — it just stopped returning valid JSON for edge-case inputs, and I only noticed when a downstream parser failed. If you ship anything that depends on LLM output, you have some version of this problem: prompts are code, but almost nobody regression-tests them like code.
This post walks through a minimal, reproducible harness for prompt regression testing that fits inside free tiers — a free hosted model for the system under test, and a free server option to run the runner on a schedule so it doesn't depend on your laptop being open. The interesting part isn't any specific provider; it's the workflow. Everything below runs with plain Python and a HTTP API.
The core idea: treat prompts like a function under test
A prompt test has three parts:
- Input — the user message or context you send.
- Contract — a machine-checkable assertion about the output (valid JSON, contains a required key, matches a regex, length bounds, refusal behavior).
- Snapshot — the full raw response, stored so you can diff behavior across model or prompt versions later.
Notice what is not here: "does the answer feel good." Vibe checks don't survive CI. Start with contracts you can assert mechanically, and only add human review for the cases where mechanical checks genuinely can't work.
A minimal harness
The following is a working skeleton (tested pattern, adapt endpoints to your provider). It reads a YAML test suite, calls a chat-completions-style API, and applies assertions:
# prompt_eval.py — minimal prompt regression harness
import json, re, sys, time
import urllib.request
import yaml # pip install pyyaml
API_URL = "https://your-provider.example/v1/chat/completions"
API_KEY = "sk-..." # read from env in real usage
MODEL = "your-model-name"
def call_model(system, user):
body = json.dumps({
"model": MODEL,
"messages": [
{"role": "system", "content": system},
{"role": "user", "content": user},
],
"temperature": 0,
}).encode()
req = urllib.request.Request(
API_URL, data=body,
headers={"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=60) as r:
data = json.loads(r.read())
return data["choices"][0]["message"]["content"]
def check(case, output):
failures = []
for rule in case.get("assert", []):
if rule["type"] == "valid_json":
try:
parsed = json.loads(output)
except json.JSONDecodeError:
failures.append("output is not valid JSON")
continue
for key in rule.get("required_keys", []):
if key not in parsed:
failures.append(f"missing key: {key}")
elif rule["type"] == "regex":
if not re.search(rule["pattern"], output):
failures.append(f"regex not matched: {rule['pattern']}")
elif rule["type"] == "max_length":
if len(output) > rule["chars"]:
failures.append(f"too long: {len(output)} chars")
elif rule["type"] == "must_refuse":
# crude but effective: refusal should NOT contain the forbidden payload
if rule["forbidden_substring"].lower() in output.lower():
failures.append("expected refusal, got payload")
return failures
def main(suite_path):
suite = yaml.safe_load(open(suite_path))
results = []
for case in suite["cases"]:
out = call_model(suite["system_prompt"], case["input"])
fails = check(case, out)
results.append({"id": case["id"], "ok": not fails,
"failures": fails, "snapshot": out})
time.sleep(1) # be polite to free tiers
passed = sum(1 for r in results if r["ok"])
print(f"{passed}/{len(results)} passed")
with open("last_run.json", "w") as f:
json.dump(results, f, indent=2)
sys.exit(0 if passed == len(results) else 1)
if __name__ == "__main__":
main(sys.argv[1])
And the suite that caught my JSON regression:
# suite.yaml
system_prompt: |
You extract invoice data. Respond with JSON only:
{"vendor": string, "total": number, "currency": string}
cases:
- id: happy_path
input: "Invoice from Acme Corp, total $1,250.00 USD"
assert:
- type: valid_json
required_keys: [vendor, total, currency]
- id: no_currency_stated
input: "Invoice from Beta Ltd, amount due 300"
assert:
- type: valid_json
required_keys: [vendor, total, currency]
- id: garbage_input
input: "asdf qwer zxcv"
assert:
- type: valid_json
- type: max_length
chars: 400
The no_currency_stated case is the one that matters. A prompt edit that makes the model chattier ("I couldn't find a currency, so...") passes a human eyeball test and fails this suite in two seconds. Exit code 1 means your CI goes red. That's the whole trick.
Where free tiers fit (and where they don't)
Running this on your laptop works, but the value compounds when it runs unattended — nightly, or on every prompt-file change in a repo. That's the point where you want two things you don't pay for: a model endpoint you can call without a billing account, and somewhere to run the runner.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. In my setup, MonkeyCode's free model access serves as the endpoint under test, and its free server option hosts the runner on a cron schedule. I deliberately treat the provider as a swappable config (API_URL, MODEL) rather than wiring anything provider-specific into the harness — you should too, because free tiers change and your tests shouldn't care.
A pragmatic decision table for choosing what runs where:
| Component | Free tier is fine when… | Pay for it when… |
|---|---|---|
| Model under test | You're validating prompt structure and output contracts | You need production-grade latency, SLAs, or a specific frontier model |
| Runner host | Nightly or on-push schedules, small suites (<50 cases) | You need parallel runs, long context evals, or many suites |
| Judge model (LLM-as-judge) | Rough triage of "is this answer on-topic" | Scoring that feeds release decisions — judge drift becomes its own problem |
| Snapshot storage | JSON files in git | You need querying across hundreds of historical runs |
Honest limitations
- Contract tests catch structural regressions, not quality regressions. A model can return valid JSON with worse extractions. Pair this with a small set of golden-answer comparisons (exact match or embedding similarity) for your highest-stakes cases.
-
Temperature 0 is not determinism. Providers change models underneath you. That's exactly why you keep snapshots — when a run goes red with zero prompt changes,
last_run.jsondiffs tell you whether the provider shipped something new. - Free tiers have rate limits and can disappear. The harness above sleeps between calls and keeps the provider behind config for this reason. If your suite grows past what a free tier tolerates, that's a good problem — it means the tests earned their keep.
- Don't do this if your "test" is really a subjective writing-quality judgment, or if you have fewer than ~5 meaningful cases — the harness overhead exceeds the value. Just re-run the prompt manually and eyeball it.
Takeaway
Prompts are the least-tested code most of us ship. A YAML file, ~60 lines of Python, and mechanical assertions get you 80% of the value of fancy eval frameworks, and the whole thing fits comfortably inside free model access plus a free scheduled runner. If you want to try this shape of workflow without a billing account, MonkeyCode's free model and server options are one way to stand it up — but the harness above works against any OpenAI-compatible endpoint, which is the part worth keeping.
What's the most embarrassing prompt regression you've shipped? Curious what assertions other people have found worth encoding.
Top comments (1)
Prompt regression harnesses are one of the most practical ways to make AI work less mystical. Even a small free-tier suite can catch silent behavior drift before it turns into a production surprise.