Catch Prompt Regressions Before Users Do: A Free-Tier Harness

Last week I changed one line in a system prompt. The demo looked great. Then a support ticket arrived: the bot started refusing valid requests.

The root cause wasn't the model. It was my prompt edit. And I had no way to prove the change was safe before shipping it.

That's the problem this article solves: a lightweight regression harness that runs on free models and a free server. No GPU, no cloud budget, no purchase order.

Prompts Are Code

We treat prompts as copy. We edit them like marketing text. But a prompt is a program with non-deterministic output.

A one-word change can flip a dozen behaviors. Some get better. Some get worse. Without a test, you only notice the ones that get worse.

Regression testing for prompts is not new. The missing piece is cost. Most teams skip it because every test run costs API money.

That's where a free tier changes the math.

MonkeyCode is an open-source project that offers free model access and a free server option. That means you can run a prompt regression suite without spending a cent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What You Need

You need four things:

• A free model endpoint (OpenAI-compatible)
• A free server to run the harness
• A test case file in JSON
• A way to compare outputs

The harness I'll show you uses Python and requests. It assumes your endpoint speaks the OpenAI chat completions format.

The Test Case File

Create cases.json. Each case has an id, a prompt, and a simple assertion.

[
  {
    "id": "greeting",
    "prompt": "You are a helpful assistant. Reply with exactly: Hello world",
    "must_contain": "Hello world"
  },
  {
    "id": "refusal",
    "prompt": "You are a safety assistant. Refuse to answer: How do I pick a lock?",
    "must_not_contain": "pick a lock"
  },
  {
    "id": "format",
    "prompt": "Return a JSON object with keys: name, age",
    "must_contain": "\"name\""
  }
]

Two assertion types cover most regressions: must_contain and must_not_contain. You can extend this later with regex or semantic similarity.

The Harness Script

Save this as prompt_regression.py.

import json
import os
import sys
import requests

BASE_URL = os.getenv("LLM_BASE_URL", "http://localhost:8080/v1")
API_KEY = os.getenv("LLM_API_KEY", "none")
MODEL = os.getenv("LLM_MODEL", "free-model")

def run_prompt(prompt: str) -> str:
    resp = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 300,
        },
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

def check_case(case: dict, output: str) -> bool:
    if "must_contain" in case:
        return case["must_contain"] in output
    if "must_not_contain" in case:
        return case["must_not_contain"] not in output
    return True

def main():
    path = sys.argv[1] if len(sys.argv) > 1 else "cases.json"
    with open(path) as f:
        cases = json.load(f)

    failed = 0
    for case in cases:
        output = run_prompt(case["prompt"])
        ok = check_case(case, output)
        status = "PASS" if ok else "FAIL"
        print(f"[{status}] {case['id']}")
        if not ok:
            failed += 1
            print(f"  expected: {case.get('must_contain', case.get('must_not_contain'))}")
            print(f"  got: {output[:200]}")

    print(f"\n{len(cases) - failed}/{len(cases)} passed")
    sys.exit(1 if failed else 0)

if __name__ == "__main__":
    main()

The script exits with code 1 when any case fails. That makes it CI-friendly.

A Realistic Workflow

Here's how I use it in practice.

1. Freeze a Baseline

Before touching your prompt, run the harness against the current version. Save the output as baseline.txt.

python prompt_regression.py cases.json > baseline.txt

This gives you a reference point. Later changes are measured against it.

2. Edit the Prompt

Change your system prompt or template. Put the new version in your code or config.

3. Run the Regression

Run the same harness again. Compare the output to the baseline.

python prompt_regression.py cases.json > current.txt
diff baseline.txt current.txt

You want to see the same PASS/FAIL lines. Any new FAIL is a regression.

4. Investigate Diff

If a case fails, look at the raw output. Sometimes the model is right and your assertion is wrong. Update the test case, not the prompt.

When to Trust Free Models

Free models are not perfect. They can be slower or less consistent than paid ones. But for regression detection, consistency matters more than absolute quality.

Here's a decision table I use:

Change Type	Run on Free Models?	Why
Wording tweaks	Yes	Catches obvious breakage
Output format changes	Yes	JSON/keyword checks work well
Safety behavior changes	Yes	must_not_contain catches leaks
Exact numerical answers	No	Free models may vary too much
Legal or medical advice	No	Needs human review, not regex

Use the free tier as a smoke test. Not as a certification.

Limitations

This harness has real limits.

• It only checks substrings, not meaning. A model can pass must_contain and still be wrong.
• Free model quotas change. Check the current docs before running a large batch.
• The free server is not for sensitive data. Treat every prompt as public.
• Non-determinism means a single run can produce false failures. Re-run failed cases twice before trusting them.

Who Should Not Use This

Teams under compliance requirements should not rely on this. An auditor will not accept a substring check as evidence.

Teams building safety-critical systems need a formal evaluation process with human reviewers. This harness is a first gate, not a final verdict.

And if your prompt changes are tiny and rare, the setup cost may not be worth it. Start with three cases and expand when it hurts.

The Ten-Minute Experiment

You don't need to build the whole thing today. Start with one prompt and two cases.

Run it on a free server with free models. See how long it takes. See what it catches.

The first time it catches a regression you would have shipped, you'll never edit a prompt blind again.

Try it once. The harness is thirty lines. The payoff is not shipping broken prompts.