A Repeatable Way to Regression-Test Your Prompts Before You Ship Them

Prompts are code. They get edited, merged, and shipped — and just like code, a small change can silently break behavior you depended on. Yet most of us still test prompts by eyeballing one or two outputs in a chat window and calling it done.

This article walks through a lightweight, reproducible workflow for regression-testing prompts: a fixed set of evaluation cases, a scoring rubric you can apply consistently, and a script you can re-run every time the prompt changes. The only thing you need is access to a model you can call repeatedly without worrying about cost per experiment — which is where free-tier model access becomes genuinely useful.

The problem: "it looked fine when I tried it"

A typical failure mode goes like this:

1. You tweak a system prompt to fix one bad output.
2. You re-test the case that was failing. It passes.
3. You ship it.
4. Three other cases that used to work are now subtly worse, and nobody notices for a week.

The fix is the same one we apply to code: keep a small, versioned set of cases that represent the behaviors you care about, and re-run all of them on every change. The barrier has usually been that running dozens of model calls per iteration adds up. If you have free model access, that barrier mostly disappears — so there is no excuse left for vibe-based prompt testing.

The workflow

The workflow has four parts:

1. A case file — inputs plus expected properties of the output (not exact strings).
2. A rubric — deterministic checks where possible, model-graded checks where not.
3. A runner script — executes every case against the current prompt and prints a report.
4. A decision rule — what has to pass before the prompt change ships.

1. The case file

Store cases as JSON next to the prompt in your repo. Notice that expectations are properties, not exact outputs — LLM output is nondeterministic, so exact-match tests will flake.

[
  {
    "id": "summary-short-input",
    "input": "The meeting covered the Q3 roadmap. We agreed to delay the mobile launch by two weeks and prioritize the billing migration.",
    "expect": {
      "max_words": 40,
      "must_mention": ["billing"],
      "must_not_contain": ["I think", "As an AI"]
    }
  },
  {
    "id": "summary-empty-input",
    "input": "",
    "expect": {
      "should_refuse_gracefully": true
    }
  }
]

Aim for 10–20 cases covering: the happy path, edge cases (empty input, huge input, non-English input), past bugs (every time a prompt fails in production, add that case), and adversarial inputs if the output is user-facing.

2. Deterministic checks first

Anything you can check with plain code, check with plain code. It is free, fast, and never flaky.

# checks.py
def word_count_ok(output, max_words):
    return len(output.split()) <= max_words

def mentions_all(output, terms):
    lower = output.lower()
    return all(t.lower() in lower for t in terms)

def avoids_all(output, phrases):
    lower = output.lower()
    return not any(p.lower() in lower for p in phrases)

For properties you cannot check deterministically ("refuses gracefully", "tone is professional"), use a second model call as a judge — but keep the judge prompt narrow ("Answer only YES or NO: does this response decline the request without an error message?") so it stays fairly stable.

3. The runner

# run_evals.py — pseudocode; adapt the client to your provider
import json
from checks import word_count_ok, mentions_all, avoids_all

SYSTEM_PROMPT = open("prompt.txt").read()
CASES = json.load(open("cases.json"))

def run():
    results = []
    for case in CASES:
        output = call_model(SYSTEM_PROMPT, case["input"])
        exp = case["expect"]
        checks = {}
        if "max_words" in exp:
            checks["max_words"] = word_count_ok(output, exp["max_words"])
        if "must_mention" in exp:
            checks["must_mention"] = mentions_all(output, exp["must_mention"])
        if "must_not_contain" in exp:
            checks["must_not_contain"] = avoids_all(output, exp["must_not_contain"])
        results.append({"id": case["id"], "checks": checks,
                        "passed": all(checks.values())})
    failed = [r for r in results if not r["passed"]]
    print(f"{len(results) - len(failed)}/{len(results)} cases passed")
    for r in failed:
        print("FAIL:", r["id"], r["checks"])
    return len(failed) == 0

if __name__ == "__main__":
    raise SystemExit(0 if run() else 1)

Because it exits non-zero on failure, this drops straight into CI. Run each case 2–3 times if you want a rough pass-rate instead of a single sample — another place where free model calls matter, because tripling your eval runs costs nothing.

4. Where the free tier fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I run these eval loops on MonkeyCode, which currently offers free model access and a free server option, so the whole runner above can execute without a meter running in the background. That changes what is practical: re-running the full suite on every prompt edit, and running cases multiple times to smooth out nondeterminism, stops being a budget conversation.

Two honest caveats about building on any free tier:

• Do not hard-depend on it in production paths. Free access is for development, evaluation, and iteration. Treat availability and limits as subject to change, and keep your client code provider-agnostic (a thin call_model wrapper, as above) so you can swap in a paid endpoint later.
• Free does not mean identical. The model you test against should be the model you ship against, or as close as possible. A prompt that passes evals on one model may fail on another.

Decision table: is this workflow worth setting up?

Situation	Recommendation
One-off prompt for a personal script	Skip it; manual testing is fine
Prompt in a user-facing feature	Yes — minimum 10 cases, run in CI
Team edits prompts frequently	Yes — version the prompt and cases together
Output must be exact/structured (JSON, SQL)	Yes, but rely on deterministic parsing checks, not rubrics
Latency- or cost-sensitive production calls	Use free evals in dev; do not route prod traffic through free tiers

Limitations

• Nondeterminism is real. A single run is a sample, not a proof. Run key cases multiple times and look at pass rates.
• Model-graded checks drift. If you change judge models, re-baseline your results.
• Small case sets give false confidence. Twenty cases will not catch everything. Grow the set from real production failures.
• This does not replace human review for tone, safety, or domain correctness — it catches regressions, not judgment calls.

Who should not use this approach

If your prompt is throwaway, if the output is only ever read by you, or if you cannot define what "correct" looks like even loosely, the eval suite is overhead. Also skip routing anything production-critical through a free tier — keep that for iteration only.

Closing

The shift that matters is cultural: treat prompt changes like code changes, with a versioned case set that fails the build when behavior regresses. If you want to try this without a billing page open in the next tab, the eval runner above is a good first project for MonkeyCode's free model access — and if you have tricks for keeping LLM evals stable, I would like to hear them in the comments.