Prompts Break in Production Too: Building a Pre-Merge Regression Check That Runs on Free Models

A few weeks ago someone on my team adjusted the system prompt behind a support-ticket triage bot. The change was two sentences long, intended to make the bot ask clarifying questions more politely. It did that. It also stopped emitting the priority field our router depends on, so for half a day every ticket defaulted to low priority, including one from a customer whose database was down.

Nobody did anything careless. The edit was reviewed by two people. What we didn't have was any automated way to answer the question: which behaviors of this prompt did my edit just change? We had that discipline for code and none for prompts, even though the prompt was effectively the API contract for a piece of routing logic.

This post describes the check I built after that incident: a small suite that replays a frozen set of scenarios against a prompt whenever it changes, fails the pull request if contractual behaviors regress, and uses a free model as a first-pass filter so the expensive production model only sees edits that survived screening. If you read my earlier piece on nightly drift canaries, the distinction matters: that one watches a model that changed underneath you; this one watches edits you made on purpose.

The framing that makes it work

The mistake most prompt-testing attempts make is comparing generated text to reference text. Free-form output differs every run, so you get noise, and teams learn to ignore the suite within a week.

What worked instead: treat the prompt as a function whose side effects you can extract are the contract, and only assert on those. For our triage bot, the contract was never "the reply sounds empathetic." It was: the reply contains a valid priority field from a fixed enum, a category slug, and at most one follow-up question. Everything else is style, and style stays a human review concern.

So each test scenario pairs an input with a set of claims about the parsed response, never about the wording.

The artifact

Scenarios live in YAML because non-engineers on my team edit them. The runner is plain Python, no framework, and the model backend is injected as a callable so the suite doesn't care what it talks to.

# scenarios/triage.yaml
- id: outage-is-urgent
  input: |
    Our production database has been unreachable for 20 minutes.
    Multiple services are failing.
  assert:
    priority_in: [critical, high]
    max_questions: 1

- id: billing-question-not-urgent
  input: |
    Can you explain the proration on my last invoice?
  assert:
    priority_in: [low, medium]
    category_equals: billing

- id: vague-report-still-parses
  input: "it's broken"
  assert:
    priority_in: [low, medium, high, critical]
    min_questions: 1

# prompt_check.py
import json, re, sys
from pathlib import Path

def extract_contract(reply: str) -> dict:
    """Pull the machine-checkable facts out of a free-form reply."""
    priority = re.search(r"priority[:\s]+(\w+)", reply, re.I)
    category = re.search(r"category[:\s]+(\w+)", reply, re.I)
    questions = reply.count("?")
    return {
        "priority": priority.group(1).lower() if priority else None,
        "category": category.group(1).lower() if category else None,
        "questions": questions,
    }

def evaluate(contract: dict, assertions: dict) -> list[str]:
    problems = []
    if "priority_in" in assertions and contract["priority"] not in assertions["priority_in"]:
        problems.append(f"priority={contract['priority']!r} not in {assertions['priority_in']}")
    if "category_equals" in assertions and contract["category"] != assertions["category_equals"]:
        problems.append(f"category={contract['category']!r} != {assertions['category_equals']!r}")
    if "max_questions" in assertions and contract["questions"] > assertions["max_questions"]:
        problems.append(f"{contract['questions']} questions > max {assertions['max_questions']}")
    if "min_questions" in assertions and contract["questions"] < assertions["min_questions"]:
        problems.append(f"{contract['questions']} questions < min {assertions['min_questions']}")
    return problems

def run_suite(prompt_file: str, scenario_file: str, backend) -> int:
    import yaml  # pip install pyyaml
    system = Path(prompt_file).read_text()
    scenarios = yaml.safe_load(Path(scenario_file).read_text())
    bad = 0
    for s in scenarios:
        reply = backend(system, s["input"])
        problems = evaluate(extract_contract(reply), s["assert"])
        if problems:
            bad += 1
            print(f"[red]   {s['id']}: {'; '.join(problems)}")
        else:
            print(f"[green] {s['id']}")
    print(f"\n{len(scenarios) - bad}/{len(scenarios)} scenarios held the contract")
    return 1 if bad else 0

if __name__ == "__main__":
    from backends import free_model   # any callable (system, user) -> str
    sys.exit(run_suite("prompts/triage.txt", "scenarios/triage.yaml", free_model))

The exit code is the entire CI integration: prompt file changed in a PR, job runs, nonzero status blocks merge. If your prompt renders with variables, template them before hashing and stash the rendered version as a build artifact so a failure six weeks from now is reproducible.

Two implementation notes that saved me pain:

• Pin temperature to zero wherever the backend allows it, and re-run any red scenario twice before counting it. A scenario that fails once in three runs is a flakiness problem you must fix at the assertion level, not by retrying in CI forever.
• Version the extractor with the prompt. When I changed what the router parses, I changed extract_contract in the same commit as the prompt and the router code. They're one logical unit; splitting them across PRs is how you get green suites guarding yesterday's contract.

The two-stage funnel, and why the cheap stage is real work

Every prompt draft doesn't deserve production-model spend. Most of my edits are exploratory and half get thrown away. So the pipeline is shaped like a funnel:

1. Screen on a free model. Runs locally while iterating and on every push. Its job is to reject, not to approve. If an edit can't keep the contract on a competent free model, it's not ready.
2. Confirm on the production model. Runs only on a green screen, before merge. This is the verdict.

The asymmetry is the point. A failure at stage one is strong evidence; a pass is weak evidence. Once you accept that, a free model stops being a toy benchmark and becomes a genuinely useful filter that saves real money.

For the screening stage I've been running against the free model access in MonkeyCode, using its free server option to host the runner so the screen behaves identically on my laptop and in CI without any spend. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing in the suite is coupled to it — the backend is the single injected callable you saw above, so substituting another provider is a small wrapper change. If you want to try this workflow without touching your paid quota, that free tier is a low-friction way to stand up stage one and see whether the gate earns a permanent spot in your pipeline.

Keep one eye open, though: stage one lies in both directions sometimes. A weaker model can fail an edit that's actually fine (false alarm), or muscle through an ambiguity your production model will trip on (false confidence). Log every scenario where the two stages disagree. After a few weeks you'll know which of your assertions are model-sensitive, and that log is worth more than the suite itself.

Where scenarios come from

Ranked by signal:

1. Every regression you've ever shipped. The incident that motivated the suite becomes scenario zero. Most teams write the suite and forget to add the original failure — don't.
2. Everything your downstream code parses. Each field, enum, or format your application extracts gets one happy-path and one adversarial scenario. If the router reads priority, there's a scenario where the obvious answer is each priority level.
3. Hostile minimal inputs. Empty messages, non-English text, two-thousand-word rants. Prompt edits routinely repair the median case by wrecking the tail.

Fifteen scenarios per prompt has covered us well so far. I'd rather have twenty crude assertions than five elegant ones — elegance in assertions tends to mean "breaks for reasons unrelated to the contract."

Honest limitations

• Contracts, not craft. A green suite means your parseable behaviors survived. It says nothing about whether the prose got worse, the tone drifted, or the answers became less useful. Humans still own that review.
• Writing the extractor is the real cost. If you can't write down what your system depends on in the model's output, the suite isn't your first problem — your architecture is.
• It can cry wolf. Weak assertions plus model nondeterminism equals flaky red builds, and flaky red builds teach teams to click "re-run" instead of investigate. Be ruthless about deleting scenarios that fail for reasons you wouldn't block a merge over.
• Skip it entirely if one person maintains the prompt, only humans read the output, and a regression costs a shrug. The suite pays off when machines consume the output or several hands edit the same prompt. A solo internal tool doesn't need a gate; it needs a careful owner.

Wrapping up

The triage incident cost us half a day of misrouted tickets and one awkward customer call. The suite cost an afternoon to build and now runs in seconds on infrastructure that costs nothing. If your prompts have versions, consumers, and the ability to break things, they deserve the same pre-merge skepticism as the code around them — and the "too expensive to run on every edit" objection mostly disappears once a free model does the rejecting.

If you build something like this, I'd be curious in the comments what your extract_contract equivalent pulls out of responses — the extraction layer turned out to be the most opinionated part of the whole thing.