Prompt Versioning and Release Gates for LLM Apps

The fastest way I know to break a working LLM feature is to "just tweak the prompt" on a Friday. Nothing crashes. The output simply gets a little worse, and nobody notices until a customer does.

So I've stopped treating prompts as config strings. They're code. They get versions, tests, and a gate in CI that can say no.

Put prompts in files, not in strings

The first step is boring and does most of the work. Every prompt lives in its own file, with a version and a short changelog note.

# prompts/ticket_summary.yaml
id: ticket_summary
version: 7
model: your-model-id
changelog: "Ask for the customer's requested outcome as a separate line."
template: |
  Summarize this support ticket in 3 bullet points.
  Then add one line: "Requested outcome: ..."
  Ticket:
  {ticket_text}

Now a prompt change shows up in a pull request, with a diff, a reviewer, and a version bump. That alone kills most silent regressions, because someone has to look.

Keep a small, honest eval set

Next, collect real cases. Not synthetic ones, at least not at first. I start with 30 to 50 examples pulled from production logs, each with what a good answer must contain.

# evals/ticket_summary_cases.py
CASES = [
    {
        "ticket_text": "My invoice for March was charged twice. Please refund one.",
        "must_include": ["refund", "Requested outcome"],
        "must_not_include": ["I don't know"],
    },
    # ... more real cases
]

String checks are crude, and that's fine for a gate. They catch the big breaks: missing sections, wrong format, refusals. For subtler quality, add a scored review later. I wrote about scoring beyond pass/fail in this piece on AI agent evaluation, which covers rubrics for actions as well as answers.

The release gate

The gate runs every case against the new prompt version and fails the build if the pass rate drops below the current version.

# evals/run_gate.py
import sys
import yaml
from evals.ticket_summary_cases import CASES

def call_model(prompt: str) -> str:
    """Wrap your LLM provider's SDK here."""
    raise NotImplementedError

def check(output: str, case: dict) -> bool:
    has_all = all(s.lower() in output.lower() for s in case["must_include"])
    has_none = not any(s.lower() in output.lower() for s in case["must_not_include"])
    return has_all and has_none

def pass_rate(template: str) -> float:
    passed = 0
    for case in CASES:
        output = call_model(template.format(ticket_text=case["ticket_text"]))
        passed += check(output, case)
    return passed / len(CASES)

if __name__ == "__main__":
    spec = yaml.safe_load(open("prompts/ticket_summary.yaml"))
    baseline = float(open("evals/baseline_ticket_summary.txt").read())
    score = pass_rate(spec["template"])
    print(f"v{spec['version']}: {score:.0%} (baseline {baseline:.0%})")
    sys.exit(0 if score >= baseline else 1)

Run it in CI on any change under prompts/. If the new version scores lower than the stored baseline, the merge is blocked. When it scores higher, update the baseline in the same PR.

A note on cost: running 50 cases on every commit adds up. I only trigger the gate on changes to the prompt folder or the model setting, not on every push.

Model upgrades go through the same gate

This is the part people skip. A new model version is a release too. Run the same eval set before switching, and keep the old model ID in config so rollback is a one-line change.

I've seen "better" models fail format checks that the old one passed every time. The gate catches that in minutes instead of a week of confused tickets.

Version the whole bundle, not just the prompt

A prompt rarely works alone. It depends on the model ID, the temperature, the tool definitions, and sometimes a system message that lives somewhere else. If you version only the template, you can still ship a "same prompt, different behaviour" change without noticing.

What I do now is treat these as one bundle. The YAML file holds the template, the model setting, and a hash of the tool schemas the prompt expects. The gate fails if any of them changes without a version bump. It sounds strict, but it's the difference between "the prompt didn't change" and "nothing changed", and only the second one is useful at 2am.

Practise the rollback

Rolling back is a one-line change only if someone has actually done it. Once a month, I revert the prompt to the previous version on a staging environment and run the gate. It takes ten minutes, and it proves the old version still passes against today's cases.

What this doesn't solve

String checks won't tell you whether a summary is actually useful. For that, you need a second layer: a sampled human review, or a scoring rubric run by a judge model, checked against human labels now and then.

Start with the gate anyway. It's an afternoon of work, and it turns "I think the prompt is fine" into a number.

How do you test prompt changes today, and has a model upgrade ever surprised you?

I'm Darius Tran, and I work on putting AI agents into production at AI Hive.