Treat prompts like code: versioning, testing and review

Somebody changes one sentence in a prompt on a Friday afternoon. On Monday, half the support summaries are in the wrong format and nobody knows why.

This happens because many teams treat prompts as settings, not code. They live in a dashboard, a database row or a string pasted into a function. They change without review, and there's no record of what they said last week.

A prompt decides what your product does. That makes it code. The fix is not a new platform. It's three habits you already use for everything else: keep it in the repo, test it, and review changes.

1. Keep prompts in the repo, as files

Put each prompt in its own file, next to the code that uses it. Plain text or Markdown is fine. Give it a version in the file itself so logs can say which version produced an answer.

app/
  prompts.py
prompts/
  summarise_ticket/
    prompt.md
    cases.jsonl
tests/
  test_summarise_ticket.py

prompt.md:

---
name: summarise_ticket
version: 3
---
You summarise customer support tickets for an internal dashboard.

Return JSON with exactly these keys:
- "summary": one sentence, under 30 words
- "category": one of "billing", "bug", "how-to", "other"
- "urgent": true or false

Ticket:
{ticket_text}

Load it with a small helper rather than copying strings around:

from pathlib import Path

PROMPTS = Path(__file__).resolve().parent.parent / "prompts"   # repo-root prompts/


def load_prompt(name: str) -> tuple[str, int]:
    text = (PROMPTS / name / "prompt.md").read_text(encoding="utf-8")
    _, header, body = text.split("---", 2)
    version = next(
        int(line.split(":", 1)[1])
        for line in header.splitlines()
        if line.startswith("version:")
    )
    return body.strip(), version


def render(name: str, **values: str) -> tuple[str, int]:
    body, version = load_prompt(name)
    return body.format(**values), version

This assumes the helper lives in app/prompts.py and the prompts/ folder sits at the repo root.

Now every change to the prompt is a commit. You can see who changed it, when, and why, and you can roll it back with one revert.

One caution: str.format treats curly braces as placeholders. If your prompt contains literal braces (for example a JSON example), double them ({{ and }}) or use a different templating approach.

2. Test against a small, fixed set of cases

You don't need a large evaluation suite to start. Twenty real examples, with what a good answer must contain, will catch most regressions.

cases.jsonl (one case per line):

{"id": "refund-late", "ticket_text": "I was charged twice this month and want a refund.", "expect": {"category": "billing"}}
{"id": "login-bug", "ticket_text": "The login button does nothing on Safari.", "expect": {"category": "bug"}}
{"id": "site-down", "ticket_text": "Your checkout is down and we are losing orders right now.", "expect": {"category": "bug", "urgent": true}}

Test what you can check reliably: valid JSON, the right keys, allowed values, and the fields that must match. Don't test exact wording; it will change from run to run.

import json
from pathlib import Path

import pytest

from app.llm import call_model          # your existing model wrapper
from app.prompts import render

CASES_FILE = Path(__file__).resolve().parent.parent / "prompts/summarise_ticket/cases.jsonl"  # adjust to your tests folder
CASES = [json.loads(line) for line in CASES_FILE.read_text(encoding="utf-8").splitlines() if line.strip()]
ALLOWED = {"billing", "bug", "how-to", "other"}


@pytest.mark.llm
@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_summarise_ticket(case):
    prompt, _version = render("summarise_ticket", ticket_text=case["ticket_text"])
    raw = call_model(prompt, temperature=0)
    data = json.loads(raw)

    assert set(data) == {"summary", "category", "urgent"}
    assert data["category"] in ALLOWED
    assert isinstance(data["urgent"], bool)
    assert len(data["summary"].split()) <= 30
    for key, value in case["expect"].items():
        assert data[key] == value, f"{key}: expected {value!r}, got {data[key]!r}"

A few practical notes:

• These tests call a real model, so they cost money and are slower than unit tests. They're marked with @pytest.mark.llm (registered below) and skipped by default; run them with pytest -m llm when a prompt or model changes, not on every commit.
• Setting temperature to 0 reduces variation but doesn't remove it. If a case fails now and then, run it three times and treat two out of three as a pass, or rewrite the case so the expected answer is clearer.
• Add a case every time you find a bad answer in production. That's how the set grows into something valuable without a big upfront effort.
• If your wrapper doesn't use the provider's JSON mode, strip code fences before parsing.

Register the marker so pytest knows it, and skip these tests by default:

# pyproject.toml
[tool.pytest.ini_options]
markers = ["llm: calls a real model; run with -m llm"]
addopts = "-m 'not llm'"

3. Review prompt changes like code changes

Prompt changes go through the same pull request as any other change. Two rules make the review useful:

1. Bump the version in the file header whenever the wording changes.
2. Paste the test results in the pull request: which cases passed before and after.

A reviewer should be able to answer three questions in a couple of minutes:

• What behaviour is this change meant to fix?
• Which case proves it's fixed? (If none exists, add one.)
• Did any other case get worse?

That's it. It sounds too simple, but most prompt incidents come from changes nobody looked at, not from changes that were reviewed and still went wrong.

4. Log the version with every answer

When something goes wrong in production, the first question is "which prompt produced this?" Log the prompt name and version with each call (not the full text, and not the user's data):

import logging

log = logging.getLogger("llm")

prompt, version = render("summarise_ticket", ticket_text=ticket.text)
raw = call_model(prompt, temperature=0)
log.info("llm_call", extra={"prompt": "summarise_ticket", "prompt_version": version, "ticket_id": ticket.id})

extra fields only show up if your formatter or JSON log handler includes them. With plain logging.basicConfig, put them in the message instead: log.info("llm_call prompt=%s version=%s ticket=%s", "summarise_ticket", version, ticket.id).

Now a bad answer leads you straight to a commit.

5. Change the model the same way

Switching to a newer or cheaper model is also a behaviour change. Keep the model name in config, run the same cases against the new model before switching, and record the result in the pull request. The cases you built for prompts protect you here too.

What this doesn't solve

Twenty test cases won't catch everything. They won't tell you whether summaries are good, only whether they're valid and correct on the points you check. For quality, a person still needs to read a sample of real answers from time to time. The aim of this setup is narrower: no prompt change reaches production without someone seeing it, testing it and being able to undo it.

A one-afternoon checklist

1. Move each prompt into its own file in the repo, with a version number.
2. Load prompts through one helper instead of copied strings.
3. Write ten to twenty cases from real inputs.
4. Add a marked test that checks structure and the fields that must match.
5. Require a version bump and test results in every prompt pull request.
6. Log prompt name and version with every model call.

Boring, and that's the point. The less exciting your prompt changes are, the fewer surprises your users get.

---

Ahmed Vaseer is Founder & CEO - Ingenious Concepts, an AI-first technology and digital growth agency. 80+ projects, ISO 27001 certified via Global CB.

Comments

[AIFix]

Putting a version number in the prompt file so logs can say which version produced an answer is a neat touch. Do you keep a small set of saved test cases per prompt, to catch format changes like the Friday-afternoon example?

[arcwrightai]

The undo story is what makes me trust a tool enough to use it daily.