Treat prompts like code: versioning, testing and review

Ahmed Vaseer — Mon, 26 Oct 2026 06:08:40 +0000

Somebody changes one sentence in a prompt on a Friday afternoon. On Monday, half the support summaries are in the wrong format and nobody knows why.

This happens because many teams treat prompts as settings, not code. They live in a dashboard, a database row or a string pasted into a function. They change without review, and there's no record of what they said last week.

A prompt decides what your product does. That makes it code. The fix is not a new platform. It's three habits you already use for everything else: keep it in the repo, test it, and review changes.

1. Keep prompts in the repo, as files

Put each prompt in its own file, next to the code that uses it. Plain text or Markdown is fine. Give it a version in the file itself so logs can say which version produced an answer.

app/
  prompts.py
prompts/
  summarise_ticket/
    prompt.md
    cases.jsonl
tests/
  test_summarise_ticket.py

prompt.md:

---
name: summarise_ticket
version: 3
---
You summarise customer support tickets for an internal dashboard.

Return JSON with exactly these keys:
- "summary": one sentence, under 30 words
- "category": one of "billing", "bug", "how-to", "other"
- "urgent": true or false

Ticket:
{ticket_text}

Load it with a small helper rather than copying strings around:

from pathlib import Path

PROMPTS = Path(__file__).resolve().parent.parent / "prompts"   # repo-root prompts/


def load_prompt(name: str) -> tuple[str, int]:
    text = (PROMPTS / name / "prompt.md").read_text(encoding="utf-8")
    _, header, body = text.split("---", 2)
    version = next(
        int(line.split(":", 1)[1])
        for line in header.splitlines()
        if line.startswith("version:")
    )
    return body.strip(), version


def render(name: str, **values: str) -> tuple[str, int]:
    body, version = load_prompt(name)
    return body.format(**values), version

This assumes the helper lives in app/prompts.py and the prompts/ folder sits at the repo root.

Now every change to the prompt is a commit. You can see who changed it, when, and why, and you can roll it back with one revert.

One caution: str.format treats curly braces as placeholders. If your prompt contains literal braces (for example a JSON example), double them ({{ and }}) or use a different templating approach.

2. Test against a small, fixed set of cases

You don't need a large evaluation suite to start. Twenty real examples, with what a good answer must contain, will catch most regressions.

cases.jsonl (one case per line):

{"id": "refund-late", "ticket_text": "I was charged twice this month and want a refund.", "expect": {"category": "billing"}}
{"id": "login-bug", "ticket_text": "The login button does nothing on Safari.", "expect": {"category": "bug"}}
{"id": "site-down", "ticket_text": "Your checkout is down and we are losing orders right now.", "expect": {"category": "bug", "urgent": true}}

Test what you can check reliably: valid JSON, the right keys, allowed values, and the fields that must match. Don't test exact wording; it will change from run to run.

import json
from pathlib import Path

import pytest

from app.llm import call_model          # your existing model wrapper
from app.prompts import render

CASES_FILE = Path(__file__).resolve().parent.parent / "prompts/summarise_ticket/cases.jsonl"  # adjust to your tests folder
CASES = [json.loads(line) for line in CASES_FILE.read_text(encoding="utf-8").splitlines() if line.strip()]
ALLOWED = {"billing", "bug", "how-to", "other"}


@pytest.mark.llm
@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_summarise_ticket(case):
    prompt, _version = render("summarise_ticket", ticket_text=case["ticket_text"])
    raw = call_model(prompt, temperature=0)
    data = json.loads(raw)

    assert set(data) == {"summary", "category", "urgent"}
    assert data["category"] in ALLOWED
    assert isinstance(data["urgent"], bool)
    assert len(data["summary"].split()) <= 30
    for key, value in case["expect"].items():
        assert data[key] == value, f"{key}: expected {value!r}, got {data[key]!r}"

A few practical notes:

These tests call a real model, so they cost money and are slower than unit tests. They're marked with @pytest.mark.llm (registered below) and skipped by default; run them with pytest -m llm when a prompt or model changes, not on every commit.
Setting temperature to 0 reduces variation but doesn't remove it. If a case fails now and then, run it three times and treat two out of three as a pass, or rewrite the case so the expected answer is clearer.
Add a case every time you find a bad answer in production. That's how the set grows into something valuable without a big upfront effort.
If your wrapper doesn't use the provider's JSON mode, strip code fences before parsing.

# pyproject.toml
[tool.pytest.ini_options]
markers = ["llm: calls a real model; run with -m llm"]
addopts = "-m 'not llm'"

3. Review prompt changes like code changes

Prompt changes go through the same pull request as any other change. Two rules make the review useful:

Bump the version in the file header whenever the wording changes.
Paste the test results in the pull request: which cases passed before and after.

A reviewer should be able to answer three questions in a couple of minutes:

What behaviour is this change meant to fix?
Which case proves it's fixed? (If none exists, add one.)
Did any other case get worse?

That's it. It sounds too simple, but most prompt incidents come from changes nobody looked at, not from changes that were reviewed and still went wrong.

4. Log the version with every answer

When something goes wrong in production, the first question is "which prompt produced this?" Log the prompt name and version with each call (not the full text, and not the user's data):

import logging

log = logging.getLogger("llm")

prompt, version = render("summarise_ticket", ticket_text=ticket.text)
raw = call_model(prompt, temperature=0)
log.info("llm_call", extra={"prompt": "summarise_ticket", "prompt_version": version, "ticket_id": ticket.id})

extra fields only show up if your formatter or JSON log handler includes them. With plain logging.basicConfig, put them in the message instead: log.info("llm_call prompt=%s version=%s ticket=%s", "summarise_ticket", version, ticket.id).

Now a bad answer leads you straight to a commit.

5. Change the model the same way

Switching to a newer or cheaper model is also a behaviour change. Keep the model name in config, run the same cases against the new model before switching, and record the result in the pull request. The cases you built for prompts protect you here too.

What this doesn't solve

Twenty test cases won't catch everything. They won't tell you whether summaries are good, only whether they're valid and correct on the points you check. For quality, a person still needs to read a sample of real answers from time to time. The aim of this setup is narrower: no prompt change reaches production without someone seeing it, testing it and being able to undo it.

A one-afternoon checklist

Move each prompt into its own file in the repo, with a version number.
Load prompts through one helper instead of copied strings.
Write ten to twenty cases from real inputs.
Add a marked test that checks structure and the fields that must match.
Require a version bump and test results in every prompt pull request.
Log prompt name and version with every model call.

Boring, and that's the point. The less exciting your prompt changes are, the fewer surprises your users get.

Ahmed Vaseer is Founder & CEO - Ingenious Concepts, an AI-first technology and digital growth agency. 80+ projects, ISO 27001 certified via Global CB.

Driving AI-Powered Digital Growth: Lessons from Building Tech Solutions at Ingenious Concepts

Ahmed Vaseer — Tue, 27 Jan 2026 21:28:27 +0000

Hello Dev.to community! I’m Ahmed Vaseer, CEO & Founder of Ingenious Concepts, a company dedicated to building software, web and mobile applications, games, and AI-driven digital marketing solutions. Over the years, I’ve learned that digital growth isn’t just about adopting new tools — it’s about integrating technology in ways that create real business impact.

In this article, I want to share some of the key lessons we’ve learned while helping businesses leverage AI and technology for growth.

1. AI is a Growth Engine, Not Just a Buzzword

Artificial intelligence can be intimidating for many businesses, but the truth is that it’s most effective when applied practically. At Ingenious Concepts, we’ve helped companies automate repetitive tasks, analyze customer behavior, and optimize marketing campaigns using AI tools. The result? More efficient operations and smarter decision-making.

Tip: Start small. Automate one process at a time and measure results before scaling.

2. Build Digital Products That Solve Real Problems

Many startups focus on flashy features but neglect whether their product actually solves a real problem. We always ask: “Does this feature help our client achieve a measurable goal?”

Whether it’s a SaaS platform, a mobile app, or a game, product design should always be purpose-driven. That mindset has helped our clients reduce costs, improve engagement, and increase revenue.

3. Integrate Marketing and Technology Early

Digital marketing and technology often work in silos. The companies that win are those that integrate these two from day one. For example, when developing an e-commerce platform, we build automation into the sales and marketing workflows so campaigns can run efficiently while data drives decisions.

Pro Tip: Analytics, automation, and AI-driven insights should be part of your product from the start — not an afterthought.

4. Teamwork and Leadership Matter More Than Tools

Even the best AI and software tools won’t succeed without a strong team. Leading cross-functional teams of developers, designers, and strategists requires clarity, vision, and an environment where experimentation is encouraged.

At Ingenious Concepts, we focus on building multidisciplinary teams that can execute quickly without sacrificing quality — a critical factor in today’s fast-paced digital landscape.

5. Focus on Sustainable, Scalable Growth

Technology moves fast, but sustainable growth is a marathon, not a sprint. Our philosophy is to build systems and processes that scale, rather than chasing short-term trends. This approach allows businesses to evolve without constant overhauls, saving time and money while maximizing long-term impact.

Final Thoughts

Digital transformation is no longer optional — it’s a necessity. But it’s not enough to adopt AI or new software for the sake of it. Real impact comes from integrating these tools thoughtfully, with a focus on measurable growth, user experience, and scalability.

At Ingenious Concepts, we continue to explore innovative ways to merge technology, marketing, and strategy to empower businesses worldwide.

If you’re curious about AI, digital growth, or building scalable tech products, feel free to connect or leave a comment below — I’d love to share more insights from our journey.

About the Author:
Ahmed Vaseer is CEO & Founder of Ingenious Concepts, a tech company specializing in software, web and mobile apps, game development, and AI-powered digital marketing solutions. He helps businesses leverage technology for growth, efficiency, and global expansion.

DEV Community: Ahmed Vaseer