Version Your AI Feature: Prompts and Evals Before Model Config

You shipped the AI feature. The model name is pinned in a Slack thread. The prompt that actually drives behavior lives in a random PR — or worse, only in the running container.

Then quality drops.

Support tickets get worse summaries. The agent starts inventing policies. Latency looks fine. Cost looks fine. Nobody can answer the only useful question: what changed?

If you cannot diff the prompt, you cannot debug the drop.

The real source of truth is not the model string

Model swaps are noisy. Someone posts "we moved to X," configs update, dashboards spike, people notice.

Prompt edits are silent. A one-line tweak to the system message can rewrite product behavior with no deploy note, no tagged release, and no rollback button. Eval datasets are the same story: if the golden examples live in a spreadsheet on someone's laptop, you have no regression signal when the prompt drifts.

Treat the prompt as code. Treat the eval set as the test suite. Treat model and provider config as infrastructure-as-code that comes after those two.

AWS Prescriptive Guidance puts it plainly: prompts are critical software artifacts that need versioning, a store or registry, PR-style review, and environment promotion — not ad-hoc text files. See Managing the generative AI prompt lifecycle.

A practical workflow that fits a normal repo

You do not need a platform rewrite. You need three artifacts next to the feature.

1. Prompts in git (or a registry with git history)

Keep the prompt text in the repo so every change is a diff and every release is a tag. If you outgrow flat files, use a prompt registry that still versions commits — Langfuse documents organizing, versioning, and testing prompts at scale in Prompt management at scale.

2. Golden evals versioned beside the prompts

A small JSONL (or YAML) set of inputs plus expected properties is enough to start. Run it on every prompt PR. Grow it from production failures. Without this, "looks better" is your only test.

3. Model + provider config as code

Pin model IDs, temperature, timeouts, and fallbacks in a config file the app loads. Easy to log. Easy to roll back. Still secondary to prompts and evals — because swapping the model without a fixed prompt and eval suite just moves the noise around.

4. PR review for prompt changes

Same bar as application code: reviewer, CI check, merge. Langfuse's Prompt CI/CD write-up frames the loop as version → gate → rollout. That is the right mental model even if your "registry" is just a folder.

Folder sketch

Something this simple already beats Slack archaeology:

ai/
  prompts/
    support_summarizer.v3.md
  evals/
    support_summarizer.jsonl
  config/
    support_summarizer.yaml

support_summarizer.v3.md — the system/user template, with a short changelog header.

support_summarizer.jsonl — one object per line: input ticket, expected fields, maybe a rubric label.

support_summarizer.yaml — model id, provider, temperature, max tokens, fallbacks.

On a prompt PR, CI loads v3 + the JSONL, scores the run, and fails the build if recall/precision/rubric drops past a threshold. Reviewers read the prompt diff the way they read a service change — because it is a service change.

Example eval line:

{"id":"t-1042","input":"Customer wants refund after 45 days; policy is 30.","expect":{"tone":"firm_empathetic","must_cite_policy":true,"must_not_invent_exception":true}}

Example config:

feature: support_summarizer
prompt: prompts/support_summarizer.v3.md
eval: evals/support_summarizer.jsonl
model:
  primary: openai/gpt-4.1-mini
  fallback: anthropic/claude-haiku-4-5
temperature: 0.2

Bump the prompt version when behavior changes. Keep the eval set growing from real misses. Log the model string every call so ops can still correlate provider incidents — without pretending the model string is the product.

What usually goes wrong

Prompt only in application code strings. Hard to review. Hard to A/B. Easy to "fix" in a hotfix and forget.

Evals that never ship. A notebook on one laptop is not a gate. Put the file in the repo and run it in CI.

Versioning the model first. Useful, but incomplete. You can pin gpt-4.1-mini forever and still ship a silent prompt regression on Tuesday.

No ownership. If anyone can edit production prompts without a PR, you will eventually ship a Monday morning mystery.

Start this week

Pick one AI feature. Move its prompt into prompts/. Add 20–50 golden examples from the last month of complaints. Wire a cheap CI job that fails on a clear regression. Require a human on the prompt PR.

That is not bureaucracy. That is how you get a diff when quality drops.

Poll — comment below

What's the first artifact you put under version control for an AI feature?

• A) Prompts
• B) Eval datasets / golden examples
• C) Model + provider config
• D) We don't version any of it yet

My lean: A and B together. Model IDs are easy to log later. If you cannot diff the prompt, you cannot debug the drop.

Which letter matches your team today — and what broke the last time quality slipped?