A prompt that runs in production is code. It belongs in version control, it gets a version id that shows up in your logs, it passes a regression suite before it ships, and it can be rolled back in a minute without a deploy. I still walk into teams where the prompt lives in a database column that someone edits from an admin panel on a Friday evening, and then everyone spends the following week wondering why the output "feels different."
Why does a prompt edit break things weeks later?
The short answer is that prompt edits look harmless and their effects are statistical. A wording change does not break a build or throw an exception. It shifts the distribution of outputs by a few percent, and that shift only becomes visible weeks later, in a support complaint nobody traces back to the edit.
On one project we changed a single instruction from "summarize the ticket" to "briefly summarize the ticket." Reasonable edit. Shorter output, lower cost. What we didn't notice was that the JSON field holding action items started coming back empty in roughly one out of every twenty responses, because the model read "briefly" as permission to drop the list. Downstream, an empty list was valid, so nothing errored. We found out from a support lead ten days later, and it took a further afternoon to connect the complaint to the edit.
That afternoon is the real cost, and it's decided entirely by where the prompt lives.
| After a bad week | Prompt in an admin panel | Prompt in the repo |
|---|---|---|
| What changed? | Nobody can say | A diff |
| Who changed it, and when? | Maybe an updated_at column |
Commit author and timestamp |
| Was it reviewed? | No | A PR approval |
| Did it pass anything first? | No | The suite that ran in CI |
| Roll it back | Retype it from memory | Check out the previous version |
What does "prompt as code" look like in practice?
Prompt-as-code means the prompt text, the model id, and the sampling settings live in the repo as files, ship through a pull request, and carry a version id that appears in every log line. Rolling back is a checkout or a config flip, never a database edit.
Our setup is deliberately unexciting.
- Prompts are files in the repo, one folder per use case, with a template and explicit variables (
prompts/ticket_triage/system.md,prompts/ticket_triage/user.md). - The model id and the sampling settings sit next to the prompt in a small config file, so a "prompt version" means the whole calling contract, and a model swap is a visible diff.
- Every LLM call logs a content hash of the rendered template plus the config version. If a log line says
triage@v14, I can check out that exact text. - Prompt changes ship through the same pipeline as code: branch, PR, CI, deploy.
None of this needs a special tool. A prompt-management SaaS can be pleasant for non-engineers, but if it doesn't give you diffs, review, a version id in the logs, and a rollback, it's a fancier admin panel.
How do you regression-test a prompt?
Build a small suite of thirty to fifty real inputs with known-good outputs, and assert on structure rather than exact wording: the JSON parses, required fields exist, enums are valid, length stays inside bounds, forbidden phrases are absent. Run it in CI whenever a file under prompts/ changes.
For the fuzzy parts (is this summary faithful to the ticket) a judge model with a rubric works, as long as the judge's own prompt is versioned too. Fifty cases on a mid-tier model costs cents. Block the merge on failure. That is the entire mechanism that would have caught our "briefly" incident before it reached a customer.
Who gets to edit prompts?
Anyone with the domain knowledge, through the same gate as code. Product managers and domain experts should be able to change prompts, and they should do it through a pull request or a staged environment that runs the regression suite. The reviewer and the green suite stay mandatory either way.
This is where I see engineering leadership matter most right now: as more of the codebase gets written by tools, the job shifts toward governing what ships rather than typing it, and prompts are the most under-governed part of the AI systems I get shown. Versioning them is the first thing we put in place on generative AI builds we inherit mid-flight, and at Shanti Infosoft we treat a prompt PR exactly like a code PR: a reviewer, a green suite, an owner, and a rollback path, no matter who wrote the words.
Where do your production prompts live right now, and could you tell me what changed in them last month?
Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 team of 80+ engineers who put AI systems into production and then have to maintain them.
Top comments (0)