Versioning Your System Prompt: The AI Config Change Nobody Logs

#llm #ai #node #typescript

Book: AI That Answers
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Someone edits a sentence in the system prompt on a Tuesday afternoon. The diff
is one line. It reviews in nine seconds. It ships.

Every AI answer your product gives from that moment behaves differently. There
is no migration, no feature flag, no gradual rollout, and — in most codebases
— nothing in any log that says which version of that string produced any given
answer.

A prompt is the most consequential config in the system and it is usually the
only config with none of the discipline.

The version that changes nothing

export const SYSTEM = `
You are a support assistant for Acme.
Be concise. Never promise refunds.
`.trim();

Then a week later:

-Be concise. Never promise refunds.
+Be concise and friendly. Never promise refunds without checking the policy.

Support quality shifts. Someone notices in a fortnight. The question "when did
this change?" is answerable from git, but "did this specific bad answer come
before or after?" is not, because the stored answer has no link to the string
that produced it.

Make it an object with an identity

export type Prompt = {
  readonly id: string;
  readonly version: number;
  readonly text: string;
};

function prompt(id: string, version: number, text: string): Prompt {
  return { id, version, text: text.trim() } as const;
}

export const SUPPORT = prompt("support", 7, `
You are a support assistant for Acme.
Be concise and friendly. Never promise refunds without checking the policy.
`);

Then every call records which one it used:

export async function ask(p: Prompt, messages: MessageParam[]) {
  const res = await client.messages.create({
    model: MODEL, max_tokens: 1024, system: p.text, messages,
  });
  logger.info("model_call", {
    promptId: p.id,
    promptVersion: p.version,
    promptHash: hash(p.text),
    model: MODEL,
    costUsd: costOf(MODEL, res.usage),
  });
  return res;
}

Both the version and a hash. The version is what a human reasons about; the
hash is what catches the edit where someone changed the text and forgot to
bump the number, which is the common case, because bumping is a discipline and
editing is a reflex.

const hash = (s: string) =>
  createHash("sha256").update(s).digest("hex").slice(0, 8);

Make forgetting impossible

Better than remembering: fail the build.

// prompts.test.ts
import { SUPPORT, TRIAGE, SUMMARISE } from "./prompts";

const LOCKED: Record<string, Record<number, string>> = {
  support:   { 7: "a3f19c22" },
  triage:    { 3: "77b0e415" },
  summarise: { 12: "e0c4a8d1" },
};

it.each([SUPPORT, TRIAGE, SUMMARISE])("$id v$version is unchanged", (p) => {
  const known = LOCKED[p.id]?.[p.version];
  expect(known,
    `${p.id} v${p.version} is not locked — add ${hash(p.text)} to LOCKED`,
  ).toBeDefined();
  expect(hash(p.text),
    `${p.id} text changed without a version bump`,
  ).toBe(known);
});

Now editing the text without bumping the version is a red test with a message
that says exactly what to do. The lock table becomes a readable history of
every prompt version you have ever shipped, in one file, reviewable.

This is the single highest-value thing in this post. It takes twenty minutes
and it converts an invisible change into one that cannot merge silently.

Store the version with the output

Logs roll off. If you persist AI output, and most products do, in a
conversation history — persist what produced it:

await db.message.create({
  data: {
    conversationId,
    role: "assistant",
    content: text,
    promptId: SUPPORT.id,
    promptVersion: SUPPORT.version,
    model: MODEL,
    createdAt: new Date(),
  },
});

Four extra columns. They answer the question that otherwise cannot be answered
at all: a user complains about an answer from three weeks ago, and you can say
which prompt version and which model wrote it.

Without them, an investigation into a past answer starts by guessing from
timestamps against the git log, and gets the wrong answer whenever a deploy
lagged a merge.

Roll it out like config, because it is

A prompt change is a behaviour change for 100% of users the moment it deploys.
Nothing else in your stack ships that way.

export function promptFor(user: User): Prompt {
  return flags.enabled("support-prompt-v8", user.id) ? SUPPORT_V8 : SUPPORT_V7;
}

Both versions exported, both locked, both logged. Now the comparison is real:

SELECT prompt_version,
       count(*)                          AS answers,
       avg(user_rating)                  AS rating,
       avg(cost_usd)                     AS cost,
       sum(escalated::int)::float/count(*) AS escalation_rate
FROM ai_messages
WHERE created_at > now() - interval '7 days'
GROUP BY prompt_version;

That query is the entire argument for this post. It is impossible without the
version column and trivial with it.

Note cost in there. Prompt changes move cost as often as they move quality —
a longer system prompt is billed on every request, and an instruction that
makes answers more thorough increases output tokens.

Keep the whole thing out of a database

Prompts belong in the repo, not in a config service where they can be edited
live. They are code: they need review, they need to roll back with a deploy,
and they need to be diffable next to the code that depends on their output
shape.

A prompt in a database is a production change with no review, no test run, and
no correlation to the deploy that "caused" the regression. It sounds
convenient until the first incident.

The exception worth allowing is a kill switch — a flag that reverts to the
previous version without a deploy. That is an operational control, not prompt
authoring.

Tool descriptions are prompts too

Everything above applies to your tool definitions, and almost nobody applies
it there.

export const TOOLS = {
  id: "support-tools",
  version: 4,
  defs: [ /* ... */ ],
} as const;

A model chooses tools from their names and descriptions. Editing a description
changes tool selection — the same class of change as editing the system
prompt, with the same invisibility. Version and log it the same way, and
include it in the lock test.

The minimum

If you do one thing: add promptVersion to your model-call log line and to
your stored output. That is two fields, it takes ten minutes, and it turns
"quality dropped at some point" into a query.

The lock test is the second thing, and it is what keeps the first one honest.