Feature-Flagging a Prompt Change So Rollback Is Instant

Feature-flagging a prompt separates two things that are usually welded together: the release that put the new text on the servers, and the decision to use it. Once they are separate, turning the change off costs an evaluation-rule write, and nothing has to be rebuilt.

The flag is not an if statement

The naive version is a boolean read from an environment variable, and it fails for a specific reason: an environment variable is part of the deployment, so changing it restarts processes. You have moved the rollback from “build and deploy” to “restart everything”, which is faster but still couples the switch to the infrastructure and still drops in-flight work.

What you want is an evaluation performed per request against a rule set that can change without touching the process. That is what a feature flag system provides, and the three properties that matter for prompts are:

• String variants, not booleans. A boolean gives you two states and no room. A string flag resolving to a prompt version identifier lets you move between three versions, or roll forward to a fix, without a code change.
• Targeting. Percentage rollouts, allow-lists for internal users, and exclusion of specific accounts. The targeting key needs care — see below.
• Local evaluation. The rules are held in the process and refreshed in the background, so a flag read is not a network call and the flag service being slow does not become your latency.

OpenFeature is a vendor-neutral way to express this, so the page can show real calls without pinning you to one product. Its JavaScript server SDK is published as @openfeature/server-sdk and documents OpenFeature.setProviderAndWait(), OpenFeature.getClient() and per-type evaluation methods such as getStringValue(key, defaultValue, context), with a details variant returning value, variant and reason.

Wiring the prompt selection

// llm/prompt-selection.ts
import { OpenFeature } from "@openfeature/server-sdk";
import { getPromptVersion } from "./prompt-store";

const client = OpenFeature.getClient();

export type Selected = {
  version: string;
  body: string;
  variant?: string;
  reason?: string;
};

export async function selectPrompt(
  promptId: string,
  ctx: { userId: string; accountId: string; locale: string },
): Promise<Selected> {
  const details = await client.getStringDetails(
    `prompt.${promptId}.version`,
    "v6",                       // default: the known-good version, not the new one
    {
      targetingKey: ctx.accountId,
      locale: ctx.locale,
      userId: ctx.userId,
    },
  );

  return {
    version: details.value,
    body: await getPromptVersion(promptId, details.value),
    variant: details.variant,
    reason: details.reason,
  };
}

Three decisions in that snippet are the whole design.

The default value is the old version. If the flag service is unreachable, if the flag was deleted, if the SDK failed to initialise — every failure path lands on the version that was already working. Defaulting to the new version means a flag outage silently deploys your change to everyone, which is the opposite of what the flag was for.

The targeting key is the account, not the request. A random key per request splits a single conversation across two prompt versions, which produces incoherent multi-turn behaviour and makes any comparison meaningless. Key on whatever unit your quality is judged in — usually an account or a conversation. Percentage rollouts are then stable: the same account stays on the same side across requests, which is also what makes a canary comparison valid.

The flag returns a version, not a prompt. Prompt text lives in your versioned store, and the flag only says which version to use. Putting the text itself in the flag payload seems convenient and costs you review, diffing and history all at once.

Recording the variant that served the request

A flag you cannot see in your telemetry is a flag you cannot reason about. Put the resolved variant on the same log line as everything else about the request, alongside the prompt hash:

{"event":"llm.completion","request_id":"req_01J9Z6",
 "prompt_id":"support.triage","flag_key":"prompt.support.triage.version",
 "flag_variant":"v7","flag_reason":"TARGETING_MATCH",
 "prompt_template_hash":"9f2c81ad0b41","model":"your-model",
 "finish_reason":"stop","latency_ms":1840}

The reason field is the one people drop, and it is the most diagnostic. It distinguishes “this account matched a targeting rule” from “this is the default because evaluation failed”. During a rollout, a rising count of default-reason resolutions means your flag provider is degrading and your rollout percentages are fiction — a condition that looks like nothing at all if you only log the resolved value.

With variant on the log line, every quality and cost metric can be grouped by it, which is what turns the flag from a switch into a measurement. That is the mechanism underneath prompt A/B testing, and it is worth having even when you are not running an experiment.

Kill-switch semantics

Rollback under a flag is setting the rule to serve the old version to everyone. Three details decide whether that is actually instant.

• Propagation delay is a number you should know. Locally-evaluated SDKs poll or stream rule updates on an interval. Whatever that interval is, it is your rollback time; find it in your provider’s configuration rather than assuming it is instant, and treat it the way you would a cache TTL.
• Prefer a separate kill switch to editing the rollout. A dedicated boolean checked before the version flag, defaulting to “not killed”, is one unambiguous action under stress. Editing a percentage rule means reading a rule editor at 2 a.m. and hoping you understood the precedence order.
• In-flight requests are unaffected. A request that already resolved its flag keeps the version it resolved. That is correct behaviour and it means your recovery is not complete until the longest in-flight request finishes — the in-flight middle state covers what to do about it.

The flag has a lifecycle, and it ends

Flags left in place forever stop being a rollback path and become complexity. Once a prompt version has been at 100% for long enough to cover your slowest feedback loop — which for quality complaints is usually weeks rather than days, since users report bad answers slowly — remove the flag and make the version the new default.

Two failure modes are worth guarding against explicitly. The first is the stale flag whose old branch no longer works: version 6 is still selectable by the flag, but the calling code has since changed the variables it substitutes, so flipping back produces an unrendered placeholder. Guard against this with a test that renders every version the flag can currently resolve to and asserts all of its variables are supplied. That test failing is the signal to retire the flag, not to fix the test.

The second is flag interaction. Two prompt flags rolling out at the same time on the same request path produce four combinations rather than two, and any comparison that ignores the other flag is confounded: an account in the treatment arm of one flag is not equally likely to be in the treatment arm of the other unless both use the same targeting key. That is the subject of interaction effects between concurrent prompt tests. At minimum, log every relevant variant on the request so the combination is recoverable afterwards, and prefer serialising prompt rollouts on a shared code path over running two at once.

Related

• Rolling Back a Bad Prompt Deploy in Under a Minute
• Choosing a Canary Percentage for a Prompt Change
• A/B Testing Prompts in Production