An automatic rollback is worth having only if it fires when something is wrong and stays quiet when nothing is. The obvious version of the trigger — error rate on the canary above some multiple of baseline — fails the second half badly on a small canary, and the failure rate is derivable before you ship it.
The trigger everyone writes first
It reads: every five minutes, compute the error rate on the canary arm; if it exceeds twice the baseline rate, flip the flag back. That is a sensible-sounding rule, it is two lines of PromQL, and on a 1% canary it will wake somebody up before lunch.
The reason is sample size. A five-minute window on a 1% canary of a service doing twenty requests a second contains 20 × 0.01 × 300 = 60 requests. At a true error rate of 2%, the expected count of errors in that window is 1.2, and the observed count is an integer. There is no such thing as a 2.4% observation on 60 requests: the possible values are 0%, 1.67%, 3.33%, 5% and so on. Half the achievable values are above the threshold.
How often it fires by chance
Make it more generous — a window with 100 canary requests, a true error rate of 2%, and a threshold of 5% observed, well above the 4% that a doubling would produce. The count of errors is binomial with mean 2, close enough to Poisson(2) for this:
P(X >= 5 | mean 2) = 1 - P(X <= 4)
e^-2 = 0.135335
sum k=0..4 of 2^k / k! = 1 + 2 + 2 + 1.3333 + 0.6667 = 7.0
P(X <= 4) = 0.135335 * 7.0 = 0.9473
P(X >= 5) = 0.0527
windows to a false alarm = 1 / 0.0527 = 19
19 windows * 5 min = 95 minutes
A false rollback every ninety-five minutes, on a perfectly healthy prompt. Nobody keeps that trigger armed for a day. It gets disabled, and then the next real regression runs unattended, which is a strictly worse position than having written no trigger at all.
The mechanism is not subtle and it is worth naming because it recurs: a threshold on a ratio computed from a small count is a threshold on noise. The canary percentage that keeps blast radius small is exactly the percentage that makes the denominator too small to threshold.
Three fixes and what each costs
Gate on sample size
Do not evaluate the rule until the window contains a minimum number of canary requests. This converts the guarantee from “every five minutes” to “every N requests”, which is what you actually want. With 300 requests and a true rate of 2%, the mean count is 6 and P(X ≥ 15), the equivalent 5% observed threshold, is under one in a thousand. The cost is latency to detection when traffic is low — precisely when you least want to wait.
Require consecutive breaches
Fire only when two windows in a row breach. If breaches are independent, the false-alarm probability squares:
0.0527 ^ 2 = 0.00278
1 / 0.00278 = 360 windows
360 * 5 min = 30 hours
Thirty hours between false alarms rather than ninety-five minutes, for the cost of one extra window of exposure before the rollback fires. This is usually the best value of the three, and it is one field in a Prometheus alerting rule. The independence assumption is the weak point: if errors arrive in bursts because an upstream provider is degraded, consecutive windows are correlated and the improvement is smaller than the arithmetic suggests. It still helps.
Compare the arms, not the arm to a constant
The strongest version tests the canary against the concurrently running baseline rather than against a hard-coded rate, and requires the lower bound of the difference’s confidence interval to be above zero. This is immune to the entire class of incident where the provider degrades and both arms rise together — a hard threshold rolls back your prompt for someone else’s outage. The cost is that it needs the sample size derived in choosing a canary percentage, which on a small canary is not available in five minutes. In practice this tier runs on a longer window and fires a page rather than a rollback.
The instant tier
All of the above is about the subtle regime. The catastrophic regime needs the opposite treatment: no windows, no statistics, act on the first few observations. These signals justify it because their baseline rate is genuinely zero, so a single occurrence is information:
- The prompt failed to render. A missing template variable, a truncated file read, a null where a name should be. This is a bug in your code, not a model behaviour, and it will hit every request.
- HTTP 400 from the provider. A prompt over the context window, a malformed tool schema, an invalid parameter. A 400 attributable to the canary arm and not the baseline is deterministic; one is enough.
- Schema-validation failure above 50% over 20 requests. Not a rate shift — a broken instruction. Twenty requests is a second or two of traffic.
- Empty content on every request in the last ten. Usually a content filter or a stop sequence that now matches immediately.
Keep these two tiers as separate rules with separate thresholds and separate names. Merging them produces a single rule that is either too twitchy for the rate metrics or too slow for the deterministic ones.
Building it
The pieces are a counter, an alerting rule, a webhook and a flag store. Prometheus is used here because its alerting rule syntax is stable and documented by the Prometheus project; the same shape works in any system that can evaluate a query on a schedule and call a URL.
-
Emit one counter, labelled by arm and outcome. The arm label has to come from the same assignment function that routed the request, not from a second lookup, or a mid-request flag flip will attribute the response to the wrong arm.
llm_requests_total{variant="canary",prompt_version="v7",outcome="error"} llm_requests_total{variant="baseline",prompt_version="v6",outcome="ok"} -
Write the rate rule with a sample gate and a duration. The
forclause is the consecutive-breach fix: the condition must hold continuously for that long before the alert moves from pending to firing.
groups: - name: prompt-canary rules: - alert: CanaryErrorRateHigh expr: | sum(rate(llm_requests_total{variant="canary",outcome="error"}[5m])) / sum(rate(llm_requests_total{variant="canary"}[5m])) > 2 * sum(rate(llm_requests_total{variant="baseline",outcome="error"}[5m])) / sum(rate(llm_requests_total{variant="baseline"}[5m])) and sum(increase(llm_requests_total{variant="canary"}[5m])) > 300 for: 10m labels: severity: rollback annotations: summary: canary error rate above 2x baseline over 300+ requests -
Write the instant rule separately, with no
forclause and a much smaller gate, keyed on the deterministic outcomes rather than on the aggregate error rate.
- alert: CanaryHardFailure expr: | sum(increase(llm_requests_total{variant="canary",outcome="render_error"}[1m])) > 0 or sum(increase(llm_requests_total{variant="canary",outcome="http_400"}[1m])) > 2 labels: severity: rollback -
Point the receiver at a rollback endpoint that does exactly one thing: set the canary percentage to zero. It must not redeploy, must not run CI, and must be idempotent, because the alert will repeat while it resolves.
export async function POST(req: Request) { const body = await req.json(); const firing = body.alerts.filter((a) => a.status === "firing"); if (firing.length === 0) return new Response("no firing alerts", { status: 200 }); const version = firing[0].labels.prompt_version; // Idempotent: setting an already-disabled variant to 0 is a no-op. await flags.set("prompt.canary.percent", 0); await flags.set("prompt.canary.blocked_version", version); await audit.write({ action: "auto_rollback", version, alert: firing[0].labels.alertname }); return new Response("rolled back", { status: 200 }); } Record the blocked version, and check it on deploy. This is the step people skip and it is the one that causes the second incident. Without it the next merge re-enables the same prompt, because the rollback lived in a flag store and the repository still contains the bad version as head.
What happens after it fires
A rollback sets the split; it does not do anything about the requests currently in flight against the old assignment, and for a streaming endpoint those can be alive for a minute. What the reader should expect there is covered in in-flight requests during a prompt rollback, and the human procedure around it in a rollback runbook.
Two habits make the trigger trustworthy over time. Log every firing with the window’s raw numerator and denominator, not just the computed rate, so that a post-hoc review can tell a real regression from a small-sample artefact — this is the only way you will ever discover that your gate is set too low. And review the false-alarm count monthly against the derivation above: if the trigger has never fired in six months, the threshold is probably so loose that it would not catch the regression it was written for either.
Prometheus rule syntax, the for semantics and the receiver payload shape are all current-at-time-of-writing surfaces of one product. The arithmetic in the second section is not, and it is the part worth carrying to whatever you actually run.
Top comments (0)