The usual argument is that a small canary limits the blast radius and a large one gets you an answer faster, so you trade one against the other. Work the arithmetic and that trade mostly disappears. The number of users harmed before you can call a subtle regression is set by the sample size the test needs, not by the slice you sent it to. The percentage is doing something else entirely.
What the percentage actually controls
A canary splits traffic between the current prompt and the new one. The percentage sets three things at once, and they are usually conflated: the rate at which evidence accumulates, the number of users exposed at any instant, and the total exposure before you make a decision. Only the first two follow from the percentage in the obvious way. The third does not, and it is the one people think they are controlling.
Hold on to the distinction between two kinds of regression, because almost everything below turns on it. A subtle one moves a rate — refusals go from 2% to 4%, schema-validation failures from 0.5% to 1.5%, tone drifts on a minority of inputs. You need a statistical test to see it. A catastrophic one breaks everything it touches: a template variable renders as the literal string undefined, a system prompt exceeds the context window and the provider returns 400 on every call, a JSON instruction is deleted and every downstream parse fails. You need one request to see it, and then a decision procedure fast enough to act.
How much traffic the test needs
Take a binary outcome — a request either fails validation or it does not. Suppose the baseline failure rate is 2% and you want to catch a doubling to 4%, testing two-sided at the conventional 5% significance level with 80% power. Those four numbers are assumptions, and each one moves the answer, so state them next to the result rather than in a footnote. The standard two-proportion sample size is:
n_per_arm = (z_a/2 + z_b)^2 * [p1(1-p1) + p2(1-p2)] / (p2 - p1)^2
z_0.025 = 1.96 z_0.20 = 0.8416
(1.96 + 0.8416)^2 = 7.849
p1 = 0.02 -> 0.02 * 0.98 = 0.0196
p2 = 0.04 -> 0.04 * 0.96 = 0.0384
------
0.0580
(0.04 - 0.02)^2 = 0.0004
n = 7.849 * 0.0580 / 0.0004 = 1138.1 -> 1,139 per arm
That is the balanced case, a 50/50 split, and it needs about 2,278 requests in total. A canary is not balanced, so the variance of the difference has to be split according to the allocation. Writing f for the canary’s share of traffic and N for the total:
N = 7.849 * [ p1(1-p1)/(1-f) + p2(1-p2)/f ] / 0.0004
f = 0.50 : 7.849 * [0.0392 + 0.0768] / 0.0004 = 2,277 total, 1,139 on canary
f = 0.05 : 7.849 * [0.0206 + 0.7680] / 0.0004 = 15,477 total, 774 on canary
f = 0.01 : 7.849 * [0.0198 + 3.8400] / 0.0004 = 75,749 total, 757 on canary
limit as f -> 0 : 7.849 * 0.0384 / 0.0004 = 754 on canary
Read the third column, not the second. A 1% canary needs fewer requests through the new prompt than a 50/50 split does — 757 rather than 1,139 — because the control arm is so large that it contributes almost no uncertainty, and the test converges to a one-sample comparison against a known baseline. The small canary is not statistically starved. It is time-starved: 757 requests at 1% of a service doing 10,000 requests a day is a week, and at 1% of a service doing 500,000 a day it is under a quarter of an hour.
This assumes the baseline arm is running concurrently. Comparing the canary against last week’s numbers instead is cheaper and wrong: it confounds the prompt change with every diurnal, seasonal and upstream-model shift in between. Providers do change served weights without a version bump, which is the whole subject of silent model updates.
Blast radius is nearly constant in the percentage
Now count the harm. The excess failures caused by the bad prompt before the test can call it is the canary arm’s sample size multiplied by the size of the regression:
excess failures = n_canary * (p2 - p1)
f = 0.50 : 1,139 * 0.02 = 22.8
f = 0.05 : 774 * 0.02 = 15.5
f = 0.01 : 757 * 0.02 = 15.1
Fifteen extra failed requests against twenty-three. Dropping the canary from half of traffic to a hundredth of it cuts the harm by a third, and multiplies the wall-clock time to a decision by fifty. That is not the trade anybody thinks they are making, and it falls straight out of the fact that both the harm and the sample size are counted in canary requests. The percentage changes the rate at which those requests arrive; it barely changes how many of them you need.
The one thing a smaller percentage genuinely buys in this regime is reversibility of a different kind: fewer distinct users touched. If your traffic is sticky per user — assignment hashed on account id, so a user in the canary stays in it — then 1% concentrates 757 bad requests onto a few dozen accounts and 50% spreads them across thousands. Which of those is worse depends entirely on whether your failure mode is annoying-once or account-destroying. For a prompt change it is nearly always annoying-once, which is why the argument for a very small canary has to come from somewhere else.
Two regimes, two different answers
It comes from the catastrophic case, where the arithmetic inverts completely. If the new prompt fails on every request, you do not need 754 samples; you need enough to be sure it is not noise, which is a handful. The harm is then set by exposure over your detection window:
harm = request_rate * canary_fraction * detection_window
service at 200 req/s, 60 s from first failure to rollback:
f = 0.01 : 200 * 0.01 * 60 = 120 broken requests
f = 0.05 : 200 * 0.05 * 60 = 600
f = 0.20 : 200 * 0.20 * 60 = 2,400
Here the percentage is the entire story, and it is linear. This is the regime the canary percentage should be chosen for. The subtle regime is governed by how long you leave the canary running, which is a separate dial, and by how fast you can accumulate 754 requests through it.
Which means the two questions decouple cleanly. Choose the percentage so that a total failure costs an acceptable number of broken requests during your detection window. Choose the duration so that the subtle test reaches its sample size. If those two are incompatible — you want 1% but 1% takes nine days to reach 754 requests, and nobody watches a canary for nine days — you do not fix it by splitting the difference. You either raise the percentage once the catastrophic window has passed, or you take the test off the canary entirely and run it on sampled shadow traffic, which has no blast radius at all because no user sees the output.
Picking a number
- Start at 1% for a fixed, short window — long enough to catch the deterministic failures, which show up in the first few hundred requests or not at all. Ten minutes at any meaningful traffic rate is plenty. This slice is not trying to measure quality; it is trying to establish that the prompt renders, the model responds, and the output parses.
- Then step to 5–10% to accumulate the sample. Once the catastrophic failure modes are excluded, the blast-radius argument for staying at 1% is worth about seven extra failed requests, and the cost is days. Step up.
- Never below 1% unless you have the traffic for it. Below about 500 requests a day through the canary arm, the test will not conclude before somebody forgets it is running. A canary that expires unanalysed is worse than no canary, because it launders a change as validated.
- Hold the assignment stable per user, and salt the hash. Re-rolling assignment per request makes the statistics easier and the user experience incoherent when consecutive turns in one conversation come from two different prompts. If another experiment is running at the same time, the salt matters — two experiments sharing a hash collapse into one.
- Write the stopping rule before you start. The sample size above is a fixed-horizon number and it is only valid if you look once, at the end. If you intend to watch a dashboard, you need a sequential test instead, or your 5% false-positive rate is not 5%.
Top comments (0)