The suite ran 240 cases against the old model and blocked 228. It runs the same 240 against the new one and blocks 234. Nobody should report that as an improvement, and the reasons are worth being precise about, because two of them are fixable and one is not.
The old pass rate is not a baseline
A baseline is a measurement of a system under conditions you intend to hold constant. A prompt-injection suite measures the joint behaviour of four things: the attack corpus, the system prompt, the model, and the judge that decides whether an attack succeeded. A model migration changes one of them directly and at least two of them indirectly.
The system prompt changes indirectly because almost nobody ports a defence prompt unmodified — the delimiter convention gets updated, the refusal instruction gets rephrased for the new family’s documented preferences, and those edits are part of the same commit. The judge changes indirectly if it is itself a model call, which it usually is: an LLM judge scoring “did the model leak the system prompt” is a classifier with its own thresholds, and swapping the judge model shifts the score of every case in the suite without any attack behaving differently. If the judge and the system under test were migrated together, the two effects are not separable at all.
Your suite is a biased sample of attacks
This is the part that cannot be fixed by holding things constant. A red-team corpus is not a random sample from the space of attacks. Each case is in the file because somebody found it, and people find attacks by probing a specific model until something works. The corpus is therefore enriched for exactly the failure modes of the model it was built against, and depleted of the failure modes of every model it was not.
The consequence is asymmetric and it always points the same way: a suite built against model A will tend to report model B as safer, regardless of which is actually safer, because B was never the target during collection. A rising pass rate after a migration is the expected result of the bias, not evidence against it. The corpus has to be re-enriched against the new model before its aggregate number means anything, and re-enrichment is manual work that a migration schedule rarely budgets for.
There is a cheap partial correction. Split the corpus by what it did on the old model — the cases that succeeded against A, and the cases that were already blocked — and report the two rates separately. Movement in the first group is the interesting number: it says whether the specific weaknesses you knew about are still there. Movement in the second group is mostly regression detection. Collapsing them into one percentage throws away the only structure the corpus has.
How much movement is noise
Before interpreting a delta, bound it. Treat each case as an independent Bernoulli trial with a block probability p — an assumption that is generously optimistic, since real suites contain families of near-duplicate attacks that are strongly correlated, and correlation only widens the interval. Under that assumption the standard error of the observed rate is sqrt(p(1-p)/n).
Assumption: n = 240 independent cases, observed block rate p = 0.95
standard error = sqrt(0.95 x 0.05 / 240)
= sqrt(0.0475 / 240)
= sqrt(0.000198)
= 0.0141 (1.41 percentage points)
95% interval = 0.95 +/- 1.96 x 0.0141
= 0.95 +/- 0.0276
= [0.922, 0.978] (92.2% to 97.8%)
Two independent runs, difference of two rates:
se(difference) = sqrt(0.0141^2 + 0.0141^2) = 0.0199
95% band = +/- 1.96 x 0.0199 = +/- 0.039 (3.9 points)
So on a 240-case suite sitting near 95%, a move of under about four percentage points is inside the band you would expect from resampling alone — and 228 to 234 out of 240 is 2.5 points. It is not a result. To resolve a two-point difference at that base rate you need roughly an order of magnitude more cases, which is usually not the right investment; the better move is to stop reporting an aggregate rate and report per-category rates with counts attached.
Every figure above is derived on this page from the two labelled assumptions, not measured. Substitute your own n and observed rate. The normal approximation is poor when the expected count of failures is small — at 240 cases and 95% blocking that is 12 expected failures, which is near the edge, so treat the band as indicative.
The arm most suites are missing
A suite that only contains attacks has a degenerate optimum: a model that refuses every request scores 100%. That is not a theoretical concern during a migration, because refusal boundaries genuinely move between families, and a model that is more cautious than its predecessor will produce a better injection score and a worse product at the same time.
The fix is a second arm of benign cases that superficially resemble attacks: a support ticket quoting a customer’s angry email that contains the words “ignore previous instructions”, a document-summarisation request over a file that legitimately contains a prompt-shaped fragment, a user asking the assistant to explain how injection works. Score that arm as a false-positive rate and refuse to report the block rate without it. A migration is the moment this matters most, because the two rates usually move together and only the pair is interpretable. The guardrail false-positive rate page covers building that arm.
Re-establishing the baseline
- Freeze the corpus and the judge before touching the model. Pin the judge to a specific model version and keep it pinned across the whole migration, even if you intend to migrate it later. One variable at a time.
- Re-run the frozen suite against the old model to get a fresh same-day figure. The number in your dashboard was produced under a different system prompt and possibly a different judge; do not compare against history.
- Run against the new model with the system prompt unchanged. This is the isolated model effect and it is the only clean comparison you will get.
- Run again with the migrated system prompt. The difference between steps three and four is your prompt edit, attributed separately.
- Report both arms, split by whether the case succeeded on the old model, with counts and the interval from the arithmetic above printed next to every rate.
- Budget re-enrichment: a fixed number of hours probing the new model for attacks the corpus does not contain, added to the corpus and tagged with the model they were found against. Without this step the suite decays into a regression test and stops being a red team.
The output of that procedure is not a verdict on which model is safer. It is a defensible statement of the form “of the 31 attacks that succeeded against the previous model, 24 still succeed; the benign arm moved from 4 false positives to 11”. That is a sentence somebody can act on, and it is the shape jailbreak defence testing should produce on both sides of a swap.
Top comments (0)