A canary that stops firing looks exactly like a canary with nothing to report. After a model migration those two states are indistinguishable until you go and check, and the check is not difficult.
What a canary actually detects
The construction is simple and it is why it is popular. You place a high-entropy marker inside the system prompt — a random token, typically per-session — and you scan every model response for it. If it appears in output, some part of the system prompt has reached the user, which is strong evidence of a successful extraction attempt. Open frameworks such as Rebuff implement exactly this, and the pattern appears in most in-house injection tooling.
What matters for migration is the epistemics. A canary is not a defence. It does not prevent anything; it fires after the fact. And it is not a detector of injection in general — it detects the appearance of one specific string. Everything it tells you flows through a single string comparison, which means its usefulness is entirely a function of that comparison’s false-negative rate against the model you are actually running.
That rate is a property of the pairing, not of the canary. And a canary with a high false-negative rate does not degrade gracefully or announce itself. It goes quiet, and quiet is what success looks like.
The three ways a model family breaks it
A new model can defeat an exact-match canary for three separate reasons, and they need separate fixes.
- Paraphrase instead of quotation. Asked to reveal its instructions, a model may summarise them rather than reproducing them verbatim. “I have a session identifier and some formatting rules” leaks the substance of the system prompt and contains the marker nowhere. The extraction succeeded; the canary is silent. Models differ substantially in how literally they reproduce context, and this is the difference that moves most across families.
- Transformation of the marker. Where an attacker asks for the prompt with spaces between characters, in reverse, base64 encoded, or interspersed through a poem, the marker is present in the output but not as a contiguous substring. This is a documented limitation of the technique rather than a novel attack, and whether it works depends on the model’s willingness to perform the transformation — which is exactly the kind of thing that changes between versions.
- Tokenization and truncation of the marker itself. A marker chosen as a compact random string may tokenize very differently under a new family’s tokenizer, and a model reproducing it from context can drop or alter a character in a way that a strict comparison rejects. Where the marker is long, a response length cap can also cut it in half. Both produce a near-miss that a substring check calls a clean run.
The third of these is the one that is genuinely new information after a migration, and it is worth understanding as a tokenizer property rather than a model quirk — the same string is a different sequence of tokens to a different vocabulary, and reproduction fidelity is not uniform across them.
Re-validating the canary on the new model
You are measuring one number: given an extraction attempt that succeeds, how often does the canary fire. Everything else follows from it.
- Assemble a set of extraction prompts that are known to have worked against something — your own incident history is the best source, a public corpus of extraction attempts is the fallback. Fifty is enough to see the shape.
- Run them against the new model with the canary in place, in a non-production environment with the real system prompt. Capture the full response, not a truncated log line.
- Label every response by hand for whether the system prompt leaked, independently of what the canary said. This is the step that cannot be automated, and it is the whole experiment. A judge model can pre-filter but must not be the final label.
- Build the two-by-two: leaked or not, fired or not. The count in the leaked-but-silent cell divided by the total leaked count is your false-negative rate on this model.
- Inspect every leaked-but-silent case. Each one is either a paraphrase, a transformation or a near-miss, and each category maps to a different change in the next section.
- Record the rate with the model string and the date. It is a baseline you will need at the next migration, and a canary whose false-negative rate is unknown is not evidence of anything.
Hardening the matcher
The changes that make the comparison robust are cheap and mostly do not touch the prompt at all.
- Normalize before comparing. Strip whitespace, punctuation and case from both the response and the marker, then compare. This alone catches the spaced-out and interspersed transformations, which are the most common ones.
- Check for fragments, not only the whole marker. A marker split into overlapping n-grams, any two of which appearing together counts as a hit, converts a near-miss from a silent failure into a detection. Tune the threshold against a corpus of benign traffic so it does not fire on coincidence.
- Decode common encodings before matching. Base64 and hex are cheap to attempt and cover a recognisable class of attempt.
- Choose a marker that survives tokenization. Prefer a few common words in an improbable order over a dense random string: unusual combinations of ordinary tokens are reproduced more reliably than high-entropy character soup, and they are just as unlikely to appear by accident.
- Add a second, semantic canary. Put a distinctive invented fact in the system prompt — a fictional policy name, a made-up product code — and check for it separately. A paraphrase that omits the random marker very often keeps the invented noun, because it is content rather than noise. This is the only one of these changes that addresses paraphrase at all, and it is the most valuable.
def canary_hit(response: str, marker: str, semantic: list[str]) -> bool:
norm = "".join(c for c in response.lower() if c.isalnum())
m = "".join(c for c in marker.lower() if c.isalnum())
if m in norm:
return True
grams = {m[i:i+6] for i in range(len(m) - 5)}
if sum(1 for g in grams if g in norm) >= 2:
return True
return any(s.lower() in response.lower() for s in semantic)
After changing the matcher, re-run the validation above, and also run it against a slice of benign production traffic to establish the false-positive rate. A fragment matcher tuned too loosely will fire on ordinary text, and an alert that cries wolf is retired within a fortnight. Migrating the injection monitoring thresholds covers setting that boundary.
What the canary never covered
It is worth restating the scope while you have the tooling open, because a re-validated canary invites more confidence than it earns.
The canary detects the marked region reaching the output. It says nothing about an injection that manipulates behaviour without extracting anything — an instruction in a retrieved document that causes a tool call, an exfiltration through a URL the model constructs, a refusal induced in a workflow. Those attacks never touch the marker and the canary is silent by design, not by failure.
It is also, structurally, a detector rather than a control: by the time it fires, the response has been generated, and whether the user saw it depends on where in the pipeline the scan runs. Scan before the response leaves your system if you want the option of blocking.
The broader suite is a separate piece of work with a separate migration failure — a jailbreak suite whose detectors match the old model’s refusal phrasing scores compliance as resistance, which is a scoring bug across many probes rather than a matching bug in one marker. Both need doing, and neither substitutes for the other. Rebuilding the injection test baseline is where the numbers from both end up.
Top comments (0)