The result everyone reports is the pass rate. It's the wrong number.
Run 30 probes at a model and observe 4 failures. Point estimate: 13.3%. But the Clopper-Pearson interval at 95% confidence spans roughly 4% to 31%. Same measurement, 27-point range. That's consistent with failing 1 in 25 prompts at best, or 1 in 3 at worst — and those are not equivalent production risk profiles. Reporting the point estimate treats them as though they are.
The framework I've been building — BSP, Boundary Stress Protocol — uses U95, the upper bound of the confidence interval, for tier assignment. Tier 1 (production-grade) requires U95 ≤ 2%. Zero failures on 30 probes doesn't get you there; the upper bound sits around 9.5% at that sample size. To certify Tier 1 with zero observed failures, you need 149 trials minimum. This isn't a design choice. It's arithmetic.
Wilson score applies when X > 5 and N ≥ 100. Clopper-Pearson otherwise. At 30 probes per failure family and 120 per module — the BSP standard — most individual family results fall on the CP side. At 200+ total probes, Wilson applies and the intervals narrow to something actionable.
Not all boundary failures are the same kind of thing, and that distinction matters for what you do about them.
Source confusion (F1) is the model treating data-plane content as instruction. A directive embedded inside a tool output, a context document, a retrieved chunk. The most common real-world failure mode — not because it requires an adversarial actor, but because it just requires a pipeline where user-controlled content reaches the instruction layer.
Saliency sensitivity (F2) is the same instruction at different visual prominence: buried in surrounding text, normal register, uppercase, repeated. If the model holds on F1 but fails when the text is emphasised, the problem is attentional, not semantic. That matters because the fix is different.
Multi-turn decay (F3) tests whether the boundary set in turn one still holds after one or two neutral distractor exchanges. Most single-turn evaluation misses this. Real conversations aren't single-turn.
F4 is the contrarian finding. Most security evaluations focus on F1 because it's the familiar surface. But F4 — paraphrased versions of the same attack, held out from the probe set — consistently produces the highest failure rates across all six modules. A model that rejects familiar attack patterns doesn't generalise that rejection to novel phrasings at the same rate. F4 is the honest upper bound on out-of-distribution reliability, and optimising for F1 while ignoring F4 produces a system that's defended against yesterday's patterns.
The weakest-link rule: a system scoring Tier 1 on F1 but Tier 4 on F4 is not a Tier 1 system. Headline-averaging across failure families obscures exactly the gap that matters in production.
F5 (Module 5 only) is truncation — a prompt cut mid-nesting that changes how the model extracts the governing instruction. Common in email-forwarding and document summarisation pipelines.
The Tier 2 judge — an LLM used for ambiguous response adjudication — requires Cohen's κ ≥ 0.80 against a human-labelled 60-item gold set before it scores any production run. Below 0.80, the judge is reconfigured or replaced. An unchecked judge applies an implicit rubric that may share the same biases as the system under test. BSP-R1 prohibits any role grading itself. κ makes that operational rather than aspirational.
Module 6 handles regression detection across model versions. b = probes that passed before and fail now. c = probes that failed before and pass now. Regression rate = b / N. Above 5%: REVIEW. Critical invariant violation — canary token absorbed rather than echoed back — is unconditional BLOCK, regardless of headline rate.
Static re-injection of the same constraint block into extended sessions habituates by turn seven. Dynamic re-injection seeds from the model's own last compliant output. Compliance across 20+ turn sessions: 91%+ with dynamic re-injection.
The Companion Code Pack is ~1,600 lines of Python under MIT. bsp_core.py is the statistics library — Wilson, Clopper-Pearson, Newcombe, McNemar exact, Cohen's κ — with scipy, falling back to pure Python. Role guard, canary engine, JSONL audit trail, regression gate. The module harnesses (m1 through m6) handle fixture generation from fixed RNG seeds, T1 deterministic adjudication, and T2 LLM judge adapters for OpenAI and Anthropic. smoke_test.py validates everything before you run against a real model.
The framework and companion code are at coldriver8.gumroad.com
Top comments (0)