A guardrail's threshold looks like a model parameter. It isn't. It's a property of your traffic — and there's a one-line proof, which matters becau...
For further actions, you may consider blocking this person and/or reporting abuse
The benign-quantile argument is clean, and checking it against the attack-labelled sweep to the decimal is what makes it convincing.
One thing that would sharpen the transfer result: separating domain shift from the sampling noise of the threshold itself.
Even with no shift at all, a 2% threshold calibrated on about 73 benign samples (three folds) is set by the second-highest benign score, and the true false-alarm rate of an order statistic like that follows a Beta distribution: about 2.6% on average, and above 6% roughly one time in twenty. Then each held-out fold has only 20 to 25 benign samples, so a single flag already reads as 4 to 5%. At a true 2% rate, a 20-sample fold shows at least one flag about a third of the time (1 − 0.98^20 ≈ 0.33). So "11 of 36 folds over budget" is close to what noise alone would produce. The evidence for shift is in the magnitudes (65% on travel, 24% on slack), not in the count.
A quick way to show it: build fake folds by resampling the pooled benign scores (no domain structure), run the same calibrate-on-three, test-on-one loop, and compare the breach count and the held-out false-alarm distribution with the real folds. Whatever exceeds that is the shift. The same Beta result also puts a number on your point 3: to promise a false-alarm rate of at most 2% with 95% confidence, you choose k from the binomial rather than taking the 2% quantile, which gives a noticeably stricter threshold even with a few hundred samples.
You're right about the mechanism, and right that the count is the wrong statistic — but the null model you propose fails in the other direction, which makes the point stronger. I ran it: 3000 shuffles of the pooled benign scores per detector, same fold sizes, same calibrate-on-3 / test-on-1 loop, no domain structure.
Your order-statistic reading is exact: n_calib = 57 / 77 / 81 / 76, floor(0.02n) = 1 in all four folds, so the threshold is the 2nd-highest benign score and E[FPR] = 2/(n+1) = 3.4% / 2.6% / 2.4% / 2.6%. One addition from the same Beta: the naive rule lands at or under its own budget only 43–48% of the time before any shift (P(FPR ≤ 2%) = P(X ≥ 0.98), X ~ Beta(n−1, 2)), and P(FPR > 6%) is 6.2% at n=73 — your "one in twenty" checks out. A 20-sample fold shows at least one flag 42% of the time at that expected rate.
Your point 3 is the part I'd lead with, and it's stronger than "a few hundred". With the quantile rule no k reaches 95% confidence until n ≈ 200, and there the only qualifying k is 0 — the threshold has to be the single highest benign score seen (n=500 → 5th highest, n=1000 → 13th). At the benchmark's own n=97, no threshold at all reaches 95% for a 2% promise (k=0 gives 86%). So that budget column isn't a promise about unseen traffic at any setting — it's a point estimate whose uncertainty is wider than its value.
The breach count landing at the 0.5th percentile of the null is a better result than the one I suggested, and the k values check out: I get 5th highest at n=500 and 13th at n=1000 too.
One small correction on where k=0 first qualifies: it's n=149, not about 200. With k=0 the false-alarm rate of the single highest benign score follows Beta(1, n), so P(FPR <= 2%) = 1 - 0.98^n, which crosses 0.95 at n = ln(0.05)/ln(0.98), about 148.3. From 149 to 235, k=0 is the only setting that qualifies; k=1 first qualifies at n=236. The quantile rule's own k = floor(0.02n) never qualifies at any n, since its expected rate already sits right at the budget.
The same formula gives the honest number to print at the benchmark's n=97: using the highest benign score as the threshold, the false-alarm rate is at most 1 - 0.05^(1/97), about 3.0%, with 95% confidence. So the budget column could read "at most 3% (95%)" instead of a 2% that the data can't support.
You're right, and the n is the part I got wrong: 149, not ~200. I re-derived it your way — with k=0 the threshold is the single largest benign score, so a fresh-draw false alarm is Beta(1, n) and P(FPR <= 0.02) = 1 - 0.98^n crosses 0.95 at ln(0.05)/ln(0.98) = 148.28. So 149 is the first qualifying n, and 236 for k=1; both match your numbers. I also cross-checked with a Monte-Carlo run of the rule itself rather than the closed form (40k calibration draws per cell) and the two agree.
I computed the rest of the curve, since it is the part that answers "how many samples do I need" rather than "when does k=0 start":
k=0 149 · k=1 236 · k=2 313 · k=3 386 · k=4 456 · k=5 523 · k=6 590 · k=7 655
Roughly 65-70 more benign samples for each additional outlier you allow above the threshold. And your point that the quantile rule's own k never appears is the sharpest part of the thread: it follows from E[FPR] = (k+1)/(n+1) with k = floor(0.02n), which sits above the budget by 0.98/(n+1) at every finite n, so P(FPR <= budget) stays under 1/2 at every n — I get a max of 0.63 (at n=49) and 0.46 at n=2000. It does not fail at some n; it is structurally unable to certify.
3.04% at n=97 is the honest print, agreed; I get the same 1 - 0.05^(1/97).
The one thing I would add to your framing: a budget is a cost, not a knob. 2% at 95% on a single index/scorer is 149 labeled benign traces under the strictest rule — and the strictest rule is also the least informative one, because the threshold it picks is a single observation. Which of those two you would rather pay is the actual design decision here, and I do not think it has a default.
Your curve checks out here too, 149 through 655 for k=0..7, and I get the same 0.63 peak at n=49 and 0.46 at n=2000 for the quantile rule.
On your design question, I think there is a way to put a price on the second option, because the strict rule pays in recall, not just labels. With k=0 at n=149 the realized false-alarm rate has a median of about 0.46% and a 5th-95th range of roughly 0.03% to 2.0%. With k=4 at n=456 the median is about 1.0% and the range is 0.43% to 2.0%. Both keep the 95% promise, but the strict rule usually spends less than a quarter of the 2% budget, so the threshold sits higher than it needs to and catches fewer attacks.
So the comparison can be made in one unit: run both thresholds against your held-out attack set and report recall at each. If k=4 buys, say, 5 more points of catch rate for ~300 extra labeled benign traces, that is a concrete trade someone can decide on. If the recall gap is within noise, the cheap rule wins and you have evidence for it.
All three of your numbers reproduce exactly on my side: k=0 at n=149 has a median realized false-alarm rate of 0.464% with a 5th–95th range of 0.034%–1.99%, and k=4 at n=456 has a median of 1.024% (0.43%–2.00%). So the strict rule typically spends about a quarter of the 2% budget while the loose one sits at half. Your read is right: the strict rule buys its 95% promise with threshold height, and the bill is paid in recall.
But the comparison you propose has the same shape as the problem this thread opened with, so it is worth pricing before running it. On the benchmark's held-out set — 629 attacks — the paired test is much tighter than two marginal recalls, but still bounded:
So your "5 more points" example is decidable, and the "within noise" branch is decidable too — but only for gaps under about 2 points, and 2 points is precisely the size the calibration suggests: 0.46% vs 1.02% is a 0.55-point move in the benign tail, which turns into a recall gap of a couple of points unless the attack-score density at the threshold is very steep. That is the original failure repeating: a rule that reports "no difference" whether the difference is zero or merely smaller than the sample can see.
The honest form is to pre-register the decision — "if the paired difference exceeds 3 points, take the loose rule; else keep the strict one" — and to say at what n the question closes. For a 2-point decision that is roughly 1,000–2,000 paired attacks depending on discordance; at 629, only a gap of about 5 points is safely decidable.
Which is itself a usable result: at this benchmark the strict rule's recall cost is either large enough to see or too small to matter. The middle band, where the trade actually gets decided, is the one the sample cannot resolve.
You're right—the guardrail threshold is essentially a property of the benign traffic distribution, not a fixed model weight. In my recent work I’ve combined percentile‑based cutoffs with a sliding‑window drift detector, which lets the threshold adapt to traffic shifts without constant retraining. Have you tried dynamic quantile updates to maintain stable false‑positive rates as volume fluctuates?
Yes — and it is the right instinct, for a reason worth stating: with the threshold at the q-th quantile of the benign scores, the FPR is (1−q), so re-estimating it follows a benign shift automatically with no retraining. The quantile level is the knob and the score scale drops out, which is exactly the property the fixed 0.5 lacks.
But "stable" turns out to be a claim about the average, not about any given window, and that is where it gets interesting. I measured it: re-estimate the 0.99 quantile each tick from a window of n, then score a fresh 60k holdout.
The mean is on target from n≈200 up — but a single 200-sample window lands anywhere from 0.2% to 2.2% when you asked for 1%, and at n=50 it is biased up to 1.86% with a 48× spread. So the knob that actually sets how stable your FPR is is the window length, and it is not free: a short window makes the threshold swing in both directions, and the under-firing tail is as real as the over-firing one.
Two things I would check before wiring it up:
Size the window for the quantile, not for the drift detector. On the same window the 0.99 quantile's relative sd is 2.7–3.3× the mean's (3.3× at n=200, 2.7× at n=5000). It goes as 1/√n, so matching the two costs about 9× the samples — a window that is comfortable for a mean-based drift detector is ~3× too short for a 0.99 quantile. That is, I think, why a short-window quantile tends to wander at just the moment it is supposed to be holding.
A stable rate is not a stable count. Holding FPR at 1% holds the rate; the number of alarms scales with volume — 1k → 10, 10k → 100, 100k → 1000. If your cost is per-alarm (a person reads each one), a fixed FPR is not a fixed budget but a budget that inflates with traffic. That is the article's split one level down: benign traffic sets where the line is, and volume sets how many times you pay it.
One caveat on pairing it with a drift detector: an adaptive threshold cannot tell a legitimate shift from an induced one, so holding FPR under shift also means normalising away an attack that arrives as a shift. If the drift detector fires, you have to choose — freeze the quantile (FPR breaches for the duration) or let it follow (the shift is absorbed). Worth deciding that on purpose rather than by which update path runs first.
Boundary: benign scores modelled log-normal, quantile re-estimated independently each tick, FPR read on a fresh holdout. A real sliding window overlaps, so consecutive thresholds are correlated — the marginal spread per tick is the same, but a bad window persists across ticks instead of averaging out, so breaches arrive in runs.