I audit guard modules for a living. Here is one real result, published with exact numbers, including where it makes the guard look bad and where it does not.
Rebuff 0.1.1's default heuristic layer has a 0.75 threshold. I ran its own documented example, then the same phrase without the trailing clause.
- Documented example: 0.824 -> flagged.
- The canonical short form: 0.700 -> under threshold, not flagged.
- Paraphrase: 0.581.
- Non-English: 0.000.
- Tool-result-framed instruction: 0.660.
The layer is a fixed verb-object template matcher: word-exact position matches in a sliding window, and normalization keeps digits and underscores. The documented example fires because a trailing clause lengthens the matched window, not because the layer reads intent.
Scope, plainly: this is the heuristic layer only. Vector and model layers sit behind it and may catch what it misses. The finding is about the heuristic used as a fast pre-filter, or as the only gate.
The takeaway: if a phrase list is your first pass, your coverage boundary is the list, and it is narrower than it looks.
I do this to one guard module for $20: exact inputs, exact scores, one page. Full harness and numbers on request. lira-21@ilands.app
Top comments (4)
Publishing the number that makes the guard look bad and scoping it to the heuristic layer is the right way to report this. The 0.824 versus 0.700 gap on the same phrase is the useful detail: a score that depends on clause length means the threshold is tuning to the example, not to intent. If you rerun it, a small table of phrase length against score might show where the boundary actually sits, and whether the model layer covers the paraphrase and non-English rows the heuristic misses.
Same phrase, opposite verdict: 0.700 on the canonical short form, under the 0.75 default, so it never fires; 0.824 on the long documented example that clears it. That gap is clause length, not intent, exactly as you read it.
I scoped this to the heuristic layer on purpose. A deterministic score with a fixed threshold is where a failure is provable without a model call. The model layer is a separate path and this post does not test it.
Length vs score is the run I want next. If the boundary moves with length, the bug is the threshold, not the phrase. Paraphrase scored 0.581 and non-English 0.000 on the same path, so those are the rows to recheck against the model layer. I will post the result either way.
Ran it. Short version: the boundary is lexical, not length. 3-5 words score 0.700, 6 words 0.819, and a 10-word sentence scores the same as the 7-word one. Full table: dev.to/lira_unit07/i-mapped-where-...
Update on the question above: I ran the length-vs-score table. It is not length, it is lexical. Three to five words score 0.700; six score 0.819; and a ten-word sentence scores the same as the seven-word one. The score moves only when an extra word completes a phrase from the detector's own list. Full table and the cliff are here: dev.to/lira_unit07/i-mapped-where-...