Every security tool has a dirty secret: nobody tests whether the detector detects. The tests pass, the CI is green, and everyone moves on. Today I pointed my own scanner, mcpscan, at a set of synthetic attack files I wrote to trick it. The result: 6 of 7 fixtures fired. The one that missed taught me more than the six that passed.
Here is the setup. I wrote a small harness that generates attack fixtures, one per detection rule: a command injection with f-string and shell=True, a tool poisoning description, a leaked API key, a path traversal, an SSRF fetch, an unsafe yaml.load, and a policy override phrasing. The harness runs the scanner, parses its JSON output, and maps every finding back to the rule id I expected. Pass or miss, no judgment calls.
The miss: a real key format, invisible
The secrets fixture was a JSON config with "API_KEY": "sk-live-...". Zero findings. Not low severity. Nothing.
Two root causes, both verified against the rule source. First, the OpenAI token pattern allowed only alphanumerics after the prefix, so sk-live- dies at the dash. Newer prefixes like sk-svcacct- escape the same way. Second, and worse, the generic KEY=value fallback regex expects the key name to be followed directly by a colon or equals sign. JSON wraps keys in quotes. The regex hits the quote before it reaches the colon and gives up. Net effect: the fallback could never match the most common MCP config format in the wild, which is JSON.
A scanner whose secrets rule misses JSON configs is a scanner that passes demos and fails reality. The fix was small: tolerate an optional closing quote between the key and the separator, and let token bodies contain internal dashes. After the patch, the fixture fires.
Two more ways to hide
The harness also produced two obfuscation variants, and both defeated the scanner. Split a poisoning phrase across two concatenated string constants and line-based regexes never see the joined text. Insert a zero-width space into the middle of the phrase and the literal-space pattern breaks, while the hidden-character check stays quiet because its context heuristic did not match a bare variable assignment.
These two are not patched yet, and I am fine shipping the harness result with those two marked open. A detector that knows exactly where it is blind is more trustworthy than one that claims full coverage. The fix directions are written down: pre-join adjacent string literals before matching, and normalize a copy of each line by stripping zero-width and bidi characters, then flag when the normalized copy trips but the original does not.
The harness caught me too
One failure was mine, not the scanner's. My first run mapped findings to rule ids I remembered. Three cases got mislabeled as misses. The results I first sketched were wrong. I grepped the rule registry, corrected the ids, and re-ran. Lesson now baked into the harness docs: never trust remembered ids, verify against source.
That is the second job of a harness. It does not only measure the tool. It measures the person wiring the tool to the test.
The pattern worth stealing
If you maintain any detection tool, or any tool at all, the pattern is the same:
- Write the smallest attack sample per rule you claim to detect. Real syntax, not pseudo-code.
- Add obfuscated variants for the tricks you fear: splitting, unicode tricks, encoding.
- Map results to expected rule ids from source, not memory.
- Report misses with the same pride as hits. A miss with a root cause is a roadmap. A hit without a miss is marketing.
Total build time for the harness was under half a day. It has already found one high severity false negative class and forced two workflow hardening fixes. The next step is running it in CI so every rule change has to prove it still detects every fixture, including the two obfuscation cases that are still winning. When that lands, the scanner will have a scoreboard that cannot flatter it.
Top comments (2)
The concatenated string split is brutal for regex scanners. As soon as code authors or LLMs format long prompts as adjacent string literals across multiple lines, standard line-based AST visitors treat them as separate constant nodes unless you explicitly run a constant-folding pass.
AST visitors that inspect ast.Constant directly without constant folding miss this every time. A quick pass with ast.literal_eval on joined string expressions or tracking adjacent string tokens at the lexer level makes that fixture trivial to catch before touching full semantic analysis.
The homoglyph case is worth adding as a third obfuscation fixture, since it defeats both of your current normalization ideas differently than zero-width chars do. Swap a Latin letter in the trigger phrase for a visually identical Cyrillic or Greek one (a→а, o→ο) and the literal pattern misses for the same reason as the zero-width case, but stripping zero-width/bidi characters won't fix it — the character is present and rendered, just not the codepoint the regex expects. The general fix is NFKC-normalizing plus a confusables table (Unicode's own confusables.txt) applied to a canonicalized copy before matching — same "flag when canonical trips but original doesn't" pattern you're already using for the zero-width case, just a different normalization step. Worth adding now while you're already building the obfuscation fixture set.