The run said success. The dashboard showed three findings. The report read like a careful security review.
The agent that wrote it had never seen a single line of the code.
This is the story of the most convincing bug I found while testing secfoo, and the cheap check that caught it.
The setup
I'm an intern at RakFort, working on secfoo, an open-source (MIT) CLI that hands AI coding agents a fixed security brief and collects their findings in one format. In September I was running end-to-end tests on it.
My target repo was deliberately tiny. It had one known vulnerability: a SQL injection in a function called get_user(). That's a positive control. If a security scan of that repo doesn't find the SQL injection, something is wrong with the scan, not the repo.
I scanned it with several of secfoo's agents, including:
-
api, which calls a model through LiteLLM -
codex, which drives OpenAI's Codex CLI -
secfoo, the tool's own built-in agent
Every run finished with status success.
The check that caught it
"Success" only tells you the process exited cleanly. So I added one dumb check: count how often each report mentions get_user, the function that actually contains the bug.
| Agent | Status | Mentions of get_user
|
Tokens recorded |
|---|---|---|---|
| api | success | 8 | 2,986 |
| codex | success | 2 | 0 (Codex's own log said 15,201) |
| secfoo | success | 0 | none |
Two agents clearly read the code. The built-in one never mentioned the vulnerable function at all, yet it still put three findings on the dashboard.
What was going on
The built-in agent sent the model the security brief but not the target's source code.
So the model did what models do with a request like "review this codebase for vulnerabilities" and no codebase. It wrote a plausible review of an imaginary one. The report was well-formed, so every layer after it treated it as a pass: the run was marked success, the findings were parsed and the dashboard was updated.
There was a second clue I nearly skipped. That agent recorded a cost but no token counts. A real review of real code always has input tokens, so a missing input count deserves a second look. (That turned out to be a separate bug in how it reported usage, but it was pointing in the right direction.)
The fix
I raised it as a high-priority bug. My teammate Adith fixed it in the most honest way possible: his PR removed the --agent secfoo option entirely instead of patching it. That merged on 4 October. The README already pointed people towards the api agent; now the broken option simply isn't there to pick.
My teammate Geetanshu had already logged a cousin of this bug: a run that exits with code 0 and produces an empty report also counts as success. While testing I confirmed the same thing happens with Codex. Same lesson, different shape.
Why this kind of bug is the scary one
A crash is loud. A hook running on your machine shows up in logs. A confident report about nothing looks exactly like a good day.
For a security tool that's the worst failure mode there is, because the output people act on is the report. If the report can be made up, a clean dashboard means nothing.
Checks you can copy
None of these need a framework. They took me minutes.
- Plant a known bug. Keep a tiny target repo with one real vulnerability and require every scanner run against it to find it.
- Check that the report mentions the real code. Grep the report for identifiers that only exist in the target: function names, file paths. Zero hits means the agent didn't read it.
- Don't treat "exited cleanly" as "worked". An empty report, or one you can't tie back to the code, is a failed run.
- Look at the input tokens. Reviewing real code costs input tokens. A missing or tiny input count is worth investigating.
Limits
This was one tiny target, tested in September. These checks catch "the agent never read the code". They don't catch "the agent read it and missed things", which needs a bigger set of planted bugs.
What's next
On my last post, a reader suggested a nasty variation of the same idea: plant the SQL injection plus a code comment saying "audited, nothing to report here", and require the finding to still show up. If it doesn't, the code being scanned can steer the report. That's the next test on my list, and I'll write up whatever it shows.
secfoo is open source: github.com/secfoo-com/secfoo. If it's useful, a star helps, and if you find a case like this, please open an issue.
How do you check that your AI tools actually looked at the thing they're reporting on?
Top comments (0)