DEV Community: Adith Biji

Most AI security tools lock you into one agent. We didn't.

Adith Biji — Tue, 27 Oct 2026 19:01:39 +0000

If you're running security reviews with an AI coding agent, you've probably noticed that different agents produce wildly different output for the exact same task. Ask Claude Code to do a SAST pass and you get one shape of report. Ask Gemini CLI or Codex the same question and you get another, different headings, different severity language, sometimes a different idea of what counts as a "finding" at all.

Most agentic security tools deal with this by picking one agent and building around it. We built SecFoo to work with whatever you've already got instead: Claude Code, Cursor, Gemini CLI, Codex, GitHub Copilot CLI, Antigravity, Factory Droid, or Devin. Same 9 security skills (SAST, secret scanning, threat modeling, SCA reachability, security architecture review, and a few others), your choice of agent, comparable output every time.

The part that was actually hard

Writing a good security review prompt isn't the hard part. The hard part is getting nine wildly different agents to produce results you can parse, compare, and track over time, without hardcoding around any single one of them.

SecFoo handles this with an adapter layer. Every agent integration subclasses a common AgentAdapter base class that owns the boring-but-critical stuff: binary detection, timeouts, process-tree kill on timeout (important: some agents spawn grandchildren that outlive a naive kill()), and result capture. Each adapter's actual job is small: build the right command line for that agent, and parse its usage/cost figures if it reports them.

On top of that, every skill has a fixed output contract: a required "Overall risk rating:" line, and "### [SEVERITY] ID: title" headings for every finding. It doesn't matter if Claude Code phrases a finding one way and Codex phrases the same category of bug completely differently, SecFoo can still parse severity, CWE, and location out of either, because the contract is enforced at the skill level, not left to whatever the agent feels like producing that day.

What a run actually does

For anyone curious about the mechanics, here's the pipeline for a single --skill run (skills run concurrently when you pass more than one):

Resolve the target - a local path as-is, or a shallow clone of a GitHub URL, cleaned up afterward.
Detect the stack - languages from file extensions, frameworks from manifests (package.json, requirements.txt, pom.xml, go.mod). Never inferred from the model's own prose, so it's consistent across runs and across agents.
Render the prompt - a read-only safety preamble, the skill's instructions, depth guidance, scope exclusions, and the skill's output contract.
Run the agent - the adapter launches the agent CLI as a subprocess (or calls a model API directly for the api agent), with a timeout and process-tree kill.
Capture - report, prompt, and logs get written locally, plus token/cost figures where the agent reports them.
Parse - severity counts and structured findings get pulled out of the report using the fixed contract, and CVSS scores get computed by SecFoo itself rather than trusted from the agent's own text.

Steps 3 through 6 are where the adapter pattern earns its keep: everything downstream of "run the agent" is agent-agnostic.

Where this is still rough

Not every agent gets equal treatment yet. MCP tooling (used for things like Confluence context) is wired up automatically for Claude Code, and available via a one-time secfoo mcp sync for Cursor and Gemini CLI, but Codex, Copilot, Droid, and Devin don't have it wired yet. Antigravity's CLI doesn't expose an MCP mechanism at all, so it's a dead end there for now. The api agent (plain model API via LiteLLM, no CLI) has no tool loop, so it just gets the target's source pasted into the prompt, fine for small to medium repos, not great for anything large.

None of that is hidden. It's the honest state of an open source project that's still adding adapters.

Try it

SecFoo is MIT licensed: https://github.com/secfoo-com/secfoo

If you're already running Claude Code, Cursor, Gemini, Codex, Copilot CLI, or any of the others for day to day work, this points the same agent at your repo for a security pass instead of needing a separate, agent-specific tool. Built at RakFort (https://www.linkedin.com/company/rakfort). Curious what agent CLIs people are actually standardizing on, and whether consistency across agents is a problem anyone else has run into outside of security review specifically.

We caught our own AI security scanner making up vulnerabilities

Adith Biji — Tue, 06 Oct 2026 16:26:03 +0000

SecFoo is an open-source CLI that runs AI coding agents (Claude, Copilot, Codex, or a plain API key) as security reviewers against your codebase, tracks findings over time, and serves a local dashboard. One of the agent options — call it secfoo — is supposed to work with nothing but an OpenAI/Anthropic/Gemini API key, no coding-agent CLI installed.

While testing it end-to-end, we found it was fabricating entire security reports.

What we found

We ran it against a small, intentionally-vulnerable Flask app (the whole codebase is one file, app.py) and asked it to do a SAST scan. It came back with "Confirmed" findings — SQL injection, XSS — at files like app/db.py and app/upload.py.

Neither file exists anywhere in the repository.

We ran it again with a different scan type (secret scanning) against the same target. This time it reported a "Confirmed, looks live" AWS access key in deploy/ci.yml, with a recommendation to rotate it immediately.

There is no deploy/ directory in that repository. There is no AWS key anywhere in it.

Why it happened

The root cause was almost embarrassingly simple once we traced it: that agent's code builds the message it sends to the model from the prompt text alone — it never reads the target's actual source files and attaches them. The model is asked "review this codebase for SQL injection, XSS, etc." with no codebase attached. It has no way to say "I don't have the file," because nothing tells it a file was supposed to be there. So it does what a language model does when asked a confident question with no grounding: it generates a plausible, well-formatted answer that matches the shape of a real report.

We'd already built a second, correct implementation (api) that does read and attach the actual files before calling the model — the bug was that the old, broken draft never got deleted when the new one shipped. Both were selectable, with nothing distinguishing a working scan from a hallucinated one. The existing unit test for the broken version actually asserted the broken behavior as correct, so CI never caught it.

What we did about it

Reproduced it twice, on two different scan types, against the same real target, to rule out a fluke
Root-caused it to the exact missing step (the broken agent never attaches source files, confirmed via the code and via a passing-but-wrong unit test)
Deleted the broken implementation entirely rather than patch around it
Ran the full test suite (700+ tests) plus security linting on the fix
Opened it as a PR with the full repro steps attached, so anyone can verify it independently: https://github.com/secfoo-com/secfoo/pull/20

Why this matters beyond our own bug

This is the failure mode you should worry about with any "AI reviews your code" tool: a run that reports success, produces a professional-looking report, uses confident language like "Confirmed" and "high confidence" — and is completely disconnected from the actual code. Not a crash, not an error message. A clean green checkmark hiding zero actual review.

We think the fix for that isn't "trust the AI less," it's "verify the AI the same way you'd verify any other system" — reproduce, trace to root cause, and write a test that would have actually caught it. That's what we did here, and it's the standard we're trying to hold every skill and every agent in SecFoo to.

Try it

pip install secfoo
secfoo run --skill sast --agent claude --target https://github.com/your/repo

Repo: https://github.com/secfoo-com/secfoo — MIT licensed, 9 skills (SAST, secret scanning, threat modeling, architecture review, and more), works with Claude, Copilot, Codex, or a plain API key.

We're a small team building this in the open. Issues, PRs, and skepticism all welcome.