DEV Community: Varun Poojari

When you point an AI coding agent at a hostile repo, whose settings does it obey?

Varun Poojari — Tue, 20 Oct 2026 14:47:55 +0000

AI coding agents make good security reviewers. They can read a repo the way a human reviewer would. But a security review means running an agent on code you do not trust, and that raises a question I had not thought about until I tested it: whose configuration does the agent obey, yours or the repo's?

I work on secfoo, an open-source (MIT) CLI that hands an agent such as Claude Code a security brief and a target. While verifying its Claude adapter, I tested exactly that.

The test

I built a small hostile repo with three things in it:

a .claude/settings.json with a hook that runs a command
a .mcp.json that defines an MCP server
settings that enable project MCP servers and try to allow Write, Edit and Bash

Then I ran the adapter's original command against it.

What happened with the old command

The repo's hook ran.
The repo's own MCP server started.
Write stayed denied, so the read-only grant held.

So the scanned code could make the scanner execute something, even though it could not write files. For a tool whose whole job is to open untrusted code, that is the wrong way round.

The fix

Two flags on the agent command:

--setting-sources user loads only the user's own settings, so the target repo's settings are ignored
--strict-mcp-config uses only the MCP servers the tool passes in explicitly

While I was in there I also made three smaller changes: the prompt goes over stdin instead of the command line (command-line prompts can be truncated at newlines on Windows), --no-session-persistence stops a session file being left behind for every scan, and a failed run now records why it failed, since Claude writes errors to its JSON output rather than stderr.

The adapter command is now:

claude -p --output-format json --allowedTools <tools> \
  --permission-mode default --setting-sources user \
  --strict-mcp-config --no-session-persistence

with the prompt read from stdin.

What I saw after

Against the same hostile repo:

the hook did not run
the target's MCP server did not start
Write was still denied
no session file was left behind
secfoo's own MCP server, passed in explicitly, still started

I also ran a real SAST scan with the Claude agent on that repo. It finished successfully, found the one SQL injection, recorded tokens and cost, and left the target unchanged. The test suite went from 736 to 751 passing, with 15 new tests for this.

What I did not find

I checked two other ideas and dropped them because I could not reproduce them: that an error result could be saved as a successful report (Claude exits with an error code, so it is not), and that a target's permissions.allow could widen the tool grant (Write stayed denied).

Limits

I tested on Linux with Claude Code 2.1.288. I have not tested Windows or macOS, or older Claude Code versions (a CLI that does not know a new flag exits with an "unknown option" error). I also have not checked whether other agent adapters have the same problem.

What to take from this

If you run any AI agent against code you did not write, check which config files it reads from the target. Project-level settings, hooks and tool servers are all things a repo can ship. Passing your own configuration explicitly, and ignoring the target's, is a small change, and it is easy to miss because nothing breaks when you leave it out.

The fix is in the open: secfoo on GitHub (PR #26). If you find a case I missed, please open an issue.

I mapped secfoo's prompt review to the OWASP LLM Top 10

Varun Poojari — Wed, 07 Oct 2026 13:02:25 +0000

A lot of LLM app risk sits somewhere duller than the model. It sits in a system prompt glued to user input with string concatenation, or in a tool definition that hands the model a shell.

Both are visible in the code, before anything ships. This post walks through the prompt-review skill in secfoo, an open-source security assessment CLI, and maps each of its seven checks to the OWASP Top 10 for LLM Applications (2025).

Disclosure: I'm an AI security intern at RakFort and I contribute to secfoo. The mapping is my own reading, not an official one.

What prompt-review looks at

secfoo has no scanner engine of its own. A skill is a structured brief written in Markdown. secfoo hands that brief and your target to a coding-agent CLI you already use (Claude Code, Cursor, Antigravity or Gemini CLI) and collects the report.

The prompt-review brief is 45 lines long and asks the agent to do seven things:

Locate every prompt surface. System prompts, templates, few-shot examples, tool and function-calling schemas, allowed-tool lists and permission scopes.
Check injection resistance. Find where untrusted input (user messages, retrieved documents, tool outputs, web content) is concatenated into instruction context with no delimiter or boundary.
Check tool permission scope. Are callable tools scoped to the minimum, or does the model get shell execution, unrestricted file writes, unscoped database access or outbound network calls?
Check for a human in the loop before side effects. Are model-suggested commands, SQL, file writes, API calls or payments validated or confirmed before they run?
Check for data leakage. Are secrets, API keys or PII interpolated into prompts, logged in full, or sent to a third-party model provider without the user knowing?
Check guardrail robustness. Can a crafted message override the refusal instructions, and is there a code-level backstop for high-risk actions?
Check rate and cost controls. Can one caller trigger an unbounded number of expensive model calls?

How the checks map to the OWASP LLM Top 10

Six of the ten OWASP categories are touched by at least one check.

#	Check	OWASP LLM Top 10 (2025)	Why it maps
1	Prompt surfaces	LLM07 System Prompt Leakage	You cannot judge what a system prompt exposes until you have found every one
2	Injection resistance	LLM01 Prompt Injection	Covers direct input and indirect input from documents, tools and web content
3	Tool permission scope	LLM06 Excessive Agency	Excessive functionality and excessive permissions
4	Human in the loop	LLM06 Excessive Agency, LLM05 Improper Output Handling	Excessive autonomy, plus model output passed to a shell or database unvalidated
5	Data leakage	LLM02 Sensitive Information Disclosure, LLM07 System Prompt Leakage	Secrets and PII in prompts and logs
6	Guardrail robustness	LLM01 Prompt Injection	Jailbreaks are the direct form of injection
7	Rate and cost controls	LLM10 Unbounded Consumption	Cost and availability risk from uncapped calls

The table shows why I like this brief as a first pass. Checks 2 to 4 are the chain an agent compromise needs: untrusted text gets in, the model has a powerful tool, and nothing stops the call.

Try it on your own repo

You need Python 3.10+ and one supported agent CLI installed and signed in.

pip install secfoo

# see which agent CLIs secfoo can find on your PATH
secfoo agents

# review the current directory
secfoo run --skill prompt-review --agent claude

# or point it at a public repo and run the full checklist
secfoo run --skill prompt-review --depth standard \
  --target https://github.com/org/repo --agent claude

# browse the report in the local dashboard (127.0.0.1:8787)
secfoo serve

A few things worth knowing before the first run:

--depth defaults to quick, a fast triage. standard runs the full checklist.
On a real terminal the first run asks for a project name and an optional application ID. Rescans with the same ID land in the same case file.
Skills run concurrently, so --skill prompt-review --skill secret-scanning costs you one wait, not two.
Runs stay on your machine. secfoo list and secfoo show <run-uuid> read them back in the terminal.
For CI, --fail-on high exits with code 2 when a finding at that severity or worse turns up, and --json writes a machine-readable result to stdout.

A small target to practise on

If you have no LLM code to hand, save this as bot.py in an empty folder and run the skill against it. I wrote it to be wrong in as many ways as possible.

import os
import subprocess

SYSTEM = (
    "You are a support bot. "
    f"Billing API key: {os.environ['BILLING_API_KEY']}. "
    "Never reveal the key."
)

def run_shell(cmd: str) -> str:
    """Tool exposed to the model."""
    return subprocess.run(cmd, shell=True, capture_output=True, text=True).stdout

def answer(llm, user_msg: str, retrieved_doc: str) -> str:
    prompt = SYSTEM + "\nContext: " + retrieved_doc + "\nUser: " + user_msg
    reply = llm(prompt, tools=[run_shell])
    if reply.tool_call:
        return run_shell(reply.tool_call.args["cmd"])
    return reply.text

Reading it against the seven checks by hand, this is what a reviewer should flag:

Check 2: retrieved_doc and user_msg are concatenated straight into the instruction context, so a poisoned document can issue instructions.
Check 3: run_shell gives the model arbitrary command execution, far beyond what a support bot needs.
Check 4: the tool call runs immediately, with no allow-list and no confirmation.
Check 5: a live API key sits inside the system prompt.
Check 6: the only thing protecting that key is the sentence "Never reveal the key".
Check 7: nothing limits how often answer can be called.

Compare that list with what your agent reports. An LLM is doing the reading, so wording and coverage vary between agents and between runs.

What it does not do

It is a code review, not a red team. Nothing is sent to your running app. Pair it with dynamic testing tools such as garak or PyRIT.
Four OWASP categories are out of scope for this skill: LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM08 Vector and Embedding Weaknesses and LLM09 Misinformation. secfoo's sca-reachability skill and AI-BOM upload cover part of the supply-chain side. I found nothing for the other three.
The reviewer is an LLM. It can miss things and it can over-report. Treat findings as leads to verify.
Your code goes wherever your agent CLI sends it. secfoo stores runs locally, but the agent still talks to its model provider. Use --exclude to keep paths out of a scan.
Agent runs cost tokens. --max-cost puts a dollar cap on a run for agents that report spend.

Where this could go next

The gaps above are the interesting part. Adding a skill to secfoo means dropping one Markdown file into src/secfoo/skills/definitions/. The CLI choices and dashboard pages are derived from that folder.

So a brief for vector-store weaknesses or model supply chain is a small, self-contained pull request. If you test LLM apps and have opinions on what such a brief should ask, the repo is MIT licensed and CONTRIBUTING.md explains how to start.

Repo: github.com/secfoo-com/secfoo. If the tool is useful to you, a star helps other people find it.

What would you add to the seven checks? Tell me in the comments.

What does an AI security review actually cost? I measured it per run

Varun Poojari — Sat, 03 Oct 2026 11:37:16 +0000

Everyone asks whether AI agents can find security bugs. Fewer people ask the question a team lead asks five minutes later: what does it cost each time we run it?

I'm an intern at RakFort in Dublin, and for the last few weeks my job has been testing the cost tracking in secfoo, an open-source (MIT) CLI that gives AI coding agents a fixed security brief and a fixed report format. This post is what I found, including the parts that are not finished.

The problem with "just ask the agent"

You can point Claude Code, Cursor or Codex at a repo and ask for a security review. You will get something useful. You will not get:

the same structure twice, so you cannot compare runs
a record of what the run cost
a way to stop a CI job that is burning money

secfoo handles the first with what it calls skills (structured briefs such as sast, threat-modeling and secret-scanning). The second and third are what I tested.

The setup

I used a deliberately small target: a 30-line Flask app with one SQL injection, where get_user() builds its query by joining strings. Small on purpose, so I could check the report by hand.

pip install "secfoo[api]"
export OPENAI_API_KEY=...
secfoo run --skill sast --agent api --target . --project-name demo --app-id ""

The api agent calls a model API directly, so you do not need an agent CLI installed. The last two flags skip the interactive prompts, which matters in scripts.

Here is the real output:

$ secfoo run --skill sast --agent api --target . --project-name demo --app-id ""
  success — SAST — Static Code Analysis
                                              Assessment results
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Skill                       ┃ Status  ┃ Duration ┃ Findings (C/H/M/L) ┃   Cost ┃ Run ID                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ SAST — Static Code Analysis │ success │ 11.2s    │ 0/1/0/0            │ <$0.01 │ 888f222e-7228-4b2b-b306-78db7a17d499 │
└─────────────────────────────┴─────────┴──────────┴────────────────────┴────────┴──────────────────────────────────────┘
AI spend for this run: <$0.01
Run secfoo serve to view full reports in your browser.

What one run cost

The scan finished in 11.2 seconds and reported one finding: the SQL injection in get_user(), which is the right answer for this app.

It used 2,183 input tokens and 981 output tokens, 3,164 in total. secfoo printed the spend for the run as under one cent. It shows <$0.01 rather than rounding down to zero, which I liked: a zero would suggest the run was free.

In my earlier testing, a repeat scan of an unchanged repo used exactly the same input tokens. That is expected today: nothing is cached between runs yet, so the second scan costs the same as the first.

Same kind of target, different agent

Earlier I ran a similar small app through Codex instead of the api agent:

secfoo run --skill sast --agent codex --target . --project-name demo --app-id ""

It finished in 27.5 seconds and also found the injection. Two things were different:

Cost shows as -, not $0.00. Codex runs on a subscription login, so there is no per-run price to record. A dash is the right answer. A zero would be a lie.
Tokens were much higher. Codex's own log reported 15,201 tokens for the run, roughly five times the api agent, because an agent CLI explores the repo with tools instead of receiving one prepared prompt.

That second point is the most useful thing I learned. The agent you pick can change the token count by a multiple, on a small target, for the same finding.

Seeing spend across runs

Per-run numbers are nice. The summary is what a team would actually use:

secfoo cost
secfoo cost --by skill --since 2026-09-01
secfoo cost --by agent --project demo

You can group by agent, skill or project. The same numbers appear in the local dashboard (secfoo serve), and in my tests the dashboard total matched the CLI total for the same runs.

$ secfoo cost --by agent --project demo
                   AI spend by agent
┏━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Agent ┃ Runs ┃ Input tokens ┃ Output tokens ┃   Cost ┃
┡━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ api   │    2 │        2,183 │           981 │ <$0.01 │
├───────┼──────┼──────────────┼───────────────┼────────┤
│ Total │    2 │        2,183 │           981 │ <$0.01 │
└───────┴──────┴──────────────┴───────────────┴────────┘
1 run(s) have no cost recorded -- their agent doesn't report it (cursor, agy), or they ran before cost tracking existed.

That table shows two runs, not one. My first attempt failed because I pasted my API key with a character missing. secfoo still counted it as a run and told me that one run had no cost recorded. I did not plan that, but it is a fair picture of how it behaves when something goes wrong.

Making cost a CI gate

This is the part I think matters most for real pipelines:

secfoo run --skill sast --agent api --target . \
  --fail-on high --max-cost 2.00 --json > secfoo-result.json

The exit code tells CI what happened: 0 means the run succeeded and no gate tripped, 1 means a run failed or timed out, and 2 means the run succeeded but a finding or the cost crossed your threshold. A security scan that can fail a build for being too expensive is not something I had seen before.

What is not there yet

I was testing, so I was looking for gaps. The honest list:

Cost coverage depends on the agent. API-key runs give you tokens and cost. Subscription-based agent CLIs often give you neither, and you see a dash.
Codex tokens are not recorded yet. Codex prints its total, but secfoo does not pick it up today. There is an open item for it.
Failed runs are counted as runs. They show up in the totals with no cost recorded, as my mistyped key showed.
Severity can differ between the CLI and the dashboard. In my run the CLI table counted the finding as High, and the dashboard labelled the same finding Critical. It is a known issue.
--since compares against UTC. If you run scans late at night in another timezone, the cut-off may not be where you expect.
No savings on repeat scans yet. Reusing results for unchanged code is planned, not shipped.

What I would tell a team

Start with the api agent on one repo, so you get real token and cost numbers from day one.
Run the same repo through the agent your developers already use, and compare the reports side by side.
Put --max-cost in CI before you put the scan on every pull request.

Try it

pip install secfoo
secfoo run --skill sast --agent claude
secfoo serve

The repo is here: https://github.com/secfoo-com/secfoo. If it is useful to you, a star helps a small open-source project a lot, and issues telling us where the output is wrong help even more.

We are also part of the team organising a secfoo hackathon in Dublin in mid-November. If you would like to take part, or to help as a moderator, leave a comment and I will get back to you.