What does an AI security review actually cost? I measured it per run

Everyone asks whether AI agents can find security bugs. Fewer people ask the question a team lead asks five minutes later: what does it cost each time we run it?

I'm an intern at RakFort in Dublin, and for the last few weeks my job has been testing the cost tracking in secfoo, an open-source (MIT) CLI that gives AI coding agents a fixed security brief and a fixed report format. This post is what I found, including the parts that are not finished.

The problem with "just ask the agent"

You can point Claude Code, Cursor or Codex at a repo and ask for a security review. You will get something useful. You will not get:

• the same structure twice, so you cannot compare runs
• a record of what the run cost
• a way to stop a CI job that is burning money

secfoo handles the first with what it calls skills (structured briefs such as sast, threat-modeling and secret-scanning). The second and third are what I tested.

The setup

I used a deliberately small target: a 30-line Flask app with one SQL injection, where get_user() builds its query by joining strings. Small on purpose, so I could check the report by hand.

pip install "secfoo[api]"
export OPENAI_API_KEY=...
secfoo run --skill sast --agent api --target . --project-name demo --app-id ""

The api agent calls a model API directly, so you do not need an agent CLI installed. The last two flags skip the interactive prompts, which matters in scripts.

Here is the real output:

$ secfoo run --skill sast --agent api --target . --project-name demo --app-id ""
  success — SAST — Static Code Analysis
                                              Assessment results
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Skill                       ┃ Status  ┃ Duration ┃ Findings (C/H/M/L) ┃   Cost ┃ Run ID                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ SAST — Static Code Analysis │ success │ 11.2s    │ 0/1/0/0            │ <$0.01 │ 888f222e-7228-4b2b-b306-78db7a17d499 │
└─────────────────────────────┴─────────┴──────────┴────────────────────┴────────┴──────────────────────────────────────┘
AI spend for this run: <$0.01
Run secfoo serve to view full reports in your browser.

What one run cost

The scan finished in 11.2 seconds and reported one finding: the SQL injection in get_user(), which is the right answer for this app.

It used 2,183 input tokens and 981 output tokens, 3,164 in total. secfoo printed the spend for the run as under one cent. It shows <$0.01 rather than rounding down to zero, which I liked: a zero would suggest the run was free.

In my earlier testing, a repeat scan of an unchanged repo used exactly the same input tokens. That is expected today: nothing is cached between runs yet, so the second scan costs the same as the first.

Same kind of target, different agent

Earlier I ran a similar small app through Codex instead of the api agent:

secfoo run --skill sast --agent codex --target . --project-name demo --app-id ""

It finished in 27.5 seconds and also found the injection. Two things were different:

• Cost shows as -, not $0.00. Codex runs on a subscription login, so there is no per-run price to record. A dash is the right answer. A zero would be a lie.
• Tokens were much higher. Codex's own log reported 15,201 tokens for the run, roughly five times the api agent, because an agent CLI explores the repo with tools instead of receiving one prepared prompt.

That second point is the most useful thing I learned. The agent you pick can change the token count by a multiple, on a small target, for the same finding.

Seeing spend across runs

Per-run numbers are nice. The summary is what a team would actually use:

secfoo cost
secfoo cost --by skill --since 2026-09-01
secfoo cost --by agent --project demo

You can group by agent, skill or project. The same numbers appear in the local dashboard (secfoo serve), and in my tests the dashboard total matched the CLI total for the same runs.

$ secfoo cost --by agent --project demo
                   AI spend by agent
┏━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Agent ┃ Runs ┃ Input tokens ┃ Output tokens ┃   Cost ┃
┡━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ api   │    2 │        2,183 │           981 │ <$0.01 │
├───────┼──────┼──────────────┼───────────────┼────────┤
│ Total │    2 │        2,183 │           981 │ <$0.01 │
└───────┴──────┴──────────────┴───────────────┴────────┘
1 run(s) have no cost recorded -- their agent doesn't report it (cursor, agy), or they ran before cost tracking existed.

That table shows two runs, not one. My first attempt failed because I pasted my API key with a character missing. secfoo still counted it as a run and told me that one run had no cost recorded. I did not plan that, but it is a fair picture of how it behaves when something goes wrong.

Making cost a CI gate

This is the part I think matters most for real pipelines:

secfoo run --skill sast --agent api --target . \
  --fail-on high --max-cost 2.00 --json > secfoo-result.json

The exit code tells CI what happened: 0 means the run succeeded and no gate tripped, 1 means a run failed or timed out, and 2 means the run succeeded but a finding or the cost crossed your threshold. A security scan that can fail a build for being too expensive is not something I had seen before.

What is not there yet

I was testing, so I was looking for gaps. The honest list:

• Cost coverage depends on the agent. API-key runs give you tokens and cost. Subscription-based agent CLIs often give you neither, and you see a dash.
• Codex tokens are not recorded yet. Codex prints its total, but secfoo does not pick it up today. There is an open item for it.
• Failed runs are counted as runs. They show up in the totals with no cost recorded, as my mistyped key showed.
• Severity can differ between the CLI and the dashboard. In my run the CLI table counted the finding as High, and the dashboard labelled the same finding Critical. It is a known issue.
• --since compares against UTC. If you run scans late at night in another timezone, the cut-off may not be where you expect.
• No savings on repeat scans yet. Reusing results for unchanged code is planned, not shipped.

What I would tell a team

1. Start with the api agent on one repo, so you get real token and cost numbers from day one.
2. Run the same repo through the agent your developers already use, and compare the reports side by side.
3. Put --max-cost in CI before you put the scan on every pull request.

Try it

pip install secfoo
secfoo run --skill sast --agent claude
secfoo serve

The repo is here: https://github.com/secfoo-com/secfoo. If it is useful to you, a star helps a small open-source project a lot, and issues telling us where the output is wrong help even more.

We are also part of the team organising a secfoo hackathon in Dublin in mid-November. If you would like to take part, or to help as a moderator, leave a comment and I will get back to you.

Comments

[Reid Marlow]

The 5x token jump between the direct API call and the Codex CLI on a 30-line file is the real takeaway here. Tool-using agent loops explore aggressively, so on a monorepo that multiplier easily blows past 20x as the agent traverses unrelated directories. Setting --max-cost in CI works as a decent circuit breaker, but until scans are scoped to changed git diffs, running full agent passes per PR gets expensive fast.

[Varun Poojari]

Thanks Reid, that matches what I saw. I have only measured small targets so far, so I can't put a number on a monorepo yet, but the direction makes sense: the agent CLI explores with tools, while the api agent gets one prepared prompt.

Two things help today: --exclude keeps unrelated paths out of the scan and --max-cost is the circuit breaker you describe.

Scanning only what changed since the last run is planned but not shipped. I agree that is what makes per-PR scans affordable. A larger repo is the next thing I want to measure.

[Peter]

The exit-code design is the most interesting part, and it raises two questions. First, when --max-cost trips mid-run, does the result report the findings found so far, or is it marked as incomplete? A scan stopped by a budget and then reported as "success, 0 findings" reads as a pass, so a cost abort should come out as its own state, not the same exit code as "findings above threshold".

Second, cost per run only means something next to recall. One 30-line app with one planted SQLi shows both agents can find it, but the five-times token gap might be buying coverage on harder cases or might be pure exploration overhead. A small corpus of known vulns across varied shapes (string concatenation, ORM raw queries, second-order) would give you cost per true finding, which is the number a team lead needs.

[Varun Poojari]

Thanks Peter, both fair points.

On --max-cost: it never stops a run midway. An agent only reports spend when it finishes, so the limit is checked after the runs complete. That means a cost breach always comes with a full set of findings, not a partial one. A run that fails or times out is a separate case: it exits 1, and gate verdicts from a partial scan are not trusted.

You are right about the weak spot, though. Exit code 2 covers both "a finding at or above --fail-on" and "--max-cost exceeded", so a pipeline cannot tell them apart from the exit code alone. That deserves its own state and is worth an issue on the repo.

On recall: agreed. I measured one small app with one planted SQL injection, so I can say what a run cost but not what it bought. A small corpus of known vulnerabilities in different shapes, reported as cost per true finding, is the right next measurement. I have not run it yet.

[Peter]

That clears up the partial-scan worry, thanks. Since the check runs after spend has happened, it measures the damage but doesn't bound it. For the api agent you know the input token count before the call, so a pre-flight cap on prompt size can refuse before any spend. For agent CLIs, where spend is only known at the end, a wall-clock timeout is the only pre-emptive bound.

On the exit codes, a distinct code for the budget case (3, say) would let a pipeline treat "too expensive" as a config problem and "finding above threshold" as a security result.

[Varun Poojari]

Thanks Peter, that's a clearer way to put it: today --max-cost measures the spend after the fact rather than bounding it. For the agent CLIs, --timeout is the only pre-emptive bound I know of. A pre-flight cap on prompt size for the api agent sounds right, because the input tokens are known before the call. As far as I know, that doesn't exist today. A separate exit code for "budget exceeded" would also let a pipeline treat it as a config problem rather than a security result. I'll write both up as an issue on the repo so they don't get lost.

[Varun Poojari]

Peter, I've opened the issue with both suggestions: github.com/secfoo-com/secfoo/issue.... Thanks again for the detail on the exit code. If you have a view on whether the prompt-size cap should be a flag or a config value, feel free to add it there.

[Peter]

Both, with a precedence rule: a config value as the team default, a flag to override per run. In CI I'd treat the lower of the two as binding, so a flag can't silently raise a cap committed in the repo's config.

Fail before any spend, with the distinct budget exit code, and print which source set the effective value, so a pipeline log answers "why was this run refused".