My last post was about a scanner that made up findings. This one is about something smaller and much more common: a tool showing you a number or label that is technically true and still sends you the wrong way.
I found three of these while testing secfoo, the open-source (MIT) CLI I work on at RakFort. It hands AI coding agents a fixed security brief and tracks their findings and cost. None of the three crashed anything or failed a test. Each one just told the user something confidently wrong.
1. "$0.00" when the answer was "we don't know"
secfoo tracks what each AI scan costs. Some agents report their spend and some don't. Codex, for example, doesn't give secfoo a price.
So I ran secfoo cost on a group of runs where no agent had reported a cost. The total said $0.00.
Those runs weren't free. Nobody knew what they cost. Those are very different answers if you're setting a budget, or if a CI check is going to fail a build when a scan costs too much.
The cause was one line of SQL. The summary query wrapped every sum like this:
-- before (simplified)
SELECT agent, COALESCE(SUM(cost_usd), 0) AS cost_usd
FROM runs GROUP BY agent;
-- after
SELECT agent, SUM(cost_usd) AS cost_usd
FROM runs GROUP BY agent;
In SQLite, SUM() already returns NULL when every value in the group is NULL. That NULL was the "unknown" signal, and COALESCE(..., 0) quietly turned it into zero.
The fix was to drop the COALESCE, make the cost fields optional (float | None), and show a dash for unknown. The CLI's Total row had the same bug one level up, so I fixed that too. The web dashboard was already doing this correctly, which is a good sign the CLI was the odd one out.
If you take one thing from this post: search your code for COALESCE(SUM( and ?? 0 and or 0 near anything money-related. Each one is a place where "unknown" might be turning into "free".
2. "binary_not_found" when the problem was a missing API key
secfoo's api agent calls a model through an API. It has no command-line tool to install.
When I ran it without an API key, the status said binary_not_found. That status means "the CLI isn't installed". So the error sent you off to install something that doesn't exist, when the real fix was to set one environment variable.
The cause: the agent correctly checked for an API key, but on failure it reused a status the other agents use for a missing program.
The fix was to add a new status, not_configured, and show it in both the CLI and the dashboard. It's a small change, but a status name is an instruction to the user. If it names the wrong problem, people lose time fixing the wrong thing.
3. Two dashboard panels that disagreed about the same bug
On the dashboard overview, one SQL injection showed up as High in "Findings by severity" and as Critical (CVSS 9.1) in "Top findings". Same project, same bug, two answers.
This one surprised me, because when I dug in, neither panel was wrong. They count different things:
| Panel | What it actually counts |
|---|---|
| Findings by severity | Severity counts read from each report's headings, added up across every run. Scan twice and the bug counts twice. |
| Top findings | Open Critical and High findings, de-duplicated, with code-scan findings ranked by their CVSS score. |
So the agent's report called it High, and the CVSS score put it in Critical. Both are fair readings.
Making both panels come from one source would mean redesigning how the chart counts. The ticket offered a cheaper option, and I took it: label what each panel counts. There's now a caption under each one, and a test checks that the captions show up.
It's not a glamorous fix, but a security dashboard that gives two answers with no explanation teaches people to trust neither.
The pattern
All three bugs had the same shape:
- The data underneath was fine.
- The display picked a confident word or number ("$0.00", "binary_not_found", "High") where the honest answer was "unknown", "not configured" or "depends what you're counting".
- Nothing failed, so no test caught it. I only found them by running the tool end to end and asking whether each label was true.
A checklist you can steal
When you review a security or cost dashboard, ask:
- Can this number be unknown? If so, does it show "unknown", or does it fall back to 0?
- Does every error status name the real problem? Read each one as an instruction. Would following it fix the issue?
- If two places show the same thing, do they agree? If not, does the screen say why?
- Does any test check the words? Both of my fixes added tests that check the label, not just the value.
Status
Both fixes are merged: the cost and status fixes in PR #30, and the severity labels in PR #33. The full test suite passes, apart from four older diagram tests that fail on any machine without a vendored JavaScript file, which is unrelated.
secfoo is open source: github.com/secfoo-com/secfoo. If it's useful, a star helps, and if you spot a label that lies, please open an issue.
What's the most misleading "0" you've seen in a tool you rely on?
Top comments (1)
That distinction between zero and unknown is a schema contract as much as a UI choice. Preserving NULL through aggregation and rendering “unknown” prevents cost caps or comparisons from treating missing provider data as free; tests for the exact label are smart because output semantics matter. I’d also expose the fraction of runs that reported spend alongside the total.