DEV Community: Samuel Rojas

Fixing one bug in my security tool would have hidden every alert in my test project

Samuel Rojas — Thu, 12 Nov 2026 23:53:40 +0000

I maintain depquiet. It sits on top of pip-audit, uv audit and npm audit and tries to cut down the noise. The one rule I care about is that it should never hide something real.

This week I found out how close I came to breaking it with what looked like a simple bug fix.

The setup

depquiet had an EPSS filter. EPSS is a score from FIRST.org that estimates how likely a CVE is to get exploited in the next 30 days. My defaults were:

under 10%: filter it out
under 30%: move it to "review"

I picked those numbers because they felt sensible. Most CVEs never get exploited, so why bug people about them?

Then I noticed the filter basically never ran on Python projects. EPSS only works with CVE ids, and pip-audit gives you ids like PYSEC-2023-74. uv audit gives you GHSA-.... The CVE is there, but as an alias, and depquiet wasn't looking at aliases. No CVE, no score, no filtering.

Easy fix, right? Look up the alias.

But that would turn the filter on for every Python user at once, so I stopped and thought about what it would actually throw away. I didn't like the answer, so I shipped the alias fix and also stopped EPSS from filtering by default. More on why below.

Then I ran it on real data

After the release I made a tiny project with three old pins:

requests==2.25.0
urllib3==1.26.4
jinja2==2.11.2

pip-audit found 22 vulnerabilities. With the alias fix, all 22 got an EPSS score. The highest was 4%. The lowest was 0.2%.

So if I had only fixed the alias and kept my 10% default, the filter would have thrown away all 22. Every single one. That list includes requests leaking your .netrc credentials and urllib3 forwarding Proxy-Authorization headers. The 30% rule would have pushed all of them down to "review" too.

I knew it would filter a lot. I didn't expect it to filter everything. The tool would have told me the project was fine.

What I got wrong

I was treating EPSS like it says something about my app. It doesn't. It tells you how likely a CVE is to be exploited somewhere on the internet. A 0.5% score means nobody is spraying that bug at scale right now. It doesn't tell you if your code passes credentials through a proxy you don't control.

Using that number to hide alerts means a global statistic gets to decide what you see about your own project. That's not what I want this tool to do.

What I changed

In v0.42:

EPSS doesn't filter anything by default anymore. It just sorts. Known exploited (CISA KEV) go first, then highest EPSS, then severity.
If you still want the filter, --epss-threshold 0.05 works. You just have to turn it on yourself.
Lookups use the CVE alias now, so Python and npm findings actually get scores.
Tiny scores show as 0.4% instead of 0%. Seeing "0%" next to a credential leak felt wrong. Going through the whole release afterwards also turned up two other things I'm not happy about. depquiet init was still writing the old 10% threshold into new config files. And if you passed a --kev-file that didn't exist, depquiet crashed with exit code 1, which is the same code it uses for "found vulnerabilities". So a CI job that tolerates findings would just pass, without ever checking KEV. Both are fixed in 0.42.1.

If you build tools like this

Run your filter on real data and look at what it throws away, not just what it keeps. My numbers looked fine on paper. On an actual project they would have hidden everything.

pip install depquiet, or use the GitHub Action samuelrojas-dev/depquiet@v0. If you think I'm wrong about EPSS, I'd like to hear why: github.com/samuelrojas-dev/depquiet

I audited my own security tool for every place it said "you're safe" when you weren't. I found 10.

Samuel Rojas — Thu, 12 Nov 2026 16:31:35 +0000

I maintain depquiet, an open-source (MIT) triage layer for pip-audit, uv audit and npm audit. It drops the alerts that only reach your dev tooling and tells you which of the rest you can actually fix.

A triage tool has one job it must never get wrong: it must not hide a real alert. So this week I went through every feature with one question: where can depquiet say "you're protected" or "this doesn't matter" when that isn't true?

I found 10 places. All are fixed in v0.40.4. Here they are, because if you build security tooling, you probably have some of the same.

1. `--fail-on high` let every Python alert through

pip-audit and uv audit report no severity. depquiet filled the gap with a placeholder, moderate, and then compared that placeholder against --fail-on high as if a scanner had measured it. On a real project (pallets/flask), two ACTION alerts: exit 1 without the flag, exit 0 with --fail-on high. And the README recommended --fail-on high.

Now an alert with no severity counts as meeting any threshold, with a note on stderr.

2. The same placeholder was shown everywhere

Even after the gate fix, "moderate" still appeared in the terminal, the Job Summary, the HTML report, the JSON, GitHub annotations, SARIF and the exported SBOM's CycloneDX ratings. SARIF marked those alerts as warning, so code scanning merge protection set to block on errors let them through.

Now it says unknown everywhere, and an ACTION alert with no severity is a SARIF error.

3. `--fix` "fixed" vulnerabilities by installing into the wrong environment

In a uv project scanned with pip-audit, --fix ran pip install pkg==fix with depquiet's own interpreter. The project's uv.lock did not change, and the report said ✅ Fixed.

Now it runs uv lock --upgrade-package and says what blocks the fix when uv refuses. Python projects without a lockfile are never pip-installed; you get the requirement to raise instead.

4. `--license` checked depquiet's environment, not your project

For Python, the license scan listed what pip list saw where depquiet runs. On flask it reported 187 packages, including playwright, pytest and depquiet itself. A GPL dependency of your project that was not installed next to depquiet was never read, and the check could PASS.

Now it reads your lockfile (78 packages on flask, matching its uv.lock) and each license from your own virtualenv or PyPI for that exact version. An unread license is UNKNOWN and fails.

5. `--kev` never matched uv audit findings

The CISA KEV catalog lists CVE ids. uv audit names advisories by GHSA id and pip-audit by PYSEC or GHSA, with the CVE as an alias, and depquiet dropped the aliases. So an actively exploited vulnerability in a dev-only package was never escalated.

Now aliases are kept and KEV matches any of them.

6. A comment turned a policy rule into "filter everything"

action: filter
match:
  severity: [low, info]  # noise only

The trailing comment broke the list, the severity condition was dropped, and the rule filtered a critical alert with EPSS 0.9. The same happened with a typo (severty:), an unreadable epss_below, and severity: medium, the value my own README documented (internally it is moderate).

Now comments are stripped, medium means moderate, and a rule with anything unreadable is skipped with a warning instead of widening.

7. Temporary suppressions became permanent

In .depquiet-ignore, expires: 2025-06-01 # temp, 2025/06/01 or 2025-13-01 was read as "no expiry". Now an unreadable date counts as expired.

8. `.depquiet.yaml` slips weakened the scan

kev: true # exploited turned KEV off. always_flag: [lodash] # core left lodash unprotected. epss_threshold: 5 (meant as 5%) was accepted, which filters every alert with an EPSS score. Now comments are stripped, EPSS must be between 0 and 1, and unknown keys are reported.

9. Fix versions from the wrong release line

Scanners list every release line that has the fix (0.3.81, 1.2.22). depquiet took the first one listed, which could be older than what you have installed. Now it takes the nearest fix above your version.

10. The PR comment cleanup deleted human comments

The GitHub Action removed any PR comment starting with "## 🔍 depquiet", whoever wrote it. Now it only removes its own.

And one number I had to correct

Two days ago I published that 54% of the fixes scanners suggest can't be installed. That was an upper bound: my first run checked only one fix per package. The exact per-alert figure, on 18 public uv projects, is 502 of 991 (51%). The more useful finding came with it: in 58 of the 72 blocked upgrades (81%), the blocker is a pin in the project's own pyproject.toml. "Update to X" usually really means "widen your own pin". Data, script and every resolver message.

The pattern

Almost every bug had the same shape: a missing or unreadable value was replaced by a default that sounded safe. "No severity" became "moderate". "Unreadable date" became "no expiry". "Unknown key" became "no condition". "Not installed here" became "not a dependency".

The fix was the same each time: when the tool doesn't know, it has to say so, and treat "unknown" as the risky case, never the safe one.

If you build security tooling, it's worth grepping for your defaults.

pip install depquiet, or the GitHub Action samuelrojas-dev/depquiet@v0. Issues and disagreements welcome: github.com/samuelrojas-dev/depquiet.