DEV Community

Cover image for Trivy Image Scanning: The CI Breaking Threshold
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

Trivy Image Scanning: The CI Breaking Threshold

Wiring a container image scanner into CI really is a ten-minute job. You add five lines of YAML, a colourful table appears on the first run, and everyone agrees you now have a security gate. The hard part is deciding where in that table the build should break. Skip that decision and you land at one of two extremes: either the scanner stops nothing and you ship vulnerable images under a green check for years, or the pipeline breaks on every run and within three weeks the team finds a way around the scan. Both end in the same place — a report nobody reads.

I have argued the break-or-warn question before, in the abstract, around dependency security; this piece is that argument at tool level, down to which flag filters which finding. It uses Trivy to work through how the threshold should be designed. I ran every behaviour described here myself, with version 0.74.0, while writing: the numbers below come from a real scan performed on 26 August 2026, not from an illustration.

Start With This Table

I took a popular base image, python:3.11-slim, and scanned it with the vulnerability scanner only:

trivy image --scanners vuln --format json -o py.json python:3.11-slim
Enter fullscreen mode Exit fullscreen mode

The image I scanned has digest sha256:be1575ed968de893bd54f4c56315ff7c4736ce522c1bca08fd521731aafc0d76, and the distribution underneath is Debian 13.6. The resulting JSON held 160 findings, distributed like this:

Severity Findings With a fix available
CRITICAL 3 0
HIGH 18 5
MEDIUM 54 8
LOW 61 4
UNKNOWN 24 21

It is worth sitting with this table for a moment, because almost everything you need to know about CI thresholds is hiding in it.

The industry's reflex threshold is "break on HIGH and CRITICAL". Apply it here and the image fails on 21 findings. But only five of those 21 have a published fix. For the remaining 16 there is nothing you can do that morning: the package maintainer has not shipped a patch. More awkward still, all three CRITICAL findings belong to perl-base and none of them is fixable — two are affected, one is fix_deferred. The three loudest alarms in the report are, by definition, the ones you cannot close today.

The five "fixable" findings are also less crowded than they look. Three of them are the same CVE (CVE-2026-14456) counted across three OpenSSL packages; the other two sit on the Python side. Where exactly those two live is instructive in its own right:

usr/local/lib/python3.11/site-packages/setuptools/_vendor/wheel-0.45.1.dist-info/METADATA
usr/local/lib/python3.11/site-packages/setuptools/_vendor/jaraco.context-5.3.0.dist-info/METADATA
Enter fullscreen mode Exit fullscreen mode

Neither finding is on a package installed into the image directly — both are copies vendored inside setuptools. You will not see those versions in pip list, and pip install --upgrade wheel will not fix them; fixing them requires setuptools itself to ship a new release. The cost and the benefit of a scanner that reads the filesystem rather than asking the package manager sit in the same line: you see what is actually there, but working out who owns it is your job. So underneath a 160-line wall of alarm there are three distinct problems that anyone could actually act on. Counting findings and counting work are not the same thing, and dashboards usually count findings.

The Quiet Trap: The Scanner Does Not Break Builds by Default

Now for the real trap. I scanned the same image filtered to CRITICAL only:

trivy image --scanners vuln --severity CRITICAL python:3.11-slim
echo $?   # 0
Enter fullscreen mode Exit fullscreen mode

Three CRITICAL findings on screen, and the command exits zero. To CI, that reads as success. Trivy's CLI reference defines the flag plainly: --exit-code "specify exit code when any security issues are found" — meaning that unless you set it, Trivy prints the findings and exits happily with zero. This is not a bug; scanning and gating are deliberately separated. But the consequence is real: a pipeline whose author forgot --exit-code will show a green check for months while shipping a red image.

I felt slightly embarrassed the first time I saw this, because I remembered writing a "security step" that behaved exactly that way. We had installed the door and forgotten the lock.

The locked version:

trivy image --scanners vuln --severity HIGH,CRITICAL --exit-code 1 python:3.11-slim
echo $?   # 1
Enter fullscreen mode Exit fullscreen mode

What Does --ignore-unfixed Actually Ignore?

The flag that turns that 21-finding failure into something actionable is --ignore-unfixed. The documentation describes it as a shorthand: it is equivalent to --ignore-status affected,will_not_fix,fix_deferred,end_of_life, so only vulnerabilities with a published fix are displayed. On the same image the 21 dropped to 5 and the build still failed — but this time what I was asking of the team was concrete: bump OpenSSL to the next patch, update two Python packages.

The trade-off deserves to be stated honestly. --ignore-unfixed does not mean "an unpatched vulnerability is harmless"; it means "an unpatched vulnerability is not a reason to break a build". Those three CRITICAL issues in perl-base are still in the image; silencing the scanner does not change that. Closing them is not a CI job — it means changing the base image, removing the package, or accepting the risk on the record. If it were me, I would write down which of the three I chose, because the person asking "why is perl still here" six months from now is usually me.

Whose Severity Is It?

The moment you build a threshold on severity, an implicit question appears: who decided this was HIGH?

Trivy's answer is: not NVD. The documentation states plainly that it prefers the vendor's rating — "the severity from vendors is more accurate" — and gives CVE-2023-0464 as an example: HIGH in NVD, marked Low by Red Hat, shown as Low by Trivy. The logic is sound, since the party that knows the compile options and the default configuration is the party doing the packaging. When no vendor rating exists, Trivy falls back to a CVSS mapping (Low 0.1-3.9, Medium 4.0-6.9, High 7.0-8.9, Critical 9.0-10.0), then to NVD, then to UNKNOWN.

That is also why 24 findings in the table are UNKNOWN. And 21 of those UNKNOWNs have a fix available — so the bucket you excluded from the gate for being unrated may hold most of the work that is actually closeable. A severity filter tells you what to deal with first; it does not tell you what is genuinely dangerous.

The same warning is written down on the NVD side: CVSS base metrics describe the innate characteristics of a vulnerability, not its impact in your environment. A CRITICAL can be practically irrelevant in a cron container with no inbound network; a MEDIUM can be catastrophic on the authentication path of your only internet-facing service. Rather than making severity the sole axis, adding evidence of exploitation as a second axis helps: a CVE listed in CISA's Known Exploited Vulnerabilities catalogue sits in a different box for me regardless of its score. The detail of that prioritisation was the subject of a separate piece.

The Threshold Is Not One Number

Reducing the breaking threshold to a single severity level is a common simplification, but the scanner hands you several different kinds of information at once, and they do not all belong at the same gate.

The first place I noticed this was the default value of --scanners: vuln,secret. When you scan an image, leaked secrets are searched for by default alongside vulnerabilities, while misconfiguration (misconfig) and licence (license) scanning are off by default. It reads like a footnote, but it matters for threshold design: an AWS key that slipped into the image and an unfixable perl-base CVE do not belong in the same bucket. The first is an unarguable stop; the second is a calendar item. I keep secret findings in a separate step that breaks unconditionally, independent of the severity filter:

trivy image --scanners secret --exit-code 1 ghcr.io/org/app:$GITHUB_SHA
Enter fullscreen mode Exit fullscreen mode

Similarly, the default for --pkg-types is os,library — both operating system packages and application dependencies are scanned. That is why OpenSSL and wheel appear side by side in the table above. When you build the gate, note that in most teams those two have different owners: OS packages are fixed by upgrading the base image, libraries by updating application dependencies. You can apply one threshold to both, but if you do not separate who performs the fix, findings will keep living in the gap between them.

The third axis, and the most frequently skipped, has nothing to do with severity: the lifetime of the base image. Trivy has a separate --exit-on-eol flag, documented as exiting with the specified code when the OS reaches end of service or life. I consider that a more valuable signal than the CVE threshold, because seeing no findings on an end-of-life distribution is not good news: the vendor has stopped publishing advisories, so the data source has gone quiet. A silent table and a clean table are not the same thing, and the EOL check is the only thing separating them.

There is also the question of branches. Applying the same threshold to pull requests and to the main branch looks consistent at first, but in practice it holds the author of a PR responsible for a finding they did not introduce. If it were me, I would keep the PR run narrow — only HIGH and CRITICAL with a fix — and scan the full inventory in a separate weekly scheduled run. That way the gate does not block a developer, and accumulating debt does not become invisible either.

The Two-Stage Run

Most threshold arguments come from expecting one command to produce both a complete report and a narrow gate. Separate those two jobs and the argument largely dissolves. The pattern recommended in the README of Trivy's own GitHub Action is exactly this: first a run that reports everything and breaks nothing, then a narrowly filtered run that breaks.

Diagram

The GitHub Actions equivalent looks roughly like this:

- name: Inventory (does not break)
  uses: aquasecurity/trivy-action@v0.36.0
  with:
    scan-type: image
    image-ref: ghcr.io/org/app:${{ github.sha }}
    format: sarif
    output: trivy-results.sarif
    exit-code: '0'

- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v4
  with:
    sarif_file: trivy-results.sarif

- name: Gate (breaks)
  uses: aquasecurity/trivy-action@v0.36.0
  with:
    scan-type: image
    image-ref: ghcr.io/org/app:${{ github.sha }}
    severity: HIGH,CRITICAL
    ignore-unfixed: true
    exit-code: '1'
    skip-setup-trivy: true
Enter fullscreen mode Exit fullscreen mode

The skip-setup-trivy: true on the second call is a small but pleasant detail: the action installs Trivy on the first call, so there is no point repeating the install. There is a similar saving on the database side — caching is on by default in the action and the README recommends leaving it on, not primarily for speed but to avoid registry rate limits. Trivy pulls its database from mirror.gcr.io/aquasec and then ghcr.io/aquasecurity; for closed environments, --db-repository accepts multiple addresses so you can point it at your own mirror registry.

This split is the natural continuation of accepting images by component inventory rather than by CVE list alone, which I covered in an SBOM-based image admission gate with Syft and Grype. Reducing false-positive load with VEX was the subject of another piece; in Trivy the --vex flag is still marked experimental, so think twice before putting it at the centre of your gate logic.

Same Commit, Red Tomorrow

The first complaint you will get after installing this gate is almost certainly: "I changed nothing, it was green yesterday."

And they are right. The vulnerability database updates continuously; when an advisory that did not exist yesterday is published today, the same image, same commit and same Dockerfile turn red. That is not a malfunction, it is what the gate is: you are measuring not the code but the currently known state of the code. Fail to tell developers this in advance, though, and the gate looks arbitrary — and arbitrary gates get switched off.

Three adjustments make it liveable in practice. First, freeze the database for pull request runs: with --skip-db-update the run uses the cached database and the refresh moves to a separate scheduled job, so every PR opened that day is measured against the same data. Second, move the surprise: refresh the database in a nightly run that raises a notification in the morning, and a new finding lands in front of the team rather than in front of whoever opened a PR. Third, write the distinction down — when the thing that broke the gate is "the world changed" rather than "your change", it should be obvious whose job that is.

There is also a neat trap sitting in the same area. SARIF output does not care about your severity filter: per the trivy-action documentation, the SARIF format by default enforces output of all vulnerabilities regardless of the configured severities, and changing that requires limit-severities-for-sarif: true (default false). In the two-stage run that is exactly what you want — let the inventory stage see everything. But reverse the two stages and upload the gate step's SARIF, and the code scanning tab fills with a list far wider than your narrow filter, and you will spend half a day working out where it came from.

Exceptions: A Promise With an Expiry Date

Every gate produces exceptions. What matters is that they do not become permanent.

Trivy's .trivyignore file accepts one finding ID per line and supports expiry dates in the form CVE-2019-14697 exp:2023-01-01. The tidier YAML format supports id, paths, purls, expired_at and statement fields. In fairness, that format is still marked experimental in the documentation, which is exactly why you have to pass its path explicitly with --ignorefile; the default filename remains the plain-text .trivyignore. Having just said "think twice" about --vex, I owe the same standard here: I do use the YAML format, but I keep a step that tests this behaviour whenever I bump the Trivy version.

To see whether this really works I prepared two files and rescanned the same image. The one with a valid date:

vulnerabilities:
  - id: CVE-2026-14456
    statement: "OpenSSL update lands with the next base image bump"
    expired_at: 2026-12-31
Enter fullscreen mode Exit fullscreen mode

With that file the OpenSSL findings dropped out of the table, leaving only the two Python ones. When I moved the same entry's date into the past (2026-01-31), the findings came straight back. So expired_at is not decorative: an expired exception does not quietly disappear, it closes the gate again.

The statement field is not used for filtering — it is there purely for humans. Even so, I think it is the most important line in the file. If the answer to "why are we ignoring this CVE" is not written down in the repository, that exception becomes folklore nobody dares touch within six months.

The Scanner's Own Supply Chain

There is one subject I cannot skip here, because it happened to the very tool you install to protect your supply chain.

The Trivy ecosystem was compromised on 19 March 2026. According to the project's own security advisory (GHSA-69fq-xp46-6x23, CVE-2026-33634), an attacker used stolen credentials to publish a malicious v0.69.4 release and force-pushed the version tags in the aquasecurity/trivy-action repository to credential-stealing code. The tags stayed malicious for roughly 12 hours. In the same wave all seven tags in aquasecurity/setup-trivy were replaced too — a detail that matters, because trivy-action calls that action internally to install Trivy. So even if you pin the one action in your own workflow to a SHA, that action may be pulling a second one you never pinned; installing Trivy yourself and passing skip-setup-trivy: true is one way to shorten the chain. The injected payload pulled secrets out of the GitHub Actions runner process memory (/proc/<pid>/mem), swept more than 50 filesystem paths for everything from SSH keys to cloud credentials, and shipped what it collected to attacker infrastructure. A few days later a second wave targeted the DockerHub images.

The painful lesson is not quite in that first paragraph: the problem was that tags are mutable. A workflow pinned to @v0.34.0 is not guaranteed to run the commit that tag pointed at yesterday. The advisory's own guidance says as much — pin actions to full, immutable commit SHA hashes rather than mutable version tags. In this blog's own repository every action, actions/checkout included, is pinned to a full SHA; I used tags in the YAML above for readability, but write SHAs in production. I covered the reasoning and the mechanics in pinning GitHub Actions to commit SHAs. That piece argued repository takeover as an abstract threat scenario; the March incident is the same scenario, actually carried out.

The second lesson is more uncomfortable: your scanning gate is a dependency too. If you never ask "what happens when this tool is compromised" while installing it, the thing you added as a security control is really a new attack surface with privileged access.

A Decision Framework

When I add a gate to a new repository, I work through these in order:

  • What breaks the build? My default is --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1. On internet-facing services I add MEDIUM to the breaking set, but only together with --ignore-unfixed.
  • What gets reported? Everything. The non-breaking run uploads the full inventory as SARIF; findings outside the threshold are not lost, they just do not stop the build.
  • Who watches the unfixable CRITICALs? Since the gate does not see them, someone has to. A base-image upgrade schedule is where that belongs.
  • Does every exception carry a date? Without expired_at it is not an exception, it is a silent policy change.
  • Is the scanner pinned? Action at a full SHA, Trivy version stated explicitly, database cache enabled.
  • How often does the gate break? If it breaks a few times a month and each break ends in a real fix, the threshold is right. If it breaks on every PR, the threshold is wrong — and if you do not fix it, the team will neutralise it by their own means.

Closing

A breaking threshold looks like a technical setting, but it is really a commitment. When you write --severity HIGH,CRITICAL you are promising your team that a finding at that level will stop the work until it is fixed. Choose a threshold you cannot honour and the system breaks the promise for you: the ignore file swells, continue-on-error lines appear, and eventually someone removes the step altogether.

So pick the threshold by asking "which finding will actually make me stop", not "how secure would I like to be". The 160 the tool showed is not asking you to be frightened, it is just counting the inventory. The decision is yours: which line of that inventory changes a human being's day?

Official Sources

Top comments (0)