DEV Community

Cover image for I pinned every image by digest. That froze 97 vulnerabilities in place.
samuel mbabhazi
samuel mbabhazi

Posted on

I pinned every image by digest. That froze 97 vulnerabilities in place.

Shipkeep is an open-source platform to deploy apps on your own servers. The control plane runs as a Docker Compose stack: the control plane image, PostgreSQL and Caddy.

Every image in that stack is pinned by digest. Every GitHub Action is pinned to a commit. CI runs CodeQL, Semgrep, gosec, gitleaks, and a Trivy scan of the repository. All green, every day.

This week I pointed Trivy at what an installation actually runs, the images, instead of at the repository. The PostgreSQL image alone had 97 fixable high or critical findings.

Pinning freezes the fixes out

Pinning by digest is the right call for a supply chain: a tag can be moved to anything, a digest cannot. But a digest also means the image never changes. Not when someone pushes something malicious to the tag, and not when Debian or the PostgreSQL project ships a fix.

The pin was set when the stack was written. The world kept patching. My compose file did not.

Here is what the first scan found, counting only vulnerabilities with a published fix (--ignore-unfixed, high and critical):

Image Findings What they were
postgres:17 (17.11) 97 62 distinct CVEs: 34 in PostgreSQL itself (reported once for the server package, once for the client), 26 in the Go standard library compiled into gosu, plus apt and zlib
node:22-bookworm-slim, base of the control plane image 10 all in the npm that ships with Node
caddy:2.11.9-alpine 1 golang.org/x/net, compiled into Caddy

The repository scan had nothing to say about any of this. It reads lockfiles. None of these packages is in a lockfile of mine.

The bot that was configured and never ran

The repository has a renovate.json. It pins Docker digests (docker:pinDigests), groups updates, and opens security updates at any time. It is a good config.

Renovate never opened a single pull request. The Renovate app had never been installed on the GitHub organization. A config file in a repository does nothing on its own, and nothing ever told me it was being ignored.

That is the whole story of how a pinned image ages: the pin works, the update mechanism is silently off.

What each finding turned into

PostgreSQL. The same major version had a new image: 17.12. Moving the pin removed all 34 PostgreSQL CVEs, apt and most of the rest: 97 findings went down to 27. I tested the upgrade on an installed server, not only in CI: data kept, migrations applied, an account created before the upgrade still signs in.

npm. The Node base image was already the newest one, so there was nothing to update. But nothing in the image runs npm: dependencies are installed with pnpm through corepack, and the app starts with node main.js. So npm is removed after the install:

RUN corepack enable \
  && corepack prepare "pnpm@${PNPM_VERSION}" --activate \
  && pnpm install --prod --frozen-lockfile \
  && rm -rf /root/.local/share/pnpm /root/.cache \
    /usr/local/lib/node_modules/npm /usr/local/bin/npm /usr/local/bin/npx
Enter fullscreen mode Exit fullscreen mode

10 findings, gone, and less to ship. The control plane image now scans clean: no fixable high or critical vulnerability and no secret, with no exception.

What nobody has fixed yet. The remaining 27 PostgreSQL findings and the Caddy one have no fixed image upstream. Debian has fixed zlib, but the official image has not picked it up. golang.org/x/net is fixed, but not in a Caddy release. The 26 Go findings live in gosu, a tiny binary that switches from root to the postgres user when the container starts and reads no input.

Those are accepted, but not forever. Trivy's ignore file supports an expiry date:

vulnerabilities:
  - id: CVE-2026-78661
    statement: >-
      golang.org/x/net in the Caddy binary (caddy 2.11.9). Fixed in x/net 0.62.0,
      not yet in a Caddy release.
    expired_at: 2026-12-18
Enter fullscreen mode Exit fullscreen mode

On December 18 the scan fails again, and someone has to look. I checked that it does, by moving a date to yesterday.

Scanning what ships, in CI

The image scans now run on every pull request and every week:

  • The control plane image is built from the branch and scanned for vulnerabilities and secrets, with no exception. That check is required to merge.
  • The PostgreSQL and Caddy images that compose.yml pins are scanned with the ignore file above. Those checks are not required, on purpose: a flaw disclosed upstream would otherwise block every pull request until a new image exists. The weekly run reports it instead.
  • The release workflow scans the pushed image, for amd64 and arm64, before it signs it. A finding stops the release before anything points to that image.

One thing the secret scanner flagged in the PostgreSQL image: a private key, /etc/ssl/private/ssl-cert-snakeoil.key, generated when the image is built. PostgreSQL TLS is off in this stack and the database only listens on the private Compose network, so third-party images are scanned for vulnerabilities only. The image I build is still scanned for secrets.

The same week, licenses

The project's rules allow MIT, Apache-2.0, BSD, ISC and MPL-2.0 for dependencies, with exceptions approved one by one. Nothing enforced that. So a new required job runs Trivy's license scanner on the production dependencies and checks the result against an allow-list.

Two things I did not know about Trivy's license scan:

  • It does not read licenses from lockfiles. For npm it reads node_modules, for Go the module cache. CI installs dependencies (--ignore-scripts, so no package code runs) and downloads the Go modules before scanning.
  • A package with no detected license simply has no entry. The check treats "no license found" as a failure, not a pass.

The first run found four production dependencies outside the list: tslib (0BSD), postgres and robust-predicates (Unlicense), and elkjs under EPL-2.0, a weak copyleft. That last one comes from Prisma Studio, which is part of the Prisma CLI, which ships in the image because migrations run with prisma migrate deploy. I would never have guessed that a migration tool brings a graph layout library into production. They are approved now, with the path that pulls each one in written down.

It also found two packages it could not classify, busboy and streamsearch. Both are MIT, declared in the old "licenses": [...] array of package.json, which Trivy does not read. They are listed as checked by hand.

The first diagnosis was wrong

The license job then failed in CI on one Go dependency, golang.org/x/text in the CLI module, while it passed on my machine.

The first explanation, written into a commit message, was that CI restored a Go module cache from other jobs. Reasonable, and wrong. Rerunning the job with an empty cache, the same Trivy version and the same commands failed the same way.

The real cause: the CLI's go.mod still named x/text v0.42.0, while the module it depends on needs v0.43.0, so every build used v0.43.0. Trivy reads the version from go.mod and looked for the license of a copy that no build ever downloads. go mod tidy was the fix. The license check had found a stale go.mod that the compiler never complained about.

The check now prints the version it looked for, which would have made that obvious in the first place.

Where it stands

  • Published prerelease: v0.1.0-alpha.2. It still uses the old PostgreSQL pin and asks for an SMTP account at install.
  • On main, not released yet: these scans and the updated images, and an installation that needs no email at all. The installer creates the first account on the server, and the sign-in screens stop offering sign-up and password reset when no mail can be sent.
  • Renovate is installed now. The first update pull requests should show up on Monday.

Where you can help

  • You have a disposable Ubuntu 24.04 VM and a domain name: follow the installation guide and tell me where it breaks. Nothing has run on a real server with a public certificate yet.
  • First contribution: the good first issues each say what to change and when it is done.

And a question for the comments: when did you last scan the images you deploy, rather than the code you wrote?

Top comments (0)