Nikolay Kuziev

Posted on May 7

Why Fixed Container Image Versions Matter: Lessons from the Trivy Supply Chain Attack

#security #cicd #docker #vulnerabilities

A CI/CD pipeline is only as trustworthy as the code and tools it pulls during execution.

That sounds obvious, but it is easy to forget.

Most supply chain conversations start with application dependencies: Maven artifacts, Gradle dependencies, npm packages, base images, operating system packages, and third-party libraries.

But CI/CD tools are dependencies too.

The image used by a pipeline job is a dependency. The scanner image used in a security stage is a dependency. The GitHub Action used to scan containers is a dependency. The script downloaded with curl | sh is a dependency. The latest tag is also a dependency, but one that can change without a pull request.

That is the part I want to focus on here.

The Trivy supply chain incident is a useful case study because Trivy is not a random tool. It is a widely used security scanner. Many teams run it in CI/CD specifically to improve supply chain security.

That is the uncomfortable lesson:

a security scanner can also become a supply chain dependency

This does not mean "do not use Trivy".

It does not mean "security scanners are bad".

It means we need to treat CI/CD tooling with the same discipline we expect from application dependencies.

The minimum practical step is simple:

do not use floating image tags for security-critical CI/CD tooling

Use fixed versions. For stronger integrity, use image digests. For GitHub Actions, use full commit SHAs.

What happened with Trivy

In March 2026, Aqua Security published an advisory for a Trivy ecosystem supply chain compromise.

According to the Aqua Security advisory, on March 19, 2026, a threat actor used compromised credentials to publish a malicious Trivy v0.69.4 release, force-push most aquasecurity/trivy-action version tags to credential-stealing malware, and replace aquasecurity/setup-trivy tags with malicious commits.

The same advisory says that on March 22, 2026, malicious Docker Hub images were published for Trivy v0.69.5 and v0.69.6.

Docker's write-up adds the image-consumer view. Docker reported that Docker Hub customers may have been affected if they pulled aquasec/trivy with tags 0.69.4, 0.69.5, 0.69.6, or latest during the affected window between March 19 and March 23, 2026. Docker also described the latest tag being re-pointed during the incident.

That is exactly why mutable tags matter.

If a pipeline used this:

image: aquasec/trivy:latest

then the pipeline was not really saying:

use the Trivy version I reviewed

It was saying:

use whatever the latest tag points to at runtime

That difference is small in YAML and huge in security.

The problem is not only latest

The obvious bad example is latest.

But the deeper problem is mutability.

Docker's own build best practices explain that image tags are mutable: a publisher can update a tag so it points to a different image later. Docker also notes the downside for consumers: the same build can use different image content over time, and the exact version used becomes harder to audit.

So this is risky:

image: aquasec/trivy:latest

But this is not perfect either:

image: aquasec/trivy:0.69.3

It is better because the intended version is visible. It narrows the update surface. It makes the pipeline more readable.

But it is still a tag.

A tag can move.

The stronger form is:

image: aquasec/trivy:0.69.3@sha256:<pinned-image-digest>

The tag keeps the file human-readable. The digest pins the image content.

That is the practical difference:

tag:
  human-readable pointer that can change

digest:
  content identifier for a specific image

For security-sensitive CI/CD jobs, that difference matters.

The task: controlled updates instead of silent updates

The task is not "never update images".

That would create a different security problem.

The task is:

use controlled updates instead of silent updates

I want CI/CD tools to change through reviewable pull requests, not by silently pulling a changed tag during a pipeline run.

That applies to:

CI job images;
scanner images;
Dockerfile base images;
GitHub Actions;
Kubernetes workload images;
sidecar containers;
init containers;
build containers;
release tooling.

The goal is to make toolchain changes visible.

If Trivy changes from one version to another, I want that change to appear in Git history. If a digest changes, I want a pull request. If a scanner is upgraded, I want the pipeline diff to show what changed.

Not because every update must be slow.

Because invisible updates are hard to audit during an incident.

Why CI/CD images are high-trust dependencies

A scanner image running in CI/CD is not just another container.

It often has access to the repository checkout. It may read dependency files, Dockerfiles, Kubernetes manifests, lockfiles, SBOMs, source code, and build outputs.

Depending on the pipeline design, it may also run in an environment where secrets exist.

GitHub's security guidance warns that a compromised action inside a workflow can be significant because actions may access repository secrets and use the GITHUB_TOKEN. The same trust model applies to containerized CI tools.

If a CI job pulls a scanner image and runs it inside the pipeline, that image becomes part of the trusted execution path.

That is why I do not like treating CI tool versions casually.

A scanner is still code.

A scanner image is still a dependency.

A scanner action is still third-party execution inside a trusted pipeline.

The bad pattern

A common GitLab CI job might look like this:

trivy:image:
  image: aquasec/trivy:latest
  stage: security
  script:
    - trivy image "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

This is simple, but it has two problems.

First, the version is not controlled.

The pipeline may use different Trivy image content tomorrow without any change in the repository.

Second, incident response becomes harder.

If someone asks this:

Which exact Trivy image did this pipeline use last Tuesday?

the Git repository alone cannot answer it. The team may need registry logs, runner logs, caches, timestamps, and a lot of luck.

That is a weak audit trail.

A better pattern: fixed version

A better first step is to use a fixed version tag:

trivy:image:
  image: aquasec/trivy:0.69.3
  stage: security
  script:
    - trivy image "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

This is already better than latest.

It makes the intended version visible in Git. It reduces surprise. It makes updates explicit:

- image: aquasec/trivy:0.69.3
+ image: aquasec/trivy:0.70.0

For many teams, fixed tags are a practical starting point. They are readable and easy to adopt.

But for security-critical CI/CD tools, I would treat fixed tags as the minimum, not the end state.

The stronger pattern: version plus digest

The stronger pattern is to use both a readable version and a digest:

trivy:image:
  image: aquasec/trivy:0.69.3@sha256:<pinned-image-digest>
  stage: security
  script:
    - trivy image "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

This gives two benefits.

The tag keeps the file readable:

aquasec/trivy:0.69.3

The digest pins the content:

sha256:<pinned-image-digest>

Now the repository says:

use this exact image content

If the tag moves later, the digest still points to the committed image content.

Docker's documentation makes the same point for base images: pinning an image to a digest helps ensure the same image is used even if the publisher updates the tag. Docker also notes the tradeoff: pinned digests require an update workflow so teams do not miss security fixes.

That tradeoff is real.

Digest pinning gives control, but it also creates responsibility.

You need a process to update pins.

What the Trivy incident teaches here

The Aqua advisory explicitly says users were not affected if they used Trivy images referenced by digest.

That does not mean digest pinning solves every possible supply chain attack.

It means digest pinning changes the failure mode.

With a floating tag, your pipeline may silently start using newly pushed content.

With a pinned digest, your pipeline keeps using the exact content that was reviewed and committed.

That is a huge difference during incident response.

When something like this happens, teams need answers:

Did we pull the affected image?
Which pipelines used it?
Which version did we use?
Which digest did we use?
Which secrets existed in those jobs?
Which runners executed those jobs?
Do we need to rotate credentials?

Pinned versions and digests make those questions easier to answer.

Floating tags make them harder.

But pinned images get old

This is the strongest argument against pinning.

And it is not wrong.

If a team pins an image and forgets about it for two years, that is also bad.

Pinning is not supposed to mean:

never update this dependency

It means:

do not update this dependency invisibly

There is a big difference.

The workflow should look more like this:

pin image
scan image
run pipeline
create automated update pull requests
review digest changes
merge updates regularly
keep the audit trail

Docker's docs describe the same tradeoff: digest pinning improves reproducibility, but teams should have tooling that detects outdated pins and raises pull requests for controlled updates.

So the real rule is not:

pin and forget

The real rule is:

pin and update intentionally

CI/CD should not change without Git history

This is the main engineering point for me.

A pipeline should be reproducible enough that I can look at a commit and understand the tools that ran for that commit.

If the pipeline uses:

image: aquasec/trivy:latest

then the same commit can run with different Trivy contents on different days.

That is not reproducible.

If the pipeline uses:

image: aquasec/trivy:0.69.3@sha256:<pinned-image-digest>

then the tool is part of the commit state.

A change requires a diff. A diff can be reviewed. A review can be audited.

That is what I want from security tooling.

Not because every update needs a meeting.

Because CI/CD is part of the software supply chain, and supply chain changes should leave a trail.

Dockerfile base images have the same problem

This is not only about scanner images.

Base images have the same issue.

A Dockerfile like this is convenient:

FROM eclipse-temurin:17

But eclipse-temurin:17 is still a moving tag.

A more controlled version is:

FROM eclipse-temurin:17.0.11_9-jre

A stronger version is:

FROM eclipse-temurin:17.0.11_9-jre@sha256:<pinned-image-digest>

The same idea applies:

floating major tag:
  easy, but changes silently

specific version tag:
  better, but still mutable

version plus digest:
  stronger and more reproducible

For production images, I prefer the third form when the team has automation to update it.

Without update automation, teams often resist digests because they feel manual and annoying.

That is fair.

The solution is not to abandon digest pinning.

The solution is to automate digest updates through reviewable pull requests.

GitHub Actions need SHA pinning

Container images are pinned by digest.

GitHub Actions are pinned by full commit SHA.

This is the same principle in another format.

Weak:

- uses: aquasecurity/trivy-action@master

Better:

- uses: aquasecurity/trivy-action@0.35.0

Stronger:

- uses: aquasecurity/trivy-action@<full-commit-sha>

GitHub's own documentation says pinning an action to a full-length commit SHA is the only way to use an action as an immutable release. GitHub also provides repository and organization policies that can require full SHA pinning.

This matters because the Trivy incident affected both container images and GitHub Actions.

So the policy should cover both:

container image:
  pin by digest

GitHub Action:
  pin by full commit SHA

What I would standardize in CI/CD

If I were standardizing this across repositories, I would start with a small policy.

Not a huge platform rewrite.

Just a clear baseline:

no :latest in CI/CD jobs;
security scanner images must use fixed version tags;
high-trust CI/CD tooling should be pinned by digest;
GitHub Actions should be pinned by full commit SHA;
Dockerfile base images should use specific versions and preferably digests;
updates should happen through pull requests, not silent tag movement;
CI/CD should fail when new floating tags are introduced.

This is small enough to adopt.

But it changes the trust model.

The pipeline stops saying:

pull whatever this tag means today

and starts saying:

use this reviewed version of this tool

Example: GitLab CI with pinned Trivy image

A simple version:

variables:
  TRIVY_IMAGE: "aquasec/trivy:0.69.3"

trivy:image:
  image: "$TRIVY_IMAGE"
  stage: security
  script:
    - trivy image "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

A stronger version:

variables:
  TRIVY_IMAGE: "aquasec/trivy:0.69.3@sha256:<pinned-image-digest>"

trivy:image:
  image: "$TRIVY_IMAGE"
  stage: security
  script:
    - trivy image "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"

I prefer keeping the image reference in a variable.

That makes updates easier to review:

 variables:
-  TRIVY_IMAGE: "aquasec/trivy:0.69.3@sha256:<old-digest>"
+  TRIVY_IMAGE: "aquasec/trivy:0.70.0@sha256:<new-digest>"

The diff clearly shows that the scanner changed.

That is exactly what I want.

Example: Docker run with digest

Some projects run Trivy through Docker directly:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  aquasec/trivy:latest \
  image my-service:local

I would avoid this in CI.

A better version:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  aquasec/trivy:0.69.3@sha256:<pinned-image-digest> \
  image my-service:local

The important part is not the exact command.

The important part is that the tool image is not resolved through a moving tag.

Example: Kubernetes workloads

The same problem exists in Kubernetes manifests.

Weak:

containers:
  - name: api
    image: registry.example.com/payment-api:latest

Better:

containers:
  - name: api
    image: registry.example.com/payment-api:1.14.2

Stronger:

containers:
  - name: api
    image: registry.example.com/payment-api:1.14.2@sha256:<pinned-image-digest>

For production Kubernetes workloads, I would also want policy enforcement:

reject :latest;
require image digests in production namespaces;
allow version tags in development namespaces;
require images from approved registries;
require signed images for critical workloads.

This is where Kyverno, OPA/Gatekeeper, admission controllers, and registry policies become useful.

But the first step is still cultural:

stop treating image tags as immutable versions

They are not.

How to detect the bad pattern

A simple first check can be basic.

For GitLab CI:

grep -R "image: .*:latest" .gitlab-ci.yml .gitlab-ci/ || true

For Dockerfiles:

grep -R "FROM .*:latest" Dockerfile* . || true

For Kubernetes manifests:

grep -R "image: .*:latest" k8s/ deploy/ helm/ || true

This is not a perfect policy engine.

It will miss some cases and produce false positives.

But it is a cheap starting point.

Later, a team can replace this with proper policy-as-code:

Semgrep rules for CI files and Dockerfiles;
OPA/Rego policies;
Kyverno policies;
GitLab or GitHub required checks;
admission control in Kubernetes;
registry allowlists and blocklists.

The practical rollout path is:

detect first
warn next
block later

Trying to block everything on day one usually creates too much friction.

What I would block immediately

There are a few patterns I would block immediately in production or security-sensitive CI/CD:

image: something:latest
docker run something:latest
FROM something:latest
uses: owner/action@master
uses: owner/action@main
curl ... | sh without version or checksum

These patterns make incident response harder.

They also make reproducibility weaker.

A pipeline should not depend on a moving target when it handles source code, secrets, cloud credentials, releases, or production artifacts.

What I would allow temporarily

There are cases where strict digest pinning may be too much at the beginning.

For example:

local experiments;
throwaway sandbox repositories;
early proof-of-concepts;
non-sensitive demo pipelines;
temporary development branches.

Even there, I would avoid latest when possible.

But security work is about risk and rollout.

The strongest policy should start where the risk is highest:

production deploy pipelines;
release pipelines;
cloud credential access;
container publishing jobs;
security scanning jobs;
jobs with secrets;
jobs that run on protected branches.

This makes adoption more realistic.

What pinning does not solve

Pinning is useful, but it is not magic.

It does not protect you if you pin a malicious digest.

It does not replace scanning.

It does not replace signature verification.

It does not replace least-privilege CI/CD tokens.

It does not replace secret rotation.

It does not replace egress controls.

It does not replace runner isolation.

It does not replace review of third-party tools.

It does not guarantee that a previously trusted image stays safe forever.

Pinning solves one specific problem:

it prevents silent movement from reviewed content to different content

That is valuable, but it is only one layer.

The bigger CI/CD hardening picture

The Trivy incident is not only a lesson about tags.

It is a lesson about how much trust we put into CI/CD.

A stronger pipeline should combine multiple controls:

pin images by digest;
pin GitHub Actions by full commit SHA;
use least-privilege tokens;
avoid long-lived credentials;
use OIDC for cloud access where possible;
restrict egress from runners;
separate trusted and untrusted workflows;
avoid running untrusted code with secrets;
rotate credentials after suspicious execution;
keep audit logs;
use short-lived runners for sensitive jobs;
scan artifacts before deployment.

Aqua's advisory also includes remediation themes such as rotating exposed secrets, auditing Trivy versions, auditing action references, and checking for exfiltration artifacts.

That is the broader lesson.

Pinning helps reduce unexpected dependency movement.

But CI/CD hardening also needs identity, isolation, permissions, monitoring, and incident response.

How this connects to my build tooling

In my Java secure build tooling articles, the main problem was security build drift.

The Gradle plugin article focuses on moving SonarQube, Dependency-Check, CycloneDX, and coverage conventions into the build system instead of duplicating scanner logic across CI/CD YAML.

The Maven extension article focuses on making normal Maven lifecycle commands security-aware so the build extension owns the security wiring and CI/CD stays smaller.

This article is about a different layer.

Even if the build system owns the security workflow, CI/CD still runs inside images.

Those images are part of the trust boundary.

For example, a GitLab CI job might use:

image: eclipse-temurin:17

That is readable and convenient, but it is still a tag.

For a stronger supply chain posture, the same principle applies:

image: eclipse-temurin:17@sha256:<pinned-image-digest>

The build system can reduce security configuration drift.

Pinned images reduce toolchain drift.

Both ideas point in the same direction:

security behavior should be explicit, versioned, and reviewable

Fixed versions vs fixed behavior

This is the part I think matters most.

Using fixed image versions is not only about security.

It is also about behavior.

If a scanner changes silently, findings can change silently.

If a base image changes silently, builds can change silently.

If a CI action changes silently, pipeline behavior can change silently.

That creates confusion:

Was this new finding caused by new code?
Was it caused by a new scanner version?
Was it caused by a changed vulnerability database?
Was it caused by a changed image?
Was it caused by a moved tag?

Security teams need signal.

Floating tags add noise.

Fixed versions and digests make changes easier to explain.

The workflow I prefer

For important CI/CD images, I prefer this workflow:

1. Choose a specific version.
2. Resolve its digest.
3. Commit image:tag@sha256:digest.
4. Scan the image.
5. Run the pipeline.
6. Automate update pull requests.
7. Review updates like dependency changes.
8. Merge intentionally.
9. Keep old pipeline history understandable.

This is not complicated.

It just treats CI tooling as real dependencies.

That is the mindset shift.

A practical team policy

A reasonable policy could look like this:

development:
  fixed version tags are recommended
  latest is discouraged

CI/CD with no secrets:
  fixed version tags are required
  digests are recommended

CI/CD with secrets:
  digests are required
  GitHub Actions must use full commit SHA
  no latest/main/master references

production:
  digests are required
  approved registries only
  automated update PRs required
  image scanning required

This is easier to roll out than one hard rule for everything.

It also matches risk.

The more privileged the environment, the less acceptable floating dependencies become.

A simple checklist

For every repository, I would ask:

Do CI jobs use :latest?
Do scanner jobs use fixed versions?
Do scanner jobs use digests?
Do Dockerfiles use floating base image tags?
Do GitHub Actions use tags instead of full SHAs?
Are image updates visible in pull requests?
Do we know which image digest ran in a past pipeline?
Can we block unsafe image references?
Can we rotate credentials quickly if a CI tool is compromised?
Are CI tokens least-privilege?

If the answer to most of these is "no", the pipeline is probably too trusting.

Not broken.

But too trusting.

What I learned from the Trivy incident

The biggest lesson is not that one specific version was bad.

The bigger lesson is that CI/CD pipelines often trust moving references.

The Trivy incident made that visible because the compromised component was a security tool.

That makes the situation feel ironic, but the engineering lesson is broader:

security tools are still dependencies
dependencies need version control
version control needs review
review needs an audit trail

A mutable tag is not an audit trail.

A digest is much closer to one.

Closing

I do not see fixed image versions as a paranoid practice.

I see them as normal supply chain hygiene.

If a pipeline pulls latest, the tool can change without a commit.

If a pipeline uses a fixed version, the intended version is visible.

If a pipeline uses a digest, the actual content is pinned.

That is the progression:

latest:
  convenient but risky

fixed version:
  better and easier to adopt

version plus digest:
  stronger and more reproducible

The Trivy supply chain incident showed why this matters in a very practical way.

A tool used to improve security became part of the attack path.

That does not mean we should stop using security tools.

It means we should run them like any other high-trust dependency:

pinned
reviewed
updated intentionally
auditable
least-privileged

Security tooling should not silently change under our feet.

Especially when that tooling runs inside CI/CD.

DEV Community