Rez Moss

Posted on Jul 17

Your CI Security Passed. Could You Prove What It Checked?

#security #cicd #devops #containers

Affiliation disclosure: I maintain the open-source repository used in this walkthrough.

A security job went green. Six weeks later there's an incident, and someone asks a simple question: what did that job check? You open the run, and the log is gone. The scanner printed something useful to a temporary file, the job exited 0, and the evidence left with the runner.

That gap is the problem, and adding more scanners does not close it. A CI security workflow is easy to make long and hard to make trustworthy.

Four questions decide whether a workflow is trustworthy during an incident or a release review:

Which findings stop the build?
What evidence survives after the runner disappears?
Did the SBOM and the vulnerability scan look at the image the build produced?
Could pull-request code get the same signing identity as a trusted release?

I built the workflow to answer those four, so I treated it as an evidence pipeline rather than a scanner checklist. The result is a GitHub Actions baseline with four parallel source gates, one immutable image handoff, CycloneDX SBOM generation, image scanning, and keyless Cosign signing. The tagged workflow and its outputs are public. You can apply the same structure with different tools.

Start with an output contract

Before picking any tool, decide what evidence each control must leave behind after the console output is gone.

Stage	Blocking policy	Retained evidence
Secrets	Any Gitleaks finding	SARIF
SAST	Any configured Semgrep error	SARIF
Dependencies	Any published OSV vulnerability	JSON
Infrastructure config	HIGH or CRITICAL Trivy misconfiguration	SARIF
Build	Every source gate must pass	Image tar and digest
Package	Fixed HIGH or CRITICAL image vulnerability	CycloneDX SBOM and SARIF
Trusted release	Signing or identity failure	Sigstore bundle

This contract heads off the most common failure mode. A scanner writes a helpful report to a temp path, the job turns green, and the file vanishes with the runner. You can't reconstruct the check later.

SARIF is useful because GitHub code scanning reads it and shows findings in the Security tab. But I keep each scanner's native output too. Fork pull requests can't upload SARIF with a write token, and native JSON is often easier to archive or feed into another tool later. Every job in the baseline uploads its report as a workflow artifact with a 14-day retention, so the evidence exists whether or not the SARIF upload runs.

The contract also forces an honest decision about blocking. A control that never fails the build is documentation, not a gate. Each row in that table names the exact condition that stops the pipeline, so there's no ambiguity about which findings are advisory and which ones are load-bearing.

One dependency graph, not one giant job

Secrets, SAST, dependency, and configuration checks all read source code. None of them needs a container image. So they run in parallel:

Gitleaks ─────┐
Semgrep ──────┤
OSV-Scanner ──┼──> Build image once ──> SBOM + image scan ──> Sign + verify
Trivy config ─┘

The build waits for all four source gates. Packaging consumes the image the build produced. Signing waits for packaging. In GitHub Actions, needs makes that contract visible in the file itself:

jobs:
  secrets:       # Gitleaks
  sast:          # Semgrep
  dependencies:  # OSV-Scanner
  iac:           # Trivy configuration scan

  build:
    needs: [secrets, sast, dependencies, iac]
  package:
    needs: build
  sign:
    needs: package

This shape gives developers fast feedback on the source code, because the four gates start at once and the slowest one sets the pace. It also keeps the released artifact on an ordered chain: nothing gets built until the source is clean, nothing packaged until it's built, nothing signed until it's packaged. You read the dependency list and see what each stage waits on, without tracing step logic.

Deny permissions first

The workflow opens with no token permissions at all:

permissions: {}

From there each job asks for only what it needs. Read-only dependency scanning needs the repository contents and nothing else:

permissions:
  contents: read

A job that publishes SARIF to code scanning also needs security-events: write:

permissions:
  contents: read
  security-events: write

Only the signing job recieves an OIDC token:

permissions:
  contents: read
  id-token: write

The last block is the one to get right. It's tempting to grant id-token: write at the top of the file because Cosign shows up somewhere below, but a job that processes untrusted pull-request content should not share release-signing authority with the job that stamps a trusted identity onto an artifact. Scope the token to the one job that earns it.

The SARIF upload also treats forks differently from same-repository pull requests:

if: >-
  ${{ always() &&
      (github.event_name != 'pull_request' ||
       github.event.pull_request.head.repo.full_name == github.repository) }}

The scan still runs for a fork. Only the write to code scanning is skipped, because a fork PR can't hold that write token anyway. The native report stays available as a workflow artifact, so a maintainer reviewing the fork still has the full output. You keep the evidence and lose only the Security-tab integration for that one case.

Build once, move the artifact forward

One mistake breaks integrity: rebuilding the image separately in the scan job and again in the signing job. Same Dockerfile, same commit, and you can still get different bytes. Timestamps, downloaded packages, and changed base-image layers all drift. Now your SBOM describes one image, your scan describes a second, and your signature covers a third. The evidence stops agreeing with itself.

The fix is to build one time and pass that exact artifact forward. The build job exports a single image tar and records its digest:

- name: Build and export image
  id: image
  uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
  with:
    context: examples/security-demo/app
    file: examples/security-demo/app/Dockerfile
    platforms: linux/amd64
    tags: ${{ env.IMAGE_TAG }}
    outputs: type=docker,dest=${{ runner.temp }}/security-demo.tar

- name: Record immutable image digest
  run: printf '%s\n' '${{ steps.image.outputs.digest }}' > "$RUNNER_TEMP/image-digest.txt"

The SHA on that action is a real pin. Use immutable commit SHAs for every third-party action, and re-resolve them yourself rather than trusting a version this article froze in time. A moving tag like @v6 can point at new code tomorrow; a SHA can't.

The package job downloads that exported tar and loads it. It does not rebuild anything:

- name: Download exported image
  uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
  with:
    name: security-demo-image
    path: artifacts/image

- name: Load image
  run: docker load --input artifacts/image/security-demo.tar

Syft generates the CycloneDX SBOM from that loaded image, and Trivy scans the same tag. Both retain output even when the policy finds nothing:

- name: Generate CycloneDX SBOM
  uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0
  with:
    image: ${{ env.IMAGE_TAG }}
    format: cyclonedx-json
    output-file: artifacts/security-demo.cdx.json

- name: Scan built image
  uses: aquasecurity/trivy-action@c07df6fec6fa692e6fd1200d50aaa1fdd66f03c8 # resolved SHA
  with:
    scan-type: image
    image-ref: ${{ env.IMAGE_TAG }}
    ignore-unfixed: "true"
    severity: HIGH,CRITICAL
    format: sarif
    output: artifacts/trivy-image.sarif
    exit-code: "1"

ignore-unfixed: "true" is a policy decision. This baseline blocks fixed HIGH and CRITICAL findings, the ones you can act on today by bumping a version. A production policy will likely need more: exploitability, reachability, an exception process, ownership, and remediation deadlines. Copy the mechanism, then set the threshold to your own risk model.

Treat signing as an event-policy decision

Keyless signing gets rid of the long-lived private key, which removes a whole class of key-management problems. It does not decide which events deserve a trusted identity. That's still your call, and it belongs in the workflow.

The signing job runs only for a push to main or a version tag. Manual runs and pull requests skip it:

sign:
  if: >-
    ${{ github.event_name == 'push' &&
        (github.ref == 'refs/heads/main' ||
         startsWith(github.ref, 'refs/tags/v')) }}
  needs: package
  permissions:
    contents: read
    id-token: write

Cosign signs the exported tar, then verifies the exact GitHub identity that produced the signature:

cosign sign-blob \
  --bundle artifacts/image/security-demo.bundle.json \
  artifacts/image/security-demo.tar

identity="https://github.com/${GITHUB_REPOSITORY}/.github/workflows/security-baseline.yml@${GITHUB_REF}"

cosign verify-blob \
  --bundle artifacts/image/security-demo.bundle.json \
  --certificate-identity "$identity" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  artifacts/image/security-demo.tar

Verifying only that "Sigstore signed this" is far too loose. Anyone can get Sigstore to sign anything. The certificate's identity has to bind the artifact to the expected repository, the expected workflow file, and the expected Git reference. For the v1.0.0 release the certificate Subject Alternative Name was:

https://github.com/rezmoss/awesome-security-pipeline/.github/workflows/security-baseline.yml@refs/tags/v1.0.0

If any part of that string is wrong, verification fails, which is the behavior you want. The attached bundle also carries a transparency-log entry and the SHA-256 signature material, so the signature is auditable after the fact.

Test failure without shipping a vulnerable app

A green demo only proves the clean path works. It says nothing about whether the gates fail when they should. To exercise failure safely, keep the broken examples out of the application build context entirely.

The repository stores synthetic fixtures in a seperate fixtures/ tree, well away from the scanned and packaged app/:

a synthetic, non-credential secret pattern;
a deterministic shell-command injection sample (exec.Command("sh", "-c", userInput));
an old test dependency with published advisories (dgrijalva/jwt-go@v3.2.0);
an insecure Dockerfile that runs as root;
a privileged Kubernetes pod (privileged: true, runAsUser: 0).

The normal workflow scans and packages only the clean application. To demonstrate a blocking result, copy one fixture into the matching scanned path on a throwaway branch, watch the gate fail, then revert. None of the fixtures is compiled into or copied into the release image. You see red without shipping the thing that caused it, and failure stays reproducible on demand.

What the tagged run produced

The v1.0.0 tag workflow completed on GitHub-hosted runners. Per job:

Job	Duration
Gitleaks	13s
OSV-Scanner	8s
Trivy configuration	16s
Semgrep	36s
Immutable image build	23s
SBOM and image scan	36s
Cosign sign and verify	12s

Don't add those numbers up. The four source jobs run at the same time, so the wall-clock cost is roughly the slowest source gate plus the build, package, and sign chain, not the sum. Your own critical path will differ with repository size, languages, network and cache state, vulnerability database freshness, and runner availability. Treat the table as a rough shape.

The release keeps the exported image, the image digest, the CycloneDX SBOM, the Trivy SARIF, and the Cosign bundle attached. Thier GitHub-recorded SHA-256 digests match the workflow artifacts, so the files on the release page are the same bytes the pipeline produced. That's the whole point of the output contract: months later the evidence is still there and still verifiable.

Reproduce it, then adapt it

The quickest way to see this work is the 10-minute quick start. It needs a GitHub account and Actions, no cloud account, secret, or registry:

Fork the repository and enable GitHub Actions in the fork.
Run Actions → Security Baseline → Run workflow on the default branch.
Confirm the source gates, build, and package jobs pass.
Inspect the retained SARIF, JSON, image, digest, and SBOM under Artifacts.
Push to the fork's main or a v* tag when you want to exercise keyless signing.

Run the baseline unchanged first. Then swap the demo paths for your application and adjust the thresholds. Doing it in that order keeps two questions apart: "does the reference workflow work" and "do our application-specific edits work." When something breaks after you adapt it, you already know the baseline itself was fine.

The repository's curation and verification methodology documents the evidence hierarchy, the maintenance classifications, and the testing levels used across the wider catalog, if you want the reasoning behind the tool choices rather than the recipe alone.

What a green run does not prove

This workflow is a reproducible baseline. It is not a compliance claim, and it is not a guarantee of security.

A green run does not prove:

that the selected rules catch every relevant weakness;
that an SBOM models every component or dependency relationship;
that a reported vulnerability is reachable or exploitable;
that the chosen thresholds fit your threat model;
that a valid signature makes unsafe bytes safe; or
that build-time controls replace admission and runtime enforcement.

The design carries over even when the tools change: explicit gates, permissions denied by default and granted per job, one artifact built and handed forward, evidence retained whether or not a finding appears, and a signature bound to an exact identity. Swap Gitleaks for TruffleHog or Trivy for Grype and the structure holds. Keep that structure, and a green checkmark becomes something you can stand behind six weeks later when someone asks what it checked.