DEV Community

Gaberial Sofie
Gaberial Sofie

Posted on

Green Tests Aren't a Safe Supply Chain: A GitHub Actions Security Audit Gate That Blocks Vulnerable Deploys

#security #cicd #devops #github

The exposure

Consider the timeline. Friday: all checks pass, PR merged, deploy succeeds, everyone goes home. The following Thursday a critical advisory drops for one of the gems we were running in production, with exploits already circulating — and the way we learned our own app was vulnerable was a Hacker News thread, not our pipeline.

That is a detection gap, and it is worth naming precisely because CI actively disguises it. Green tests prove your code behaves; they say nothing about whether your dependencies are safe. A pipeline that treats "tests passed" as "safe to ship" is conflating two very different claims, and the space between them is exactly where a known-vulnerable dependency reaches production unopposed. A github actions security audit gate closes that space by making a red audit block the deploy. The design principle we anchored on is the one GitHub itself pushes in its security hardening guide for Actions: make the safe path the default path, enforced by the pipeline rather than remembered by a human. For the workflow shape and the first-month rollout I also worked from a thorough third-party writeup of this audit-gate pattern →.

Threat model

  • Known-vulnerable dependency reaching production. A gem with a published CVE flowing through a green build into prod is the primary exposure — and because the test suite cannot see it, nothing in the default pipeline stops it.
  • Post-merge advisory window. New advisories drop after code merges. A dependency that was clean at merge becomes vulnerable in place while sitting in production, and a merge-time-only check never revisits it. This is a mean-time-to-detect problem: ours was roughly seven days, measured from disclosure to a human noticing.
  • License / legal exposure. A copyleft-licensed transitive dependency slipping into a commercial codebase is a distinct risk that neither tests nor CVE scanners address, and it surfaces at the worst time — during due diligence, not development.
  • The gate itself as supply-chain surface. Adding third-party Actions to enforce security introduces its own risk: an unpinned marketplace action is mutable and can be re-pointed at hostile code. The control must not become the vulnerability.

Controls we added

Control 1 — a red audit blocks the deploy

The core control is one line. An audit job runs the scanners; the deploy job declares needs: [audit], so it will not start unless the audit is green. No audit pass, no deploy — the safe path stops being something a human remembers and becomes what the pipeline enforces.

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with: { ruby-version: .ruby-version, bundler-cache: true }
      - run: gem install bundler-audit license_finder
      - run: bundle audit check --update
      - run: license_finder --quiet
      - uses: google/osv-scanner-action@v1
        with: { scan-args: '--recursive --skip-git .' }

  deploy:
    needs: [audit]   # deploy won't start until audit is green
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... deploy steps
Enter fullscreen mode Exit fullscreen mode

Control 2 — three scanners for defense in depth

We chose three complementary tools rather than one, because they cover different blind spots and layering them is cheap:

  • bundler-audit — RubySec's patch-level verification, scanning Gemfile.lock against the ruby-advisory-db for known CVEs, and flagging insecure http:///git:// gem sources. Deepest on Ruby advisory data.
  • license_finder — reports dependencies outside your approved-license policy, covering the legal risk the CVE scanners ignore entirely.
  • Google's OSV-Scanner, backed by OSV.dev across many ecosystems — broadest coverage, so it catches vulnerabilities in polyglot repos, not just Ruby.

Overlapping controls is deliberate: no single advisory source is complete, so two independent scanners plus a license check reduce the residual chance that a known issue slips past all of them.

Control 3 — trigger strategy that closes the post-merge window

The triggers are a control in their own right:

on:
  push: { branches: [ main ] }
  pull_request: { paths: ['Gemfile*', '*.gemspec'] }
  schedule:
    - cron: '0 8 * * MON'   # weekly re-audit
Enter fullscreen mode Exit fullscreen mode
  • pull_request scoped to dependency files so the gate only fires when dependencies actually change — which keeps it off most PRs and prevents the resentment that gets security gates disabled.
  • The weekly schedule cron is the one that addresses the post-merge window directly: it re-audits already-deployed code against advisories that did not exist at merge, moving mean-time-to-detect from about a week to same-day or next-scheduled-run.
  • push to main as the last check before production.

Control 4 — hardening the gate itself

Because the gate depends on third-party Actions, we treated that dependency as supply-chain surface. GitHub's hardening guide is explicit that pinning to a full-length commit SHA "is currently the only way to use an action as an immutable release," so third-party actions are SHA-pinned rather than tag-referenced. When the tagged OSV action broke on a stale reference, we installed the scanner as a binary instead — vendor-neutral and not subject to a mutable tag:

curl -sSL https://raw.githubusercontent.com/google/osv-scanner/main/scripts/install.sh | sh -s -- -b /usr/local/bin
osv-scanner --recursive --skip-git .
Enter fullscreen mode Exit fullscreen mode

We also adopted a few Bundler settings to reduce source-substitution surface (bundle config set disable_multisource true, cache_all true, clean 'true'), layered Trivy alongside OSV for further depth, and put a quarterly dependency-hygiene day on the calendar.

Verifying the control, not just shipping it

A gate is only real if it fails when it should and the team keeps it on. The first run failed — we had two existing advisories on merge — and that failure was the point: triaging them was the whole value, not something to route around by disabling the step. Two design choices kept the gate trusted rather than resented. Because the audit runs on pull_request only when Gemfile* changes, most PRs never see it. And because bundle audit prints the advisory URL and the fixed version, "unblock" almost always meant a one-line version bump rather than a research project. After the team saw the red build point at a real, patchable issue every time — no false-positive noise — trust went up, not down. By week four a red audit was simply "bump the gem and move on." A couple of operational notes that matter: bundle audit needs --update or it scans a stale advisory DB and gives false comfort, and license_finder will surprise people (ours flagged a transitive GPL dependency nobody had noticed), so set your allowed-license policy up front.

Residual risk / what we're still watching

The gate reliably keeps known-vulnerable dependencies out of production, but it is not a complete supply-chain defense, and being honest about the edges is the point.

  • Scanners find known vulnerabilities only. A zero-day or a brand-new typosquat with no advisory yet passes clean. The gate lowers mean-time-to-detect for disclosed issues; it does not detect the undisclosed. Human review of new dependencies still carries that load.
  • Gate-bypass is a cultural failure mode. The most likely way this control dies is a developer commenting out the audit step to unblock a feature. We defend it by keeping it fast, legible, and low-false-positive so bypassing feels unnecessary — but a determined bypass is always possible, so we also protect the workflow file in review.
  • Third-party action drift. SHA-pinning freezes an action, but a frozen action also stops receiving fixes; we periodically review pinned SHAs so "immutable" does not become "stale and unpatched."
  • Coverage gaps beyond gems. The gate audits dependencies, not our container base images or the runner environment. We are extending the same pattern to image scanning next, and rolling the gate out as a reusable workflow so every repo inherits it rather than reimplementing it.
  • The advisory databases are themselves a dependency. Our detection is only as current as OSV.dev and the ruby-advisory-db; a gap or delay in their data is a gap in ours. Running two independent sources mitigates but does not eliminate that.

The measurable outcome that mattered: mean-time-to-detect a known-vulnerable dependency dropped from about a week to same-day, and twice the Monday cron flagged a fresh advisory on already-deployed code before any customer or news cycle did. What we keep watching is the gap the gate cannot close — the undisclosed vulnerability and the human tempted to route around a red build.

Sources & further reading

Top comments (0)