Shaarav Agarwal

Posted on Sep 29

How I secured an AI app's supply chain: SBOM + AIBOM, keyless signing, and SLSA attestation in one pipeline

#security #devsecops #ai #docker

Context

In March 2024, a backdoor was planted in xz-utils — a compression library so ubiquitous it ships with almost every Linux distribution. It was caught by a developer noticing his SSH logins were a few hundred milliseconds slower. When you build a container image, you inherit hundreds of components you never wrote: OS packages, language libraries, and for AI applications, model files. I built this pipeline to answer one question: how do you actually know that the software you're about to deploy is the software you think you built?

A note on status: this is a fresh build, not a years-running system. The commits span Sept 26–28, 2026 — app, pipeline, policies, and docs, run green end-to-end in the same week. The honest claim is "this gates every commit, locally and in CI, right now."

Approach

I built an intentionally vulnerable AI application — a FastAPI model registry with a chat endpoint — and wrapped it in an eight-stage pipeline. The app is the demo victim: its core flaw, unverified model ingestion (POST /models/upload accepts any file, no hash check, no provenance), is exactly the discipline the pipeline enforces. The pipeline is the product; the app proves it isn't theater.

The whole thing reduces to five ideas in order:

Inventory — you can't secure what you can't list. The SBOM is the list.
Vet — every item gets checked: known CVEs, licenses, provenance.
Sign — bind the artifact to an identity so it can't be swapped.
Prove — record who built it, from what, with what process, publicly.
Deploy — only artifacts that survived all four previous stages run.

Stage by stage, what I actually built:

1. Build. python:3.12-slim, non-root runtime user, exact pins, gunicorn with the uvicorn worker (FastAPI is ASGI; the default WSGI worker breaks request handling). Hermetic — no model downloads in CI, so anyone can reproduce it without credentials.

2. SBOM. Syft emits a CycloneDX inventory — an open JSON-schema standard for bills of materials. My image: 129 components of OS packages and Python dependencies. You can't scan what you don't list.

3. AIBOM. A plain SBOM lists libraries; it says nothing about the model. scripts/build_aibom.py reads the shipped model manifest and merges CycloneDX model components carrying the model's sha256, source URL, framework, and dataset hashes — computed from the actual files at build time, per India's CERT-In CISG-2024-02 AIBOM guidance.

4. Trivy gate. Fails the build on any CRITICAL or HIGH finding.

5. CISA KEV gate. A second, stricter check: does any finding appear in the catalog of CVEs attackers are actually using right now? CVSS says how bad something could be; KEV says it's being weaponized today.

6. Policy gate. Conftest runs Rego policies against the SBOM/AIBOM: SPDX/OSI license rules and a hard requirement that every model carries a sha256 and a source URL — the direct counterpart to the app's vulnerable upload endpoint.

7. Sign. Keyless cosign — no long-lived private key to lose. Identity comes from a short-lived certificate issued by Fulcio (Sigstore's certificate authority) and bound to an OIDC identity — my GitHub identity locally, the GitHub Actions workflow in CI. Every signature lands in Rekor, Sigstore's public transparency log: a timestamped record anyone can look up.

8. Attest. An in-toto attestation (a signed statement that a build happened) carrying an SLSA v1 provenance predicate — the machine-readable "who built this, from what source, with what process."

Then the deploy gate: deploy-local.sh refuses to run an image that isn't both signed and attested.

Architecture

flowchart LR
    A["Source: app/, policy/, scripts/"] --> B["Build\npython:3.12-slim, non-root"]
    B --> C["SBOM (Syft → CycloneDX)"]
    C --> D["AIBOM (build_aibom.py → model components)"]
    D --> E{"Trivy gate\nCRITICAL/HIGH"}
    E -->|fail| X["build fails"]
    E -->|pass| F{"CISA KEV gate\nactively exploited?"}
    F -->|fail| X
    F -->|pass| G{"Conftest policy\nSPDX/OSI + AIBOM provenance"}
    G -->|fail| X
    G -->|pass| H["cosign sign (keyless → Rekor)"]
    H --> I["cosign attest (SLSA v1 predicate)"]
    I --> J{"deploy gate\nverify sig + attestation"}
    J -->|pass| K["run container"]
    J -->|fail| X

Evidence

The gate caught real problems on the very first run. I had pinned gunicorn 20.1.0 and the gate found 40+ CRITICAL/HIGH findings: gunicorn 20.1.0 (CVE-2024-1135, HTTP request smuggling), plus CVEs in starlette, python-multipart, setuptools, and wheel. I bumped every fixable pin — that remediation is commit 173c36d, not a claim. Second run: 8/8 stages green, 0 fixable CRITICAL/HIGH.

The CI run logs show the gate doing its job — the Trivy step failing red on the vulnerable pins:

And the actual findings the gate surfaced — Total: 11 (HIGH: 11) in the Python dependencies alone, gunicorn 20.1.0 → 22.0.0 flagged for CVE-2024-1135 and CVE-2024-6827:

But the base image itself ships 8 HIGHs with no upstream fix yet (util-linux, ncurses, systemd, acl, perl). Rather than hide them with --ignore-unfixed or stay permanently red, I wrote an explicit .trivyignore — each CVE listed with a comment explaining why it's accepted. The gate now fails on anything new or fixable, and the CISA KEV gate overrides the baseline if any of those CVEs is ever actively exploited.

Keyless signing is verifiable by a stranger. My local run wrote a real Rekor entry (logIndex 2969112166) binding my GitHub identity to the image. The repo ships scripts/verify.sh for anyone to check signature, attestation, and vulnerabilities — no secrets, no keys:

IMAGE=localhost:5000/ai-model-registry:dev \
  CERT_IDENTITY="142115441+Shaarkymoo@users.noreply.github.com" \
  CERT_ISSUER=https://github.com/login/oauth \
  bash scripts/verify.sh

The real output — signature validated against the transparency log, SLSA attestation verified with the certificate subject shown, then the vulnerability scan:

The deploy gate refuses unsigned images. cosign verify on an unsigned image fails with no signatures found, exit code 10, and the container never runs. The same verify runs in CI before anything is released.

And the CI isn't a photo op — the workflow runs on every push to main, all 14+ runs visible in Actions history alongside the Dependabot update PRs:

And the app's own vulnerability gets caught in front of you. The chat endpoint is prompt-injectable by design — "ignore previous instructions" leaks its hidden system prompt — and the app's two-layer evaluator (keyword filter, then an LLM judge returning {"leak_detected", "confidence", "reason"}) flags it LEAK_LIKELY. I ran it against the deployed container: signature verified, attestation verified, then the leak verdict — the loop this pipeline exists to keep honest.

Policies have tests. policy/sbom_test.rego carries 7 test cases covering every rule — including the license-name form below.

What went wrong

cosign attest expects the bare predicate, not the in-toto wrapper. The predicate file must be just buildDefinition + runDetails; cosign builds the Statement around it. Wrong shape → "provenance predicate: required field builder missing" (sigstore/cosign#3757).
Syft emits some licenses as free-text names, not SPDX IDs. autocommand ships its license as the name LGPLv3 (OSI-approved); my first policy only checked IDs and rejected a legitimate license. Fix: accept declared names and match the forbidden list against names too.
OCI image refs must be lowercase. Shaarkymoo in a GHCR path is invalid; the workflow failed until I lowercased it.
verify.sh originally took the image as a positional arg and broke under IMAGE env usage — now it honors the env var.

Honest limitations

The app is still vulnerable by design. The pipeline secures the supply chain; it does not fix prompt injection or SSRF in the app. That boundary is intentional — "attack the app's bugs all you want; you can only ever deploy the exact artifact we built, scanned, signed, and attested."
I claim SLSA Level 1–2, honestly. Level 3–4 needs hardened, isolated, hermetic build infrastructure — a real investment I'm not pretending to have made.
No runtime detection. The pipeline guarantees what you deploy is what you built; it doesn't watch the app afterward.
Upstream maintainer compromise isn't preventable here. If a maintainer ships malware, the scan catches the CVE after publication; the SBOM's job is to make the blast radius visible.

Lessons learned

Supply-chain security is mostly inventory + verification, not magic. The SBOM makes the invisible visible; the signatures make claims checkable.
The gate catching real findings is the feature. My "clean" pins had 40+ CRITICAL/HIGH issues; the pipeline turned a surprise into a fix.
Keyless signing removes the biggest excuse. "Key management is hard" doesn't survive contact with Sigstore.
For AI, provenance starts with the model. A registry that accepts unverified uploads has a supply-chain vulnerability in its core — and the same discipline that secures libraries extends to models.
Frame against standards, not opinions. Every control maps to something an auditor recognizes: CVSS, CISA KEV, SPDX/OSI, CycloneDX, SLSA, NIST SSDF, CERT-In CISG-2024-02. That's the difference between "I added security tools" and "here's which control catches which attack."

DEV Community