Mohit Kumar

Posted on Aug 19 Originally published at mohitkumar1.hashnode.dev

You Can't Govern What You Can't See - Building an AI Bill of Materials

#ai #security #agents #devops

Author: Mohit Kumar

Project: Bulwark – An open-source security stack for AI agents

GitHub: mk12002/Bulwark

This article is part of the Bulwark series, where I explore practical approaches to securing AI systems, agentic workflows, and the AI software supply chain.*

The EU AI Act is here. Auditors will ask what your AI system is made of. Most teams can't answer. Here's how to fix that in one command.

An auditor walks into your ML team and asks a deceptively simple question: "What is this AI system actually made of, where did each piece come from, and can you prove it's safe?" For most organizations the honest answer is a shrug — a pile of models, datasets, MCP servers, prompt templates, notebooks, and dependencies with no single record of what's inside. With the EU AI Act now in force and NIST AI RMF the default reference for AI governance, that shrug is becoming a liability. So I built a tool that turns the shrug into a document.

TL;DR

Software has SBOMs. AI systems need AI-BOMs — a bill of materials for models, datasets, MCP servers, prompts, tools, notebooks, and dependencies.

Manifest discovers all of it statically, resolves provenance/license/version, checks deps against OSV, and emits a standards-based CycloneDX or SPDX AI-BOM.

It doesn't just inventory — it folds in security risk by calling Airlock (on models/MCP) and Warden (on agents), and maps every gap to NIST AI RMF and EU AI Act articles.

Plus manifest diff shows AI-BOM drift between versions, so you gate CI on unexpected change.

Why this matters

Governance, risk, and compliance for AI is having its moment — not because it's fashionable, but because regulators made it mandatory. The EU AI Act imposes documentation and transparency obligations. NIST's AI RMF (Govern / Map / Measure / Manage) is the framework everyone maps to. And the through-line of both is the same: you must know what your system is composed of.

Here's the uncomfortable truth: the SBOM (software bill of materials) world is mature, but AI components — a model from the Hub, a dataset with an unclear license, a prompt template nobody version-controls — mostly fall outside traditional SBOM tooling. The AI-BOM practice is emerging (CycloneDX added ML-BOM support; OWASP launched an AIBOM project) but tooling that actually generates one from a real repo — with integrated security risk, not just a component list — is scarce.

That's the gap Manifest fills, and it's the one your compliance team will thank you for.

The core concept: inventory first, govern second

You cannot recall, patch, or audit a component you don't know you have. So step one is always discovery — and the analogy is exactly the physical one. A car manufacturer keeps a bill of materials listing every part, its supplier, and its batch number. When a supplier's airbag turns out to be defective, the BOM is what makes a targeted recall possible instead of a blind panic.

An AI-BOM is that document for your AI system. When the next malicious-model disclosure or dependency CVE lands, the AI-BOM is the difference between "we're checking…" and "we use that model in two services, patching now."

How it actually works

Discover → resolve → assess → govern

Manifest runs four stages. Discoverers are independent and additive — adding a new component type never touches the others — so the inventory grows by contribution, not rewrite.

The nine governance findings (B-codes)

Code	Governance finding
B1	Undeclared / unpinned component
B2	Missing provenance
B3	License risk (restrictive / copyleft / unknown)
B4	Known-vulnerable dependency (via OSV)
B5 ⭐	High-risk component — imported from Airlock / Warden, surfaced inline
B6	Dataset governance gap
B7	Secret / credential reference
B8	Unversioned / untracked prompt template
B9	Control gap (NIST AI RMF and EU AI Act mapping)

The differentiator: inventory plus risk, in one artifact

Most AI-BOM generators stop at "here's a list of components and their licenses." Manifest goes further: --scan-risk calls Airlock and Warden as libraries and attaches their findings to the exact components they belong to, as B5. The bill of materials and the security assessment become one document:

$ manifest scan ./project --scan-risk --govern

  HIGH     B4  Known-vulnerable dependency (pyyaml 5.3.1)     # OSV: CVE-2020-14343
  HIGH     B7  Secret/credential referenced in the project    settings.py
  CRITICAL M1  Pickle references a shell/exec callable  ◀── Airlock, inline on the model component (B5)
  MEDIUM   B3  Restrictive license (cc-by-nc-4.0)             model/
  MEDIUM   B1  Component used without a pinned version         transformers
  MEDIUM   B6  Dataset lacks documented governance            data/train.csv
  → CycloneDX 1.5 written · NIST AI RMF: GOVERN/MAP/MEASURE gaps · EU AI Act: Art.10/13/15 gaps

Standards in, standards out

CycloneDX 1.5 (--format cyclonedx) — the ML/AI-BOM standard; drops into existing SBOM tooling and Dependency-Track.
SPDX 2.3 (--format spdx) — for pipelines standardized on SPDX.
--govern maps every finding to NIST AI RMF (Govern/Map/Measure/Manage) and EU AI Act articles (advisory — transparent and sourced, never a conformity claim), and emits a risk register (component → risk → severity → action) — the exact artifact a GRC reviewer wants.

Drift is a first-class citizen

Governance isn't a one-time scan; it's continuous. manifest diff ./v1 ./v2 shows what changed between two versions — components added, removed, re-licensed, or version-bumped — and exits non-zero on any change. Gate CI on it and an unreviewed model swap or a silent license change becomes a failed build, not a surprise in an audit.

What can go wrong (the governance attack surface)

The unpinned model swap. A repo references org/model with no version pin or hash. An attacker (or a well-meaning maintainer) pushes a new revision. Your build silently pulls different weights. Detection: B1 (unpinned) at scan time; manifest diff catches the swap between releases.

The license landmine. A model ships under cc-by-nc-4.0 (non-commercial) and someone wires it into a paid product. Detection: B3 classifies license risk and flags the conflict before legal does.

The invisible dependency CVE. A transitive dep has a known OSV advisory. Detection: B4 cross-checks dependencies against OSV and attaches the advisory to the component.

The notebook nobody audited. A data scientist's explore.ipynb quietly from_pretraineds a model and !pip installs a package that never made it into requirements.txt. Detection: the notebook discoverer parses .ipynb cells and inventories both, with the exact cell as the location.

Defenses that actually work

Generate an AI-BOM every release. manifest scan ./project --scan-risk --govern --format cyclonedx as a build artifact. This is the single highest-leverage governance action you can take.
Gate on drift. Wire manifest diff into CI so an unreviewed component change fails the build.
Pin everything. Models, datasets, and deps get exact versions and hashes. Reproducibility is security.
Map to a framework early. --govern gives you the NIST AI RMF and EU AI Act mapping now, so the first audit isn't a fire drill.
Fold in risk, don't bolt it on. An inventory without security findings is a spreadsheet; --scan-risk makes it a risk artifact.

Hot take: "AI governance" is drowning in slideware — maturity models and principles decks that never touch a repo. The entire field reduces to one unglamorous engineering task: generate an accurate, risk-annotated bill of materials, automatically, on every build. Do that and 80% of your AI Act documentation writes itself. Skip it and no framework will save you.

Key takeaways

Inventory is the prerequisite for governance — you can't recall, patch, or audit what you can't see.
Emit standards (CycloneDX + SPDX) so your AI-BOM plugs into the SBOM ecosystem you already have.
Fold security risk into the BOM by calling dedicated scanners — inventory + risk in one artifact.
Map to NIST AI RMF and the EU AI Act transparently, and produce a risk register a GRC reviewer can use.
Treat drift as a gate — diff between versions turns silent component changes into failed builds.

DEV Community