In my last post I made the broader case that agentic AI breaks every governance playbook built for traditional software and traditional models. That argument was deliberately wide. This one is narrow on purpose.
I have spent years building and reviewing MLOps systems, including model inventories, validation pipelines, drift monitoring, and approval workflows across enterprise and financial services environments. None of that gets thrown out when an organization moves to LLMs and agents. What changes is more specific than most people assume, and that’s what this post is about.
Picture a model risk team getting a routine intake request. Their form asks the standard questions: what data was this trained on, what’s the version number, what’s the approval status, when was it last checked for drift. They’ve processed multiple of these. Then they read the description of the system in front of them: four agents, six tools between them, a retrieval layer that refreshes daily, and a decision that comes out the other end of a chain where no single step was wrong on its own. The form has no field for that. McKinsey’s 2026 AI Trust survey of roughly five hundred organizations found something consistent with what that team is about to discover: only about a third of enterprises report governance maturity levels adequate for the autonomous agents they are already running. Most of the rest are filling out the same intake form and hoping it’s close enough.
It usually isn’t. Here’s specifically where it stops being close enough, and what closing that gap actually requires.
What MLOps governance is, and how it fits into MLOps
Start with what the intake form already gets right.
MLOps governance is the set of controls that make sure a traditional machine learning model stays trustworthy from training through retirement. In practice that means model versioning so you always know which model is in production and why, drift monitoring that watches for performance degradation as input data shifts, data lineage back to what a model was trained and validated on, approval gates before anything reaches production, and the CI/CD and change management discipline any production software system needs. Software governance sits inside this, not off to the side as a separate track. None of these controls enforce themselves, though. Someone has to own each one, and someone else has to check that the owner actually did the job.
It helps to be precise about who is responsible for each of these controls, and the model I use is the three lines of defense, a framework banking and other regulated industries have used for years to assign accountability. One accuracy note: the Institute of Internal Auditors renamed this the Three Lines Model in 2020, shifting the emphasis toward value creation rather than pure defense. The older term has stuck in finance and model-risk circles regardless, so I use it here, but it is worth knowing the name has technically moved on even if the industry has not.
Applied to a traditional MLOps program, the first line is product, data, and engineering: they own model development, testing, deployment, and documentation. The second line is compliance and risk: they set model use policy, monitor for bias, and review high-risk applications before anything ships. The third line is internal audit, and their job is narrower than either of the other two: independently verify that the first two lines are doing what they claim, not build or approve anything themselves. This structure is mature, well understood, and it works, for the systems it was designed around: models with a fixed structure that gets validated once and monitored between review cycles.
What AgentOps governance is, and how it fits into AgentOps
AgentOps governance starts from the same three-line structure as MLOps, but the substance each line has to check is different, because the systems themselves behave differently.
A traditional model is one artifact. The four-agent system has the underlying model, four system prompts, six tool definitions, and a retrieval layer that updates on its own schedule. The first line now has to version and test all of that, not just the model. In practice this looks less like a model registry entry and more like treating prompts and tool schemas as code: stored in version control, reviewed through pull requests, and run through canary tests against a holdout set before a change is promoted to production, the same discipline used for any other production software change. Version the model alone in a system like this and you have covered maybe a quarter of what actually determines behavior.
A traditional model is deterministic given the same input, which is what makes validation straightforward. An LLM can give two different, both-defensible answers to the same prompt on two different runs. The second line’s review process has to shift from checking whether an output is correct to checking whether it falls within an acceptable range most of the time. Concretely, that means an evaluation gate with explicit thresholds, for example requiring a defined pass rate against a labeled test set before promotion, plus a trigger for human review whenever an agent’s output confidence or evaluation score drops below a set floor. Writing that threshold, and getting compliance to sign off on what “acceptable most of the time” means for a specific use case, is its own project. It is not a policy update you make in an afternoon.
A single model making a single prediction has a clear owner. The four-agent system does not. When the outcome is wrong and no individual step was wrong on its own, tracing which step actually caused it requires a decision log that records not just the final output but the intermediate handoffs: what each agent received as input, what it decided, and what it passed to the next agent in the chain. Without that log, “which of the four agents is accountable” is a question nobody can actually answer, and the first line and second line end up arguing about it after the fact instead of before deployment.
The third line’s job changes too. Drift monitoring watches whether predictions are degrading against a known baseline. It does not watch what an agent actually does while it runs: which tools it called, what actions it took, whether it stayed inside its intended scope. A newer category of agent observability platforms has emerged specifically for this: they capture the full execution trace of an agent run, not just the final output, including which tools were called, what parameters were passed, and how one agent’s output became another agent’s input. Many build on open instrumentation standards like OpenTelemetry so traces stay portable across tools rather than locked into one vendor’s format. For our four-agent system, that trace is the only artifact that would actually let an auditor reconstruct what happened after the fact. Without it, the third line is auditing a black box and calling it independent verification.
Last, the retraining cadence itself breaks. Traditional models retrain on a calendar: monthly, quarterly, whenever there is enough new data to justify it. A system built on RAG or continuous fine-tuning can shift its effective behavior as often as the knowledge base underneath it updates, which can be daily. If the four-agent system’s retrieval layer refreshes every night, a third line built around a quarterly review cadence is checking the system roughly ninety times less often than the system is actually changing.
The comparison, the gap, and what closes it
Laid side by side, the two programs share a spine and diverge on substance. Both assign the same three lines. Both need versioning, testing, review, and independent audit. What differs is granularity, frequency, and what each control actually has to look at.
The gap is not that MLOps governance is wrong for the four-agent system. It is that MLOps governance answers a set of questions that assumed a model’s behavior stays fixed between review cycles, and a system with a nightly-refreshing retrieval layer does not hold still long enough for that assumption to work. Every one of the five deltas above traces back to that single broken assumption: a model that could change monthly is now a system that can change behavior daily, sometimes without anyone changing the model itself.
Closing the gap does not mean building an entirely new governance function from scratch. It means extending the existing three lines with capabilities they did not previously need, and being honest about what each one costs. The first line needs prompt and tool version control with canary testing before promotion, which is mostly an engineering practice change, not a new team. The second line needs an evaluation gate with explicit acceptance thresholds and a documented escalation path for chained decisions, which is a genuine policy-writing project that will take real negotiation with legal and compliance the first time through. The third line needs runtime observability tooling plus the skill set to audit a decision log rather than a static model card, and for most model risk functions that is a new hire or a real training investment, not a checkbox added to an existing review template. None of these are optional if the goal is to actually answer, credibly, which of the four agents made the call.
Where the regulators stand
No single regulatory framework has fully caught up to agentic AI yet, but three existing models, from three different industries, are worth watching for where this is heading.
The EU AI Act takes a horizontal approach that applies across sectors. Its Article 50 transparency obligations take effect August 2, 2026 regardless of other developments. Its heavier Annex III high-risk obligations were originally set for the same date, but a proposed Digital Omnibus package, agreed in principle in May 2026 but not yet formally adopted, would push those specifically to December 2027. Worth treating as pending rather than settled if you are building a compliance calendar around it.
The FDA’s approach to AI-enabled medical devices is a genuinely different model: vertical, product-based, and built around a mechanism called a Predetermined Change Control Plan. A PCCP lets a manufacturer pre-authorize specific future model changes within defined performance bounds as part of the original approval, paired with Good Machine Learning Practice guidance and quality system alignment toward ISO 13485. It is essentially a change-control gate built directly into the regulatory pathway rather than a compliance requirement layered on after the fact.
Finance offers a different kind of case, partly because of what it does not cover. I have written elsewhere in this series about SR 26–2, the interagency guidance that replaced SR 11–7 in 2026, and its explicit exclusion of generative and agentic AI from formal scope. That gap is itself instructive. A mature, well-established regulatory framework is acknowledging that the tools it built for one generation of models do not automatically extend to the next. Enterprises operating under it are, in practice, already building governance for agentic systems ahead of formal supervisory guidance catching up to them.
None of these three is a template to copy wholesale. Together they point at where things are heading: pre-authorized change boundaries, tiered risk classification, and an acknowledged gap between existing rules and what agentic systems actually do.
A checklist for both generations
Back to the model risk team. They don’t need to throw out their intake form. They need a second one for the multi-agent system, mapped against the same three lines they already trust, and they need to accept that filling it out honestly is real work, not a formality. If you are running LLM or agent-based systems through an existing model risk function, I am curious which step in this roadmap your governance program has actually caught up to first. My guess is versioning gets solved early, because it looks like a version control problem engineers already know how to solve. Accountability for chained decisions is the one I would bet stays unresolved the longest, because it is an org design problem wearing a technical costume, and no evaluation gate fixes an org design problem.
References
- McKinsey & Company. State of AI Trust in 2026: Shifting to the Agentic Era. AI Trust Maturity Survey of approximately 500 organizations, 2026.
- European Commission. Timeline for the Implementation of the EU AI Act. AI Act Service Desk.
- EU AI Act Update: Timeline Relief, Targeted Simplification, and New Prohibitions. Inside Global Tech, May 28, 2026.
4. IntuitionLabs. FDA AI/ML SaMD Guidance: Complete 2026 Compliance Guide.
5. Institute of Internal Auditors, Three Lines Model (2020 revision).
Top comments (0)