I stopped writing about runtime AI governance and tried to build a piece of it. Here's what I learned.

A few weeks ago I wrote in a blog post on LinkedIn ("Runtime Governance for Agentic AI: Why Existing AI Governance Frameworks Are No Longer Enough", https://www.linkedin.com/feed/update/urn:li:activity:7477072384773079040) that AI governance is shifting from documents to runtime — from committees approving use cases to policy embedded in architecture. The post resonated, and the comments pushed my thinking further.

So, I did something I am very excited about: I stopped writing about the architecture and tried to build a piece of it.

I'm a governance and privacy person, not primarily an engineer. But I wanted to feel where the hard parts actually are — not where the frameworks say they are. A few weeks in, what surprised me most is that the hardest problems were not the ones most governance frameworks spend time discussing.

Governance Is Fundamentally a Decision Problem

Most AI governance discussions focus on policies, principles, controls, and requirements. But when you try to operationalize governance, those things are only inputs.

The real question becomes:
How does a system reach a decision that is explainable, repeatable, evidence-based, and accountable?

Everything else starts to look different once you view governance through that lens.

And the more I worked on the problem, the less governance looked like policy management and the more it looked like a decision pipeline:

Classify → Determine Applicability → Collect Evidence → Evaluate Sufficiency → Decide → Assign Accountability

Frameworks provide the inputs. Governance happens in the translation.

1. The Hardest Question Isn't "What Are the Controls?" It's "Which Controls Apply?"

There is no shortage of controls. NIST AI RMF, ISO/IEC 42001, ISO 23894, the EU AI Act, OWASP's work on LLM and agentic systems—every framework contains requirements, safeguards, and recommendations.

The difficult part is not finding controls. The difficult part is determining which controls apply to a specific application and being able to explain why.

Treat every control as universally applicable and you overwhelm teams with noise. Let a language model decide applicability on its own and you introduce inconsistency, unpredictability, and audit challenges.

So, what surprised me is that applicability itself emerges as a first-class architectural component. Most frameworks assume someone has already determined that a control applies. But in practice, that determination is often undocumented, inconsistent, and heavily dependent on institutional knowledge.

Once governance becomes software, applicability logic becomes part of the system design.

2. The Moment You Automate, the Temptation Is to Let the Model Decide

And that is where many governance demos become misleading.
It is remarkably easy to build an "AI compliance assistant" that reviews an application, asks an LLM whether requirements are satisfied, and returns a compliance score.
It demos beautifully.

But it is also difficult to defend: The outcome is not necessarily reproducible. The confidence score is often not meaningful.
Most importantly, no auditor, regulator, or accountable executive will accept "the model said so" as a governance decision.

The most important design decision turned out to be surprisingly unglamorous: The verdict must be deterministic.

The language model is exceptionally useful as an analyst. It can read documents, summarize evidence, map artifacts to controls, identify gaps, and draft assessments. But the model should not be the judge. The decision itself should come from explicit, explainable rules that humans approve and can defend.

3. Not Everything Should Be Solved With an LLM

Another lesson emerged quickly. Not all controls work the same way.

Some are deterministic:

• Is the endpoint approved?
• Is the data classified?
• Are permissions appropriately scoped?

Others require behavioral validation:

• Can the agent be prompt-injected?
• Can sensitive data be exfiltrated?
• Can tools be misused?

Others remain fundamentally human questions:

• Is the use proportional?
• Is the risk acceptable?
• Is the assessment adequate?
• Is the outcome fair?

Governance programs often struggle because every problem gets routed to the same evaluation mechanism.

In practice, controls need different methods of verification.
One of the most useful design patterns I discovered was simply categorizing controls by how they are assessed:

1. Deterministic
2. Behavioral
3. Documentary
4. Human judgment

That simple distinction improved both trustworthiness and explainability more than additional model sophistication ever could.

4. Evidence Quality Matters More Than Evidence Presence

This was another lesson I did not fully appreciate until I started building.

Many governance reviews effectively ask whether an artifact exists.

• Does a policy exist?
• Does a DPIA exist?
• Does documentation exist?
• Does someone say the review occurred?

But the existence of evidence is not evidence of control effectiveness.
Operational governance requires something stronger.

It requires evidence that a control was actually implemented, tested, and remains effective.

A security review document is not security. An AI risk assessment is not risk management. A governance committee meeting is not governance.

The more I worked through assessment logic, the more I realized that governance depends on linking controls to meaningful evidence—not simply collecting files and checking boxes.

5. Design-Time and Runtime Governance Are the Same Thing

This was probably my biggest surprise. When I wrote about runtime governance, I imagined it as the next generation of governance.
Building a design-time governance system changed my perspective.
The core decision is identical.

Whether governance is evaluated before deployment or before an agent executes an action, the system is ultimately answering the same question:

1. Allow
2. Deny
3. Modify
4. Escalate

The difference is not the decision. The difference is where the decision runs.

Design-time governance is simply the safer starting point. The environment is controlled, the risks are lower, and the reasoning can be tested.
Runtime governance is the same governance engine moved closer to the moment of action.

The logic does not fundamentally change. Its position in the architecture does.

The more I explored the problem, the more runtime governance started to look less like a separate capability and more like the natural destination of a governance system that has already learned how to make defensible decisions.

6. The Bottleneck Was Never More Policy

The last lesson was perhaps the most important.
Organizations are surprisingly good at creating policies.
What they struggle with is connecting those policies to evidence and accountability.

Writing another control is easy. Establishing how that control is verified, identifying the evidence that demonstrates compliance, and recording who made the decision is hard.

The real work sits in tracing a requirement to evidence and then linking that evidence to an accountable human decision. Without that chain, governance becomes difficult to verify and even harder to trust.

What Changed My Thinking

At some point I realized that governance without evidence is just documentation. And governance without accountability is just advice.

The thing I did not expect was how quickly this stopped being a governance project and became an architectural one.

The interdisciplinary nature of AI governance is often discussed as a collaboration challenge between governance, legal, privacy, security, and engineering teams. Trying to build governance made me realize it is something more fundamental than that.

The moment governance has to operate, it becomes a systems design problem.

I don't think the answer is that governance professionals must become engineers. But I do think we need to get close enough to implementation to distinguish what is theoretically desirable from what is actually enforceable. More importantly, we need to distinguish between a compelling demo and a governance decision that can withstand audit, regulatory review, or executive accountability.

That was a humbling realization, and probably the most valuable one.

The Emerging Challenge

I came away from this exercise more convinced than before that the next phase of AI governance is not primarily about creating new frameworks.
Frameworks remain essential. But frameworks alone do not determine whether an action should be permitted.

Someone—or something—must determine applicability, evaluate evidence, assess whether that evidence is sufficient, apply decision logic, and ensure accountability.

That translation layer is where I increasingly believe the future of AI governance lives. The more I work on the problem, the less governance looked like a collection of policies and the more it looked like a decision engine.

Or perhaps more accurately:
a decision pipeline that turns governance requirements into deterministic, evidence-based, human-accountable decisions.

Runtime governance is simply the next step in that evolution: moving those same decisions closer to the moment an AI system acts.

Curious whether others working between governance, privacy, security, and engineering are seeing the same thing. Has governance become less of a policy problem and more of a decision-engine problem in your own work?