AI agent governance fails at the intersection of governance and permission. What appears to be granted by way of permission to use a tool in a narrow fashion can rapidly devolve. In the end, what failed governance? A permission spreadsheet with attached vibes recorded in a spreadsheet.
Live execution path is the space where governance happens. Runtime security for autonomous agents, a space Microsoft is actively tackling in the open with the Agent Governance Toolkit, is the space of deterministic policy enforcement, identity, isolation, audit, reliability controls, etc. for a variety of agent frameworks. All of this is open-source, MIT-licensed. Governance is moving out of the prompt and into the runtime.
The action is only half the question
What has traditionally been thought of as access control, a simple yes or no as to whether a subject can take an action on a resource, becomes rapidly more complex as the same action becomes, following a series of steps, a vastly different thing. As an example, a verified billing change for a customer may cause the sending of a single email to that customer. A scraping of a long-dormant escalation queue by an AI agent could cause the sending of 4,000 such emails, potentially each with its own complexity, and be a wildly different event. Similarly, a verified alert causing a remediation script to be run is a vastly different action from an untrusted prompt injection that caused the alert in the first place.
The paper Runtime Governance for AI Agents: Policies on Paths gives a more refined definition of what one would expect from a runtime governance system. Policy should map: 1) identity of the AI agent; 2) partial execution path of the AI agent; 3) proposed next action of the AI agent; 4) organizational state to a probability of policy-violation.
The policy decision belongs on the execution path, before the side effect.
Prompts can shape behavior. They cannot enforce policy.
There is a key distinction here between prompt-level safety and the safety of the application code. Even stochastic systems can be designed to request safe actions. When we write application code that invokes tools, we can, as a rule, intercept the tool call on the wire before it actually executes, i.e. implement deterministic safety as opposed to stochastic safety of the prompt. The Agent Governance Toolkit README quick-start does just this, and governs tool calls by wrapping them in a function, for example govern(my_tool, policy="policy.yaml"), which logs information about the call, makes the decision, and returns GovernanceDenied when policy blocks the action.
This is why agent identity has to be workload identity, not a shared key. “The agent did it” does not help during incident review. Which agent? Which run? Which delegated subagent? Which credential? Which policy version? Which approval? Shared credentials turn the trail into mush.
The harness is the governance surface
The Agent Systems with Harness Engineering paper describes harness as a set of infrastructure supporting the execution of workflows, using memory, using skills, orchestrating groups of agents, error handling, and adaptive context management. This exactly matches what we see in real builds today. The harness provides the tool registry, memory lookup, retry strategy, subagent delegation, context, approval, traces, and handling of side effects.
As it currently stands, a model can plan out the actions of an agent including choosing the tools to be used, passing in the appropriate arguments (e.g. file paths, email addresses, etc.), obtaining credentials for the actions, which the model requests but the runtime can mint as a scoped token with an expiration time and trace information, and repeating loops of behavior. The harness can stop the agent from continuing down a particular path if the agent’s behavior crosses a policy boundary.
The harness is therefore becoming the lock-in layer for agent governance. The number of components, including runtime policy, identity, memory, tool routing, and evals, that get to accumulate on the harness is far greater than on a model. Microsoft has a large architecture deep dive into the components that form the Agent OS, including Agent Mesh, Agent Hypervisor, Agent Runtime, Agent SRE, and Agent Compliance. As with SRE domains, the areas of concern map exactly.
The runtime sits between the model and the application
The AI Runtime Infrastructure paper describes AI Runtime Infrastructure as execution-time infrastructure that observes, reasons, and intervenes in the behavior of an AI agent for goals related to task success, latency, token efficiency, reliability, and safety. Governance typically occurs after a model has determined a course of action and before an application is invoked to execute the resulting action.
I like the frame because it forces the boring implementation question. Where does the action pause? Who signs the decision? What trace carries the receipt? At Focused, we keep coming back to that seam because it is the last safe place to turn intent into governed execution before the agent touches production.
A runtime policy gate needs to see more than just the name of the tool, e.g. “run SQL.” It needs to know the identity of the agent, the task that the agent is currently trying to complete, the current path that the agent is following, the retrieved context, previous denials, the current set of approvals, the current resource budget, the customer boundaries, the data classification, and the organization state. To make a decision, a customer-service agent can refund an order of up to $200 as long as the account has been verified successfully and no unrelated support notes have been retrieved on the current path. A coding agent can open a pull request after the tests for the associated code change have passed successfully. It cannot merge auth code without first receiving approval from the relevant security department.
Trace evidence makes governance inspectable
Blocking an action safely is better than allowing it, and turning that block into an eval case is how a system will improve. OpenTelemetry’s GenAI agent semantic conventions specify spans for creating an agent, invoking an agent, invoking a workflow, and executing a tool. A trace should have all the run information, tool attempted, policy version, decision, reason for decision, approval state, any subsequent approval’s revised decision, and result of downstream actions.
Honeycomb’s Q&A for AI-assisted development correctly observes the AI agent as a program making network calls, reading and writing data, with latency characteristics and failure modes, to be observed with the same tools that observe software: traces and logs. Therefore, the trace for a blocked tool call contains a receipt, as does a risky approval, a policy override, and so on. These are used for incident review, as eval cases, to change the harness. The data from these observations is accurate only to the degree that it is observable, i.e., turned into evidence, not into an averaged metric displayed on a dashboard. That is how accuracy becomes observable evidence.
Governance gets better when denied actions and risky approvals become eval and policy evidence.
Governance has to own the boring artifacts
Name the agents. Scope the credentials. Register the tools. Sign the skills. Record the policy version. Log the decision. Attach the trace. Route the approval. Preserve the receipt. Turn bad paths into evals. Roll out policy changes like software changes.
Skills are also important to highlight in this framework, because skills are executable, have tool expectations, have hidden assumptions, and have an upgrade path. I wrote a post recently explaining why agent skills are a software supply-chain surface. For exactly the same reasons that we run software through test cases and sanity checks before promoting it through the supply chain, a skill can change what an agent tries to do before policy ever sees the tool call, so it too must be governed.
The governance boundary follows the side effect
To determine the boundaries of governance for an AI system, test the agent by seeing where it can do damage. The areas where an AI can cause harm are tool calls, sending messages, writing files, updating databases, making payments, deploying software, merging pull requests, and launching subagents. Governance needs to occur right before the side effect of each of these. As such, the determination about taking a particular action by an agent needs to travel with the resulting evidence of that action.
AI agent governance follows the execution path. The risk of actions by an AI agent follows the execution path of that AI agent. Thus, the company’s governance of an AI agent follows the execution path of that AI agent. In other words, governance is determined for the next action, in that run, after those prior steps, under that policy, with that identity, before that side effect. Follow the execution path and own that path, or do not govern the agent and let it decide what to do with that path.
Top comments (0)