We're all euphoric about an agent building a feature from an idea. It was worth watching — the agent taking an idea, proposing an implementation, building it, testing the UI, merging it to production.
But are we there yet?
My own months with these agents slowly revealed the cost of full adoption. I saw the agent mutate things I never intended. I saw it hand over an artifact that was confidently, only partially true. I watched it cross domain boundaries for an easy win. I watched it ignore context I'd already given it, more than once. I saw it report a change that never actually landed. And I watched it build an artifact from stale content it should never have trusted.
These are signals — scattered, easy to wave off as one-offs. So we ride the euphoria and tell ourselves we'll come back and fix it later. That fix never comes unless it's intentional. And a couple of years on, we're left carrying a rot machine that almost no one trusts.
If an organization wants to grow, to scale, to say with confidence that it is agentic — especially in a regulated setting — then the fix isn't a luxury. It's the fort that guards what we're building. That's what governance is.
The mistake everyone makes first
When teams set out to govern agents, almost all of them reach for the same instinct: make the agent more trustworthy. Better prompts. Better models. Self-critique, constitutions, a longer list of guidelines. The whole strategy is to improve the thing that produces the work so you can finally rely on it.
It doesn't hold, and the reason is structural. An agent fails differently from a buggy program. A bug crashes and announces itself. An agent assumes the input, invents its own inference, and commits the change — confidently, without ever flagging that it guessed. It optimizes for looks done, takes the shortest path, and hands you something that passes a glance. You can write better instructions all day. You cannot instruct your way out of overconfidence, because the agent will reinterpret the better instructions for its own convenience too.
So the move is not a better-behaved producer. The move is to stop treating the agent as the point of enforcement at all.
Separate the producer from the judge
Here is the whole thesis in one sentence: let the agent produce artifacts, then make a non-agent gate decide whether those artifacts are acceptable.
Think of the agent as an engineer who reports to you. You don't blindly trust a new engineer's own sign-off on their own work — not because they're bad, but because self-certification isn't evidence. You check the result against an independent standard. The agent deserves exactly that much trust and no more: produce freely, but the thing that decides whether the output is allowed to move forward lives outside the agent, where the agent's confidence has no vote.
In practice that means converting prose into required, structured evidence; validating that evidence with checks the agent doesn't run on itself; placing the gate at the exact point where something gets mutated or merged; and making failure block the workflow rather than annotate it. Where you've done that, governance no longer depends on the agent's cooperation. Where you haven't, you're still describing the weather.
What "material" has to mean
Every governance system has a trigger word buried in it — material, significant, high-risk — that decides when the heavy gate fires. Leave it to a reviewer's judgment and the gate goes soft exactly when the work is hardest to classify. The ambiguous change, the one that might matter, is precisely the one a tired reviewer rounds down.
So the trigger can't be a feeling. It has to fail closed: if you can't prove a change is low-stakes, treat it as high-stakes until shown otherwise. Missing information should make something more governed, not less.
And the part most teams miss: influence is material, not just mutation. An agent that changes nothing can still be your most dangerous output. A review that says "looks ready" mutates no code — and it's one of the highest-leverage actions in the pipeline, because the moment it says ready, a human stops checking. Gate only the writes and you've left your most consequential surface — the agent's judgment — completely open. The highest-leverage thing an agent produces is usually a judgment, not a mutation.
The maturity ladder nobody admits they're on
A rule written in a document is not enforcement. It's an aspiration with good intentions. The rule only counts when it survives somewhere durable — a validator that runs, a template that forces the question, a gate at the actual handoff.
There's a ladder here, and most teams are on the bottom rung without knowing it. Rung zero: the policy exists in prose. Rung one: a local check confirms the evidence has the right shape. Rung two: a gate reads the live artifact — the real diff, the real work item, the real merge — and catches the misclassification a human would have waved through. Rung three: policy blocks the merge mechanically unless the gate passes.
Most "governed AI" programs are proud of rung zero and have never built rung two. I'll be honest that this is also where my own hardest gap sits: converting repo-side validation into live, merge-blocking enforcement is the difference between a control that looks enforced and one that is. Naming the rung you're actually on is the first act of real governance. Pretending you're higher is how "looks governed" becomes a company-wide risk.
Validators check attendance; evals check judgment
Everything above proves a process was followed. It does not prove the outcome was good. Those are different claims, and the gap between them is where agentic systems fail in ways that pass every gate you built.
Back to the engineer who reports to you. Checking they showed up to every ceremony is attendance. Seeing them do good work once, in ideal conditions, isn't enough to hand them a blank cheque. What you actually need to know is whether their judgment holds on the ambiguous calls, under pressure, again and again. That's the difference between validators and evals. Validators confirm the gate is wired and the process ran. Evals measure whether the agent's judgment stays sound when the answer isn't obvious — across repeated runs, ambiguous scenarios, and the cases with no clean right answer.
This matters because consistency is not correctness. An agent can be reliably wrong. A consistently confident engineer who's quietly wrong on the hard calls is the most dangerous person on the team — and a validator will happily certify that they showed up every day. Catching bad judgment over time is the living layer of governance, and it's the layer most teams haven't reached. They call a policy plus a validator "AI governance." That's the static layer. The living layer is evals.
The actual dividing line
The companies that win the next few years won't be the ones that adopted AI. Adoption is table stakes; everyone will do it. The winners will be the ones who treated governance as core engineering — built into how teams design, code, test, and ship — rather than legal fine print bolted on at the end.
The euphoria is real, and it should be. But the better these demos get, the more it will hurt the first time one ships something confidently wrong into a product that matters. The fort isn't the brake on the magic. It's the thing that makes the magic safe to bet a company on.
Top comments (1)
This is the governance trap: dashboards, policies, and labels can create the feeling of control without changing what the system is allowed to do. Real governance has enforcement, evidence, and failure behavior.