Originally published at devopsdiary.blog. Post F-AID3 in the "Governing AI in the Enterprise" series.
A validator that fixes the thing it just flagged has quit its job. It stopped grading and started doing the homework, in the same motion, and now you can't tell which part of the result you're supposed to trust.
I keep running into this in the tooling everyone's shipping right now. The AI-dev vendors have converged on a lot of good ideas over the past year. Most of them I want to steal. One of them I'll fight about, because it quietly breaks the one property that made the whole setup governable in the first place.
Let me take the good ones first. I'm not interested in being the guy who only shows up to say no.
Three ideas worth stealing
The first is a piece of vocabulary: verification debt. Sonar's been using it to name the gap between the quality an agent produces by default and the quality a long-lived, business-critical app actually needs. That gap has always existed. We used to file it under "tech debt" and move on, which was lazy, because tech debt is what you owe after a shortcut you chose. Verification debt is what you owe after a machine made a hundred choices you never saw. Different problem. Better name. I'm adopting it.
Second: inner loop and outer loop as a way to say where a check runs. Inner loop is inside a single reasoning step, the agent second-guessing itself mid-thought. Outer loop is across the whole task, after the work is notionally done. It sounds like pedantry until you try to explain to a security team where your quality gate lives, and you realize you've been waving your hands at "somewhere in the agent." The outer loop is where the promotion gate belongs. Naming it makes the gate a thing you can point at.
Third, and this is the one with teeth: shadow testing. Run the new agent in parallel with the real one, write path disabled, and compare what it would have done against what actually happened. One payroll-automation team took an agent from 70 percent accuracy to 98 before it ever touched production, purely by running it in the dark and grading it against humans for a few weeks. That only works because the live version is frozen while the shadow runs. Same inputs, no side effects, honest comparison. It's the most disciplined testing idea I've seen come out of the agent world, and it maps cleanly onto something I already believed: don't promote what you haven't watched behave.
So far, so agreeable. That's where I get off the bus.
The one I won't take
Sonar's framework has a stage they call Solve. The validator doesn't just find the problem. It fixes it. Finds the bug, writes the patch, closes the loop, all inside the thing whose job was to tell you whether the code was any good.
I understand why it demos well. It feels like magic to watch a tool flag an issue and resolve it in the same breath. But think about what you just did to your audit trail. The finding and the fix collapsed into one event, with one author, and that author is the same component that decided the code was wrong. You've asked the grader to fix your answer before scoring it. It's going to give you an A. It has no reason not to.
In a regulated shop the cost is concrete, and it lands on a Monday morning. Something ships broken, the review board asks "what changed, who approved it, and what did the check actually catch," and you need those to be three separate facts. A validator that remediates smears them into one line: the tool found it, the tool fixed it, promoted clean. Nobody looked. There's no daylight between the judgment and the intervention, which means there's nowhere to stand and ask whether the judgment was right.
And there's a blast-radius problem hiding underneath. The second your validator can write, it's not a read-only observer anymore. Its scope just went from "look and report" to "look and change your code." That's a different security posture, a different threat model, and a different conversation with your platform team, and most people adopt it without having any of those conversations because it arrived bundled as a feature.
The honest counterargument
I don't want to pretend this is a clean win. The pull toward Solve is real, and the people building it aren't fools.
Fast feedback is how you get the good numbers. When the fix lands a half-second after the finding, iteration speed goes through the roof, and there's a genuine prize on the table: teams reporting something like nine in ten issues caught at edit time, before a human ever reviews a line. If I forbid the validate-and-remediate collapse with a hard rule, I'm leaving some of that speed on the floor. A strict "no" has a cost, and anyone who tells you otherwise is selling something too.
So the question isn't whether remediation is valuable. Obviously it is. The question is whether it belongs inside the validator, and my answer is still no.
Where AIEOS lands
The invariant I build on is that validators judge, they don't help. When I say that, people hear "never auto-fix anything," and that's not what I mean. Auto-fix all you want. Just don't let the grader hold the pen.
In practice, remediation is a separate artifact with its own author. A remediation agent consumes the validator's verdict and proposes a change. That change goes back through the same gate as any other change, from a frozen baseline, and has to clear it on its own merits. The verdict and the fix stay two events, two authors, two timestamps.
Agent output
|
v
[ Validator: judges only ]
|
+--- pass -----------------------+
| |
+--- fail --> [ Remediation |
agent: |
proposes fix ] |
| |
v v
[ Promotion gate: frozen baseline ]
|
+-----+------+
| |
clears fails --> back to remediation
|
v
Promote
The validator emits a verdict and stops. A separate remediation agent proposes the fix, which clears the same promotion gate as any other change. Two authors, two events, one audit trail.
You keep the speed. You keep the audit trail. What you give up is the thing that was never yours to keep: the idea that the check and the cure can be the same act without anyone losing track of which was which. Freeze-before-promote does the load-bearing work. The fix doesn't get to skip the line because it came from a smart tool. It clears the gate like everything else, or it doesn't ship.
The part I haven't figured out
If remediation doesn't live in the validator, where does it live? I have a working answer, a separate agent under its own governance, but I'm not convinced that's the final shape. Maybe it's a first-class layer. Maybe it's a practice folded under verification with hard rules about artifact separation. The industry hasn't settled this, and neither have I, and I'd rather say that plainly than paper over it with a diagram.
The one line I'll hold, though, is narrow enough to defend: the thing that decides whether your code is good doesn't also get to make it good. Keep those two jobs in two hands. The day they merge, you've automated away the only honest signal you had, and you won't notice until the thing you trusted to catch problems becomes the thing quietly creating them.
Todd Linnertz is a Senior Solutions Engineer with thirty years of enterprise engineering experience. He's the creator of AIEOS, an open-source AI governance system for software delivery teams. Find him at devopsdiary.blog and github.com/wtlinnertz.
Top comments (0)