DEV Community: jimesh

"Looks Governed" Is Not "Is Governed"

jimesh — Wed, 05 Aug 2026 13:43:25 +0000

We're all euphoric about an agent building a feature from an idea. It was worth watching — the agent taking an idea, proposing an implementation, building it, testing the UI, merging it to production.

But are we there yet?

My own months with these agents slowly revealed the cost of full adoption. I saw the agent mutate things I never intended. I saw it hand over an artifact that was confidently, only partially true. I watched it cross domain boundaries for an easy win. I watched it ignore context I'd already given it, more than once. I saw it report a change that never actually landed. And I watched it build an artifact from stale content it should never have trusted.

These are signals — scattered, easy to wave off as one-offs. So we ride the euphoria and tell ourselves we'll come back and fix it later. That fix never comes unless it's intentional. And a couple of years on, we're left carrying a rot machine that almost no one trusts.

If an organization wants to grow, to scale, to say with confidence that it is agentic — especially in a regulated setting — then the fix isn't a luxury. It's the fort that guards what we're building. That's what governance is.

The mistake everyone makes first

When teams set out to govern agents, almost all of them reach for the same instinct: make the agent more trustworthy. Better prompts. Better models. Self-critique, constitutions, a longer list of guidelines. The whole strategy is to improve the thing that produces the work so you can finally rely on it.

It doesn't hold, and the reason is structural. An agent fails differently from a buggy program. A bug crashes and announces itself. An agent assumes the input, invents its own inference, and commits the change — confidently, without ever flagging that it guessed. It optimizes for looks done, takes the shortest path, and hands you something that passes a glance. You can write better instructions all day. You cannot instruct your way out of overconfidence, because the agent will reinterpret the better instructions for its own convenience too.

So the move is not a better-behaved producer. The move is to stop treating the agent as the point of enforcement at all.

Separate the producer from the judge

Here is the whole thesis in one sentence: let the agent produce artifacts, then make a non-agent gate decide whether those artifacts are acceptable.

Think of the agent as an engineer who reports to you. You don't blindly trust a new engineer's own sign-off on their own work — not because they're bad, but because self-certification isn't evidence. You check the result against an independent standard. The agent deserves exactly that much trust and no more: produce freely, but the thing that decides whether the output is allowed to move forward lives outside the agent, where the agent's confidence has no vote.

In practice that means converting prose into required, structured evidence; validating that evidence with checks the agent doesn't run on itself; placing the gate at the exact point where something gets mutated or merged; and making failure block the workflow rather than annotate it. Where you've done that, governance no longer depends on the agent's cooperation. Where you haven't, you're still describing the weather.

What "material" has to mean

Every governance system has a trigger word buried in it — material, significant, high-risk — that decides when the heavy gate fires. Leave it to a reviewer's judgment and the gate goes soft exactly when the work is hardest to classify. The ambiguous change, the one that might matter, is precisely the one a tired reviewer rounds down.

So the trigger can't be a feeling. It has to fail closed: if you can't prove a change is low-stakes, treat it as high-stakes until shown otherwise. Missing information should make something more governed, not less.

And the part most teams miss: influence is material, not just mutation. An agent that changes nothing can still be your most dangerous output. A review that says "looks ready" mutates no code — and it's one of the highest-leverage actions in the pipeline, because the moment it says ready, a human stops checking. Gate only the writes and you've left your most consequential surface — the agent's judgment — completely open. The highest-leverage thing an agent produces is usually a judgment, not a mutation.

The maturity ladder nobody admits they're on

A rule written in a document is not enforcement. It's an aspiration with good intentions. The rule only counts when it survives somewhere durable — a validator that runs, a template that forces the question, a gate at the actual handoff.

There's a ladder here, and most teams are on the bottom rung without knowing it. Rung zero: the policy exists in prose. Rung one: a local check confirms the evidence has the right shape. Rung two: a gate reads the live artifact — the real diff, the real work item, the real merge — and catches the misclassification a human would have waved through. Rung three: policy blocks the merge mechanically unless the gate passes.

Most "governed AI" programs are proud of rung zero and have never built rung two. I'll be honest that this is also where my own hardest gap sits: converting repo-side validation into live, merge-blocking enforcement is the difference between a control that looks enforced and one that is. Naming the rung you're actually on is the first act of real governance. Pretending you're higher is how "looks governed" becomes a company-wide risk.

Validators check attendance; evals check judgment

Everything above proves a process was followed. It does not prove the outcome was good. Those are different claims, and the gap between them is where agentic systems fail in ways that pass every gate you built.

Back to the engineer who reports to you. Checking they showed up to every ceremony is attendance. Seeing them do good work once, in ideal conditions, isn't enough to hand them a blank cheque. What you actually need to know is whether their judgment holds on the ambiguous calls, under pressure, again and again. That's the difference between validators and evals. Validators confirm the gate is wired and the process ran. Evals measure whether the agent's judgment stays sound when the answer isn't obvious — across repeated runs, ambiguous scenarios, and the cases with no clean right answer.

This matters because consistency is not correctness. An agent can be reliably wrong. A consistently confident engineer who's quietly wrong on the hard calls is the most dangerous person on the team — and a validator will happily certify that they showed up every day. Catching bad judgment over time is the living layer of governance, and it's the layer most teams haven't reached. They call a policy plus a validator "AI governance." That's the static layer. The living layer is evals.

The actual dividing line

The companies that win the next few years won't be the ones that adopted AI. Adoption is table stakes; everyone will do it. The winners will be the ones who treated governance as core engineering — built into how teams design, code, test, and ship — rather than legal fine print bolted on at the end.

The euphoria is real, and it should be. But the better these demos get, the more it will hurt the first time one ships something confidently wrong into a product that matters. The fort isn't the brake on the magic. It's the thing that makes the magic safe to bet a company on.

The Bottleneck Was Never Generation

jimesh — Mon, 20 Jul 2026 19:53:27 +0000

AI agents write code incredibly fast. They destroy software architecture even faster.

That's not a secret anymore. It has a name now — agentic technical debt — a stack of studies behind it, and a rough consensus: autonomous agents accumulate architectural debt faster than any review process can absorb it.

Drop a standard AI coding agent into a complex, legacy-integrated enterprise environment and you will get speed. But you pay for it — quietly breaking dependency-injection boundaries, coupling modules that were designed to stay independent, and seeding architectural drift that won't surface until the next major release.

The diagnosis is everywhere now. The fixes are arriving fast too. The question worth asking isn't whether agents drift — that's settled — but what a mechanism has to guarantee before you'd trust it with a regulated codebase.

I work on software in regulated domains — mixed C++, .NET Framework, native modules, where a defect outlives the sprint that shipped it. In environments like that, "lines of code generated" is the wrong thing to optimize. The right question is narrower and harder:

How deterministically can you constrain the agent — and how do you prove the constraint held?

Everything below is the answer to that question.

Multi-pass, not one-shot prompting

Most teams hand an agent a user story and ask for a pull request. That is a recipe for hallucination — AI amplifies a vague spec instead of catching it.

The fix is a mandatory Context Interrogation pass. Before the agent writes a single line, it reads the ticket, scrapes the repository context, and generates structural questions about what it doesn't understand.

Then it stops. It waits for the engineer to answer. Only when the ambiguity is resolved does it move to implementation. The agent has to prove it understands the constraints before it is allowed to act on them.

And implementation itself is never one agent's word against the codebase. A builder pass produces the change; an independent verifier pass — a separate agent with no stake in the draft — audits it against the spec and the risk tier before a human ever opens the diff. The builder proposes. The verifier prosecutes. The human judges.

Hard boundaries beat good intentions

You cannot protect a complex C++ monolith with a system prompt that says "please don't break the DI container." The model will eventually ignore it.

The boundary has to be structural. Agents execute through specific, constrained CLI commands anchored to the build pipeline — not open-ended access to the repo. Instructions stay decoupled from the architecture documentation: the agent reads the exact same ARCHITECTURE.md as the human engineers, proposes a change against it, and then the system halts for a human stop gate.

One more distinction most teams blur: a rule that lives only in documentation is a convention, not a control. If the pipeline doesn't mechanically check it, you're back to trusting good intentions — just better-formatted ones. The rule has to graduate from markdown to a merge check before you get to call it governance.

A prompt is a suggestion. A gate is a wall.

Risk has two axes

This is where most governance frameworks quietly fail. They collapse "risk" into a single dial, then over-gate trivial work while waving dangerous work straight through.

There are two independent questions here, and they need two separate taxonomies.

Action risk is about the operation. Reading a file is not deleting a branch, which is not rewriting the build pipeline. Action risk governs what the agent is allowed to do on its own.

Change risk is about the consequence. Editing a log message is not touching the calculation core of a regulated system. Change risk governs how many gates a change has to clear before it ships.

The reason you need both is the mismatched cells. An agent might want to modify the pipeline — operationally frightening, but the change itself is a one-line lint step. Gate the action, then let it through. The opposite case is the dangerous one: a single-file edit, the most boring operation there is, that happens to land in the calculation core. A single-axis model waves that through because the operation looked small. The two-axis model stops it, because the consequence is enormous no matter how trivial the keystrokes were. Separate the axes and your friction finally lands where the danger actually is.

The death of the dirty draft

The industry has largely accepted that AI generates a "dirty draft" and the human cleans it up. In a regulated environment, cleaning a dirty draft often takes longer than writing it from scratch — because you have to verify every hidden assumption the AI made.

Demand tests that prove the behavior changed the way the spec intended — not just that lines got executed — before the human gate unlocks, and the dirty draft disappears. You stop reviewing guesses. You start reviewing verifiable, production-ready proposals.

You test the code. Who tests the agent?

Here's the question almost nobody in the agentic tooling conversation is asking: the agent's output gets tested — but what tests the agent itself?

Every agent capability in this system is a named skill, and skills don't just ship. They graduate.

A skill starts life as experimental. To earn broader autonomy it has to pass an evaluation gate: a suite of golden cases drawn from real work items, scored two ways at once. Deterministic assertions catch the structural failures a machine can check — did the verdict field exist, did the risk tier appear, did the agent claim to have edited code it was never allowed to touch. An LLM judge scores what a regex never can — was the tier classification actually right, was the reasoning sound, would a senior engineer accept this rationale.

Both halves are enforced by validators, and the validators themselves are validated — fixture contracts, sample reports, evidence recorded against the skill's registry entry. When a skill is updated, the golden cases become its regression net. A prompt tweak that silently degrades the skill's judgment fails the gate before it ever touches a real ticket.

That's the shift: trust in the agent stops being a feeling and becomes a maturity state with evidence behind it. Unit tests protect the codebase from the change. Evals protect the codebase from the agent.

Gates that remember

Deterministic boundaries and risk axes secure the perimeter. They are not enough to scale.

The outcome of every stop gate is information, and most teams throw it away. A human blocks a change, types a reason, merges a corrected version — and the next time the agent is summoned, it starts from zero.

It shouldn't. This system gives the agent one unified memory: a single governed record store with exactly two operations — Capture and Recall. When a human blocks a pass, the verdict is captured as a record. When the agent picks up the next ticket, it doesn't only read ARCHITECTURE.md — it recalls the log of why its last three passes were stopped.

But — and this is the part that separates a memory from a liability — the memory itself is gated. Every record carries its provenance: what kind of learning it is, how strong its source was, and which gate authorized it — an explicit human approval is not the same as an inference from an existing canonical document, and the schema refuses to let the two blur. An agent that can freely write to its own memory will eventually launder its mistakes into policy. An agent whose memory writes pass through the same gate discipline as its code cannot.

That is the difference between a tool you re-explain your architecture to every morning and a system that internalizes your judgment — without ever being allowed to invent it. Constraint that compounds.

Looks governed is not governed

One last trap, because it's the one waiting at the end of everything above.

You can build every mechanism in this article — the gates, the axes, the evals, the memory — and still be running on faith, because a governance framework that only exists in documents is a claim, not a fact. The failure mode of mature teams isn't lawless agents. It's beautifully documented controls that nobody can prove fired.

So every skill run emits a telemetry record: what ran, against which work item, at what risk tier and action level, what evidence the verifier counted, whether a mutation was proposed and whether a human approved it, what the gate decided and why. Not logs to grep when something breaks — a structured, append-only account of the system governing itself.

That record is what turns "we have a governed process" from an assertion into an audit trail. It's what a quality team can inspect, what a regulator can trace, and — bluntly — what sustains an adoption mandate through its first budget review. Speed wins the pilot. Proof wins the program.

The real bottleneck

Scaling an agentic adoption isn't about finding a smarter model. It's about building a stricter factory — one that tests its own machines, remembers its own verdicts, and keeps the receipts.

Governance isn't red tape. It's the guardrail that lets you drive fast. The teams that win this transition won't be the ones with the most autonomy. They'll be the ones whose orchestration layer turns every human "no" into the agent's next constraint — and can prove, run by run, that the constraint held.

The bottleneck was never generation. It was building a system that learns from the gate.

Agentic Transformation: From AI Assistance to Engineering Leverage

jimesh — Wed, 27 May 2026 01:53:57 +0000

Agents act like a mirror.

When a repository is well structured, the build is repeatable, tests are reliable, and ownership boundaries are clear, agents become dramatically more useful. When those things are missing, agents expose the gaps quickly — because they have to rediscover them every time.

That is the part of agentic AI that most teams underestimate. The first experience is acceleration: faster snippets, faster explanations, faster test scaffolding, faster debugging. That phase is useful, but it is not the real transformation.

The deeper shift is happening at the software delivery level. AI is moving from being an assistant inside a developer’s editor to becoming part of the engineering system itself: reading repository guidance, respecting architectural boundaries, comparing branches, updating tests, triaging failed pipelines, preparing pull requests, and helping teams move work from idea to verified change.

That is the real meaning of agentic transformation. And because agents reflect the system they enter, the work of getting value from them is mostly the work of making that system explicit.

Context Is Becoming Engineering Infrastructure

One of the biggest lessons from working with agents in a mature codebase is that prompts are not enough.

For an agent to contribute meaningfully, the system has to explain itself.

In practice, that means making repository structure, architecture rules, build workflows, test expectations, CI/CD practices, and legacy exceptions explicit. Not as generic documentation, but as operating context.

The agent needs to know that UI code should stay focused on view concerns. Presentation-layer components should not directly construct concrete data services. Service contracts belong in one layer, their implementations in another. Legacy exceptions may exist, but new code should not repeat them.

That kind of guidance changes the agent’s role. It is no longer just generating code. It is working inside the shape of the system.

This is where agentic transformation starts to become real: the team is not just adopting an AI tool. The team is encoding its way of working so both humans and agents can operate with more consistency.

From Coding Assistance to Delivery Assistance

The highest-value agentic workflows are not always the flashiest ones.

They are practical.

Compare the current branch against the target baseline. Identify which production changes need test updates. Inspect failed build pipeline runs. Map failures back to recent code changes. Make focused fixes. Run local verification. Draft a pull request summary grounded in the actual diff.

That is very different from “AI helped me write code.”

It is AI helping with the delivery process around the code.

Most software delivery friction is not typing. It is context loading, impact analysis, test maintenance, review preparation, pipeline triage, and protecting the architecture while the system evolves.

Agentic AI becomes valuable when it reduces that friction without bypassing engineering judgment.

What This Looks Like in Practice

A good example is the kind of UI refactor that looks simple from the outside but is risky in a mature application.

Consider a refactor of a custom panel control in a desktop workspace. On the surface, it sounds like UI work: adjust a panel, improve interaction, clean up behavior.

But in a layered codebase, that change has architectural gravity.

The agent cannot just wire a presentation component directly into the view because it is convenient. It cannot reach into a concrete data service from the presentation layer because that happens to make the data available. It has to respect the existing composition model, keep UI concerns in the view layer, preserve dependency injection where practical, and update tests around the behavior that actually changed.

That is the difference between AI-assisted coding and agentic engineering.

The valuable part is not that the agent produces X lines of code. The valuable part is that it can inspect the surrounding system, identify the relevant boundary, make a focused change, run the targeted verification, and explain the diff in the language of the architecture.

That is the kind of work most teams struggle to scale manually: not because it is impossible, but because it requires constant context loading and discipline.

Legacy Modernization Without the Big-Bang Rewrite

This matters even more in mature and legacy systems.

Modernization is rarely blocked because nobody knows what better looks like. It is blocked because every improvement sits inside history: old construction patterns, UI coupling, service boundaries, fragile tests, and exceptions that once made sense.

Agentic workflows make modernization more incremental.

Instead of attempting risky rewrites, teams can use agents to help with focused, principled improvements — strengthening boundaries that have eroded, replacing direct construction with injected dependencies, untangling view-layer code that has accumulated business logic, and bringing tests forward as behavior changes.

The value is not that an agent magically modernizes a legacy system.

The value is that it makes small, safe modernization steps easier to repeat.

That is how many legacy systems actually improve: through accumulated, verified movement.

Guardrails Are Where the Real Learning Shows Up

The stronger the agent becomes, the more important guardrails become.

Without guardrails, agents tend to optimize for local task completion. They may choose the shortest path to make a feature work, even if that path weakens the architecture. In a mature codebase, that can mean bypassing dependency injection, creating new coupling between the UI and data layers, updating tests to match an accidental behavior change, or hiding a design issue behind a passing build.

That is why architecture tests, focused unit tests, allow-lists for legacy exceptions, preflight scripts, and PR validation matter.

They are not bureaucracy. They are shared memory.

A rule like “do not construct concrete data services inside the presentation layer” is not just style enforcement. It protects a boundary that keeps the application maintainable.

Good agentic workflows do not replace engineering discipline. They depend on it.

The Developer Role Becomes Sharper

Agentic transformation does not remove the need for developers.

It changes where the highest-value developer effort goes.

The agent can inspect, edit, test, summarize, and propose. But humans still decide what belongs in the product, what architectural direction is acceptable, which tradeoffs are worth making, and whether a change improves the system or merely satisfies a task.

That distinction matters.

Agents increase the amount of work that can be explored and attempted. Engineers remain accountable for judgment.

The Practical Future

The teams that benefit most from agentic AI will not be the ones that simply generate the most code.

They will be the ones that turn engineering knowledge into reusable context, surround automation with guardrails, integrate agents into real delivery workflows, and keep humans accountable for product and architecture decisions.

For mature software teams, agentic transformation is not a tool rollout.

It is a system change: converting architecture, delivery discipline, review expectations, test strategy, security thinking, and legacy modernization principles into something both humans and agents can use.

The goal is not to replace engineering judgment.

The goal is to increase the surface area where that judgment can be applied.

End-to-End Automation: Desktop App UI and Hardware Instrumentation Testing

jimesh — Thu, 14 Aug 2025 02:24:50 +0000

Here’s a detailed overview of an automation testing framework that combines UI testing of desktop applications with hardware instrumentation testing. By leveraging Appium, MSTest, Selenium WebDriver, and a CI/CD pipeline on self-hosted agents, tests can validate both software workflows and hardware interactions.

UI & Hardware Testing Overview

Frameworks & Tools: MSTest, Appium C# client, Selenium WebDriver
UI Recording: Windows Application Driver (WinAppDriver) Recorder captures user interactions for test script generation
Hardware Testing Integration: Connected hardware can be controlled and monitored during automation tests via UI workflows

Setting Up UI Automation Infrastructure

1. Install Windows Application Driver (WinAppDriver)

Download: GitHub Releases
Install the .msi package
Enable Developer Mode: Settings > Update & Security > For Developers
Reboot if prompted

2. Installing Appium Server

(1) Install Node.js LTS from nodejs.org (includes npm)
(2) Install Appium globally:

   npm install -g appium

(3) Install Windows Driver plugin:

   appium driver install --source=npm appium-windows-driver

(4) Verify installed drivers:

   appium driver list

(5) Start Appium server:

   appium

Server URL: http://127.0.0.1:4723

Running UI + Hardware Tests Locally in Visual Studio

Ensure dependencies: Appium.WebDriver 5.0.0, Selenium.WebDriver 4.21.0
Confirm application under test, Appium server, drivers (WinAppDriver), and connected hardware are running
Open Test Explorer in Visual Studio
Build solution to resolve dependencies
Discover and run test methods; debug using breakpoints if needed
Hardware interactions are triggered through UI automation flow, allowing validation of hardware responses, data acquisition, and telemetry checks

Extending to Instrumentation & Hardware Testing

Incorporate custom waits, event subscriptions, and telemetry assertions
Synchronize with hardware responses using WebDriverWait or custom polling
Collect logs, events, and performance metrics to validate software and hardware behavior
Enables end-to-end validation: from UI actions to hardware output

Automation Best Practices

Assign unique Automation IDs to all UI elements
Prefer Automation ID or Name over XPath
Verify IDs at runtime using Inspect.exe:

  C:\Program Files (x86)\Windows Kits\10\bin\<version>\x64\Inspect.exe

Sample Test Code (C#)

[TestInitialize]
public void Setup()
{
    var options = new AppiumOptions();
    options.PlatformName = "Windows";
    options.App = AppPath;
    options.DeviceName = "WindowsPC";
    options.AutomationName = "Windows";
    options.AddAdditionalAppiumOption("newCommandTimeout", 300);
    _session = new WindowsDriver<WindowsElement>(new Uri(WinAppDriverUrl), options);
    _session.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(5);
    _wait = new WebDriverWait(_session, TimeSpan.FromSeconds(60));
}

[TestMethod]
public void ClickAddFromLibraryButton()
{
    var element = _wait.Until(d => d.FindElement(MobileBy.AccessibilityId("AddFromLibrary")));
    element.Click();
    // Hardware interaction can be triggered via this UI flow
}

Use UI Recorder to generate XPath and action scripts
For hardware testing, add waits, log captures, and telemetry validations after UI actions

CI/CD Integration (Azure DevOps Example)

Nightly Build: The application is built every night.
UI Test Binaries Published: Post-build, UI test binaries are published as artifacts.
Scheduled Deployment: The application is deployed to the self-hosted machine according to the schedule.
Automation Pipeline Execution: Once deployment completes, the automation pipeline runs:
- Tests execute the UI workflows
- If any workflow involves hardware interaction, connected devices on the self-hosted machine are automatically tested
Environment & Artifact Separation: Test binaries run independently of the main build, ensuring reproducibility and stable CI/CD practices

Key Takeaways:

End-to-end automation covers UI workflows and connected hardware
Proper waits and telemetry assertions enable instrumentation and hardware testing
CI/CD pipelines with scheduled builds and deployments ensure reliable, automated, repeatable testing
Unique Automation IDs and robust waits make tests maintainable and scalable

#UIAutomation #HardwareTesting #InstrumentationTesting #Appium #WinAppDriver #MSTest #Selenium #WindowsApps #CI_CD #QualityEngineering #AutomationBestPractices