DEV Community

Shawn knight
Shawn knight

Posted on Originally published at Medium on

2026 Agents Case Study Series: Let the Code Disagree


2026 Agents Case Study Series Logo

Why I use AI agents to investigate, challenge, and verify what my software actually does.

There’s something I haven’t really explained throughout this series.

Why do I conduct so many audits?

If you’ve been following along, you’ve probably noticed that auditing has become a pretty significant part of how I develop software.

I audit implementations, documentation and audit architectural decisions.

I even use AI to audit conversations I’ve had with other AI systems.

And sometimes, after finishing a round of implementation work, I’ll turn around and audit the same area again.

But there’s a reason for that.

I don’t know what I don’t know. And I don’t know it until I know it.

I’m building systems that involve programming languages, execution engines, memory, scheduling, governance, and a growing collection of interconnected services.

I didn’t come into this with a formal education in software architecture.

A lot of what I understand today, I’ve learned while building.

And that means there are going to be things I don’t know to ask about.

Things I haven’t considered.

Things I think I understand but don’t fully understand yet.

So I audit.

Not because I expect every audit to find something wrong.

But because I want to discover what I might be missing.

And over time, I noticed something interesting about using different AI systems to review the same work.

I’m Looking for Disagreement, Not Consensus

I’ve seen people recommend having one AI agent build something and another agent verify it.

And I understand the idea.

But that’s not quite how I approach it.

I’m not necessarily looking for the agents to agree with each other. I’m purposely looking for where they disagree.

Because that’s where I start asking questions.

Why did one agent say the implementation was complete while another identified missing functionality?

Why did one audit classify a component as runtime infrastructure while another described it as application-specific?

Why did one review say a feature existed while another claimed it hadn’t been implemented?

Sometimes the answer is that one of the agents made a mistake.

But quite often, I’ve found that the difference comes down to what each agent actually examined.

One read the documentation.

Another inspected the code.

One was reasoning from an earlier architectural description.

Another was looking at the latest implementation.

And sometimes, what I remembered intending to build didn’t match what had actually been implemented.

Those are different sources of information.

And they don’t always tell the same story.

So rather than choosing whichever explanation sounds most convincing, I investigate the disagreement.

What evidence is each conclusion based on?

Which files were examined?

Was the implementation actually tested?

Is the documentation describing current behavior or an intended capability?

And what happens when we compare those claims against the actual system?

That’s where the audits become useful.

Not simply because they identify defects.

But because they expose gaps between what different people, documents, and AI systems believe the software does.

Ask the Repository

I have a saying:

Reality is the great equalizer.

And in software development, I’ve come to apply that principle pretty literally.

I can think I’ve built one thing.

The documentation can describe another.

A chatbot might remember an earlier version of the architecture.

And a coding agent might have received instructions based on assumptions that are no longer true.

But when the question is what has actually been implemented, I want to examine the code.

I want to know what the repository says.

And when the question is whether that implementation works, I want to see what happens when we run it.

That’s why the environment matters so much to how I use coding agents.

I’m not simply asking an AI to search my GitHub repository and tell me what it finds.

I’m working with an agent directly inside the development environment.

It can inspect the source.

Trace dependencies.

Run tests.

Examine failures.

Compare implementations with documentation.

And, when appropriate, make changes and verify the results.

Now, that doesn’t mean MCP connections or browser-based tools aren’t useful.

They are.

But for this particular kind of work, I want the agent connected to the environment where the software is actually being developed.

Because there’s a difference between reading about an implementation and being able to investigate how it behaves.

And the more complicated A.I.N.D.Y. became, the more important that distinction was.

The Prompt Was Wrong

One of the clearest examples came during work on idempotency in the A.I.N.D.Y. runtime.

If you’re unfamiliar with the term, idempotency is essentially about making sure repeating an operation doesn’t produce unintended duplicate effects.

For example, if a workflow retries an action because it didn’t receive a response, you don’t necessarily want that retry to perform the same external action twice.

I’d been discussing idempotency and invariants with Copilot.

Then I brought that conversation into Claude Code and asked it to examine which ideas were relevant to the actual runtime.

That started another audit.

The agent compared the discussion against the implementation, identified gaps, and helped turn those findings into a more concrete development plan.

But during one of the implementation phases, something happened.

The prompt contained an incorrect assumption.

It described a problem with an existing retry path and proposed a change to fix it.

Except the implementation didn’t actually behave the way the prompt assumed.

And this is where the working method mattered.

The agent had been instructed to inspect the relevant code first and stop if the implementation contradicted the assumptions in the request.

So that’s what Claude Code did.

It examined the retry path and discovered that the common case was already handled correctly.

The actual problem was narrower.

There was a potential issue involving concurrent attempts, but not the general retry behavior the prompt had described.

So instead of implementing the requested change, Claude Code stopped.

It explained what it found.

It identified the difference between the prompt and the implementation.

And it returned the question to me with a recommendation.

My reaction?

I read the explanation.

Reviewed the recommendation.

And approved it.

That was it.

Because the agent had done exactly what I wanted it to do.

It didn’t blindly follow the prompt. It checked whether the prompt was right.

And when the evidence contradicted the instruction, it gave me the information I needed to make a better decision.

The resulting change addressed the actual issue without unnecessarily modifying behavior that already worked.

That’s what I mean when I say let the code disagree.

Not that the agent should argue with me.

Not that it should disregard my instructions.

But that it should be able to investigate the assumptions behind those instructions and report when they don’t match reality.

The Sandbox That Wouldn’t Stay Finished

Now, if the idempotency work was a relatively clean example of the method, the sandbox was something else entirely.

That damn sandbox.

Of all the individual components I’ve worked on while developing the runtime, I think the sandbox may have taken more of my time than almost anything else.

And it was a headache for a very particular reason.

I’d complete a round of work.

The agent would report that the requested implementation was finished.

Then I’d conduct another audit.

And that audit would identify something else that needed attention.

Sometimes I’d look at the finding and think:

Wait a minute.

Didn’t we just do that?

So I’d have the agent check the actual implementation.

And quite often, the answer was essentially:

Yes, that part was implemented.

But there’s another issue here.

Or the implementation is correct, but this other surface isn’t reporting it properly.

Or the tests cover the intended behavior, but the documentation describes a stronger guarantee than we’ve actually verified.

And then we’d investigate that.

Fix what needed fixing.

Run the tests.

Update the documentation.

And eventually, another audit would reveal another distinction that needed to be addressed.

It wasn’t necessarily that the previous agent had failed to complete its assignment.

Sometimes it had completed exactly what was requested.

But completing a task and establishing that an entire capability works as intended are not always the same thing.

And sandboxing made that painfully clear.

The Tests Said Yes. But Did Reality?

One of the more important sandbox issues involved cross-platform support.

The runtime needed to understand what kind of isolation environment was actually available.

And that wasn’t always as simple as checking whether the computer was running Windows, Linux, or macOS.

For example, a Windows machine could run Linux containers through Docker Desktop.

So the host operating system alone didn’t necessarily tell the runtime what isolation capabilities were available.

The agents worked through the relevant detection and reporting mechanisms.

Tests were created to simulate different environments and verify how the runtime would classify them.

And those tests passed.

But that raised another question.

Did passing a simulated Windows environment prove the sandbox actually worked on Windows?

No.

It proved the implementation behaved as expected under the conditions represented by the tests.

That was useful evidence.

But it wasn’t the same as testing the capability in a live environment.

So the work continued.

Eventually, the sandbox was tested on a Windows host using Docker Desktop with a Linux container backend.

The required hardening evidence was checked, and the runtime reached the certification condition defined for that environment.

Only then could the documentation be updated to reflect what had actually been verified.

And that’s the distinction I kept encountering throughout the sandbox work.

The implementation could be correct.

The tests could pass.

And there could still be another question about whether the larger claim had been established.

A simulation isn’t a certification.

Just like a completed implementation isn’t automatically proof that the entire system works.

Sometimes the Code Was Right and the Documentation Was Wrong

Another recurring source of disagreement was documentation.

This became especially important as A.I.N.D.Y. grew into a runtime with multiple interfaces, packages, and operational surfaces.

An audit might identify what looked like an implementation problem.

But when the agent examined the code, it would discover that the behavior was intentional.

The documentation was simply describing something different.

In one sandbox-related review, the documentation described a narrower capability allowance for certain trusted code than the implementation actually provided.

So we had a decision to make.

Was the code too permissive?

Or was the documentation outdated?

Those are very different problems.

And the correct answer couldn’t be determined by automatically treating either source as authoritative.

We needed to understand the intended behavior, inspect the implementation, and determine which one needed to change.

In that case, the documented contract was brought back into alignment with the intended implementation, and tests were added to help prevent the mismatch from returning.

That was another reason I kept auditing.

Because an implementation can be correct while the information describing it is wrong.

And if an agent relies on that documentation during future development, the discrepancy can become the foundation for another incorrect assumption.

So the goal wasn’t simply to make the code work.

It was to make sure the code, tests, documentation, and operational claims were telling a consistent story.

Stop, Report, Verify

I didn’t sit down one day and invent a formal methodology for all of this.

The process evolved naturally through the work.

I would audit something.

Find a discrepancy.

Ask the agent to investigate.

Compare the findings with the implementation.

Make a decision.

Then verify the result.

And over time, those expectations began appearing more explicitly in my prompts.

Read the relevant files first.

Establish what already exists.

Don’t assume the documentation is current.

If the implementation contradicts the prompt, stop and report.

If the feature is already implemented, don’t build another version.

If a proposed change affects an architectural boundary, identify the dependencies before proceeding.

And don’t describe a capability as verified until the relevant tests or live environment have actually established that claim.

Those instructions weren’t meant to make the agent less capable.

They were meant to make its work more accountable to the system it was changing.

And I think there’s an important distinction there.

I’m not trying to eliminate the agent’s ability to reason independently.

I want it to reason.

I want it to investigate.

I want it to identify problems I haven’t considered.

But I also want it to recognize when it doesn’t have enough evidence to proceed.

And when that happens, I want it to tell me.

Not manufacture certainty.

Not implement something simply because the prompt was detailed.

Stop.

Explain what it found.

And let me decide what happens next.

Disagreement Needs Evidence Too

There’s another side to this.

Just because an agent disagrees with a prompt doesn’t automatically mean the agent is right.

The agent can misunderstand the code.

It can overlook a dependency.

It can interpret a test incorrectly.

Or it can rely on documentation that doesn’t reflect the current implementation.

That’s why I don’t treat disagreement itself as proof.

I want to know why the disagreement exists.

What evidence supports it?

Which files or execution paths were examined?

Was the behavior actually tested?

Does the finding apply to the current implementation, or is it describing something from an earlier version?

And if the available evidence is incomplete, what would we need to examine next?

That’s also why using multiple AI systems can be useful.

Not because one model automatically becomes the judge of another.

But because different analyses can expose different assumptions.

And those assumptions can be investigated.

Sometimes the first agent was right.

Sometimes the second agent was right.

Sometimes both were describing different parts of the same system.

And sometimes neither had enough information to make the claim it was making.

The important thing is having a way to find out.

Let the Code Disagree

Looking back, I think this is one of the more important changes in how I work with AI agents.

Earlier in the series, I explained how I moved from conversations into the terminal because I wanted the agent to work where the project actually existed.

Then I began using audits to understand the architecture, repair implementations, and establish clearer boundaries between the runtime and application.

But as the system grew, I started recognizing something else.

The agent’s job wasn’t simply to execute my instructions.

And my job wasn’t simply to produce increasingly detailed prompts.

We needed a way to challenge the assumptions behind the work.

Because I don’t know everything about the system I’m building.

And neither does the agent.

I can remember the original intention.

The documentation can preserve the architectural decisions.

The code can show what was implemented.

And the tests and live environment can help establish whether that implementation behaves as claimed.

Those sources won’t always agree.

And when they don’t, I don’t necessarily see that as a problem.

I see it as something worth investigating.

Because that’s often where I discover what I didn’t know.

Reality is the great equalizer.

And in my development process, that means allowing the actual implementation to challenge what I think, what the documentation says, and even what the agent has been instructed to do.

Not because the code is automatically perfect.

But because claims about the system should ultimately be accountable to what the system actually does.

I don’t need an agent that agrees with me.

I need one that can show me when the code doesn’t.

What’s Next?

If you’re new to the series, you can catch up on the story so far:

  • Article 1 — I Created a Coding Language — How a simple question to ChatGPT became a programming language experiment, and what happened when I let Codex work on it overnight.
  • Article 2 — Chat Was Control — How I began directing Codex through persistent project documents, implementation constraints, and a manual agentic workflow.
  • Article 3 — What a Version Promises — How architecture analyses, version control, and AI-assisted evaluation began changing the way I preserved and verified agent-generated work.
  • Article 4 — I Built My Own n8n — How an automation question led me to build a runner with Gemini, question its place in my architecture, and begin integrating it into A.I.N.D.Y. with ChatGPT and Codex.
  • Article 5 — It’s Just a Conversation — What working across different AI systems taught me about context, tools, execution environments, and why the model alone doesn’t determine what an AI can accomplish.
  • Article 6–1 Week to Vibe Code V1 — How my attempt to finish Nodus in a week led me from Codex to Claude Code and helped turn conversational testing requirements into enforceable engineering practices.
  • Article 7 — Discovering The Arcitecture — How Gemini, ChatGPT, and Codex helped me discover where the Nodus Runner belonged and begin defining the execution architecture A.I.N.D.Y. needed.
  • Article 8 — Prototyping with AI — How I developed a repeatable AI-assisted prototyping method, used single-file implementations to preserve architectural ideas, and worked with coding agents to translate those ideas into existing software.
  • Article 9 — Where Does the Work Exist? — How auditing A.I.N.D.Y.’s Memory Bridge with Claude Code changed my development workflow, helped me understand my own architecture, and taught me to evaluate agent-generated work through repository evidence rather than blind trust.
  • Article 10 — Wasn’t I Building an App? — How auditing and integrating A.I.N.D.Y.’s existing subsystems led me to recognize the runtime emerging beneath the application and decide to develop it deliberately.
  • Article 11 — Decomposing A.I.N.D.Y. — How I learned to distinguish reusable runtime capabilities from application-specific behavior, reorganized the repository, and worked with coding agents to separate A.I.N.D.Y. into independently accountable components.

By this point, I’d developed a fairly extensive process for auditing, implementing, and verifying AI-generated code.

I’d learned to question completion reports, investigate disagreements between agents, compare documentation against actual implementations, and use tests and live environments to establish confidence in what I’d built.

But there was still an assumption I hadn’t fully challenged.

Just because something worked on my machine didn’t mean it worked on anyone else’s.

And Nodus was about to make that distinction painfully clear.

I’d already taken the language through its first major development milestones. I’d tested it, reviewed its architecture, and worked with coding agents to establish increasingly rigorous engineering practices.

As far as my own development environment was concerned, Nodus worked.

But when I eventually discovered that the language wasn’t working on other machines, I had to reconsider what I’d actually been verifying.

Was I testing the language itself, or was I testing a language inside an environment I’d spent months configuring around it?

And what did it really mean to release software if I hadn’t established that someone else could install and use it?

In the next article, I’ll explore how discovering Nodus’s portability problems changed the way I approached development, packaging, and releases — and why my verification standard evolved from “ Does this work for me? ” to “ Does this work for other people?”

The next installment of the 2026 Agents Case Study Series is coming soon.

Explore the Projects

The systems discussed throughout this series are part of the Masterplan Infinite Weave ecosystem. While not every project has appeared in the story yet, you can explore the technology, examine the source code, and follow its ongoing development.

  • Nodus — The programming language that began as the experiment documented in this series. Explore Nodus
  • A.I.N.D.Y. Runtime — The governed execution substrate supporting workflow orchestration, persistent execution, memory, and agent coordination. Explore A.I.N.D.Y. Runtime
  • A.I.N.D.Y. Platform — The application platform built on the runtime, bringing the underlying capabilities into usable applications and interfaces. Explore A.I.N.D.Y. Platform
  • Masterplan Infinite Weave — The broader ecosystem connecting these projects and their ongoing development.

Previous Work

This series builds on the ideas, experiments, and lessons documented in my earlier work.

  • 2025 ChatGPT Case Study Series — The original exploration of learning, building, and executing through sustained human-AI collaboration. Read the Series
  • 2025 Duality of Progress Series — An exploration of the relationships between constraints, creativity, iteration, and progress. Read the Series

Originally published at https://www.linkedin.com.


Top comments (0)