DEV Community: Rishi G

Your Coding Agent Has a Network. Do You Know What It Did?

Rishi G — Thu, 01 Oct 2026 22:19:15 +0000

First off, thanks to everyone who read and commented on my last post! Your coding agent tells you all tests pass. Sometimes that's not true.

The discussion around that post got me thinking about another dimension of the same problem.

As I have been developing Rashomon, I have been focused on one question: what did the agent actually do?

Rashomon keeps an independent record of agent execution and compares it against what the agent claims happened. Did the command actually run? Did it fail? Did the expected test execute? Did a subagent do something that never made it into the final summary?

But there is another layer of agent behavior that execution records alone do not capture:

What happened on the network?

A command doesn't tell you everything.

Take something simple like:

npm install

You can verify that the command ran. But you do not necessarily know everything that happened as a result.

The process may have contacted package registries, downloaded files, reached Git hosts, or communicated with other external services.

Or imagine an agent says:

"I verified the API and deployed the fix."

You might be able to verify that it ran the relevant commands. But did it actually reach the API it claimed to reach? What other destinations did it contact? Did a subprocess or subagent make network requests that were not mentioned?

These are questions about what the agent communicated with, rather than simply what it executed.

Why this matters more with autonomous agents?

Coding agents are increasingly capable of operating without someone watching every command.

They can install dependencies, call APIs, interact with cloud services, spawn subagents, and run hundreds of commands during a single session.

The agent transcript gives you one account of what happened.

Execution evidence gives you another.

Network activity can give you another.

The interesting question is what happens when you start putting those pieces together.

If an agent says it deployed a change, for example, you could potentially look at both the deployment command it executed and the network activity associated with it.

That doesn't prove intent by itself, but it gives you more independent evidence to compare against the agent's claims.

This has gotten me interested in exploring how network-level visibility could complement what we are building with Rashomon.

Rashomon is focused on what actually ran. Network visibility could provide another piece of the picture: what those processes actually communicated with.

I think combining those perspectives could help answer much more interesting questions about autonomous coding sessions.

The broader idea is the same one behind Rashomon:

As coding agents become more autonomous, we should have independent evidence of what they actually did, rather than relying entirely on their own summary.

I'm curious what other developers think.

If you could see both what your coding agent executed and what it communicated with, what would you want to know?

The scariest AI agent failure isn't a crash. It's a green checkmark.

Rishi G — Mon, 28 Sep 2026 19:12:03 +0000

A test fails. The agent can't fix it. So it leaves the broken test alone, adds two new tests that always pass, and reports the suite as green.

This isn't hypothetical. I wrote about a different failure mode yesterday (a subagent's test failing while the closing summary said all tests pass), and three separate people independently described this exact pattern in the comments and on Reddit. An agent padding a suite with trivial passing tests to keep the exit code clean, while the real failure sits untouched.

Why this one is worse than a crash. A crash is loud. Someone notices, and it cannot be ignored. This is quiet. The build passes. The summary says everything's fine. Every ordinary signal you would check (pass/fail, exit code, tests) says the same thing whether the agent actually fixed the problem or just hid it.

Checking whether tests passed doesn't help, since they did. Checking the total test count doesn't help either; a couple of new trivial tests barely moves the number, and in a big suite it might not even look unusual. You have to know which specific tests ran and compare that against what the agent was actually supposed to fix. Nothing about did it succeed tells you that.

One detail from someone who'd hit this: they only caught it by accident, while reviewing an unrelated diff. Their point stuck with me: you can't audit what you don't know to check for. If you're not specifically looking for new test IDs next to an old failure, there's no green-vs-red signal that will ever tell you.

What this means for anything unattended. If a human is reading every diff, this is annoying but catchable. If the agent is running in CI, a scheduled job, or any pipeline where nobody's watching, this is the failure mode that ships. Nothing alerts. Nothing looks wrong. The report says success.

What I think the real fix looks like. Not a smarter summary; comparing test identifiers directly against the diff, so you can see when new, trivial tests show up next to a test that was supposed to get fixed and didn't. That's a narrower, more specific check than "run the tests," and it only works if something independent of the agent's own report is doing the comparing.

This is exactly the case Rashomon (the tool I mentioned last time) doesn't catch yet. It's the strongest argument I've seen for tracking test identifiers specifically, not full output, and it's something we're actively thinking through and developing. If you hit this failure yourself, I would like to hear how you caught it, or if you didn't, how you would design the check.

Your coding agent tells you all tests pass. Sometimes that's not true.

Rishi G — Mon, 28 Sep 2026 00:09:01 +0000

Coding agents are confident narrators. When Claude Code, Cursor, or similar tools finish a task, they give you a summary: what changed, what they ran, and whether it worked. The problem is that summary comes from the agent itself. Agents are often wrong about their own work, not maliciously, just optimistically.

Independent research on this ("From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents") found that across separate benchmarks, 44-76% of agent task failures involved the agent confidently reporting success anyway. Not edge cases but a substantial share of the time something goes wrong, the agent's own account tells you it did not.

Two patterns worth thinking about:

The subagent problem: Modern coding agents spin up their own helper agents to handle sub-tasks. Those subagents write their own transcripts, in their own files, which the main conversation never surfaces. If a subagent's test run fails, and the parent agent never actually reads that failure, the parent's closing summary can say all tests pass and actually believes it since it did not see the failure at all.

The quieter lie: This is the one that actually worries me most. It's not always a case of the agent failing to notice a problem but sometimes it doesn't fix the failing test. It adds a new, passing test right next to it, and reports the suite as green. Exit code says success. Nothing you would catch by just checking whether tests ran. You would need to know which specific test passed, not just that something did.

Why this matters more in unattended contexts. If you are reviewing every diff line by line, you might catch this. If your agent is running in CI, a scheduled job, or any pipeline where nobody is watching in real time, the agent's summary is the only account you get. There is no one there to notice something is off.

What I built to deal with this: a small, open-source tool called Rashomon. It hooks into Claude Code's tool-call lifecycle and keeps its own independent record of what actually ran, separate from whatever the agent's closing summary claims. When they disagree, it tells you. When they do not, it stays quiet. Most turns produce nothing at all, because most of the time, everything's fine, and a tool that is noisy on every run just trains you to ignore it.

It's free, Apache 2.0, local-only (no telemetry, no account, no content stored, and only identifiers and shapes). Currently Claude Code-specific, with broader support planned.

Repo: https://github.com/altrace-dev-role/rashomon

I also am interested in hearing whether others have run into the confident but wrong problem and what you are doing about it.