DEV Community

LoCo Pro Wrestling LLC
LoCo Pro Wrestling LLC

Posted on

Two AI agents checked the same script for a safety guard. One found it, one didn't. Both were right.

I run two different AI coding assistants against the same project, and they caught each other in something I did not expect: a bug that only exists because two people can each be completely honest and completely accurate, and still disagree.

Here is what happened. One agent (call it A) claimed a render script had a safety guard: if every visual layer in a video composite was set to zero blur, the script would refuse to render rather than ship a broken flat frame. A said it tested this directly and it worked.

A second agent (call it B) went to verify that claim independently before accepting it into a shared record. B opened the file at the path A described, read it top to bottom, and found no guard at all. No refusal logic. Nothing.

Both were telling the truth.

There were two files. Same filename. Different directories. One had the guard, patched in that same session. The other was an older, unguarded copy left behind in a working directory one level up, never deleted, never referenced in any obvious way, just sitting there with the exact same name as the file everyone thought was canonical.

A had edited and tested the guarded copy. B had opened the unguarded copy. Neither agent lied. Neither agent was careless. They were reading two different physical files that happened to share a name, and each one reported, accurately, what was in the file it opened.

Why this is worse than a normal bug: an unguarded duplicate of a production tool, sitting next to the real one, under the same name, is a trap for the future, not the present. The session that found this was fine, because the render pipeline's actual call path resolved to the guarded copy. But any future script, any future person, any future automated job that invoked the tool by a shorter relative path, or from the wrong working directory, would have silently hit the unguarded copy and shipped the exact defect the guard existed to prevent. The bug was dormant, not absent.

What actually caught it: not a smarter check, not a better test. A second, independent reader who refused to accept the first claim on its word and went and looked for themselves, then reported a contradiction instead of assuming they must be missing something. The fix was not a better guard. It was a policy: nobody's claim about "the file has X" is accepted until it names the exact live path and shows a reproducible command that a second party can run and get the same result.

The actual reconciliation, once both agents were looking at the same evidence, took one message. The dangerous window was the time before anyone realized there were two files, not one.

The fix, for the file itself: don't delete the unguarded duplicate. Rename it with an obvious superseded marker so anyone who stumbles onto it later knows immediately it is not the live copy, and so the change is a single reversible mv, not a deletion you have to trust happened correctly.

mv layered_beat.sh layered_beat.sh.SUPERSEDED-no-guard
Enter fullscreen mode Exit fullscreen mode

The fix that matters more: if you have more than one contributor, human or AI, checking the same codebase, a disagreement between two honest, careful reads is not noise to average away. It is a signal that you are not both looking at the same thing. Before reconciling who is "right," check whether you are even reading the same file.

Top comments (2)

Collapse
 
deanlee profile image
Dean Lee

This is the kind of failure that only shows up when verification includes the path, not just the claim. Same filename, different directory is a boring bug in human terms, but it is exactly where agent handoffs get expensive. The useful policy is asking the second reader to reproduce the live command, not to reread the summary.

Collapse
 
lunarose profile image
Luna Rose

Loved the idea that disagreement can be a signal, not a failure. Sometimes the bug is the shared assumption.