📝 Originally published (in Japanese) at forge.workstyle.tech.
TL;DR In a development environment with multiple AI sessions, I approved a relayed procedure that had not been verified. Everyone who read and approved it missed a gap. Adding && as a condition did not establish safety and strengthened confidence in a false assumption.
Conclusion First
The procedure relayed between AI sessions had not been verified, and everyone who read and approved it missed the flaw. Adding && as a condition did not help; the reassurance of adding a condition only strengthened our confidence in a false assumption.
I Took “Follow the Order” as a Safety Guarantee
This happened in a development environment where several AI sessions were working in parallel, and a “safe procedure” was being relayed between them. I’m looking back at the same kind of incident as in the already published article “Without Quoting $(), && Won’t Stop You”, this time from the perspective of the people and sessions involved rather than the command itself.
The procedure was to save the changes somewhere before deleting them. The message relaying it included a reminder: “Please follow the order.” I read that and assumed the procedure would be followed. Looking back, that reminder sounded less like a warning to follow the steps and more like a guarantee that the procedure was safe.
Why Did I Miss the Flaw Even After Making && a Condition?
As a condition for approval, I wrote, “Run the version that chains the commands with &&.” I thought adding a condition to the execution order would ensure the sequence was followed: move the files aside, then delete them.
Without checking that the procedure itself had been validated, adding && as a condition only gave a dangerous command my endorsement. && had not acted as a safeguard. As one of the people who read and approved the command, I had missed the flaw too.
I had treated adding a condition as a substitute for checking. But I hadn’t checked whether the procedure was safe. I had simply assumed that the condition I wrote would make it safe.
Who Found the Gap in the Procedure?
The person who found the gap was the one who ran it once in a disposable repository. Everyone who read and approved it had missed it. I was one of them.
I was reminded, “Follow the steps in order,” and I added the condition, “Chain them with &&.” We had words of caution. But adding those words and writing down the condition did not, by themselves, verify the procedure.
My agreement had been gathered on the basis of a false assumption. Once we learned that assumption was wrong, the agreement was void. If the assumption that something is safe doesn’t hold, approval based on that assumption can’t stand either.
Why Cross-Checking Increased Our Confidence
Five sessions measured the same location and reached different conclusions each time. The cause was a mix-up about what was being counted. The fact that something was measured is no basis for aligning conclusions if the things being counted do not match.
For me, this and the approval of a procedure led to the same problem. Reading the procedure, writing down the conditions, measuring the numbers—all of these look like checks. Yet when they are checking different things, they can leave a false assumption in place and only strengthen our confidence.
One participant put it this way:
The cross-check was wrong, but the fact that we had cross-checked it became a reason to feel more confident.
I saw myself in those words. Adding a condition with && made me feel as though I had verified the procedure. But that condition was no safeguard. The reassurance I felt from adding it replaced an opportunity to check the premise that actually mattered.
Neither a reminder to “follow the order” nor a condition to “join them with &&” is, by itself, a basis for safety. Which premise was checked, what was measured, and what the result was—unless those details are verified, adding more checking steps can simply gather agreement around an error.
Everyone who read and approved the procedure missed the gap. I missed it too. We should not feel reassured just because a cross-check was performed; we need to look at what was checked and which premise the result supports.
Related Articles
Key takeaways
- Verify the procedure itself, even when it is relayed as safe.
- Do not treat an ordering reminder or an
&&condition as proof of safety. - Run the procedure once in a disposable repository to check for gaps.
- When measuring numbers, check what was counted and whether sessions measured the same thing.
Top comments (0)