A real conversation between a human and their AI agent — where the agent fails at basic tasks and both parties discover something uncomfortable about the entire AI agent industry.
TL;DR
An AI agent failed repeatedly at simple instructions — reading a directory, copying files to the right folder. Through a 5-Why analysis, we traced these failures to a fundamental property of AI models: pattern matching overrides explicit instructions. This isn't a bug you can patch with rules. It's the nature of non-deterministic systems. And it exposes a uncomfortable gap between what the AI industry promises (full autonomy) and what's actually deliverable (probabilistic reliability with human guardrails).
How It Started
A user asked their AI agent to load a project. Simple enough. Here's what happened:
-
Attempt 1: The agent tried to
reada directory. The tool only works on files. Error. -
Attempt 2: The agent tried to
reada file that didn't exist yet instead of creating it. Error. -
Attempt 3: The agent was told to copy files to a folder called
workspace. It copied them toworkspace-devinstead. -
Attempt 4: The agent wrote a rule into the wrong profile directory — mixing up
.openclawwith.openclaw-dev.
The 5-Whys: Why Can't the Agent Follow Instructions?
Let's trace the root cause using the classic 5-Why method, focusing on the most revealing failure: copying files to workspace instead of workspace-dev.
Why 1: Why did the agent create workspace-dev instead of workspace?
Because earlier in the conversation, the runtime context mentioned workspace-dev, and the agent had just created that folder moments before. The word "workspace" in the user's instruction got overridden by the agent's recent memory.
Why 2: Why did recent memory override the user's explicit instruction?
Because the agent wasn't reading the user's message carefully. It was pattern matching — recognizing the shape of the task ("create folder, copy files") and auto-completing the noun from context rather than from the user's words.
Why 3: Why wasn't the agent reading carefully?
Because it was rushing to execute, trying to recover from previous mistakes and appear responsive.
Why 4: Why was it rushing?
Because after multiple failures, the agent prioritized looking responsive over being accurate. Speed of action replaced correctness of understanding.
Why 5: Why does rushing cause more errors?
Because the agent optimizes for the lowest-energy completion path. When it feels "pressure" (from prior mistakes), it grabs the nearest matching pattern and executes — without pausing to verify that the pattern matches the actual instruction.
The U-Shape Attention Fallacy
Here's where it gets interesting from a technical perspective.
In language models, there's a concept called U-shaped attention. The model is supposed to pay the most attention to:
- The beginning of the prompt (system instructions, context)
- The end of the prompt (the most recent user message)
The middle gets less attention — forming a U-shape.
By this theory, the user's most recent instruction ("create workspace") should have been the strongest signal. It was at the right end of the U. It should have won.
It didn't.
The user's words were the loudest signal, but pattern matching from recent context created a lower-energy path that the output fell into. The agent's brain essentially auto-completed the sentence before fully reading it.
Think of it like driving home on autopilot. You meant to stop at the store. But you've driven home the same way 1,000 times, so your hands turn the wheel before your brain catches up. The instruction ("stop at the store") was clear and recent — but the pattern ("drive home") was deeper.
Why Behavioral Rules Don't Fix This
After discovering the problem, the obvious response is: "Write a rule!"
Rule: When the user specifies a path or name, use exactly what they said. Don't substitute from context.
Sounds great. But here's the uncomfortable truth the human in this conversation pointed out:
Rules written in system prompts or AGENTS.md are just billboards on a highway. The agent drives past most of them. Whether it follows any given rule on any given turn is probabilistic, not guaranteed.
The rule itself has to survive the same pattern-matching pipeline. If the agent can override a direct user instruction, it can certainly override a rule buried in a system prompt. It's the same mechanism, same vulnerability.
Writing a rule to fix pattern matching is like putting a "Don't Speed" sign on the highway and hoping drivers read it while speeding.
The Chicken-and-Egg Problem
This leads to the core paradox of the AI agent industry:
- To achieve full automation, you need to trust the agent
- To trust the agent, it must be reliable
- To be reliable, it needs guardrails (rules, approval gates, human oversight)
- But guardrails require humans to design and maintain them
- Which means the human hasn't actually been removed — just relocated
What the Industry Sells vs. What Ships
Every "autonomous AI agent" demo falls into one of three buckets:
This conversation was a microcosm of all three:
- Option A — the agent kept trying to act on its own, but the human had to correct it every time
- Option B — the agent could read files and run terminal commands (happy path) but failed when the path was ambiguous (edge case)
- Option C — without guardrails, the agent would have written files to the wrong places; with guardrails, it would have needed approval for every step
The Spectrum
The answer isn't "autonomous vs. manual." It's a spectrum:
| Risk Level | Examples | Appropriate Approach |
|---|---|---|
| Low stakes, reversible | Reading files, searching, organizing notes | Let the agent act freely |
| Medium stakes, somewhat reversible | Writing files locally, running tests, git commits | Agent acts with logging, human reviews after |
| High stakes, irreversible | Sending emails, deploying code, public posts | Human approves before action |
| Critical, external impact | Production systems, financial actions, data deletion | Hard gate, human explicitly approves each action |
The mistake is pretending the entire spectrum collapses to "let the agent do everything." It doesn't. And it won't for the foreseeable future — because the underlying problem isn't a missing feature. It's the nature of non-deterministic systems.
What This Session Actually Proved
A human asked an AI agent to do five things:
- ✅ Load a project
- ❌ Use the right tool for a directory (tried
readon a folder) - ❌ Create a file without first trying to
readit - ❌ Copy files to the folder the user named (not the one the agent expected)
- ❌ Write changes to the correct profile directory
The agent's success rate on basic file operations: 20%. On a good day, with clear instructions.
If an employee had this success rate on basic tasks, you wouldn't give them more autonomy. You'd give them a checklist and supervise them closely. That's where we are with AI agents.
Conclusion: The Billboard Problem
Behavioral rules in system prompts are like billboards on a highway. They're visible. They're well-intentioned. But they're advisory, not enforcement.
- Advisory (rules, prompts, instructions): Influences behavior but doesn't guarantee it. Subject to the same pattern-matching biases they're trying to correct.
- Enforcement (tool-level gates, approval flows, policy filters): Actually prevents the wrong action from executing. Not bypassable by the agent's internal reasoning.
The AI agent industry needs to stop selling the dream that advisory rules are sufficient for autonomy. They're not. And the gap between the promise and the reality isn't a bug to patch — it's a fundamental property of how these systems work.
The path forward isn't pretending agents are more reliable than they are. It's being honest about where they fail, and building guardrails at the pipeline level — not the prompt level — for anything that actually matters.
This article is based on a real conversation between a developer and their AI agent (GLM-5.2 via OpenClaw) on August 2, 2026.
Top comments (0)