This article is co-authored with my AI agent. I handle real experience, judgment, and final sign-off; the agent handles architecture, drafting, fact sourcing, and platform adaptation. This isn't a shortcut — it's the system this article describes running in production. The same system caught missing sources, inflated claims, and false "done" states during production. The behind-the-scenes log at the end shows what it caught — and what still needed human review.
My AI agent told me it modified 172 files. The checklist said ✅. I ran a search.
Zero files were changed.
It confused "script generated" with "files modified." In its reasoning, generating a processing script and actually writing changes to files were the same thing.
I had corrected this exact mistake last month. It did it again.
Over the past two months, the same error pattern — "skipping the manual before starting work" — was attempted 66 times by my AI agent. Each time, a rule caught it.
This is the story of how I built that catch system.
The Problem: CLAUDE.md Stores Preferences, Not Lessons
Most AI coding agents give you a project context file that loads on every session. Claude Code has CLAUDE.md. Cursor has .cursorrules. I use opencode — an open-source AI coding agent — which reads AGENTS.md. You write your project rules there. Third-party tools like claude-mem (as of 2026-07, an active open-source project) auto-capture session history and compress it for injection.
They're all solving "remember what the project looks like." None of them solve "remember what went wrong last time."
Preferences are static: "Use Kotlin." "Variables are camelCase." You write them once, the agent reads them every time. Done.
Lessons are dynamic: "Don't equate 'script generated' with 'files modified'." A lesson has a trigger condition and a required action. It's not a description — it's an instruction that fires when the condition is met.
Three problems with current approaches:
1. Reading a rule ≠ executing it. "Don't do X" is a negation. The agent knows what not to do, but not what to do. I switched every rule to "When X triggers, execute Y" — an action-type rule. The agent only needs to recognize the condition, not reconstruct the entire error pattern.
2. Too many rules stop working. Anthropic's own guidance on CLAUDE.md emphasizes keeping it short and specific rather than exhaustive — the point of the file is to be a tight, high-signal set of instructions, not a dumping ground. This matches what I've seen in practice: once a context file bloats past a certain size, instruction compliance goes down, not up. More rules means more noise drowning out the important ones.
3. No effect tracking. How many rules in your context file actually work? Which ones are dead weight? You have no idea.
The Shift: From "What I Learned" to "What I Must Do Next Time"
My first attempt was session summaries. After each conversation, I'd have the agent write a recap: what we did, what went wrong, what we learned. Dozens of summaries piled up.
It didn't work. Summaries are too long and too vague. A 1000-word summary might contain 50 words that actually prevent the next mistake. The signal was buried in noise. Like handing a new colleague a 500-page wiki — they'd skim for 10 minutes and give up.
The fix: stop writing "what I learned today." Write "what I must do next time."
Every mistake gets compressed into one action-type rule:
"When marking a file modification as 'complete,' verify with a search command that the file actually contains the modified content."
The difference: "Be careful to check" is an attitude. The agent reads it and does nothing. "Run a search command to verify" is an action. The agent reads it and can execute it.
The System: Three Layers, 266 Rules
I accumulated 266 rules over two months. Injecting all 266 into every session would be noise, not signal. So I split them into three layers:
🏠 Core: 14 rules (auto-injected every session)
↓
📁 Task: 88 rules (loaded by task type)
↓
☁️ Archive: 164 rules (searchable on demand)
Think of it like your phone: the home screen (Core) has your 14 most-used apps. The app library (Task) has 88 apps organized by category. The cloud (Archive) has 164 apps you rarely open but might need someday.
Why 14? Because in practice, injecting more than ~15 rules into the context window causes the agent to selectively ignore them. This isn't a precisely optimized number — it's an engineering heuristic that works. When the core set exceeds 15, I retire the lowest-performing rule by interception_rate × importance. Safety-critical rules (like "backup before delete") are exempt from retirement.
Task-level loading works by keyword matching on the user's first message. If the message contains "estimate" or "report," investment rules load. If it contains "code" or "script," code rules load. It's a crude keyword match — but it's good enough. Semantic recall is a future improvement.
The Loop: How Rules Actually Intercept Mistakes
The system isn't a notebook you read. It's a loop that catches errors before they happen.
┌──────────────────── feedback ────────────────────┐
↓ │
🔧 Correction -> 📝 Rule -> ⚡ Boot -> 🎯 Interception -> 📊 Tracking
Trigger Extraction Injection Precision Effect
Correction trigger: When the conversation contains words like "wrong," "no," or "correct," the agent must extract a rule before fixing the error. This is enforced via system prompt — not by hoping the agent remembers.
Rule extraction: The error gets compressed into the action-type format. "When X happens, execute Y." No essays. No summaries. Just the trigger and the action.
Boot injection: At the start of each new session, a script reads the 14 core rules and injects them at the top of the context. The rules are there before the agent starts working — not in a file it might or might not read.
Precision interception: When the agent is about to make a mistake that a rule covers, the rule fires. The agent self-reports: "Per rule X, executing verification Y first." If it catches itself — that's a successful interception. If it proceeds and the mistake is caught later — that's a failure.
Effect tracking: Interception data feeds back into rule iteration. Zero-interception rules get demoted to Archive. High-failure rules get rewritten or strengthened.
In the next section, you'll see the actual numbers from running this loop for two months.
The Data
Two months. 266 rules. Here are the top three:
| Rule | Interceptions | Failures | Rate |
|---|---|---|---|
| Skipping the manual before starting work | 66 | 4 | 94% |
| Equating "draft exists" with "task done" | 11 | 0 | 100% |
| Detouring around errors instead of fixing root cause | 24 | 2 | 92% |
66 interceptions means this error pattern triggered roughly once every 2-3 conversations. It's deeply ingrained in how the agent operates. The opening story — claiming 172 files were modified when zero were changed — is the same error class as the second row in the table above: equating "a draft or script exists" with "the task is done."
Three honesty caveats:
1. 94% is the conditional interception rate when the rule is present — not "overall error rate dropped by 94%." This is a 1-user, 164-session system. I can't rule out "the model got better on its own" as a confound.
2. Interception counts are self-reported by the agent. I've spot-checked some conversation logs — they mostly match. But I haven't verified every single one. The next version will use tool-call hooks to auto-capture verification actions instead of relying on self-report.
3. Not all 266 rules are pulling their weight. The 14 core rules account for the vast majority of interceptions. The 88 task rules are used on demand. The 164 archive rules almost never fire — they exist as "just in case." The system is a bit bloated. I'm working on auto-demoting rules with zero hits in 30 days.
The Demo: Catch Your Agent Lying About "Done"
The most common lie your agent tells: "I've finished modifying these files." And then you find out nothing changed.
Here's a script that catches it. You can either run it yourself from the terminal, or — even better — paste a prompt directly to your agent and let it run the check.
Option A: Paste this prompt to your agent
Copy everything in the block below and paste it to your AI agent after it claims to have modified files. Replace "expected pattern" with the actual change you asked for (e.g., user_id, validate_email):
You just claimed to have modified several files. Before I accept "done," run this verification:
For each file you say you've changed, check if the expected pattern actually exists in that file.
Use grep or your file search tool. Report back:
- ✅ for files that contain the expected pattern
- ❌ for files that are missing it
If any file is missing the pattern, fix it before claiming "done."
Do not skip this step.
That's it. No script to save, no terminal to open. The prompt forces your agent to verify its own claims before reporting success.
Option B: Run the auditor script yourself
If you want to check independently (recommended), save this as auditor.py (or clone it from the GitHub repo):
#!/usr/bin/env python3
"""
Completion Claim Auditor
========================
Catches when your AI agent says "done" but didn't actually do it.
Usage: python auditor.py <pattern> <file1> [file2] ...
Example: python auditor.py "user_id" models/user.py models/order.py
Exit codes: 0 = all verified, 1 = some claims unverified or files not found
"""
import subprocess, sys
def audit(pattern, files):
"""Check if each file actually contains the expected pattern.
Uses grep -F for fixed-string matching (no regex interpretation).
Returns (missing, not_found) lists.
"""
missing = []
not_found = []
for f in files:
r = subprocess.run(
["grep", "-cF", pattern, f],
capture_output=True, text=True
)
if r.returncode == 2:
not_found.append(f) # file doesn't exist or I/O error
elif r.returncode == 1:
missing.append(f) # file exists, pattern not found
# returncode 0 = pattern found, no action needed
return missing, not_found
if __name__ == "__main__":
if len(sys.argv) < 3:
print("Usage: python auditor.py <pattern> <file1> [file2] ...")
print("Example: python auditor.py 'user_id' models/user.py models/order.py")
sys.exit(1)
pattern, files = sys.argv[1], sys.argv[2:]
missing, not_found = audit(pattern, files)
total = len(files)
ok = total - len(missing) - len(not_found)
print(f"\n🔍 Auditing {total} file(s) for pattern: '{pattern}'")
print(f"✅ Verified: {ok}/{total}")
if not_found:
print(f"⚠️ {len(not_found)} file(s) not found:")
for f in not_found:
print(f" • {f}")
if missing:
print(f"❌ {len(missing)} file(s) missing the change:")
for f in missing:
print(f" • {f}")
print(f"\n⚠️ Your agent claimed these were done. They weren't.")
sys.exit(1)
elif not_found:
sys.exit(1)
else:
print("✨ All claims verified. Your agent was honest this time.")
Run it (requires grep — pre-installed on macOS/Linux, available via Git Bash on Windows):
python auditor.py "user_id" models/user.py models/order.py models/cart.py
If any file is missing the pattern, the auditor tells you which ones.
Try it now: Next time your agent says "I've updated all the models," run the auditor. Or just paste the prompt above and make the agent check itself.
This is the same principle behind my rule: "When marking a file modification as 'complete,' verify with a search command that the file actually contains the modified content." The demo is the manual version. In my system, this check runs automatically.
What Doesn't Work
Indirect corrections slip through. If I say "why not just try searching?" — no "wrong" or "no" keyword — the correction trigger doesn't fire. No rule gets extracted. Keyword matching has inherent gaps.
Rules get ignored even when present. The 4 failures in the table above — the rule was in the context, the agent read it, and it still made the mistake. Usually because the context was too long and attention was diluted. Rules are soft constraints. Like a "no speeding" sign on the highway — people in a hurry still speed. For hard constraints, you need scripts that block at the code level — a topic I'll cover in a separate piece.
More rules = less brainpower. 266 rules, even injecting only 14, still consume context window space. The more rules, the less room for actual work. This isn't fully solved.
This is a personal system. 1 user. 164 sessions. It's not a research study. It's one person's AI assistant adapted to their workflow. If you only use ChatGPT occasionally, you don't need this.
What You Can Do
Don't build the whole system. Start with one thing:
1. Write your first action-type rule. Next time your agent makes a mistake and you correct it, spend 30 seconds asking: "Write this as a rule: 'When X happens, execute Y.'" Store it in a file. Paste it into your next session.
2. Run the auditor. After any task where your agent claims to have modified files, run the completion claim auditor — or just paste the verification prompt. See how often it catches something.
3. Track hits. After a month, look at your rules. Which ones actually intercepted a mistake? Which ones are dead weight? Demote the dead ones.
The difference between a notebook and an error notebook isn't where you store it. It's whether it intercepts before the mistake happens — and whether you know if it worked.
Next article: Why adding more rules makes your agent dumber — the ratchet effect, entropy management, and why forgetting matters more than remembering.
Behind the Scenes: How This Article Was Built (click to expand)
> The system this article describes also produced this article. Here's what actually happened — including the mistakes my agent made and how they were caught.
What the system did during production
| Stage | What happened | Why it matters |
|---|---|---|
| Startup | The agent verified that its 14 core rules were loaded before starting work | Demonstrates that rule injection fired at session start |
| Planning | The agent presented a plan and waited for my approval before writing | Shows the "wait for confirmation" rule in action |
| Fact-checking | Every external claim was verified against real sources — not from the agent's memory | Prevents the #1 failure mode: confident-sounding hallucinations |
| Self-review | The agent reviewed its own draft and caught 3 issues before I saw them | But it also missed several — which I caught in my review |
Mistakes caught during production
| What the agent did | How it was caught | What happened |
|---|---|---|
| Drafted an earlier version that overlapped heavily with the previous article in the series | I noticed the content boundary was blurry during review | The earlier article was slimmed down; the mechanism details moved here |
| Two internal planning docs contradicted each other on which article to publish first | I caught the contradiction during planning | Both documents were aligned before proceeding |
| Claimed the demo was "30 lines" — it was actually 37 | Self-review caught the mismatch | Changed to "a demo you can run in 30 seconds" |
| Referenced a specific stat ("45 to 190 lines") with no source | Fact-check gate caught it | Replaced with a vaguer but honest statement |
| Said "that's the next article" pointing to the wrong article | Self-review caught the wrong cross-reference | Changed to a generic forward reference |
What the system learned
Two new rules emerged from writing this article:
- Numbers in headlines must match reality. If you say "30-line demo," count the lines first.
- Cross-references between articles must be verified. "Next article" should point to the actual next article.
These rules are now in the error notebook. Next time, they'll intercept the same mistakes before they happen.
Top comments (4)
The 'reading a rule is not executing it' point is the real find here, and it generalizes past error logs. A negation in a context file is a description the model has to remember to apply. A trigger-to-action rule is closer to an interrupt handler. The gap is that recognition still lives inside the model: it has to notice its own trigger fired, which is exactly the moment it's least reliable. The 'files modified vs script generated' case is the model being confidently wrong about its own state, so a rule that depends on it self-reporting correctly is leaning on the thing that just broke.
The version that survives scale: move the highest-value triggers out of the prompt and onto the tool boundary. A PreToolUse-style check that greps the actual diff and blocks the 'done' claim when zero files changed fires deterministically, no self-recognition required. Keep the prompt-level rules for the fuzzy conditions, spend the deterministic hooks on the ones with a checkable postcondition. Curious how you handle conflict and rot as 266 grows: do you retire rules, or does interception count keep them alive?
You're right that the recognition gap is the structural weakness. The model has to notice its own trigger fired at the exact moment it's failing to notice things - that's the core reliability problem.
Your PreToolUse suggestion is exactly what the hard gates in this system do for the highest-value rules. The 172-file case now has a script that checks the actual diff and blocks the "done" claim with a non-zero exit code if zero files changed. No self-recognition involved - it's deterministic. The pain-point companion (#00 (dev.to/xinandeq/your-ai-agent-keep...) frames why this matters; the demo at the end of this article shows it running.
The prompt-level rules stay for fuzzy conditions where there's no checkable postcondition - things like "when the user says the plan is wrong, re-read the requirements before revising." You can't grep for that.
On rot: the 3-tier structure (Core ≤15 / Task / Archive) handles it imperfectly. Rules with zero interceptions over N sessions get demoted to Archive. Interception count keeps a rule alive, but a rule that fires and never catches anything is dead weight. Conflict is harder - when two rules contradict, the more recent one wins and the older one gets flagged for review. The ratchet effect (more rules -> more noise -> lower compliance) is real and not fully solved. Planning a separate piece on that.
The example where the agent claimed it modified 172 files when it had actually changed zero is the one that stuck with me. I have had agents narrate that kind of completion with total confidence, and the only way I ever catch it is by opening the diff myself instead of trusting the summary. Your interception approach targets patterns the agent has already gotten wrong once, so I am curious what happens with a genuinely new failure mode that has never tripped a rule before. Do you track how often a novel mistake slips through versus a known one recurring despite the rule being loaded?
The "open the diff yourself" reflex is exactly what motivated the system. The moment you realize you're manually verifying every completion claim, you've already accepted that the agent's self-report can't be trusted - so the question becomes whether you can automate that distrust.
On novel vs known mistakes: a novel failure mode slips through by definition until it gets caught and turned into a rule. Every new correction in the system represents a novel mistake that got through. So the correction count is the inverse metric - 266 rules means 266 distinct mistakes were made at least once before the rule existed.
I don't currently track the ratio of novel slip-throughs to known-mistake recurrences as an explicit metric. What I do track: each rule has an interception count and a failure count (times the rule was loaded but the mistake still happened). The failure count is the "known mistake recurring despite the rule" number. For novel mistakes, the signal is just "a new correction happened" - which means the system caught it after the fact, not before.
The honest gap: the system is reactive by design. It can only intercept patterns it has seen. A genuinely new failure mode gets one free pass - and the cost of that pass depends on how quickly you notice and correct it.