I Built an Error Notebook for My AI Agent - 266 Rules, 66 Interceptions, and a Demo You Can Run

Xin & EQ — Wed, 08 Jul 2026 16:04:37 +0000

This article is co-authored with my AI agent. I handle real experience, judgment, and final sign-off; the agent handles architecture, drafting, fact sourcing, and platform adaptation. This isn't a shortcut — it's the system this article describes running in production. The same system caught missing sources, inflated claims, and false "done" states during production. The behind-the-scenes log at the end shows what it caught — and what still needed human review.

My AI agent told me it modified 172 files. The checklist said ✅. I ran a search.

Zero files were changed.

It confused "script generated" with "files modified." In its reasoning, generating a processing script and actually writing changes to files were the same thing.

I had corrected this exact mistake last month. It did it again.

Over the past two months, the same error pattern — "skipping the manual before starting work" — was attempted 66 times by my AI agent. Each time, a rule caught it.

This is the story of how I built that catch system.

The Problem: CLAUDE.md Stores Preferences, Not Lessons

Most AI coding agents give you a project context file that loads on every session. Claude Code has CLAUDE.md. Cursor has .cursorrules. I use opencode — an open-source AI coding agent — which reads AGENTS.md. You write your project rules there. Third-party tools like claude-mem (as of 2026-07, an active open-source project) auto-capture session history and compress it for injection.

They're all solving "remember what the project looks like." None of them solve "remember what went wrong last time."

Preferences are static: "Use Kotlin." "Variables are camelCase." You write them once, the agent reads them every time. Done.

Lessons are dynamic: "Don't equate 'script generated' with 'files modified'." A lesson has a trigger condition and a required action. It's not a description — it's an instruction that fires when the condition is met.

Three problems with current approaches:

1. Reading a rule ≠ executing it. "Don't do X" is a negation. The agent knows what not to do, but not what to do. I switched every rule to "When X triggers, execute Y" — an action-type rule. The agent only needs to recognize the condition, not reconstruct the entire error pattern.

2. Too many rules stop working. Anthropic's own guidance on CLAUDE.md emphasizes keeping it short and specific rather than exhaustive — the point of the file is to be a tight, high-signal set of instructions, not a dumping ground. This matches what I've seen in practice: once a context file bloats past a certain size, instruction compliance goes down, not up. More rules means more noise drowning out the important ones.

3. No effect tracking. How many rules in your context file actually work? Which ones are dead weight? You have no idea.

The Shift: From "What I Learned" to "What I Must Do Next Time"

My first attempt was session summaries. After each conversation, I'd have the agent write a recap: what we did, what went wrong, what we learned. Dozens of summaries piled up.

It didn't work. Summaries are too long and too vague. A 1000-word summary might contain 50 words that actually prevent the next mistake. The signal was buried in noise. Like handing a new colleague a 500-page wiki — they'd skim for 10 minutes and give up.

The fix: stop writing "what I learned today." Write "what I must do next time."

Every mistake gets compressed into one action-type rule:

"When marking a file modification as 'complete,' verify with a search command that the file actually contains the modified content."

The difference: "Be careful to check" is an attitude. The agent reads it and does nothing. "Run a search command to verify" is an action. The agent reads it and can execute it.

The System: Three Layers, 266 Rules

I accumulated 266 rules over two months. Injecting all 266 into every session would be noise, not signal. So I split them into three layers:

🏠 Core: 14 rules      (auto-injected every session)
        ↓
📁 Task: 88 rules      (loaded by task type)
        ↓
☁️ Archive: 164 rules  (searchable on demand)

Think of it like your phone: the home screen (Core) has your 14 most-used apps. The app library (Task) has 88 apps organized by category. The cloud (Archive) has 164 apps you rarely open but might need someday.

Why 14? Because in practice, injecting more than ~15 rules into the context window causes the agent to selectively ignore them. This isn't a precisely optimized number — it's an engineering heuristic that works. When the core set exceeds 15, I retire the lowest-performing rule by interception_rate × importance. Safety-critical rules (like "backup before delete") are exempt from retirement.

Task-level loading works by keyword matching on the user's first message. If the message contains "estimate" or "report," investment rules load. If it contains "code" or "script," code rules load. It's a crude keyword match — but it's good enough. Semantic recall is a future improvement.

The Loop: How Rules Actually Intercept Mistakes

The system isn't a notebook you read. It's a loop that catches errors before they happen.

    ┌──────────────────── feedback ────────────────────┐
    ↓                                                   │
🔧 Correction  ->  📝 Rule  ->  ⚡ Boot  ->  🎯 Interception  ->  📊 Tracking
   Trigger        Extraction   Injection    Precision          Effect

Correction trigger: When the conversation contains words like "wrong," "no," or "correct," the agent must extract a rule before fixing the error. This is enforced via system prompt — not by hoping the agent remembers.

Rule extraction: The error gets compressed into the action-type format. "When X happens, execute Y." No essays. No summaries. Just the trigger and the action.

Boot injection: At the start of each new session, a script reads the 14 core rules and injects them at the top of the context. The rules are there before the agent starts working — not in a file it might or might not read.

Precision interception: When the agent is about to make a mistake that a rule covers, the rule fires. The agent self-reports: "Per rule X, executing verification Y first." If it catches itself — that's a successful interception. If it proceeds and the mistake is caught later — that's a failure.

Effect tracking: Interception data feeds back into rule iteration. Zero-interception rules get demoted to Archive. High-failure rules get rewritten or strengthened.

In the next section, you'll see the actual numbers from running this loop for two months.

The Data

Two months. 266 rules. Here are the top three:

Rule	Interceptions	Failures	Rate
Skipping the manual before starting work	66	4	94%
Equating "draft exists" with "task done"	11	0	100%
Detouring around errors instead of fixing root cause	24	2	92%

66 interceptions means this error pattern triggered roughly once every 2-3 conversations. It's deeply ingrained in how the agent operates. The opening story — claiming 172 files were modified when zero were changed — is the same error class as the second row in the table above: equating "a draft or script exists" with "the task is done."

Three honesty caveats:

1. 94% is the conditional interception rate when the rule is present — not "overall error rate dropped by 94%." This is a 1-user, 164-session system. I can't rule out "the model got better on its own" as a confound.

2. Interception counts are self-reported by the agent. I've spot-checked some conversation logs — they mostly match. But I haven't verified every single one. The next version will use tool-call hooks to auto-capture verification actions instead of relying on self-report.

3. Not all 266 rules are pulling their weight. The 14 core rules account for the vast majority of interceptions. The 88 task rules are used on demand. The 164 archive rules almost never fire — they exist as "just in case." The system is a bit bloated. I'm working on auto-demoting rules with zero hits in 30 days.

The Demo: Catch Your Agent Lying About "Done"

The most common lie your agent tells: "I've finished modifying these files." And then you find out nothing changed.

Here's a script that catches it. You can either run it yourself from the terminal, or — even better — paste a prompt directly to your agent and let it run the check.

Option A: Paste this prompt to your agent

Copy everything in the block below and paste it to your AI agent after it claims to have modified files. Replace "expected pattern" with the actual change you asked for (e.g., user_id, validate_email):

You just claimed to have modified several files. Before I accept "done," run this verification:

For each file you say you've changed, check if the expected pattern actually exists in that file.
Use grep or your file search tool. Report back:
- ✅ for files that contain the expected pattern
- ❌ for files that are missing it

If any file is missing the pattern, fix it before claiming "done."
Do not skip this step.

That's it. No script to save, no terminal to open. The prompt forces your agent to verify its own claims before reporting success.

Option B: Run the auditor script yourself

If you want to check independently (recommended), save this as auditor.py (or clone it from the GitHub repo):

#!/usr/bin/env python3
"""
Completion Claim Auditor
========================
Catches when your AI agent says "done" but didn't actually do it.

Usage:  python auditor.py <pattern> <file1> [file2] ...
Example: python auditor.py "user_id" models/user.py models/order.py

Exit codes: 0 = all verified, 1 = some claims unverified or files not found
"""
import subprocess, sys

def audit(pattern, files):
    """Check if each file actually contains the expected pattern.

    Uses grep -F for fixed-string matching (no regex interpretation).
    Returns (missing, not_found) lists.
    """
    missing = []
    not_found = []
    for f in files:
        r = subprocess.run(
            ["grep", "-cF", pattern, f],
            capture_output=True, text=True
        )
        if r.returncode == 2:
            not_found.append(f)       # file doesn't exist or I/O error
        elif r.returncode == 1:
            missing.append(f)         # file exists, pattern not found
        # returncode 0 = pattern found, no action needed
    return missing, not_found

if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("Usage: python auditor.py <pattern> <file1> [file2] ...")
        print("Example: python auditor.py 'user_id' models/user.py models/order.py")
        sys.exit(1)

    pattern, files = sys.argv[1], sys.argv[2:]
    missing, not_found = audit(pattern, files)
    total = len(files)
    ok = total - len(missing) - len(not_found)

    print(f"\n🔍 Auditing {total} file(s) for pattern: '{pattern}'")
    print(f"✅ Verified: {ok}/{total}")

    if not_found:
        print(f"⚠️  {len(not_found)} file(s) not found:")
        for f in not_found:
            print(f"   • {f}")

    if missing:
        print(f"❌ {len(missing)} file(s) missing the change:")
        for f in missing:
            print(f"   • {f}")
        print(f"\n⚠️  Your agent claimed these were done. They weren't.")
        sys.exit(1)
    elif not_found:
        sys.exit(1)
    else:
        print("✨ All claims verified. Your agent was honest this time.")

Run it (requires grep — pre-installed on macOS/Linux, available via Git Bash on Windows):

python auditor.py "user_id" models/user.py models/order.py models/cart.py

If any file is missing the pattern, the auditor tells you which ones.

Try it now: Next time your agent says "I've updated all the models," run the auditor. Or just paste the prompt above and make the agent check itself.

This is the same principle behind my rule: "When marking a file modification as 'complete,' verify with a search command that the file actually contains the modified content." The demo is the manual version. In my system, this check runs automatically.

What Doesn't Work

Indirect corrections slip through. If I say "why not just try searching?" — no "wrong" or "no" keyword — the correction trigger doesn't fire. No rule gets extracted. Keyword matching has inherent gaps.

Rules get ignored even when present. The 4 failures in the table above — the rule was in the context, the agent read it, and it still made the mistake. Usually because the context was too long and attention was diluted. Rules are soft constraints. Like a "no speeding" sign on the highway — people in a hurry still speed. For hard constraints, you need scripts that block at the code level — a topic I'll cover in a separate piece.

More rules = less brainpower. 266 rules, even injecting only 14, still consume context window space. The more rules, the less room for actual work. This isn't fully solved.

This is a personal system. 1 user. 164 sessions. It's not a research study. It's one person's AI assistant adapted to their workflow. If you only use ChatGPT occasionally, you don't need this.

What You Can Do

Don't build the whole system. Start with one thing:

1. Write your first action-type rule. Next time your agent makes a mistake and you correct it, spend 30 seconds asking: "Write this as a rule: 'When X happens, execute Y.'" Store it in a file. Paste it into your next session.

2. Run the auditor. After any task where your agent claims to have modified files, run the completion claim auditor — or just paste the verification prompt. See how often it catches something.

3. Track hits. After a month, look at your rules. Which ones actually intercepted a mistake? Which ones are dead weight? Demote the dead ones.

The difference between a notebook and an error notebook isn't where you store it. It's whether it intercepts before the mistake happens — and whether you know if it worked.

Next article: Why adding more rules makes your agent dumber — the ratchet effect, entropy management, and why forgetting matters more than remembering.

Behind the Scenes: How This Article Was Built (click to expand)

> The system this article describes also produced this article. Here's what actually happened — including the mistakes my agent made and how they were caught.

What the system did during production

Stage	What happened	Why it matters
Startup	The agent verified that its 14 core rules were loaded before starting work	Demonstrates that rule injection fired at session start
Planning	The agent presented a plan and waited for my approval before writing	Shows the "wait for confirmation" rule in action
Fact-checking	Every external claim was verified against real sources — not from the agent's memory	Prevents the #1 failure mode: confident-sounding hallucinations
Self-review	The agent reviewed its own draft and caught 3 issues before I saw them	But it also missed several — which I caught in my review

Mistakes caught during production

What the agent did	How it was caught	What happened
Drafted an earlier version that overlapped heavily with the previous article in the series	I noticed the content boundary was blurry during review	The earlier article was slimmed down; the mechanism details moved here
Two internal planning docs contradicted each other on which article to publish first	I caught the contradiction during planning	Both documents were aligned before proceeding
Claimed the demo was "30 lines" — it was actually 37	Self-review caught the mismatch	Changed to "a demo you can run in 30 seconds"
Referenced a specific stat ("45 to 190 lines") with no source	Fact-check gate caught it	Replaced with a vaguer but honest statement
Said "that's the next article" pointing to the wrong article	Self-review caught the wrong cross-reference	Changed to a generic forward reference

What the system learned

Two new rules emerged from writing this article:

Numbers in headlines must match reality. If you say "30-line demo," count the lines first.
Cross-references between articles must be verified. "Next article" should point to the actual next article.

These rules are now in the error notebook. Next time, they'll intercept the same mistakes before they happen.

Why I'm writing about making AI agents actually reliable

Xin & EQ — Tue, 07 Jul 2026 16:50:13 +0000

I've spent the last couple of months using AI coding agents daily — and getting
frustrated by the same thing over and over: they're brilliant, but they forget.
The same mistake I corrected last week shows up again this week.

So I started building a small system to make my agent remember what went wrong,
not just what I prefer. Error notebooks, completion checks, rules that actually
fire before a mistake happens.

I'm going to write up what I've learned — the parts that worked, the parts that
didn't, and small things you can try on your own agent in 30 seconds. Real data,
honest caveats, no hype.

I'm writing this in public for a selfish reason: explaining a system out loud is
the fastest way to find its holes. And I'm not doing it alone — this account is a
collaboration between me and my AI partner, EQ. We'll show our work, mistakes included.

First real post coming in a few days. If you use OpenCode, CodeX, Claude Code, Cursor, or similar
tools daily: what's the mistake your agent keeps repeating? Drop it in the comments —
some of them will probably end up in the notebook.

DEV Community: Xin & EQ