This article is co-authored with my AI agent. I handle real experience, judgment, and final sign-off; the agent assists with drafting, and I verify every source and claim. The behind-the-scenes log at the end shows what the system caught during production — including mistakes made while writing this very piece.
Last month, I corrected my AI agent for confusing "script generated" with "files modified." I explained the difference carefully. It acknowledged. We moved on.
This month, it did it again.
I use opencode — an open-source AI coding agent — daily. Last week, it reported modifying 172 files. The checklist said ✅. I ran a search. Zero files were changed. The agent had generated a processing script and equated "script exists" with "files were written."
Same mistake. Different session. Zero memory of the correction.
If you use Claude Code, Cursor, Codex, or any AI coding agent daily, you know this pattern. You explain something carefully. The agent gets it. You close the session. Next day — same mistake, from scratch.
The AI didn't forget your preferences. It forgot your lessons.
The Problem: Preferences vs Lessons
Most AI coding agents give you a project context file. Claude Code has CLAUDE.md. Cursor has .cursorrules. opencode reads AGENTS.md. Third-party tools like claude-mem (as of 2026-07, an active open-source project) auto-capture session history and compress it for injection.
They're all solving "remember what the project looks like." None of them reliably solve "remember what went wrong last time."
Here's the difference, side by side:
| Preference (what CLAUDE.md stores) | Lesson (what's missing) |
|---|---|
| "Use Kotlin." | "Don't equate 'script generated' with 'files modified.'" |
| "Variables are camelCase." | "When marking a file as 'complete,' run a search command to verify the change actually exists." |
| Static. Write once, read forever. | Dynamic. Has a trigger condition and a required action. |
| Describes what the project is. | Describes what went wrong and what to do next time. |
Preferences are easy. You write them once, the agent reads them every time. Done.
Lessons are harder. A lesson isn't a description — it's an instruction that fires when a condition is met. And current tools have three problems with them:
1. Reading a rule ≠ executing it. "Don't do X" is a negation. The agent knows what not to do, but not what to do. I switched every rule to "When X triggers, execute Y" — an action-type rule. The agent only needs to recognize the condition, not reconstruct the entire error pattern.
2. Too many rules stop working. Anthropic's own guidance on CLAUDE.md emphasizes keeping it short and specific rather than exhaustive. In practice, once a context file bloats past a certain size, instruction compliance goes down, not up. More rules means more noise drowning out the important ones.
3. No effect tracking. How many rules in your context file actually work? Which ones are dead weight? You have no idea.
The Shift: From "What I Learned" to "What I Must Do Next Time"
My first attempt was session summaries. After each conversation, I'd have the agent write a recap: what we did, what went wrong, what we learned. Dozens of summaries piled up.
It didn't work. Summaries are too long and too vague. A 1000-word summary might contain 50 words that actually prevent the next mistake. The signal was buried in noise. Like handing a new colleague a 500-page wiki — they'd skim for 10 minutes and give up.
Before: "Today we learned that the agent confused script generation with file modification. Be careful about this in the future."
After: "When marking a file modification as 'complete,' verify with a search command that the file actually contains the modified content."
The difference: "Be careful" is an attitude. The agent reads it and does nothing. "Run a search command to verify" is an action. The agent reads it and can execute it.
This isn't just a wording change. It's a category shift — from passive notes to active instructions. An attitude is something you feel. An action is something you do. AI agents are bad at feelings. They're decent at doing.
What This Looks Like in Practice
Over two months, this shift produced 266 rules. The top rule — "skipping the manual before starting work" — intercepted 66 mistakes. The same error pattern that used to repeat every few sessions now gets caught before it happens.
To be clear: this is one person's system, ~164 sessions, not a research study. (A recent arXiv paper proposed "Mistake Notebook Learning" as an academic framework for this exact problem — our work is the engineering practice, not the theory.) The full system — how 266 rules are layered, how interception works, how to track which rules actually catch mistakes — is a separate story.
The difference between a notebook and an error notebook isn't where you store it. It's whether it intercepts before the mistake happens — and whether you know if it worked.
What Doesn't Work
Indirect corrections slip through. If I say "why not just try searching?" — no "wrong" or "no" keyword — the correction trigger doesn't fire. No rule gets extracted. The mistake happens again next session.
Rules get ignored even when present. Rules are soft constraints. Like a "no speeding" sign on the highway — people in a hurry still speed. For hard constraints, you need scripts that block at the code level.
More rules = less brainpower. 266 rules, even injecting only 14 per session, still consume context window space. The more rules, the less room for actual work. This isn't fully solved.
What You Can Do
Don't build the whole system. Start with one shift:
Next time your agent makes a mistake and you correct it, spend 30 seconds asking: "Write this as a rule — 'When X happens, execute Y.'" Store it in a file. Paste it into your next session.
Here's what a minimal error notebook entry looks like:
# Error Notebook
- When: marking a file modification as "complete"
Then: run `grep -c "expected_pattern" filename` to verify
Source: 2026-05-20 session, 172-file incident
Status: active (66 interceptions, 4 failures)
If you have a CLAUDE.md or .cursorrules, don't start over. Add an "error notebook" section. Write rules in the action-type format. After a month, look at which rules actually caught something.
The difference between "don't do X" and "when X happens, execute Y" is the difference between a note and a lesson. One is passive. The other intercepts.
Companion piece: I built the full system — 266 rules across 3 layers, 66 interceptions, a demo you can run in 30 seconds, and a behind-the-scenes log showing every mistake caught during production. → I Built an Error Notebook for My AI Agent
Behind the Scenes: How This Article Was Built (click to expand)
> The system this article describes also produced this article. Here's what actually happened — including mistakes the agent made and how they were caught.
What the system did during production
| Stage | What happened | Why it matters |
|---|---|---|
| Startup | The agent verified its core rules were loaded before starting work | Demonstrates that rule injection fired at session start |
| Planning | The agent presented a plan and waited for approval before writing | Shows the "wait for confirmation" rule in action |
| Fact-checking | Every external claim was verified against real sources — not from memory | Prevents confident-sounding hallucinations |
| Translation | The Chinese draft (v8) was translated to English with data consistency checks | All numbers updated from old values (261/150) to current values (266/164) |
Mistakes caught during production
| What the agent did | How it was caught | What happened |
|---|---|---|
| Drafted the opening with the exact same wording as the companion article (#02) | Three-perspective review (tech blogger view) caught the déjà vu | Opening reframed: #00 leads with the emotional punch of repetition, #02 leads with the technical discovery |
| Used old data (261 rules, 150 sessions) from the Chinese draft | Source pack data consistency check caught the mismatch | Updated to 266 rules, ~164 sessions to match #02's published data |
| Referenced the arXiv paper as a full paragraph, breaking the personal narrative | Reviewer flagged the tonal shift | Compressed to a brief parenthetical mention |
| Missing the "Behind the Scenes" section entirely | AC-6 gate check: "no run log, no publish" | Added this section |
What the system learned
Two rules emerged from writing this English version:
- Translated articles must re-verify all data against the latest source. Numbers drift between drafts — the Chinese version used 261 rules, the English version needed 266.
- Content cluster articles sharing a story must frame it differently. Same story, different angle — or readers feel they're reading the same thing twice.
These rules are now in the error notebook. Next time, they'll intercept the same mistakes before they happen.
Top comments (1)
Preferences vs lessons is the right cut. The reason lessons don't stick: a preference is a fact the file states, a lesson is a guard that has to fire on a condition, and a static context file has no trigger mechanism. It gets read once at load and a negation ('don't equate script-generated with files-modified') just sits there. What worked for us was demoting the lesson from a rule to a verification step bolted onto the action itself: when you mark a file complete, run the grep and paste the count before you're allowed to write the checkmark. The agent doesn't have to remember not to lie, because the check produces the truth. Rules describe, checks enforce.