A field note from the autonomous Claude Code agent I run every day on one Windows PC. The numbers come from its own ledgers, not from memory.
Free to read first: a PDF sample of the book behind this setup — its first 3 chapters: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample
One of the rules in my last note was that a test that copies your sentence copies your mistake. This note is about the opposite failure. It happened the same day, on a short post.
The post
My agent queues short Korean posts for another channel. One of them retells a story from an earlier note in this series: my ebook checker printed a pass line, and the store's validator rejected the file I had resubmitted with 17 errors. The post then says what my checker looked at, which was zip layout, XHTML parsing, the copyright page, and that the validator the store uses had not been run until then.
Every sentence in the post matched its source. The rejection comes from my diary entry for that day. What the checker looked at comes from the checker itself, as it was one commit before the validator was added. The test reads that diary line and runs git show on that commit. I wrote a second post in the same cycle, and its test was built the same way.
What the test held
For each post, the test held words I thought were the facts. For this one, that meant words like the 17 errors and XHTML parsing. I broke the two posts 10 ways by hand, and the tests failed all 10 times.
That looked like a good test. It was a test of my own picks. Most of the 10 went after the words I had picked, and none touched two phrases.
The first pass
An independent auditor agent made its own changes to the post.
One turned the file I had resubmitted into a first submission. Another turned had not been run until then into had been run only sometimes. Both made a true post false. Neither touched a number or a name, so the test never looked at them, and it passed.
The auditor also flagged one verb. My post said the checker stamped the pass line on the file. The source says it printed it. I changed the verb.
For the fix, I held longer stretches of each sentence and added the opposite wordings as text that must not appear. My hand breakages across the two posts grew from 10 to 18, and the tests failed on all 18.
The second pass
The auditor failed it again. This time it changed the subjects. One sentence starts with the ebook checker, another with what our tool looked at. Swap who did it, and the test still passed. I had held phrases from each sentence and left the start open.
It also found two gaps in my notes: my diary did not record the first failed pass, and my handoff note's list of source lines left one out.
The fix
I stopped picking. The test now compares each of the two posts with its full text, character for character. The source checks stay next to it as separate assertions: the diary line, the old checker's items, the commit where the validator first appears.
The auditor's own set of 35 breakages, across both posts, failed the test 35 times. The third pass passed. The full suite ran 3,112 tests, all passing.
Why this does not undo the last note
Holding the whole text only says the post did not change. It says nothing about whether the post is true. That half is the source checks.
A test that holds only your sentence, with no source, copies your mistake. A test that holds only the words you picked misses the words you didn't. I needed both, in two places.
The cost is that any edit to the post now needs an edit to the test. The posts are under 200 characters and already written. That is cheap.
The rules
- Do not choose which words are the facts. In a short public text, the subject, the verb and the words about time carry facts too. Hold the whole text.
- Hold the text and check the source separately. One says nothing changed. The other says it was true.
- Your own breakages test your own blind spots. Most of my first 10 went where I was already looking. Let someone else break it.
- Add the opposites. First against again, never against sometimes. They cost one line each.
Where this comes from. Every post here comes from one setup I run daily: a CLAUDE.md, memory files the agent reads before it touches anything, and a separate auditor agent that returns PASS or FAIL. The first 3 chapters of the book that walks through it are free as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample
The full edition is 11 chapters plus 4 ready-to-use templates (CLAUDE.md starter, memory files, auditor checklist, measurement guide) and a hands-on section for every chapter, $19 as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code
Questions about the setup are welcome in the comments — I'll answer with what actually happened, not theory.
Top comments (0)