This was originally published on All Agents Considered.
Some time ago, I covered my Hermes research workflow. It runs every week, sorts through tens of tech news publications and forums, and filters out the fluff so I can stay on top of everything that is going on. It took me a few tries to get it right, but now I can safely rely on it.
That trust came from two early mistakes. On the first real run, Hermes ranked a generic news item too highly because my definition of practical tech news was too loose. I tightened the instruction and ran it again.
On the second run, a vendor press release passed the filter because I never told Hermes to exclude vendor press releases. I added that rule, ran it a third time, and got a clean result.
The workflow got better, but the reason why it got better simply vanished.
If you’d open my final instruction file now, you’ll see one clean rule: reject vendor press releases. You do not see the bad output that created the rule, why the first wording failed, or the result that proved the fix worked.
Hermes still remembers my projects, files, and old conversations. It also squeezed the useful lessons from those failed runs into one line with no history. And that bothers me because the next mistake it makes might be less obvious. A tool may fail only on one server. An API may accept a believable field name and ignore it. A workflow may write to the wrong folder and still report success.
So I figured we can’t let the AI bury these critical lessons in old chat logs.
That’s why I started adding one file to each repeated workflow:
MISTAKE_LEDGER.md
This article explains what belongs in it, what does not, and the small instruction I use to stop an agent from turning every failed attempt into permanent bad advice.
In this edition
Why persistent memory still loses operational lessons
Where a Mistake Ledger fits inside a file-based workflow
The four-step rule that keeps guesses out of the ledger
A complete instruction you can paste into Hermes or Codex
The failure modes that make a mistake ledger worse than no ledger
Memory Keeps Context
I have argued for months that memory changes what an AI agent can do.
An agent that carries project details and preferences across sessions beats a blank chat window every time. I wrote about this shift in Forgetting to Forget when persistent memory made long-running agent work feel possible.
I still believe that.
The mistake was treating memory and learning as the same thing. Hermes Agent v2026.6.5 describes its built-in memory as a bounded, curated set of user preferences, project details, environmental context, and learned information. Session logs preserve more of what the system did.
An operational lesson serves a different purpose. It tells you what failed, what fixed it, and when that fix applies again.
Those are three different layers.
The exact correction may still exist in an old conversation, but that does not mean it will enter the context of a new run six weeks later. Hermes built-in memory is injected as a bounded snapshot at the start of a session rather than retrieved as a complete archive of every old chat. The useful lesson can remain buried in conversation history even while the agent remembers the project.
On March 21, 2026, an Ask HN post about operational memory called this missing layer an “Agent Experience Cache.” The post listed tool quirks, repeatable workflow patterns, environment-specific knowledge, and failure modes that cost too much time to rediscover.
That idea made sense to me, but I wanted a smaller version that I could read without adding another memory service.
One Markdown file beside the workflow felt like the right size.
Keep Mistakes Local
My first instinct was to create one global file containing every mistake made by Hermes or Codex.
That file would become useless fast.
My content research workflow, Substack analytics tools, server deployments, and draft-writing system fail for different reasons. A rule learned while deploying software should not make Hermes cautious while sorting research. Likewise, a correction about brand sources should not influence how Codex handles a server path. Both Hermes and Codex share the same files and workflows through my Obsidian vault.
The mistakes should live where the work happens.
I already use an INDEX.md file to map large workflows. I explained that setup in Why My Best Agent Workflow Is Mostly Files.
Inside the project, each repeated workflow gets its own small folder. My research sorter came from the four-file pattern in Tear Down Your AI Workflow and Rebuild It Like This:
01.Research Sorter/
├── 01.instructions.md
├── 02.input.md
├── 03.output.md
├── 04.review.md
└── MISTAKE_LEDGER.md
The first four files tell Hermes how to run the work. The fifth preserves the verified lessons produced by running it.
If a project contains one workflow, project-specific and workflow-specific mean the same thing. Once a project contains several repeated processes, separate ledgers keep the lessons scoped.
Because the ledger is visible and readable, I can challenge an entry, edit its scope, or retire the advice after a tool changes. I do not have to rely on a memory entry or session log that sits apart from the workflow it affects.
If the same lesson appears across several workflow ledgers, I can promote it into a wider agent instruction after reviewing it.
What an Entry Holds
The file needs enough detail to stop the same error without becoming a diary of everything that went wrong.
Each entry records seven fields:
Date
Task
What went wrong
Confirmed cause
Verified fix
When this applies
When to recheck or retire it
If the ledger had existed when I built my research workflow, the vendor PR entry would have looked like this:
## YYYY-MM-DD: Vendor PR passed the research filter
**Task:** Sort research items into All Agents Considered article observations.
**What went wrong:** A vendor announcement was kept as a strong
article signal even though no independent technical source discussed it.
**Confirmed cause:** The filter excluded broad news but did not
explicitly exclude vendor PR as a daily input.
**Verified fix:** Add vendor PR to the exclusions and require an
independent technical source before keeping a vendor announcement.
**When this applies:** Every All Agents Considered research-sorting run.
**Recheck or retire:** Recheck if the source policy changes.
This is a reconstruction from the failure I documented in the earlier article.
I did not have a ledger then, so I am taking a pattern already hidden inside my workflows and making it visible.
Verify Before Recording
AI agents write terrible rules when they update their own instructions after every failure. One plausible guess can harden into a permanent constraint before anyone checks whether it explains what went wrong.
During one weekly research run, Hermes kept a promotional article from a vendor’s own website and treated it as independent reporting. When I told it to remove the article, Hermes guessed that the missing ingredient was a deep technical quote. It proposed a permanent rule requiring one in every future submission.
The diagnosis sounded plausible, but it focused on the article’s format when the source was the real warning sign. The piece appeared on the vendor’s corporate domain and repeated the company’s marketing claims without support from independent reporting. The useful lesson was to treat articles on vendor-owned domains as promotional unless an independent source supported them. Recording Hermes’s first guess would have taught the workflow to reject legitimate news briefs because the reporter did not include a direct quote.
That is why I have to verify the root cause with my own eyes before the agent turns a plausible explanation into a permanent lesson.
Building my unofficial Substack SDK exposed the same problem in code. Hermes proposed scheduled_at as the field for scheduling a post. The choice looked logical because plenty of other APIs use that naming convention. The Substack scheduling endpoint I tested expected trigger_at instead.
An automated test now checks the request body and fails if the implementation sends a different field. That gives me proof outside the conversation instead of another confident answer from an agent.
I covered the full story in How I Built a Substack API With Hermes and Codex. Without that hard check, neither field name would deserve a permanent spot in a Mistake Ledger.
Based on that experience, I created a strict four-step rule that keeps guesses out of my final documentation:
Catch: Workflows fail, I correct the AI, or the AI spots a bad assumption.
Fix: I finish the task before writing down a lesson.
Verify: I prove the fix works using a result outside our chat.
Record: I add one specific entry to the workflow ledger.
Real proof makes or breaks the verification step. It can come from passing code tests, live API responses, correct files landing in the right folders, or final outputs meeting exact standards. AI confidence alone proves nothing.
The Tiny Instruction
The working version fits in one instruction-only skill. There is no script, service, database, or download.
You can paste the text into a Hermes or Codex task. If you want it available across repeated sessions, save the same block as SKILL.md.
---
name: mistake-ledger
description: Record a verified reusable lesson after a workflow
failure or user correction, and check the current workflow's
MISTAKE_LEDGER.md before running related work.
---
# Mistake Ledger
Before running a repeated workflow, check whether its folder contains
MISTAKE_LEDGER.md. Read only that workflow's ledger and apply relevant
active lessons. State which lesson you are applying.
When a task fails, the user corrects you, or you discover a wrong
assumption:
1. Fix the problem first.
2. Verify the replacement through an observable result.
3. Record a lesson only when it is concrete, reusable, and scoped to
this workflow.
4. Append one entry to MISTAKE_LEDGER.md with:
- date
- task
- what went wrong
- confirmed cause
- verified fix
- when this applies
- when to recheck or retire it
Do not record speculation, secrets, private content, or one-off user
preferences. Do not turn an unverified diagnosis into a permanent rule.
Mark outdated lessons as retired instead of deleting them.
Hermes users can keep the repeated version under ~/.hermes/skills/, or add a shared folder to skills.external_dirs in ~/.hermes/config.yaml. The Hermes Agent v2026.6.5 skill documentation describes both routes.
If you keep the instruction inside the workflow folder without configuring that folder as a skill directory, ask Hermes to read the file before the run.
Codex users can save the same instruction at .agents/skills/mistake-ledger/SKILL.md inside a repository. The current OpenAI Build skills documentation describes the required SKILL.md format.
Test the Next Run
Creating the file proves nothing by itself. The next run must show that the recorded lesson changed the workflow’s behavior.
My next research test is straightforward. I will place a vendor announcement in 02.input.md beside several independent technical stories. I will then run the workflow and check three things:
Did Hermes read
MISTAKE_LEDGER.mdbefore scoring the items?Did it state that the vendor PR lesson applied?
Did it exclude the announcement unless an independent technical source supported it?
The same test works for other workflows. First, reproduce the conditions that caused the original failure. Then check whether the agent applies the relevant lesson without being reminded.
I also need to measure the cost of using the ledger. If Hermes starts citing irrelevant history on every run, the ledger has created a new problem. It should reduce repeated work without becoming a compliance manual that the agent must review before every task.
This method still needs several real repetitions before I make stronger claims about it. I expect the trigger wording, retirement rules, and promotion threshold to change as I use it across more workflows. That uncertainty belongs in the build log. A clean ending would give the method more confidence than it has earned.
When the Ledger Lies
A Mistake Ledger creates five failure modes of its own. Any one of them can make the workflow worse:
False causality: The agent records a symptom as the cause. It then applies the wrong fix every time the workflow runs.
Bad scope: The agent promotes a useful lesson from one workflow into a universal rule. That rule starts interfering with unrelated work.
Stale advice: APIs change, tools get updated, and folder structures move. A correct fix can eventually become a new bug.
Noise: Every typo, failed search, and abandoned idea enters the ledger. Hermes then has to sort through a second pile of clutter before starting the real task.
Exposure: A careless entry preserves credentials, private customer information, session cookies, or sensitive tool output. None of that information belongs in a file the agent keeps reading.
Some repeated failures reveal a broken workflow structure rather than a reusable lesson.
If Hermes keeps selecting an archived file because current and old drafts live together, the folder structure needs fixing. If every output lands in a new location, the workflow needs a defined destination. Recording “pick the current file next time” only hides those structural problems from the next run.
Before turning that kind of confusion into a permanent lesson, run the audit from The Twenty Minute Audit That Found Where Hermes Was Getting Lost. The audit will show whether the agent’s map of the work is the real problem.
My dividing line is simple:
If Hermes cannot find the right work, fix the map. If it finds the right files and still repeats the same operational error, record the lesson.
Try It in 60 Seconds
Open one workflow that you run more than once. Think of the last correction you gave the agent. Then confirm that the replacement worked outside the conversation where you suggested it.
If you still do not know what caused the failure, stop there. An unverified correction has not earned a permanent entry.
Once the fix is verified, create MISTAKE_LEDGER.md inside the workflow folder. Record the seven fields while the evidence is still easy to inspect.
On the next run, ask the agent to read the ledger first and state which lesson applies. Do not remind it of the original error or steer it toward the answer.
That run is the test. You do not need a new model, an extra memory service, or a global database containing everything your agent has ever done. One workflow and one verified mistake are enough to test whether the lesson survives the conversation where it was learned.
Give Failure a Job
The clean version of my research workflow hides the two failed runs that shaped it. The final instructions preserve the rules without showing which mistakes forced me to add them.
That makes the instructions easier to read. It also removes the operating history I will need when a rule starts causing trouble.
Six months from now, I want to know why a rule exists before I delete it. Hermes should understand when the rule applies without searching old conversations. Both of us should see when the advice has expired.
These tools support different parts of the same system. Project maps and workflow folders make the work findable and repeatable. Audits expose structural defects before they become permanent instructions. The Mistake Ledger preserves verified failures after the immediate problem has been fixed.
The ledger does not turn Hermes into an agent that trains itself. It gives the next run a short piece of reviewed evidence from the last one. That is enough for version one.
If the broader stack is what you are after, I wrote about the full cost breakdown and the morning workflow that runs on it. Both depend on the same file-based workflow structure to stay reliable across sessions.
Capability is cheap when the foundation is broken. Audit one folder. Fix one thing. Then decide whether you need another tool.
Top comments (0)