I can save an expert correction in a file and still get the same wrong answer from an agent. The saved text records the correction. I also need to test whether the agent uses it.
That gap matters in our organizational-memory prototype. A September 5, 2026, code check found that its replay reads a proposed file and looks for expected and forbidden strings. It does not rerun the agent on the task that failed. A green result has a narrow meaning.
If you run agents against real work, you have seen the failure mode. Corrections accumulate in notes. A new session loads some of those notes, then repeats an old mistake. More stored context has not settled whether the answer improved.
What Meta's design adds
Meta Engineering described its organizational second brain on September 2, 2026. It separates knowledge files from procedures, asks experts to resolve ambiguity, and turns feedback into proposed edits. Its evaluation reruns the original scenario and judges the new answer without showing the judge the proposed change. Regression tests check other cases before an expert reviews the fix.
Anthropic's context-engineering article separately describes notes stored outside the context window and loaded in later sessions. That supports the storage pattern. It does not establish that our prototype matches Meta's evaluation results.
What our replay actually checks
Our config/feedback_compiler/replay.py opens the file named by a case. judge_output() then checks the text for required and forbidden substrings. The September 5 inspection confirmed that path in the code.
For example, a case can require the phrase recipe in a concept file. That check can catch a missing file or a missing term. It cannot tell me whether an agent now follows the recipe when choosing a model, handling a failed test, or deciding that evidence is missing.
The compiler has a similarly bounded role. It produces a report of proposals, including recurring patterns and Requests that need a human decision. It does not implement Meta's full process for creating and validating file diffs.
I keep that distinction in the completion claim. A file-content check can pass while behavioral verification remains untested. Calling both results "learned" would hide the work left to do.
Test the answer after the edit
The next check needs the original task, the wrong answer, and the expert's expected correction. Freeze those inputs before proposing an edit. Otherwise the agent can make the test easier while trying to repair its mistake.
Run the original task against the edited knowledge in a fresh session. For a local coding agent, record the GGUF file, quantization, runner version, tools, and task inputs. Keep those fixed while testing the memory edit. Otherwise a model swap and a knowledge change become one result that is hard to explain. Give a separate reviewer the new answer and the expert's acceptance criteria. Keep the edit rationale out of that review.
Then run nearby cases that should still work. A correction that fixes one answer but breaks an unrelated boundary needs another edit. Keep the human review step for changes to money, public identity, or the rules that control what an agent may do.
This is a proposed extension to our prototype. The code check above does not establish that we have built or measured that full loop. It establishes why the current file test cannot close that claim.
Keep storage and proof easy to inspect
The storage choice remains useful. My earlier post, Your AI agent needs a file, explains that choice. This post covers the separate failure mode where a test inspects saved words but never observes a new answer.
The same rule applies before a GGUF quant runs a coding agent: write down the task and acceptance check, run the changed path, and report exactly what the evidence supports.
Accompanying prompt
What the prompt does: It separates checks of a saved correction from tests of the agent's answer and drafts a bounded verification plan.
Copy/paste this prompt:
Role:
Review an agent-memory correction and its tests.
Context:
I will provide the original task, the failed answer, the expert correction,
the proposed file edit, and the current verifier code.
Task:
1. State the behavior the correction should change.
2. List what the current verifier actually observes.
3. Identify any check that reads stored text without rerunning the agent.
4. Specify a fresh-session replay and nearby regression cases.
Output:
For each claim, give its evidence, remaining gap, and smallest useful test.
Keep file-content results separate from observed answer results.
Constraints:
Do not apply edits or change acceptance criteria.
Do not invent a successful replay, model result, or reviewer verdict.
Flag missing inputs and decisions that need the owner.
Copy the block above.
Weekly measured local runs: https://bmdpat.com/5090-reports
Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=organizational-second-brain-without-fine-tuning-2026
Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.
Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.
Top comments (0)