DEV Community

Dawncoral
Dawncoral

Posted on Originally published at wallabytoken.com

AI agent memory needs a verification step

#ai #opensource #agents #memory

An AI can update MEMORY.md and still leave five other files describing the old project. We added a small CLI that makes those missed updates visible. It maps nine change types to the documents that need review, checks explicit premises, tracks contradictions, and only then marks a change verified.

The new extras/memory-governance/ pack in wallaby-agent-rules does this without a model, a database, or a service. One Python file. JSON for state. A fresh install takes about a minute.

The first run

From the repository:

python3 extras/memory-governance/memory_governance.py \
  --target /path/to/your-project init
cd /path/to/your-project
python3 scripts/memory_governance.py doctor
Enter fullscreen mode Exit fullscreen mode

A fresh install ends with:

CONTRADICTIONS open=0
RECEIPTS pending=0
DOCTOR OK
Enter fullscreen mode Exit fullscreen mode

init never overwrites an existing config or script. The core memory files stay independent, so you can add the pack to a project that already has MEMORY.md, NOW.md, and INDEX.md.

A rename is not done when one file changes

Tell the CLI which old term should disappear:

python3 scripts/memory_governance.py impact \
  --event file_move \
  --terms "old-component" \
  --record
Enter fullscreen mode Exit fullscreen mode

In our clean-project demo, it found one reference in README.md and created receipt R-000001. The first verification failed:

NOT VERIFIED R-000001: 1 unresolved and 0 unread file(s)
  README.md (1)
Enter fullscreen mode Exit fullscreen mode

That is the useful part. A log entry saying “renamed” would not have caught the second reference.

After updating the reference, verify runs the scan again. If an old mention is intentional history, you allow that path explicitly:

python3 scripts/memory_governance.py verify \
  --receipt R-000001 \
  --allow "archive/**"
Enter fullscreen mode Exit fullscreen mode

The impact and verification stay as separate records in .memory/receipts.jsonl.

Premises have failure signals, not just good intentions

Some rules depend on a relationship staying true. A required file must exist. A JSON field must keep a specific value. A forbidden artifact must stay absent.

The pack turns those into four deterministic checks: file_missing, file_exists, text_contains, and json_equals. It does not run shell commands from the config.

There are three levels:

  • P0: stop the action, exit 2.
  • P1: require review, exit 1.
  • P2: show a warning without blocking routine work.

A manual premise cannot silently pass at P0 or P1. It needs evidence.

Contradictions keep their history

When a new statement replaces an old one, you record both and the evidence:

python3 scripts/memory_governance.py contradict add \
  --old "INDEX.md is optional" \
  --new "INDEX.md changes with files" \
  --evidence "DECISION-42"
Enter fullscreen mode Exit fullscreen mode

Open contradictions make doctor return nonzero. Resolving one appends a new history entry instead of erasing the original claim.

What the scanner will not do

It does not edit your documents. It reports paths and match counts, not file contents. It skips .git, .credentials, .memory, dependency directories, and build output.

Unreadable or oversized files block verification. A file the scanner could not read must not become evidence that no old references exist.

The pack has 14 regression tests covering installation, all nine event types, premise exit codes, contradiction history, explicit scan limits, and false-green prevention. Those tests show that the checks behave as specified. They do not prove a memory-accuracy improvement across other projects.

Try it

The repository has the full README, the CLI, example JSON files, and the tests:

wallaby-agent-rules on GitHub

Start with doctor. Then give it one real rename from your project and see whether the first verification passes.

Wallaby Token builds open-weight inference infrastructure. We use file-based memory and governance on our own projects; the CLI is the part that keeps neighboring documents from being left behind.

Top comments (0)