DEV Community

Coding-Dev-Tools
Coding-Dev-Tools

Posted on Edited on Originally published at engraphis.com

The agent memory checklist: 12 questions before you trust a memory tool

#agents #coding #llm

A 12-point checklist for evaluating memory for coding agents. Use it before you trust a tool with project decisions.

The failure modes

Coding agents are useful inside a session and forgetful across weeks. Compaction drops architectural decisions. A new session re-reads files it already understood. Flat vector search returns stale and current facts side by side, with no provenance and no way to tell which fact is still true.

The checklist is not about how many memories a tool can store. It is about scope, time, evidence, and control.

The 12 checks

1. Scope

Why it matters: Scratch work should not become a repository convention.

What to test: Store a fact in one workspace and repository, then try to recall it from another. It should not leak.

2. Decision capture

Why it matters: A fact without the reason is hard to trust later.

What to test: Store a decision with its trade-off and date. Ask why it was made.

3. Temporal validity

Why it matters: Stale and current facts should not compete as equals.

What to test: Store a fact, replace it, then query the old and new states. The old fact should close, not disappear.

4. Provenance

Why it matters: An answer is only useful if you can inspect its support.

What to test: Ask for a recall and check that the returned context cites the memory it used.

5. Abstention

Why it matters: A confident guess is worse than a clear miss.

What to test: Ask about a decision that was never stored. The engine should say the evidence is insufficient.

6. Inspection

Why it matters: You need to see what the agent remembered.

What to test: Open the dashboard and inspect the memory, its scope, its history, and the recall rationale.

7. Local default

Why it matters: Project memory should not require a cloud account.

What to test: Run the core offline. Confirm it stores and recalls without a hosted service.

8. Portability

Why it matters: You should be able to back up or move the store.

What to test: Find the database file. Copy it, open the copy, and confirm the memories are intact.

9. Write review

Why it matters: Not every agent write deserves prompt eligibility.

What to test: Check whether writes can be approved, rejected, or quarantined before recall.

10. Retention

Why it matters: Some memory should expire; some must not.

What to test: Define a retention class and confirm the engine respects it.

11. Client coverage

Why it matters: One store is more useful than one store per editor.

What to test: Connect Claude Code and one other MCP client. Confirm both see the same scoped memory.

12. Claim honesty

Why it matters: Benchmarks should be bounded and reproducible.

What to test: Read the benchmark page. Check the fixture size, the metric, and the reproduction command.

The 10-minute test

Use one real repository and one decision you already know. Do not start with a large import.

  1. Install the MCP server: 
   pip install "engraphis[mcp]"
   engraphis-init
   claude mcp add engraphis -- engraphis-mcp
Enter fullscreen mode Exit fullscreen mode
  1. Ask your agent to remember a scoped decision: 
   Checkout API uses PASETO access tokens.
   The November migration replaced the legacy JWT signer.
   Scope: workspace "acme-api", repository "checkout".
Enter fullscreen mode Exit fullscreen mode
  1. Open a fresh session and ask: 
   Why did checkout move from JWT to PASETO?
   Check Engraphis memory and show the source, scope, and history.
Enter fullscreen mode Exit fullscreen mode

Pass if the new session returns the decision, the reason, the scope, and a citation. Fail if it returns a plausible answer with no source, or if the old fact is still treated as current.

What good looks like

  • Bounded: recall returns a small set of relevant memories under a token budget, not a dump of everything the tool has seen.
  • Scoped: every memory carries its workspace, repository, session, and record boundary. Cross-project recall is a choice, not an accident.
  • Temporal: superseded facts close instead of disappearing. You can ask what was believed before and why it changed.
  • Inspectable: you can open the record, see its history, and trace the recall. If the tool cannot show you that, it is a black box.

Try it

Engraphis is the local-first memory engine this checklist is based on. The core is Apache-2.0, stores memory in one SQLite file, and needs no cloud account.

If you use the checklist, tell me which check is missing or wrong. I am more interested in the critique than a vote.

Top comments (0)