DEV Community

Anish Chugh
Anish Chugh

Posted on Edited on

I tested whether memory actually helps Claude Code. Here's what happened.

I've been building Mnemolith, an open-source memory for AI coding tools. Before claiming it helps, I wanted to measure it.

The setup. Claude Code did 7 tasks in a sample codebase, 3 times each, in three setups: no memory, a CLAUDE.md file with the same facts, and Mnemolith with the same facts. Every task needed something only a teammate would know: a past decision, who owns which code, a promise to a partner, a business rule. Two needed a fact mentioned in an earlier session. An automatic check graded every run.

Results (63 runs, Claude Opus 5.5):

Setup Tasks done right Avg cost per task
No memory 9 of 21 $0.125
CLAUDE.md, same facts 18 of 21 $0.121
Mnemolith, same facts 21 of 21 $0.125

Without memory, Claude edited another team's code, changed a frozen partner API and skipped refund approval, every time. CLAUDE.md held the same facts but missed a database address mentioned in an earlier session; Mnemolith saved it on its own and used it next session. Cost per task came out about the same in all three.

What didn't work as I expected. Mnemolith's main idea is that each memory is tied to the files it depends on, and when they change it's flagged as possibly outdated. In this small repo, that flag didn't change outcomes: Claude read the code and noticed the changes on its own. I think it matters more in big repos, but I haven't proven that yet, and that's the next evaluation.

Team memory. This week I added shared memory: memories live in the repo as plain Markdown files, get reviewed in PRs, and a GitHub Action comments when a PR makes one outdated. In a small team test, team memory got 4 of 4 tasks right against 3 for a shared CLAUDE.md, but that's one run per task, so a direction, not a final score.

Everything runs locally, no account or API key, MIT licensed. You can rerun the evaluation yourself with your own Claude Code plan.

Repo: github.com/anishchugh05/mnemolith
Video (70 sec): mnemolith.dev/team.html

I'd love to hear from anyone who's tried measuring this kind of thing on a bigger codebase.

Top comments (0)