I've been building Mnemolith, an open-source memory for AI coding tools. Before claiming it helps, I wanted to measure it.
The setup. Claude Code did 7 tasks in a sample codebase, 3 times each, in three setups: no memory, a CLAUDE.md file with the same facts, and Mnemolith with the same facts. Every task needed something only a teammate would know: a past decision, who owns which code, a promise to a partner, a business rule. Two needed a fact mentioned in an earlier session. An automatic check graded every run.
Results (63 runs, Claude Opus 5.5):
| Setup | Tasks done right | Avg cost per task |
|---|---|---|
| No memory | 9 of 21 | $0.125 |
| CLAUDE.md, same facts | 18 of 21 | $0.121 |
| Mnemolith, same facts | 21 of 21 | $0.125 |
Without memory, Claude edited another team's code, changed a frozen partner API and skipped refund approval, every time. CLAUDE.md held the same facts but missed a database address mentioned in an earlier session; Mnemolith saved it on its own and used it next session. Cost per task came out about the same in all three.
What didn't work as I expected. Mnemolith's main idea is that each memory is tied to the files it depends on, and when they change it's flagged as possibly outdated. In this small repo, that flag didn't change outcomes: Claude read the code and noticed the changes on its own. I think it matters more in big repos, but I haven't proven that yet, and that's the next evaluation.
Team memory. This week I added shared memory: memories live in the repo as plain Markdown files, get reviewed in PRs, and a GitHub Action comments when a PR makes one outdated. In a small team test, team memory got 4 of 4 tasks right against 3 for a shared CLAUDE.md, but that's one run per task, so a direction, not a final score.
Everything runs locally, no account or API key, MIT licensed. You can rerun the evaluation yourself with your own Claude Code plan.
Repo: github.com/anishchugh05/mnemolith
Video (70 sec): mnemolith.dev/team.html
I'd love to hear from anyone who's tried measuring this kind of thing on a bigger codebase.
Top comments (0)