DEV Community

Calas Automations
Calas Automations

Posted on AI-assisted

I re-ran a rival's memory benchmark. It tied mine. Here is the interesting part.

I build Monomem, shared memory for AI coding agents. It records which app wrote each note and holds suspicious writes until a person looks.

On LongMemEval-S, a standard long-term-memory test, a well-known memory project (agentmemory) publishes 84.7%. I re-ran its own evaluation on the same data file and got 84.7% back, exactly.

  • Monomem scored 86.0% (404 of 470 questions).
  • Question by question, 14 were answered only by Monomem and 8 only by agentmemory.
  • That gap is not significant (exact McNemar test, p = 0.29). It is a tie.

Two more things I will not hide:

  • Monomem's ranking was tuned on half of this question set, so the fairer figure is the other half: R@5 97.5, Rall@5 86.2. The repo says so.
  • Another system, Vestige, reports a higher score than mine (its own number, which I have not re-run). It is named in the docs.

Why post a tie? Because memory search is turning into a commodity. What I think still matters is trust: who wrote a note, whether an outside web page could have poisoned it, and whether you can undo it. Monomem's own attack check (monomem check --attacks) replays 26 known poisoning attacks on a copy of your store. Two are known misses (a reworded override, and self-poisoning), and the README lists them next to the headline number. It is pattern matching, so a reworded attack it does not know can get through.

Everything runs on your computer in one file: no cloud, no account. MIT licence. Beta, not independently security-reviewed.

Reproduce the numbers: monomem bench --longmemeval ... --per-question out.ndjson (instructions in docs/BENCHMARK.md). If you can beat or break it, tell me.

Repo: https://github.com/calasdan-sketch/monomem

Using it at work? Team setup and support: https://calasautomations.com/devtools/monomem/#teams

Top comments (0)