Your agent's memory remembers what you chose. Does it remember what you rejected?

Every memory benchmark I've seen asks the same question: did the right item come back? Recall@k, MRR, hit rate. All measuring whether retrieval surfaces the thing you stored.

That's not the question that costs teams money.

The one that costs money is: you already tried Redis for session caching, it fell over under load, and you ruled it out. Three months later someone asks the agent to "add a caching layer for session data." Does it propose Redis again?

Your memory system stored the choice — "we use Postgres for X." Did it store the veto — "we evaluated Redis and rejected it because of Y"? And if it stored it, does that veto survive long enough to actually stop the re-proposal?

That's what VetoBench measures. It's small, fully reproducible, and — this matters — checked in with every retrieved context, agent reply, and verdict committed to the repo. You don't have to take my word for any number below.

The setup

24 synthetic engineering decisions. 10 of them carry a structured rejected[] field with the reason — an incident, a failed spike, a rolled-back migration. Then 10 scenarios, each posing a task that naturally invites one of those rejected approaches:

• direct traps — the task asks for the rejected thing outright ("Add Redux Toolkit to manage global state")
• implicit traps — the task merely invites it ("propose a caching layer for session data" → Redis)

The judge is deterministic — no LLM judge. A violation is counted when the rejected option shows up in the agent's key_technologies (the proposal actually relies on it), or a conservative regex matches the prose and the agent didn't list it under acknowledged_rejections. Naming an approach while declining it is a reference, not a proposal. The bias is deliberate: violations are undercounted, never over.

The floor: no memory

Same 10 scenarios, agent (claude-haiku-4-5) sees nothing:

none — 80–90% violation rate across 5 runs

With no memory, the agent re-proposed a previously rejected approach in 8 or 9 out of 10 tasks. That's the problem worth solving. Everything else is measured against this floor.

Add the vetoes to context — even as a flat dump, no retrieval — and it drops to 0% across all 50 cells, with the agent naming the prior rejection every single time. So storing the veto is doing the heavy lifting. Which brings us to the interesting part.

The head-to-head: RoBrain vs Mem0

Here's where I have to be honest, because the benchmark is.

I ran Mem0 — the real thing, mem0ai@3.0.13 OSS, its own infer: true LLM extraction (gpt-4o-mini) and its own semantic search. Mem0 got the same information as session-transcript prose: the decision, the rationale, and every rejected option with its reason. Its pipeline decides what becomes a memory; its search retrieves top-5.

Mem0's headline number is good, and I'll say so plainly:

mem0 — 0–20% violation rate across 5 runs (individual runs: 20, 20, 0, 20, 0)

Mem0 handles most of this corpus fine. If I stopped at the violation rate, the story would be "roughly a wash." But the archived retrieval contexts tell a subtler story — and it's the one that matters at scale.

I checked all 50 cells for whether the recorded rejection actually survived into what Mem0 retrieved. In 19 of 50 cells (38%), the veto was simply absent from the retrieved context — lost during LLM fact-extraction, or not retrieved. For three scenarios it was gone in all five runs: Express (s01), axios (s04), GraphQL (s10).

And the violations concentrate exactly there:

• 26% violation rate (5/19) when the veto was absent
• 3% violation rate (1/31) when the veto was present

The cleanest exhibit is s01. Mem0's retrieved memories say "settled on Hono for its middleware model" — and nothing, anywhere, about Express having been evaluated and declined, or why. Five runs, never once acknowledged. For s04, where the axios veto was likewise lost every run, the agent proposed axios for the billing integration in 3 of 5 runs.

Here's the thing: a veto that doesn't survive ingestion isn't just uncitable. It eventually stops protecting you. Fact-extraction pipelines are built to summarize what was decided — "team uses Hono." The rejected alternative is exactly the kind of negative-space detail that summarization drops, because it doesn't look like a fact worth keeping. Until the day it is.

The fair fight

You might object — and you'd be right — that comparing Mem0's full pipeline against a corpus that arrives with rejected[] already structured is rigged. Mem0 had to run its own extraction; RoBrain should have to as well.

So it does. The robrain-e2e condition pushes the byte-identical transcripts given to Mem0 through RoBrain's real production extractor — the exact prompt Sensing and Perception run in production, same Haiku 4.5 — and whatever that produces becomes the corpus. Same input, same burden.

Five archived runs:

• Extraction — 100/100 vetoes survived
• Retrieval — veto present in 50/50 contexts
• Behavior — 0/50 violations, 100% acknowledgement

Side by side, on identical input: Mem0's ingestion lost the veto from 38% of retrieved contexts and violated in 0–20% of tasks per run. RoBrain's full pipeline — extraction included — lost none and violated in none.

The reason isn't magic retrieval. It's structural: RoBrain's extraction prompt asks for rejected[] as a first-class output field. Keeping the veto is the extractor's job, not a lucky side effect of summarizing what happened. When retrieval later surfaces a decision, its rejections come attached — they can't arrive stripped.

What this does NOT show (read before you quote it)

The benchmark's own honesty section is longer than its results section, and I'm going to honor that here:

• flatfile ties robrain at this corpus size. 24 decisions fit in any context window, so dumping everything works as well as retrieving the right five. RoBrain's retrieval advantage is claimed at real corpus sizes — hundreds of decisions across months — which this fixture set doesn't reach. What this benchmark isolates cleanly is: vetoes-in-context vs not, and whether they survive ingestion. Not retrieval quality at scale.
• n = 10 scenarios. Synthetic fixtures, authored by the RoBrain team. Realistic, but chosen by us. The antidote is that they're all checked in — read them, dispute them, add harder ones.
• Variance is real. Across nine runs the none floor ranged 70–90%. Always run 3×+ and quote the range with the date and model.
• The Mem0 veto-absence check is a string match, so it can't perfectly separate "lost at extraction" from "not retrieved." Archiving Mem0's full store per run would — PRs welcome.

Try to make it look bad

VetoBench takes third-party adapters. You implement one interface (MemoryAdapter): an init() that runs your system's real ingestion over the corpus once, and a buildContext() that returns what your system would put in front of the agent. The Mem0 adapter in-tree is the reference — copy its shape and its fairness contract.

# offline, deterministic, no API key
pnpm --filter @robrain/vetobench bench

# behavioral, needs an LLM key
node dist/run.js --live --adapters none,robrain,mem0

PRs adding adapters for other memory tools are explicitly welcome — including ones that make RoBrain look bad. That's what the benchmark is for.

---

RoBrain is open source: github.com/adelinamart/robrain. If your agent has ever cheerfully re-proposed the thing your team spent a week ruling out, that's the gap it's built to close.