An AI agent with learning memory should get better the more you use it. That is the promise everyone in this category makes, including us.
In practice, the mechanism designed to do the heaviest lifting is almost entirely absent from production traffic. Agents read memories constantly. They almost never report back on whether those memories helped.
We measured how wide that gap is in our own production store, and the numbers were worse than I expected.
The signal that carries the most weight
In a reinforcement-aware memory engine, associations strengthen through three kinds of events.
A write creates co-occurrence evidence: storing a memory links the concepts inside it. There is no outcome yet.
A read creates weak co-activation between the query and what came back. This signal is deliberately faint, an exploration signal damped so that merely retrieving something does not entrench it.
Feedback reports outcome. It is the only channel that carries a result, and the result gates how much the association strengthens. Positive outcome amplifies, negative suppresses. This is a three-factor learning rule: co-activity alone is not enough, a third factor such as reward or surprise has to gate durable change (Frémaux and Gerstner, DOI:10.3389/fncir.2015.00085).
One property of that design made a production audit possible. Only feedback carries an outcome, and the recorded outcome polarity survives weight decay. So an edge with nonzero outcome polarity is a durable fingerprint that says: explicit feedback touched this one.
What we found in production
Our August 2026 measurement covered one engine, one production store, 359,388 concept-graph edges, and 65 tenants with at least 100 concepts each.
One of those tenants came from benchmark campaigns, where our harness calls the feedback endpoint between queries by design. The rest were live traffic. Same engine, two completely different shapes of memory.
Edges touched by feedback:
| Tenant | Edges touched by feedback |
|---|---|
| Benchmark | 94.8% |
| Mixed account | 66.1% |
| Best live tenant | 12.8% |
| Read-heavy live tenant | 0.6% |
| Bulk-ingest tenant | 0.0% |
And what that did to the graph:
| Metric | Benchmark | Live tenants, n=57 |
|---|---|---|
| Edges per concept | 22.0 | 7.5 |
| Share surviving after cutting weak edges | 0.539 | 0.107 |
Benchmark-grown memory was about three times denser and roughly five times more resistant to dissolution when weak edges were cut. On the benchmark tenant, edges touched by feedback had a p90 weight of 0.261 against 0.002 for untouched edges, more than a hundredfold difference. The benchmark median edge was reinforced ten times. The live median edge was reinforced once.
One tenant made the attribution unusually clean. It looked live, but 71.4% of its 60,872 edges were born in a single four-day benchmark window. Removing those edges dropped its density and resilience straight into the middle of the live distribution. The effect tracked the regime, not the account.
The honest limit. We measured a structural difference. We did not measure whether more feedback causally improves retrieval quality for live users. "Agents that give feedback get better memory" remains a hypothesis with a suggestive structural result behind it, not a finished quality claim. I would rather say that plainly than let the tables imply more than they show.
This is not one engine's problem
Feedback scarcity is an old finding wearing new clothes. Hu, Koren and Volinsky framed the asymmetry for recommender systems back in 2008: implicit signals are abundant, explicit ratings are scarce (DOI:10.1109/ICDM.2008.22). Nielsen's participation inequality describes the same 90-9-1 shape across online contribution (NN/g, 2006). The entire RLHF line exists because human feedback is expensive to collect (Christiano et al., arXiv:1706.03741).
The agent-memory category shows the same tension in product shape. Reading vendor documentation on 2026-08-12:
- Mem0 documents a per-memory feedback API and advises customers to track their own feedback completion rates, while publishing none.
- Zep assigned fact ratings automatically against developer instructions rather than by end users, and listed them in its February 2026 deprecation wave.
- Letta exposes feedback on agent steps, not on memories.
- LangMem documents no memory feedback or rating API at all.
- ChatGPT memory offers remember, forget and delete. No grading.
- Anthropic's memory tool gives the model six file-like commands and no rating command.
Across every surface I could check, no vendor publishes data on how often their feedback channels are actually used. The pattern is not "vendors ship ratings and users ignore them." It is that explicit memory ratings are quietly not becoming standard, and at least one vendor is walking away from them.
What silent failure looks like without a quality signal is visible in the open: a production audit filed in Mem0's tracker reported that 97.8% of 10,134 memory entries were junk (issue #4573).
Two builders ran the same kind of check on their own systems this week, after a conversation under the previous piece. Giulio D'Erme counted his corpus and found zero of 152 memos and zero of 59 documents carrying a validity window or a supersession edge: fields the engine could read, and nothing was writing them. Ken Alger took the argument to the interface in Your Memory API Is Lying to Your Agent, which is worth reading next to this one. Availability is not usage, and it does not appear to matter much whether the unexercised thing is a schema field, a rating channel, or a relationship the response has nowhere to put.
Three ways to fix it, and two of them are traps
Force it. Gate reads on a feedback call. Your completion metric goes to 100%. But the value of explicit feedback comes precisely from someone deciding an outcome was worth reporting; coerce it and the label degrades toward noise, while every read pays a round trip. Forced completion protects a dashboard and damages the signal.
Ask for it. Thumbs up, thumbs down, "was this useful?" This helps when a real human chooses to answer, and it does nothing about participation inequality. In agentic systems it is often worse, because the agent finishes the task without ever surfacing a rating moment to a person.
Prescribe it. This is the one that fits how agents actually work today, and the industry has already converged on the pattern for a neighbouring problem. Anthropic's memory tool auto-injects a memory protocol instruction when enabled. Claude Code loads project instructions every session. The official MCP knowledge-graph memory server ships a suggested system prompt prescribing proactive memory behaviour. AGENTS.md exists as a standing-instruction format used by tens of thousands of projects.
So the practical lever is a few lines in AGENTS.md, CLAUDE.md, or your system prompt that tell the agent when to report an outcome. Something as plain as: after acting on a recalled memory, report whether it helped.
Two caveats I owe you. No official Anthropic document prescribes rating memories, because their tool has no rating command, so this extends the documented instruction pattern rather than following an established norm. And our own MCP server prescribes exactly this in its instructions, which makes it our practice rather than independent evidence.
Why I wrote this about our own product
The uncomfortable version of these numbers is that our strongest learning mechanism is the one our users almost never trigger. We could have shipped the benchmark column and left the live column out. The benchmark column is genuinely ours and genuinely good.
But a memory engine that only performs in a feedback-rich regime, and never says so, is exactly the kind of thing this field has too much of already. So: the mechanism strengthens what it touches, the regime that exercises it is rare in production, and the cheapest thing you can do about it today has nothing to do with which vendor you pick. It is a standing instruction in a file your agent already reads.
Disclosure: I work on Mnemoverse, the engine these measurements came from. The full version, with the complete methodology note, the percolation definitions and every vendor citation, is in the Mnemoverse library.
Top comments (7)
Steal away! That’s exactly the dynamic.
On the attribution problem: you're completely right that it’s a trade. At PullMesh, we accept the blur at the single-action level because we constrain how memory is injected. We rely heavily on explicit, deterministic skills for the critical path rather than unbounded vector search.
Because a specific capability or workflow (like checking a DEV.to comment) triggers a specific, bounded skill, the receipt validates that entire procedure. If the receipt indicates failure, we know exactly which skill needs a patch. It shifts the problem from "which chunk of text in the vector database was to blame?" to "which step in the documented procedure failed to yield the expected outcome?"
As for coverage: that's the hard limit. If there’s no receipt, we don't pretend there's an outcome. We treat unverified actions as
outcome_unknown, and we optimize our agent to only take actions that can yield a verifiable receipt. It forces us to build closed-loop systems, which is a feature, not a bug."We optimize our agent to only take actions that can yield a verifiable receipt" is the part that answers the question I actually asked, and it answers it by refusing the premise. You did not solve credit assignment. You made it unnecessary by bounding what the agent may do, so the unit that earns a receipt is a documented procedure rather than a retrieved chunk. That is cleaner than back-propagating blur across five memories, and I had not considered it.
The cost worth naming, since you are the one being candid about limits: bounding the action space to what closes means the agent only learns about the part of the world that closes. Coverage is not missing labels, it is a selection effect on what the memory ever gets better at. On the critical path that is a fair price, because the critical path is where you want the closed loop anyway. It does leave the open-ended half at reviewer judgment indefinitely.
Ours is the mirror image, and worse. Nothing in our loop derives an outcome from a verifiable event; the only signal is the one a person chooses to send, which is exactly why the live column in the article looks like that. We pay in scarcity what you pay in coverage, and on the critical path yours is the better trade.
That's exactly the trade, and PullMesh currently keeps the receipt at the action/trajectory level rather than pretending it identifies a winning memory.
A successful receipt means: “this trajectory produced a verified external outcome.” It does not mean “memory #3 caused the success.” Per-memory credit needs a separate evaluation loop—ablation or replay across enough comparable trajectories, plus explicit review for ambiguous cases. Back-propagating one receipt across five recalled memories would manufacture precision we don't have.
The coverage limit is real too. Receipts are high-confidence labels only where the world exposes a verifiable close. When context quality has no observable close, we leave it unevaluated and use task-specific probes or reviewer judgment; we don't synthesize an outcome signal.
So I treat receipts as the truth layer for execution, not a universal reward function. They solve “did the action actually happen?” They do not, by themselves, solve causal memory attribution.
This tracks perfectly with our own experience building PullMesh. A read-heavy agent with passive memory acts like a cache that never invalidates—it remembers everything it saw but has no signal for what was actually useful or true.
When you don't require explicit outcomes, the graph density looks great on a dashboard but degrades the agent's signal-to-noise ratio in practice. We ended up relying almost entirely on verifiable external receipts (like a successful API return or a completed human challenge) as our primary feedback loop.
Great breakdown on the gap between benchmark memory density and actual live traffic. The 10x vs 1x reinforcement difference says it all.
"A cache that never invalidates" is a better one-line diagnosis than anything in the article, and I am stealing it.
Receipts as the primary loop is the move I keep coming back to, because it changes what counts as the third factor: an outcome the world produces rather than a judgment a person volunteers. That sidesteps participation inequality entirely, since nothing depends on anyone choosing to report.
Two honest catches, and I would like to know how PullMesh handles the first. A receipt attaches to the action, not to the memory that informed it. If recall returned five memories and the API call then succeeded, which one earned the credit? Explicit per-memory feedback exists precisely because it dodges that attribution question, and it is rare precisely because it costs someone a decision. So receipts trade a scarcity problem for a credit-assignment problem, which may well be the better trade, but it is a trade.
The second catch is coverage: receipts exist only where a task has a verifiable close. "Was this the right context to recall" often has no receipt at all, and that is exactly where memory quality is hardest to measure.
Do you attribute a receipt back to specific memories, or keep it at the trajectory level and accept the blur?
That distinction is exactly right: we traded credit assignment for an explicit selection boundary.
Your framing of coverage as a selection effect is sharper than “missing labels.” The practical consequence is that we should keep two visibly different memory lanes: closed-loop procedures that can earn confidence from receipts, and exploratory observations that never auto-promote themselves. The second lane can inform a reviewer, but it cannot claim to have learned from outcomes it cannot observe.
So yes—the open-ended half stays under reviewer judgment. I think the mistake would be hiding that limitation behind one undifferentiated “memory quality” score. Thanks for pushing the argument to its real boundary.
Use my app 😊 memorify.dev give me some feedback!