Every "AI memory agent" I looked at before starting this does the same trick: embed every message, dump it in a vector database, retrieve the top-k...
For further actions, you may consider blocking this person and/or reporting abuse
The 71 versus 95 is the most interesting number in the post, and it is currently uninterpretable in a way that is good news for you. Recall accuracy measured against everything ever said is the wrong oracle for a system whose whole thesis is that some of it should be gone. A forgetting agent scores lower on total recall than a hoarding one by construction, because the baseline's 95 includes recalling stale and superseded facts that Synapse correctly dropped. So the 24-point gap mixes two things this metric cannot separate: facts you wrongly forgot, which is the real failure, and facts you correctly forgot that the test counts as a miss because it does not know they were supposed to be gone. Some unknown share of the loss is the mechanism working, scored as the mechanism failing.
This is the same shape as judging code by does-it-run. The measurement passes straight through the exact behavior you built the system to change, so it cannot see the difference between the feature and the bug. Until the misses are split, 71 could mean the system is broken or it could mean the system is working and the metric is penalizing what it was built to do, and the number alone cannot tell you which.
The instrument that fixes it is the one you already noticed is missing: a reason on every drop. Superseded, decayed, consolidated. With that, a recall miss becomes adjudicable. Queried fact was dropped as superseded, the miss is correct behavior and leaves the denominator. Dropped as decayed but still relevant, that is the real bug and it stays. Someone upthread framed the drop-log as accountability, which it is, but it is also the thing that turns 71 from a number you cannot act on into a measurement that tells you whether the similarity-gate fix is even aimed at the right failure. Right now you are tuning the gate against a recall score that is partly counting your successes as errors.
This is a genuinely great point, and the "does-it-run" analogy is the sharpest way anyone's put it in this thread.
But I actually already checked the specific thing you're describing, back when someone else asked about the 71 number. I went into the DB for all 6 actual misses and confirmed the relevant memory was still active in every single one—not pruned, not superseded, and not decayed away. So in this particular run, the conflation you're describing wasn't actually what happened. Both failure categories were the memory sitting right there, active, and either the contradiction just never got caught (the stale Python memory was still active alongside the correct Rust one) or the correct memory lost the retrieval ranking to something that got recalled more often. Neither of those is "correctly forgot it, and the metric didn't know."
One correction too—I didn't tune the similarity gate against the 71 score at all. I measured the actual cosine similarity between the Python and Rust memories directly (it came out to 0.59 to 0.74) and set the gate below that. So that fix wasn't chasing an aggregate number, it was grounded in a measured value.
That said, I got lucky in the sense that I happened to check by hand. The metric itself genuinely can't tell the difference you're describing on its own, and at a bigger scale, or with more contradictions running at once, exactly what you're worried about could happen, and I wouldn't know unless I went digging again. Someone else upthread asked for a reason on every drop for basically this same reason — that's what would make this checkable without me doing DB archaeology every time.
The DB archaeology is the right move and the fact that it ruled out the failure I was worried about is a real result, not a lucky miss. Both categories being memory-active, not decayed, narrows the actual bug meaningfully: it's not the forgetting mechanism doing anything wrong, it's ranking or contradiction-detection, which are cheaper problems than a silently over-aggressive forget.
The reason-on-every-drop idea is the piece that scales past hand-checking. Right now the check that saved you was you personally going into the DB for six misses. That's fine at six and won't be fine at six hundred. A logged reason per drop turns your one-off archaeology into something queryable: at scale you'd grep for superseded versus decayed versus contradiction-unresolved instead of re-deriving each case by hand, and the two failure categories you found today would show up as different reason-codes instead of both reading as silent metric blindness.
The similarity gate number is the other thing I'd keep an eye on. 0.59 to 0.74 measured directly is solid, but it's one measurement on one pair. Worth a second pair from a different domain before trusting the threshold generalizes.
Treating forgetting as a first-class behavior instead of a cleanup job is the right move. The salience plus half-life split matches what usually breaks in production: ephemeral context keeps crowding out durable constraints unless decay is explicit and testable.
One thing that helps later is keeping a receipt for why a memory was reinforced, decayed, or merged. Otherwise the hard debugging question becomes "why did the agent forget this?" and you end up reverse engineering retrieval state after the fact.
Did you end up storing those memory lifecycle events alongside the benchmark runs, or only the final retrieval outcomes?
Good question, and no — only the final outcomes, not the events themselves. My per-turn records just log aggregate stuff: active memory count and context tokens at that point. Every reinforce, decay, merge, and supersede event happened for real during the run, but it only showed up as a plain logger.info() line in the console and as the final state on each row (is_active, pruned_reason) — nothing tied back to which turn actually caused it.
So when I was debugging the contradiction-detection bug, I wasn't replaying any kind of lifecycle log; I was literally querying the database by hand afterward and cross-referencing timestamps to guess what must have happened. It worked, but it was slower than it should've been.
Someone else in this thread asked for basically the same thing — a "why did it forget this" trail. Two people independently landing on the same gap is a pretty good sign I should just build it.
The category-specific lambda is the design choice that separates this from naive TTL approaches - a uniform decay rate treats a drug allergy the same as a weather mention, which is exactly the silent failure most memory systems ship with. The consolidation step is where I'd want to know more: when the LLM synthesizes two near-duplicate memories, does it propagate the higher importance_score forward, or does it re-derive salience from the merged content, because a merged memory that inherits the wrong decay rate can evaporate faster than either original. The other thing worth adding to your benchmark is a "retrieval noise floor" metric: as the corpus grows and low-salience memories accumulate before consolidation fires, measuring how often stale near-duplicates still land in top-k even with the decay applied would tell you whether the consolidation cycle frequency is the right tuning knob.
Good catch on the mechanism, and the honest answer is it doesn't inherit; it fully re-derives. When a cluster gets merged, I regenerate the summary text with Qwen, then run that new text through the importance scorer completely fresh—nothing carries over from the sources' original scores.
But there's one thing that partially protects against the "evaporates faster" case you're describing: the merged memory gets tagged memory_type='consolidated,' which uses the same long half-life as semantic facts (30 days) regardless of what half-life the source episodic memories individually had (72 hours). So even in a worst case where the fresh importance score comes out lower than I'd like, the decay rate itself is already the slow one just by virtue of being consolidated. What I haven't stress-tested is what happens if the LLM synthesis produces a vague, generic-sounding summary that scores low on specificity—that's a real gap in what I actually verified versus just assumed was fine.
The retrieval noise floor idea is genuinely good, and I didn't measure it. My benchmark tracked memory count, token cost, and recall accuracy at fixed checkpoints, but not "how often does a stale near-duplicate still land in top-k between consolidation passes." Given consolidation only fires every 20 writes, there's very plausibly a window where near-duplicates pile up and get retrieved unnecessarily before the sleep pass catches them. Would be worth actually instrumenting instead of assuming the cycle frequency is fine.
The forgetting model is the useful part here. Most “memory” agents are just append-only retrieval with a nicer label, and stale facts become trusted context.
I’d add one receipt to the decay path: when a memory is dropped, log why and what last retrieved it. Otherwise deletion bugs get very quiet.
Yeah, this is a good one. I already soft-delete with a pruned_reason field (decayed / superseded / consolidated), so the "why" at least survives. But you're right that I don't log "what last retrieved it" — just an aggregate recall_count and a last_recalled_at timestamp, no actual trace of which query touched it before it got dropped.
And honestly, the one real bug I hit during the benchmark was basically your warning happening in real time. A timestamp comparison was quietly using real wall-clock time instead of the simulated time it should've used, so the "which fact is older" logic was wrong for a while and nobody noticed — I only caught it because I went and manually queried the DB after seeing a backwards answer. If I'd had a receipt trail like you're describing, I probably would've caught it in minutes instead of digging through Postgres by hand.
Adding it for v2. Appreciate the nudge
The most convincing part was publishing the 71% recall result instead of hiding it. A forgetting system probably needs a metric that separates correctly discarded memories from genuinely missed ones. Really solid and honest write-up.
Thanks, and that's actually a sharper point than it might look at first glance. Right now recall accuracy is one blunt number—correct or not—with no way to tell 'the system forgot this on purpose and that's fine' apart from 'the system just failed.' For what it's worth, I did check this by hand: I went into the database for every wrong answer and confirmed the memory was still active, not pruned—so in this specific run, none of the failures were actually the system correctly discarding something and getting unfairly penalized for it. They were genuine bugs (a contradiction that never got caught and a ranking formula losing to a more-recalled but less relevant memory). But that was me manually checking after the fact, not something the benchmark actually measures as a first-class metric. You're right that it should be one—adding it.
I really liked that you published the failures instead of only the wins—that made the benchmark much more credible.
One thing I’d love to see in a future iteration is evaluation across multiple conversation styles rather than a single long synthetic dialogue. Memory systems often behave very differently with preference drift, ambiguous statements, long inactive periods, or multiple independent topics evolving at once. A more diverse benchmark could reveal failure modes that don’t appear in a single timeline.
Thanks, that honesty was actually the hardest part to commit to—it's tempting to just show the two charts that look good and quietly leave the third one out.
On the benchmark structure, you're right, and I'll be straight about it: it's a single 110-turn thread, one persona, one storyline. It does get some of what you're describing almost by accident—the conversation spans 40 simulated days, so long inactive gaps between mentions are baked into the decay math already. But it's not testing what you're actually pointing at: no real ambiguous statements (my two contradictions were both clean, unambiguous, deliberately easy to detect), no gradual preference drift without an explicit trigger, and no truly independent topics evolving in parallel rather than just interleaved in one conversation.
Given that literally every real bug I found in this project only showed up at scale in one specific storyline, I'd bet a more diverse benchmark suite finds a handful of failure modes I haven't seen yet, not just more of the same ones. That's a legitimate next step, not just a nice-to-have.
The distinction between a search index and actual memory is something I've been thinking about deeply — building a memory system for an AI agent myself, I ran into the exact same problem with naive vector retrieval.
Your decay math resonates with me. The asymmetry between episodic decay (72h) and semantic fact decay (30 days) maps closely to what I've implemented as a "Dream Cycle" — a nightly consolidation pass that promotes high-salience episodic memories to a core layer while letting routine conversation details fade. The idea that "the system should forget what you talked about, not what you told it matters" is a clean way to state it.
One thing I'd add: contradiction detection is harder than it looks when the "old fact" is implicit rather than stated. "I moved to Lisbon" is easy; but what about accumulated behavioral drift over months? I'm still figuring out how to handle that — curious whether you hit similar edge cases in your two days.
Honestly, no, didn't run into that one—my contradictions were all explicit (moved city, switched language), so there was always a clean sentence to point the judge at.
The way mine works is basically: cluster similar memories, then ask Qwen "does this new one kill that old one?" Works great when someone says "I moved to Lisbon." Falls apart completely for the coffee → tea thing you're describing, because there's no sentence that ever says "I don't like coffee anymore." Nothing to actually compare.
If I went back and tried to fix it, I don't think it's a contradiction problem anymore honestly, it's more like... just re-summarize each cluster every so often based on only the recent stuff, and let old claims quietly lose instead of getting formally overturned. Kind of close to what my consolidation already does, just needs to fire on a schedule instead of once.
Curious how far you've gotten with the Dream Cycle on this, sounds like you're already deeper into it than I am.
The asymmetric half-lives for episodic (72h) vs semantic (30d) is the right instinct, and it's the part most memory writeups skip. My question is on the importance_score call at write time: Qwen scoring its own future recall introduces a drift loop where whatever the model considers salient today biases what it considers salient tomorrow, and you never get to observe the counterfactual. Did you compare against a fixed heuristic baseline, or just against the naive vector store?
Good question, and no — I only ran it against the naive baseline (zero scoring, zero decay, store everything). I never actually built a third version with a dumb heuristic doing the importance scoring instead of Qwen. So really what I proved is "structured forgetting beats no forgetting," not "LLM scoring specifically is what's doing the work." Honestly a cheap heuristic (recency, maybe a bump if someone says "remember this," basic frequency) might get you most of the way there without paying for a model call on every write. I have zero data ruling that out.
The drift loop thing is the one I don't really have an answer for. Qwen scores importance when it's written, and Qwen's also the judge scoring recall accuracy in the benchmark—so yeah, it's not just biasing its own future salience; it's kind of the same model family judging itself at every step. Never built a way to check against that. Just a real gap, not something I can explain away.
If I ever get around to it, I'd want to swap in a heuristic scorer, keep literally everything else the same, and see how much of the gap vs. naive survives. Would tell me a lot. Just didn't have the hours before the deadline
Great approach effective AI memory depends on knowing what to forget, not just what to remember.
Thanks — honestly that's the whole thesis in one line.
Excellent insight. True AI memory isn't about storing everything it's about remembering what matters and forgetting what no longer does. Decay, consolidation, and contradiction handling make memory systems more scalable, reliable, and closer to how humans actually learn.
Coming at this from a very different angle — I build small web tools for
clients in Indonesia, mostly order systems and dashboards, nothing with an
agent in it. But the cost chart is the part that would sell this to my clients,
not the recall number.
When you're billing a small business a fixed price, a token cost that grows
linearly with usage is the thing that quietly eats the margin six months after
handover. Flat 130-170 versus climbing past 2000 is a maintenance story, not
just a benchmark.
Naive question, since I'm outside this space: does the decay approach need the
LLM scoring calls to work, or could a much dumber importance heuristic still
get most of the flattening? Wondering whether the idea survives being ported
to a project with no AI budget.
One aspect that's often overlooked is that memory quality matters more than memory size. As long-running AI agents become more common, selective retention, conflict resolution, and memory freshness will likely have a greater impact on reliability than simply storing every interaction. Well-managed memory is what keeps context useful over time.
'it gets slower and dumber the longer it runs because retrieval gets noisier with every near duplicate you never clean up' is the one that kills most RAG systems in production too.
we ran into this building a long session agent — by week 3 the retrieval was returning contradictory facts from different conversation windows because nothing had been consolidated. adding a nightly dedup pass cut context noise by ~40% but it still felt like duct tape.
how are you handling contradictions where the newer memory is almost certainly right but the older one is higher confidence?
I like that you measured this against a simple baseline instead of assuming more memory is always better. I've hit the same problem where old context starts getting in the way more than it helps.