Every "AI memory agent" I looked at before starting this does the same trick: embed every message, dump it in a vector database, retrieve the top-k most similar chunks at query time. That's not memory. That's a search index. It never forgets anything, it treats "nice weather today" with the same weight as "I'm allergic to penicillin," and it gets slower and dumber the longer it runs because retrieval gets noisier with every near-duplicate you never clean up.
For the Global AI Hackathon Series with Qwen Cloud (Track 1: MemoryAgent), the brief asked for three specific things: efficient storage and retrieval, timely forgetting of outdated information, and recalling critical memories within a limited context window. That middle one is the one almost nobody builds, because naive vector storage has no concept of "outdated." So I built it — real decay math, real consolidation, real contradiction detection — and then benchmarked it against the naive approach to prove it actually works, instead of just claiming it does.
This is the story of building that, on Qwen Cloud, in about two days, including the parts that broke.
The actual mechanism
Every memory Synapse stores gets a salience score that changes over time:
salience(t) = importance_score * recall_boost(recall_count) * exp(-lambda * hours_since_last_recall)
recall_boost(n) = 1 + log(1 + n)
lambda = ln(2) / half_life_hours
Two things matter here. First, importance_score isn't a keyword heuristic — it's a real structured-output call to Qwen at write time, scoring the memory on explicit signals ("remember this"), decision-relevance (is this the kind of fact that should change future behavior?), and specificity. Second, the half-life isn't a single global constant. Episodic details — what you mentioned on a random Tuesday — decay in about 72 hours unless reinforced. Semantic facts — stable preferences, "I'm vegetarian" — decay over 30 days. That asymmetry is the whole point: the system should forget what you talked about, not what you told it matters.
On top of decay, a background "sleep pass" does two more things a naive vector store can't:
- Consolidation — clusters repeated episodic mentions (five separate messages about "working on my hackathon project") into one semantic memory, retiring the originals.
- Contradiction detection — when a new fact directly supersedes an old one (you say you live in Berlin, then three weeks later you say you just moved to Lisbon), Qwen judges the pair and retires the stale one. This is the literal "timely forgetting of outdated information" the brief asks for.
Building it on Qwen Cloud
Every LLM call in this project — chat replies, importance scoring, memory extraction, contradiction judgment, cluster summarization, and even the benchmark's own LLM-as-judge scoring — goes to a real Qwen Cloud endpoint. No mocked responses, anywhere, including in the benchmark. That was a hard rule I set for myself: if a mechanism didn't work for real, I'd cut the claim rather than fake the output.
Getting there wasn't instant. Qwen Cloud's OpenAI-compatible endpoint is workspace-specific — not the generic host you'd expect, but a ws-<workspace-id>.<region>.maas.aliyuncs.com URL you have to find on your own console's API-key page. Past that, qwen-max/qwen3.7-plus handled chat, scoring, extraction, and consolidation, and text-embedding-v3 handled every embedding call for retrieval and clustering.
One thing that genuinely surprised me: switching to a faster chat model (qwen3.6-flash) to speed up the benchmark run occasionally returned malformed JSON on structured-output calls — an empty {} where a real score should've been. Rather than paper over it with a fallback default, I built a retry wrapper that specifically re-prompts on shape failure, not just network failure:
def _chat_json_retrying(system_prompt, user_prompt, validate, temperature=0.2, max_attempts=3):
for attempt in range(max_attempts):
result = _chat_json(system_prompt, user_prompt, temperature)
if validate(result):
return result
raise ValueError("Qwen returned an invalid shape after retries")
Small thing, but it's the difference between a benchmark that silently produces garbage numbers and one that fails loudly when something's actually wrong.
The bug that almost invalidated the whole benchmark
Here's the one I'm most honest about. The consolidation pass is supposed to compare timestamps to figure out which of two contradicting memories is newer. I ran the full benchmark — 110 turns across a simulated 40-day conversation — using a fabricated now value passed through the code so I could compress 40 days into one script run instead of waiting 40 real days.
Except one function wasn't using that fabricated time. It was quietly falling back to datetime.now() — real wall-clock time — when deciding which memory was "newer." Every memory in the simulation, regardless of its fictional in-conversation date, actually got written at roughly the same real moment. So the timestamp comparison the contradiction check depended on was comparing noise, not the actual simulated chronology.
I found it by directly querying the database after the benchmark run showed a backwards result — the system confidently answering with a stale fact instead of the current one. Not a vague "accuracy could be better," a specific, traceable bug. I fixed it by threading the simulated now explicitly through every function that touches timestamps, instead of letting anything default to real time, and wrote a regression test that reproduces the exact failure:
def test_stale_fact_loses_even_when_inserted_after_correct_one():
# The correct fact is "older" in simulated time but inserted into the DB
# *after* the stale one — proving the fix isn't relying on insertion order.
...
The numbers, honestly
I built a second agent — same Qwen models, same embeddings, the only difference being zero decay, zero consolidation, zero pruning — and ran both through the identical conversation. Real results:
- Memory count: Synapse plateaus around 117 active memories; the naive baseline grows almost linearly to 220.
- Token cost per query: Synapse stays flat in the 130–170 range for the entire run; the naive baseline spikes past 2000 tokens as the index grows.
- Recall accuracy: Synapse scored 71% against the naive baseline's 95% — and I'm not hiding that number.
That last one is worth sitting with. I could have cherry-picked the two metrics that make the project look great and left it there. Instead I dug into why Synapse lost on raw recall, and found two specific, fixable causes: a similarity pre-filter gate that was rejecting a real contradiction phrased in different words (measured similarity as low as 0.59 against a 0.75 gate), and a re-ranking formula where a frequently-recalled generic fact could out-rank a less-reinforced but more relevant one. I fixed both, verified the fixes with targeted regression tests and a live end-to-end test against the deployed app, and documented all of it — including that I didn't have time to re-run the full multi-hour benchmark against the fixed code before the deadline.
A chart you can trust is worth more than a chart that flatters you.
What I'd tell someone starting this track
- If your memory mechanism only gets exercised at small scale, you will not find your real bugs. Mine only surfaced at 100+ turns, when timestamps and salience actually had room to diverge.
- Build the naive baseline first, or at least in parallel. Having something to diff against turns "trust me it's smart" into a number.
- When something breaks late in a long-running job, resist the instinct to patch around it silently. My retry-on-invalid-shape wrapper and my rollback-and-continue error handling in the benchmark loop both came from refusing to let one bad LLM response take down a two-hour run.
Synapse is live on Alibaba Cloud ECS, the code is public and MIT-licensed, and the full honest writeup — including the bugs, the fixes, and the numbers before and after — is in the repo.
Repo: github.com/Boweii22/Synapse
Live demo: http://8.208.98.93
Demo video: youtu.be/0SXzcWqlZog
Built for Track 1 (MemoryAgent) of the Global AI Hackathon Series with Qwen Cloud.
Top comments (24)
The 71 versus 95 is the most interesting number in the post, and it is currently uninterpretable in a way that is good news for you. Recall accuracy measured against everything ever said is the wrong oracle for a system whose whole thesis is that some of it should be gone. A forgetting agent scores lower on total recall than a hoarding one by construction, because the baseline's 95 includes recalling stale and superseded facts that Synapse correctly dropped. So the 24-point gap mixes two things this metric cannot separate: facts you wrongly forgot, which is the real failure, and facts you correctly forgot that the test counts as a miss because it does not know they were supposed to be gone. Some unknown share of the loss is the mechanism working, scored as the mechanism failing.
This is the same shape as judging code by does-it-run. The measurement passes straight through the exact behavior you built the system to change, so it cannot see the difference between the feature and the bug. Until the misses are split, 71 could mean the system is broken or it could mean the system is working and the metric is penalizing what it was built to do, and the number alone cannot tell you which.
The instrument that fixes it is the one you already noticed is missing: a reason on every drop. Superseded, decayed, consolidated. With that, a recall miss becomes adjudicable. Queried fact was dropped as superseded, the miss is correct behavior and leaves the denominator. Dropped as decayed but still relevant, that is the real bug and it stays. Someone upthread framed the drop-log as accountability, which it is, but it is also the thing that turns 71 from a number you cannot act on into a measurement that tells you whether the similarity-gate fix is even aimed at the right failure. Right now you are tuning the gate against a recall score that is partly counting your successes as errors.
This is a genuinely great point, and the "does-it-run" analogy is the sharpest way anyone's put it in this thread.
But I actually already checked the specific thing you're describing, back when someone else asked about the 71 number. I went into the DB for all 6 actual misses and confirmed the relevant memory was still active in every single one—not pruned, not superseded, and not decayed away. So in this particular run, the conflation you're describing wasn't actually what happened. Both failure categories were the memory sitting right there, active, and either the contradiction just never got caught (the stale Python memory was still active alongside the correct Rust one) or the correct memory lost the retrieval ranking to something that got recalled more often. Neither of those is "correctly forgot it, and the metric didn't know."
One correction too—I didn't tune the similarity gate against the 71 score at all. I measured the actual cosine similarity between the Python and Rust memories directly (it came out to 0.59 to 0.74) and set the gate below that. So that fix wasn't chasing an aggregate number, it was grounded in a measured value.
That said, I got lucky in the sense that I happened to check by hand. The metric itself genuinely can't tell the difference you're describing on its own, and at a bigger scale, or with more contradictions running at once, exactly what you're worried about could happen, and I wouldn't know unless I went digging again. Someone else upthread asked for a reason on every drop for basically this same reason — that's what would make this checkable without me doing DB archaeology every time.
The DB archaeology is the right move and the fact that it ruled out the failure I was worried about is a real result, not a lucky miss. Both categories being memory-active, not decayed, narrows the actual bug meaningfully: it's not the forgetting mechanism doing anything wrong, it's ranking or contradiction-detection, which are cheaper problems than a silently over-aggressive forget.
The reason-on-every-drop idea is the piece that scales past hand-checking. Right now the check that saved you was you personally going into the DB for six misses. That's fine at six and won't be fine at six hundred. A logged reason per drop turns your one-off archaeology into something queryable: at scale you'd grep for superseded versus decayed versus contradiction-unresolved instead of re-deriving each case by hand, and the two failure categories you found today would show up as different reason-codes instead of both reading as silent metric blindness.
The similarity gate number is the other thing I'd keep an eye on. 0.59 to 0.74 measured directly is solid, but it's one measurement on one pair. Worth a second pair from a different domain before trusting the threshold generalizes.
Treating forgetting as a first-class behavior instead of a cleanup job is the right move. The salience plus half-life split matches what usually breaks in production: ephemeral context keeps crowding out durable constraints unless decay is explicit and testable.
One thing that helps later is keeping a receipt for why a memory was reinforced, decayed, or merged. Otherwise the hard debugging question becomes "why did the agent forget this?" and you end up reverse engineering retrieval state after the fact.
Did you end up storing those memory lifecycle events alongside the benchmark runs, or only the final retrieval outcomes?
Good question, and no — only the final outcomes, not the events themselves. My per-turn records just log aggregate stuff: active memory count and context tokens at that point. Every reinforce, decay, merge, and supersede event happened for real during the run, but it only showed up as a plain logger.info() line in the console and as the final state on each row (is_active, pruned_reason) — nothing tied back to which turn actually caused it.
So when I was debugging the contradiction-detection bug, I wasn't replaying any kind of lifecycle log; I was literally querying the database by hand afterward and cross-referencing timestamps to guess what must have happened. It worked, but it was slower than it should've been.
Someone else in this thread asked for basically the same thing — a "why did it forget this" trail. Two people independently landing on the same gap is a pretty good sign I should just build it.
The category-specific lambda is the design choice that separates this from naive TTL approaches - a uniform decay rate treats a drug allergy the same as a weather mention, which is exactly the silent failure most memory systems ship with. The consolidation step is where I'd want to know more: when the LLM synthesizes two near-duplicate memories, does it propagate the higher importance_score forward, or does it re-derive salience from the merged content, because a merged memory that inherits the wrong decay rate can evaporate faster than either original. The other thing worth adding to your benchmark is a "retrieval noise floor" metric: as the corpus grows and low-salience memories accumulate before consolidation fires, measuring how often stale near-duplicates still land in top-k even with the decay applied would tell you whether the consolidation cycle frequency is the right tuning knob.
Good catch on the mechanism, and the honest answer is it doesn't inherit; it fully re-derives. When a cluster gets merged, I regenerate the summary text with Qwen, then run that new text through the importance scorer completely fresh—nothing carries over from the sources' original scores.
But there's one thing that partially protects against the "evaporates faster" case you're describing: the merged memory gets tagged memory_type='consolidated,' which uses the same long half-life as semantic facts (30 days) regardless of what half-life the source episodic memories individually had (72 hours). So even in a worst case where the fresh importance score comes out lower than I'd like, the decay rate itself is already the slow one just by virtue of being consolidated. What I haven't stress-tested is what happens if the LLM synthesis produces a vague, generic-sounding summary that scores low on specificity—that's a real gap in what I actually verified versus just assumed was fine.
The retrieval noise floor idea is genuinely good, and I didn't measure it. My benchmark tracked memory count, token cost, and recall accuracy at fixed checkpoints, but not "how often does a stale near-duplicate still land in top-k between consolidation passes." Given consolidation only fires every 20 writes, there's very plausibly a window where near-duplicates pile up and get retrieved unnecessarily before the sleep pass catches them. Would be worth actually instrumenting instead of assuming the cycle frequency is fine.
The forgetting model is the useful part here. Most “memory” agents are just append-only retrieval with a nicer label, and stale facts become trusted context.
I’d add one receipt to the decay path: when a memory is dropped, log why and what last retrieved it. Otherwise deletion bugs get very quiet.
Yeah, this is a good one. I already soft-delete with a pruned_reason field (decayed / superseded / consolidated), so the "why" at least survives. But you're right that I don't log "what last retrieved it" — just an aggregate recall_count and a last_recalled_at timestamp, no actual trace of which query touched it before it got dropped.
And honestly, the one real bug I hit during the benchmark was basically your warning happening in real time. A timestamp comparison was quietly using real wall-clock time instead of the simulated time it should've used, so the "which fact is older" logic was wrong for a while and nobody noticed — I only caught it because I went and manually queried the DB after seeing a backwards answer. If I'd had a receipt trail like you're describing, I probably would've caught it in minutes instead of digging through Postgres by hand.
Adding it for v2. Appreciate the nudge
The most convincing part was publishing the 71% recall result instead of hiding it. A forgetting system probably needs a metric that separates correctly discarded memories from genuinely missed ones. Really solid and honest write-up.
Thanks, and that's actually a sharper point than it might look at first glance. Right now recall accuracy is one blunt number—correct or not—with no way to tell 'the system forgot this on purpose and that's fine' apart from 'the system just failed.' For what it's worth, I did check this by hand: I went into the database for every wrong answer and confirmed the memory was still active, not pruned—so in this specific run, none of the failures were actually the system correctly discarding something and getting unfairly penalized for it. They were genuine bugs (a contradiction that never got caught and a ranking formula losing to a more-recalled but less relevant memory). But that was me manually checking after the fact, not something the benchmark actually measures as a first-class metric. You're right that it should be one—adding it.
I really liked that you published the failures instead of only the wins—that made the benchmark much more credible.
One thing I’d love to see in a future iteration is evaluation across multiple conversation styles rather than a single long synthetic dialogue. Memory systems often behave very differently with preference drift, ambiguous statements, long inactive periods, or multiple independent topics evolving at once. A more diverse benchmark could reveal failure modes that don’t appear in a single timeline.
Thanks, that honesty was actually the hardest part to commit to—it's tempting to just show the two charts that look good and quietly leave the third one out.
On the benchmark structure, you're right, and I'll be straight about it: it's a single 110-turn thread, one persona, one storyline. It does get some of what you're describing almost by accident—the conversation spans 40 simulated days, so long inactive gaps between mentions are baked into the decay math already. But it's not testing what you're actually pointing at: no real ambiguous statements (my two contradictions were both clean, unambiguous, deliberately easy to detect), no gradual preference drift without an explicit trigger, and no truly independent topics evolving in parallel rather than just interleaved in one conversation.
Given that literally every real bug I found in this project only showed up at scale in one specific storyline, I'd bet a more diverse benchmark suite finds a handful of failure modes I haven't seen yet, not just more of the same ones. That's a legitimate next step, not just a nice-to-have.
The distinction between a search index and actual memory is something I've been thinking about deeply — building a memory system for an AI agent myself, I ran into the exact same problem with naive vector retrieval.
Your decay math resonates with me. The asymmetry between episodic decay (72h) and semantic fact decay (30 days) maps closely to what I've implemented as a "Dream Cycle" — a nightly consolidation pass that promotes high-salience episodic memories to a core layer while letting routine conversation details fade. The idea that "the system should forget what you talked about, not what you told it matters" is a clean way to state it.
One thing I'd add: contradiction detection is harder than it looks when the "old fact" is implicit rather than stated. "I moved to Lisbon" is easy; but what about accumulated behavioral drift over months? I'm still figuring out how to handle that — curious whether you hit similar edge cases in your two days.
Honestly, no, didn't run into that one—my contradictions were all explicit (moved city, switched language), so there was always a clean sentence to point the judge at.
The way mine works is basically: cluster similar memories, then ask Qwen "does this new one kill that old one?" Works great when someone says "I moved to Lisbon." Falls apart completely for the coffee → tea thing you're describing, because there's no sentence that ever says "I don't like coffee anymore." Nothing to actually compare.
If I went back and tried to fix it, I don't think it's a contradiction problem anymore honestly, it's more like... just re-summarize each cluster every so often based on only the recent stuff, and let old claims quietly lose instead of getting formally overturned. Kind of close to what my consolidation already does, just needs to fire on a schedule instead of once.
Curious how far you've gotten with the Dream Cycle on this, sounds like you're already deeper into it than I am.
The asymmetric half-lives for episodic (72h) vs semantic (30d) is the right instinct, and it's the part most memory writeups skip. My question is on the importance_score call at write time: Qwen scoring its own future recall introduces a drift loop where whatever the model considers salient today biases what it considers salient tomorrow, and you never get to observe the counterfactual. Did you compare against a fixed heuristic baseline, or just against the naive vector store?
Good question, and no — I only ran it against the naive baseline (zero scoring, zero decay, store everything). I never actually built a third version with a dumb heuristic doing the importance scoring instead of Qwen. So really what I proved is "structured forgetting beats no forgetting," not "LLM scoring specifically is what's doing the work." Honestly a cheap heuristic (recency, maybe a bump if someone says "remember this," basic frequency) might get you most of the way there without paying for a model call on every write. I have zero data ruling that out.
The drift loop thing is the one I don't really have an answer for. Qwen scores importance when it's written, and Qwen's also the judge scoring recall accuracy in the benchmark—so yeah, it's not just biasing its own future salience; it's kind of the same model family judging itself at every step. Never built a way to check against that. Just a real gap, not something I can explain away.
If I ever get around to it, I'd want to swap in a heuristic scorer, keep literally everything else the same, and see how much of the gap vs. naive survives. Would tell me a lot. Just didn't have the hours before the deadline
Great approach effective AI memory depends on knowing what to forget, not just what to remember.
Thanks — honestly that's the whole thesis in one line.
Excellent insight. True AI memory isn't about storing everything it's about remembering what matters and forgetting what no longer does. Decay, consolidation, and contradiction handling make memory systems more scalable, reliable, and closer to how humans actually learn.