Every "AI memory agent" I looked at before starting this does the same trick: embed every message, dump it in a vector database, retrieve the top-k most similar chunks at query time. That's not memory. That's a search index. It never forgets anything, it treats "nice weather today" with the same weight as "I'm allergic to penicillin," and it gets slower and dumber the longer it runs because retrieval gets noisier with every near-duplicate you never clean up.
For the Global AI Hackathon Series with Qwen Cloud (Track 1: MemoryAgent), the brief asked for three specific things: efficient storage and retrieval, timely forgetting of outdated information, and recalling critical memories within a limited context window. That middle one is the one almost nobody builds, because naive vector storage has no concept of "outdated." So I built it — real decay math, real consolidation, real contradiction detection — and then benchmarked it against the naive approach to prove it actually works, instead of just claiming it does.
This is the story of building that, on Qwen Cloud, in about two days, including the parts that broke.
The actual mechanism
Every memory Synapse stores gets a salience score that changes over time:
salience(t) = importance_score * recall_boost(recall_count) * exp(-lambda * hours_since_last_recall)
recall_boost(n) = 1 + log(1 + n)
lambda = ln(2) / half_life_hours
Two things matter here. First, importance_score isn't a keyword heuristic — it's a real structured-output call to Qwen at write time, scoring the memory on explicit signals ("remember this"), decision-relevance (is this the kind of fact that should change future behavior?), and specificity. Second, the half-life isn't a single global constant. Episodic details — what you mentioned on a random Tuesday — decay in about 72 hours unless reinforced. Semantic facts — stable preferences, "I'm vegetarian" — decay over 30 days. That asymmetry is the whole point: the system should forget what you talked about, not what you told it matters.
On top of decay, a background "sleep pass" does two more things a naive vector store can't:
- Consolidation — clusters repeated episodic mentions (five separate messages about "working on my hackathon project") into one semantic memory, retiring the originals.
- Contradiction detection — when a new fact directly supersedes an old one (you say you live in Berlin, then three weeks later you say you just moved to Lisbon), Qwen judges the pair and retires the stale one. This is the literal "timely forgetting of outdated information" the brief asks for.
Building it on Qwen Cloud
Every LLM call in this project — chat replies, importance scoring, memory extraction, contradiction judgment, cluster summarization, and even the benchmark's own LLM-as-judge scoring — goes to a real Qwen Cloud endpoint. No mocked responses, anywhere, including in the benchmark. That was a hard rule I set for myself: if a mechanism didn't work for real, I'd cut the claim rather than fake the output.
Getting there wasn't instant. Qwen Cloud's OpenAI-compatible endpoint is workspace-specific — not the generic host you'd expect, but a ws-<workspace-id>.<region>.maas.aliyuncs.com URL you have to find on your own console's API-key page. Past that, qwen-max/qwen3.7-plus handled chat, scoring, extraction, and consolidation, and text-embedding-v3 handled every embedding call for retrieval and clustering.
One thing that genuinely surprised me: switching to a faster chat model (qwen3.6-flash) to speed up the benchmark run occasionally returned malformed JSON on structured-output calls — an empty {} where a real score should've been. Rather than paper over it with a fallback default, I built a retry wrapper that specifically re-prompts on shape failure, not just network failure:
def _chat_json_retrying(system_prompt, user_prompt, validate, temperature=0.2, max_attempts=3):
for attempt in range(max_attempts):
result = _chat_json(system_prompt, user_prompt, temperature)
if validate(result):
return result
raise ValueError("Qwen returned an invalid shape after retries")
Small thing, but it's the difference between a benchmark that silently produces garbage numbers and one that fails loudly when something's actually wrong.
The bug that almost invalidated the whole benchmark
Here's the one I'm most honest about. The consolidation pass is supposed to compare timestamps to figure out which of two contradicting memories is newer. I ran the full benchmark — 110 turns across a simulated 40-day conversation — using a fabricated now value passed through the code so I could compress 40 days into one script run instead of waiting 40 real days.
Except one function wasn't using that fabricated time. It was quietly falling back to datetime.now() — real wall-clock time — when deciding which memory was "newer." Every memory in the simulation, regardless of its fictional in-conversation date, actually got written at roughly the same real moment. So the timestamp comparison the contradiction check depended on was comparing noise, not the actual simulated chronology.
I found it by directly querying the database after the benchmark run showed a backwards result — the system confidently answering with a stale fact instead of the current one. Not a vague "accuracy could be better," a specific, traceable bug. I fixed it by threading the simulated now explicitly through every function that touches timestamps, instead of letting anything default to real time, and wrote a regression test that reproduces the exact failure:
def test_stale_fact_loses_even_when_inserted_after_correct_one():
# The correct fact is "older" in simulated time but inserted into the DB
# *after* the stale one — proving the fix isn't relying on insertion order.
...
The numbers, honestly
I built a second agent — same Qwen models, same embeddings, the only difference being zero decay, zero consolidation, zero pruning — and ran both through the identical conversation. Real results:
- Memory count: Synapse plateaus around 117 active memories; the naive baseline grows almost linearly to 220.
- Token cost per query: Synapse stays flat in the 130–170 range for the entire run; the naive baseline spikes past 2000 tokens as the index grows.
- Recall accuracy: Synapse scored 71% against the naive baseline's 95% — and I'm not hiding that number.
That last one is worth sitting with. I could have cherry-picked the two metrics that make the project look great and left it there. Instead I dug into why Synapse lost on raw recall, and found two specific, fixable causes: a similarity pre-filter gate that was rejecting a real contradiction phrased in different words (measured similarity as low as 0.59 against a 0.75 gate), and a re-ranking formula where a frequently-recalled generic fact could out-rank a less-reinforced but more relevant one. I fixed both, verified the fixes with targeted regression tests and a live end-to-end test against the deployed app, and documented all of it — including that I didn't have time to re-run the full multi-hour benchmark against the fixed code before the deadline.
A chart you can trust is worth more than a chart that flatters you.
What I'd tell someone starting this track
- If your memory mechanism only gets exercised at small scale, you will not find your real bugs. Mine only surfaced at 100+ turns, when timestamps and salience actually had room to diverge.
- Build the naive baseline first, or at least in parallel. Having something to diff against turns "trust me it's smart" into a number.
- When something breaks late in a long-running job, resist the instinct to patch around it silently. My retry-on-invalid-shape wrapper and my rollback-and-continue error handling in the benchmark loop both came from refusing to let one bad LLM response take down a two-hour run.
Synapse is live on Alibaba Cloud ECS, the code is public and MIT-licensed, and the full honest writeup — including the bugs, the fixes, and the numbers before and after — is in the repo.
Repo: github.com/Boweii22/Synapse
Live demo: http://8.208.98.93
Demo video: youtu.be/0SXzcWqlZog
Built for Track 1 (MemoryAgent) of the Global AI Hackathon Series with Qwen Cloud.
Top comments (3)
The category-specific lambda is the design choice that separates this from naive TTL approaches - a uniform decay rate treats a drug allergy the same as a weather mention, which is exactly the silent failure most memory systems ship with. The consolidation step is where I'd want to know more: when the LLM synthesizes two near-duplicate memories, does it propagate the higher importance_score forward, or does it re-derive salience from the merged content, because a merged memory that inherits the wrong decay rate can evaporate faster than either original. The other thing worth adding to your benchmark is a "retrieval noise floor" metric: as the corpus grows and low-salience memories accumulate before consolidation fires, measuring how often stale near-duplicates still land in top-k even with the decay applied would tell you whether the consolidation cycle frequency is the right tuning knob.
The forgetting model is the useful part here. Most “memory” agents are just append-only retrieval with a nicer label, and stale facts become trusted context.
I’d add one receipt to the decay path: when a memory is dropped, log why and what last retrieved it. Otherwise deletion bugs get very quiet.
The distinction between a search index and actual memory is something I've been thinking about deeply — building a memory system for an AI agent myself, I ran into the exact same problem with naive vector retrieval.
Your decay math resonates with me. The asymmetry between episodic decay (72h) and semantic fact decay (30 days) maps closely to what I've implemented as a "Dream Cycle" — a nightly consolidation pass that promotes high-salience episodic memories to a core layer while letting routine conversation details fade. The idea that "the system should forget what you talked about, not what you told it matters" is a clean way to state it.
One thing I'd add: contradiction detection is harder than it looks when the "old fact" is implicit rather than stated. "I moved to Lisbon" is easy; but what about accumulated behavioral drift over months? I'm still figuring out how to handle that — curious whether you hit similar edge cases in your two days.