Building a drift-adaptive memory for a Qwen agent, and the one debugging session that turned a tie into a win.
I built a memory sidecar for an LLM agent. The pitch was simple and, I thought, a little clever: decide what to remember with no LLM call at all. A small neural memory (Titans-style, trained at inference) produces a "surprise" score for every incoming fact. Novel things are surprising, so you store them. Repeats are not, so you skip them. Zero tokens on the write path, while systems like Mem0 and Zep pay for an LLM call every single turn.
The demo looked great. Then I tried to prove it actually worked, and it fell apart in a way that taught me the real problem.
The tie
To test whether adaptive forgetting helps, I wrote a benchmark: procedurally generate multi-session conversations where the user's facts drift over time. They move from Ames to Chicago to Seattle. They switch from tea to coffee to matcha. Each policy sees the same stream and the same retrieval, so the only variable is the write-and-forget logic. Then I measure two things: does it still recall the current fact, and does it drag stale facts back into retrieval?
I ran it expecting my clever neural gate to win. It tied a five-line cosine-similarity baseline. On a couple of runs it lost.
This is the moment that decides whether a project is real. You either explain away the number or you go find out why. I went to find out why, and the answer was two separate bugs sitting on top of each other.
Bug one: the pre-filter was hiding the evidence
The forgetting logic worked like this: when a new fact arrives, find old facts it might contradict (cheap cosine similarity as a first pass), then run a tiny local NLI model to decide if it really is a contradiction. If it is, decay the old fact.
I printed the actual NLI scores. They were near perfect. "I moved from Ames to Chicago" contradicts "I live in Ames" with 0.99 confidence. "My dog loves the apartment" contradicts nothing, 0.00. The model was not the problem.
The cosine pre-filter was. I had set it to 0.75, thinking same-topic facts would be very similar. But real updates are not. "I switched to coffee" and "these days I only drink matcha" sit at 0.63 cosine. The pre-filter was throwing them out before the NLI model ever saw them, so the stale coffee fact just survived forever. The fix was to loosen the pre-filter and let the discriminative signal do its job. Stale contamination dropped by almost everything at once.
And then current recall collapsed.
Bug two: surprise cannot tell a repeat from a correction
Here is the part I did not see coming, and it is the actual idea in this whole project.
With forgetting fixed, the memory was now dropping facts it needed. I traced it and found the culprit was the admission gate, the thing I was proudest of. Watch what happened to a knowledge update:
SKIP surprise=0.38 threshold=0.40 "I moved from Ames to Chicago."
The surprise gate rejected it. Of course it did. "I moved to Chicago" is about location, and the memory already knows a lot about location. It is a familiar topic, so it scores low surprise, so the gate reads it as a redundant repeat and throws it away.
That is the whole problem in one line. Surprise measures novelty, and a correction is not novel. It looks exactly like a duplicate. A fixed threshold drops it. An adaptive threshold drops it too, because you cannot separate "I live in Ames" said twice from "I live in Ames" then "I moved to Chicago" by surprise magnitude alone. Both are low surprise. One is noise and one is the single most important thing the user has told you all week.
So what separates them? The same contradiction signal I was already using for forgetting. A duplicate is low surprise and consistent. A correction is low surprise and contradicting. I made admission read both signals: store a fact if it is novel, OR if it contradicts something you already believe. That one change means the contradiction signal does double duty. It admits the correction and it forgets the stale fact it corrects, in a single write.
It also explains, cleanly, why every static density gate in the literature fails on updates. They see a correction as a near-duplicate and drop it. They are structurally blind to the exact case memory exists to handle.
The result
Over 400 held-out drifting streams (a different random seed range from the one I used to tune the single threshold, so I am not grading my own homework), the finished controller beats the published state-of-the-art write gate by 0.35 on clean recall and cuts stale contamination by about nine times, at zero LLM tokens per write. Every confidence interval on the gap excludes zero.
The part I have to be honest about
I also ran it on LongMemEval, a real published benchmark, on the knowledge-update questions. In the oracle setting there, forgetting hurts. Store-everything scores 0.69 and my forgetting scores 0.33.
That result is real and I am not going to bury it, but it makes sense once you see why. The oracle setting hands the answer model every relevant turn of history. When the model can already read both "my old best was 27:00" and "new best, 25:50," it just picks the newer one. Forgetting the old value only takes away context it could have used. Forgetting earns its keep when there are distractors, when the stale fact would otherwise pollute retrieval, which is exactly what the drift benchmark isolates and the oracle setting removes. The small consolation: even there, my precise forgetting nearly doubles naive cosine forgetting, and it matches store-everything on the abstention questions.
Building it on Qwen Cloud
Since this was built for Qwen Cloud, a note on where Qwen actually shows up. It does four jobs, all on the read side. It answers the user. It drives recall through native function-calling, so the agent itself decides when to call a recall_memory tool and writes the query, instead of me hard-coding a retrieval step. It plays the LLM-extract baseline my gate replaces, which is how I get an honest zero-versus-N token comparison on the same model. And a stronger Qwen model judges the benchmark answers. The write decision never calls any of them, which is the entire point.
The backend runs on an Alibaba ECS instance and snapshots memory to Object Storage, so a session can close and reopen and the agent still remembers. The one deployment scar worth passing on: HuggingFace is unreachable from Alibaba's Singapore region, so the first model download hung the request forever with no error. I pre-cached the models locally, shipped them to the box, and ran fully offline. Obvious in hindsight, an hour of quiet confusion at the time.
The lesson I am keeping
The clever part of this project was never the neural memory. It was noticing that two signals were doing a job I had asked one signal to do, and that the failure only showed up when I built a benchmark honest enough to break my own demo. The tie was the most useful result I got. It is the reason the final thing works.
Flashbulb runs on Qwen for the answers and on Alibaba Cloud for hosting and persistence. It is open source under MIT. If you want the ten-second version: teach it a few facts, save the memory to the cloud, close the session, reopen it, and it still remembers. And when you tell it you moved, it actually believes you.
Code (MIT): https://github.com/somtri/flashbulb-qwen
Live demo: http://47.84.205.239/
Demo video: https://youtu.be/7KKzW_9Hlec
Top comments (0)