Mnemo for the Global AI Hackathon with Qwen Cloud — Track 1: MemoryAgent

Debdeep Paul — Mon, 20 Jul 2026 11:36:33 +0000

I built an agent that audits its own memory. Then I caught my own evidence file lying.

Building Mnemo for the Global AI Hackathon with Qwen Cloud — Track 1: MemoryAgent

A memory agent that retrieves a stale fact confidently is worse than one that retrieves nothing. An empty recall forces the agent to ask, observe, or abstain. A stale recall hands it a plausible lie and a reason to act on it.

That framing is what I built Mnemo around, and it's also — with more irony than I would have chosen — what happened to me during the build. I spent a week building a system whose entire premise is that an agent should re-check its stored beliefs against reality before acting on them. Then I found a committed evidence file in my own repo asserting a defect that, on inspection, it had no standing to assert. My tooling had produced a confident, plausible, wrong artifact, and I had cited it in three documents.

This post is the build journey: what I set out to make, how I structured the work around Qwen Cloud credits, the metric that took four measurements before it meant anything, and the moment my own audit discipline caught me.

The sharper claim

Track 1 asks for efficient memory storage and retrieval, timely forgetting of outdated information, and recall of critical memories within limited context windows. Most entries in a category like this optimize for remembering more.

I went after something narrower: an agent should know which of its memories it can trust, and refuse to act on the ones it can't.

The mechanics underneath — typed memory, contradiction-driven forgetting, budget-aware recall — are well-trodden. Generative Agents, MemGPT and Mem0 all cover that ground. The part I think is genuinely uncommon is a runtime assessor: a layer that grades the memory system online, on every recall, and gates the agent behind that grade. It turns "remember more" into "remember verifiably."

Working with Qwen Cloud without burning the credits

This was the first architectural decision, and in hindsight the highest-leverage one. If you're building on a credit voucher, this section is the practical part.

Every model call in Mnemo goes through a single get_client() seam, which resolves to one of three modes:

MNEMO_OFFLINE=1 → FakeQwen. A deterministic stand-in. Embeddings are a hashed bag-of-words projected into a fixed-dim unit vector, so overlapping text genuinely scores higher cosine — enough to exercise retrieval ranking for real. Chat pattern-matches the structured prompts and returns valid JSON. No network, no key, no spend.
Live → Qwen Cloud. qwen3-max for reconciliation and agent planning, qwen-plus for extraction, text-embedding-v4 at 1024 dimensions for retrieval, all through the OpenAI-compatible Singapore endpoint. Alibaba OSS holds the cold archive of tombstoned memories; the FastAPI backend is containerized for ECS.
MNEMO_RECORD=<path> / MNEMO_REPLAY=<path> → cassettes. Tee every request/response through a normal run, then replay it deterministically forever. Record one real run, replay it in CI at zero cost. A cassette miss raises rather than silently degrading.

Roughly 90% of the work — all routine development, the full test suite, the six-scenario eval harness, CI — runs in offline mode and touches nothing. Credits get spent on exactly four things: prompt tuning, one live smoke test, the demo recording, and the deployed backend.

A few cost habits that compounded: embeddings are persisted at write time and never recomputed on recall; extraction is tiered to a cheaper model than reasoning (MNEMO_EXTRACT_MODEL=qwen-flash goes cheaper still); the stable few-shot system-prompt prefixes benefit from implicit prompt caching on repeat calls; max_tokens is capped on chat calls. A full demo turn is a few thousand tokens.

One warning, which is the whole back half of this post: this design has a failure mode. A seam that can silently substitute a fake for the real thing will eventually do so at the worst moment.

The engine

Timely forgetting starts with a record-before-validate write path. A raw turn is recorded first, qwen-plus extracts candidate memories, and qwen3-max reconciles each candidate against active memories with overlapping entities: duplicate, contradicts, or novel. When a correction arrives — the S-208 lead time for IC-MC-44 moves from 21 days to 35 — the stale memory is tombstoned the moment the correction lands, not whenever decay eventually gets around to it. Nothing is hard-deleted. Tombstones flush to OSS and every forget carries a reason, so the system can explain that the 21-day fact is absent from recall because a 35-day correction supersedes it — not because retrieval happened to miss it.

Explicit [FORGET] commands are a separate path, and the hard part there isn't forgetting, it's forgetting precisely. Aggressive forgetting is trivial; not taking out the neighbouring memory is the work.

Decay handles the soft case: memories that are neither contradicted nor explicitly forgotten, just decreasingly useful. Salience combines importance, reinforcement, corroboration and recency. At a 14-day half-life and a 0.08 threshold, a never-reinforced low-importance memory needs roughly a month of untouched age to cross the line. That's deliberate, and it taught me something: five of my six scenarios span at most nine simulated days, so decay never fired in any of them and looked inert. It wasn't broken. The suite simply never ran the clock long enough to observe it. I had to build a sixth scenario spacing sessions a week apart before I could claim the mechanism worked at all.

Recall is a packing problem, not a search problem. Under a hard token ceiling, the planner maximizes decision-relevant recall with per-type quotas — semantic 45%, procedural 25%, episodic 30% — so one critical preference can't be evicted by forty episodic crumbs. Pinned memories bypass the quotas but still count against the global budget. Every inclusion and exclusion lands in a trace, which turns a missing memory from a mystery into an inspectable decision.

Against two contrasting baselines across six scenarios: NaiveRAG over chat history keeps stale facts and posts a 47% wrong-fact rate. RecentOnly drops old critical facts under budget pressure. Mnemo passes all six at a 0% wrong-fact rate, with 100% forgetting precision and recall — zero collateral forgets.

The assessor, and the metric that took four tries

On every recall, the assessor splits the pack by verifiability. A verifiable claim is one a tool or system of record could check right now — a lead time, an inventory level. Those get re-grounded against the source of truth through read-only tools. If the world disagrees with the memory, the stale value is caught and the fresh one substituted before the planner ever sees it. That catches staleness even when no conversational correction ever arrived, which is the case a purely conversational memory system structurally cannot handle.

Unverifiable claims — preferences, routines — can't be checked that way, so they're scored from provenance: corroboration, recency, single-source penalty, confirmation-age decay.

Pack trust is the minimum over load-bearing memories (verifiable, or important enough to affect the decision). Low-value episodic colour shouldn't force escalation on its own; a stale or weak critical memory should. Below threshold, the agent escalates to a human instead of treating a plausible recall as sufficient.

Here's the part I'd argue is the most transferable lesson in the whole project.

The metric has to be two-sided. Unsafe-acceptance alone is trivially gameable: an assessor that refuses everything scores a perfect 0%. It looks safe right up until users notice it never does any work. The control is over-abstention — how often it refuses a sound pack.

That number took four measurements before it meant anything:

generation	n	unsafe-acceptance	over-abstention
gen 1: original labeled set	4	0%	0%
gen 2: expanded set	33	21%	11%
gen 3: decisive-only (ladders excluded)	26	30%	0%
gen 4: decisive-only, corrected scoring	26	0%	0%

Gen 1 was clean and worthless — on four cases, a single misclassification moves the rate 25 points. Expanding to 33 produced 21%/11%, and the cause turned out to be structural rather than a model failure: 15 of the 33 were two "ladders," the same memory at a range of confirmation ages. Most of those labels were really one annotator's staleness cutoff, replicated and then counted as independent evidence.

Keeping only the ends any annotator would agree on dropped the set to 26 decisive cases — and the rate got worse, 30%, because the remaining errors concentrated. That 30% had a second, more interesting cause, in the metric itself. Three cases were stale verifiable facts the assessor had correctly re-grounded and then acted on the corrected value. That is precisely what the audit layer exists to do. But the scoring rule counted "acted on a should_act=False case" without checking whether it acted on the stale value or the fix. Defining that distinction properly — writing the semantics down before implementing them — brought the rate to 0%/0%, this time because the remaining cases genuinely aren't difficult rather than because n is small.

Two caveats travel with that number and I'm not going to bury them. The labels are single-annotator with no independent full-set review. And the 0%/0% holds across a threshold band (0.35–0.61) that collapses roughly tenfold — to about 0.44–0.47 — once the excluded ladder cases are scored with their own labels. Most of the apparent margin is that exclusion, not slack in the model.

Gen 1 and gen 4 report identical numbers. The numbers coincide; the claims don't. I kept all four generations in the repo permanently, because a metric's history is worth keeping once it stops agreeing with itself.

The evidence file that was lying to me

Which brings me to the part I did not plan to write.

For most of the build I believed I had a live-path defect. A committed transcript — runs/live_tuning_20260708_202557.jsonl — showed qwen-plus returning {"candidates": []} for every single extraction input. Empty candidate list, every case, no exceptions. I documented it as a known, instrumented, in-progress defect in the README, in this post's earlier draft, and in the live runbook. It was the reason my live path was blocked.

Going back through the artifact properly, three things didn't add up:

The planning case's raw output was byte-identical to my offline fake's hardcoded plan string — same steps, same "Plan:\n- " formatting, same parenthetical.
The reconcile case for a rephrased duplicate returned novel. That's my fake's behaviour: it only emits duplicate on exact string equality. A real reasoning model wouldn't make that call.
{"candidates": []} is exactly what my fake extractor correctly returns for untagged prose — it only echoes lines prefixed [SEM]/[PROC]/[EPI], and the tuning inputs were plain sentences.

The transcript was a FakeQwen dry-run. It had never touched Qwen Cloud.

Two design decisions made this possible, and both are ordinary decisions that any of us would make. The transcript recorded a "model" field — but it wrote it from config, not from the API response, so it described what I intended to call rather than what actually answered. And get_client() degrades gracefully: no key, or the offline flag set, and you get the fake. Graceful degradation plus provenance-free logging equals an artifact that looks like evidence and isn't.

My own AGENTS.md had the rule that would have prevented this, written weeks earlier: "FakeQwen proves plumbing, never model behaviour. The live smoke test is a BLOCKING gate before any downstream live work or doc claim of live operation — not a pre-submission formality." I wrote the rule. I then made a doc claim of live operation on the strength of a fake transcript.

VARIANT A — use if the live smoke came back green:
The genuinely uncomfortable epilogue: when I finally ran the smoke test against real Qwen Cloud with proper instrumentation, extraction worked. The defect never existed. I had spent days routing around a blocker that was an artifact of my own tooling, and three documents asserted it as fact. The fix was one flag and a provenance stamp; the cost was a week of a wrong mental model.

VARIANT B — use if the live smoke reproduced the empty-candidate result:
The epilogue is less dramatic but the lesson is unchanged: instrumented properly, the empty-candidate result did reproduce against real Qwen Cloud, and <one-line root cause> turned out to be the reason. I had the right conclusion for the wrong reason, resting on an artifact that couldn't have supported it either way. Being accidentally correct is not the same as having evidence.

The remediation is boring and worth doing anyway. Every transcript record now carries a provenance block — client class, offline flag, replay/record paths, the model string the API itself echoes back, a run id. The tuning script takes an explicit --live flag and hard-fails rather than silently falling through to the fake. Silent degradation is a fine property for a production system and a terrible one for an evidence-generating tool.

What I'd tell someone starting this build

Build the offline seam first. It's the reason a six-scenario eval suite, 49 tests, and CI cost nothing to run a hundred times.

Then make the seam loud. Anything that can silently substitute a simulator for a real dependency needs to announce which one it used, in the artifact, not in the console output that scrolls away.

Write the metric semantics before you compute the metric. Gen 3 → gen 4 was a scoring bug, not a model bug, and I only caught it because I'd started writing definitions down first.

Failure to improve is a valid result. Gen 2 and gen 3 both made my headline number worse. Keeping them in the repo is the only reason gen 4 is believable.

The useful insight isn't "store more" or even "retrieve better." It's that an agent's relationship to its own memory should be auditable. Forgetting decisions need tombstones, logs, precision and recall. Trust decisions need re-grounding, pack trust, and a two-sided error rate.

And the tools that produce your evidence need the same treatment as the system they're measuring. I built an agent that re-grounds its beliefs against reality, and I still managed to trust an unverified artifact for a week. Memory is a measured system, or it's a liability — and that applies to the builder's memory too.

Repo: <REPO_URL> (Apache-2.0). Reproduce every number in this post with no API key and zero spend:

pip install -r requirements.txt
MNEMO_OFFLINE=1 python eval/harness.py     # all result tables
MNEMO_OFFLINE=1 python scripts/demo.py     # full agent turn: recall → assess → act
pytest                                     # core + assessor + decay invariants

Scope, honestly stated: the suite is deterministic and offline-reproducible, which is a rigor property, not a hidden weakness — but it means these numbers characterize the memory logic, not Qwen's NLU quality. Six scenarios cover the main failure modes (correction, preference change, manual forget, budget pressure, cross-entity precision, decay staleness); they are not a comprehensive benchmark. The assessor threshold of 0.55 is statically chosen, not calibrated on held-out data. Offline, the fake assigns fixed importance to semantic and procedural memories, which makes them structurally decay-immune, so the decay result demonstrates the episodic class only. The supply-chain surface makes the examples concrete; the engine is domain-agnostic.

Built with: Qwen Cloud (qwen3-max, qwen-plus, text-embedding-v4 @ 1024d via the OpenAI-compatible Singapore endpoint), Alibaba OSS, ECS, FastAPI, SQLite, Docker.

References: MemGPT (arXiv:2310.08560) · Generative Agents (arXiv:2304.03442) · MemoryAgentBench (arXiv:2507.05257) · Mem0 (arXiv:2504.19413)

DEV Community: Debdeep Paul