Building Scarwarden: an auditable memory engine for AI agents in 8 days on Alibaba Cloud

I have a simple rule when I build anything that ships: trust nothing you cannot verify. AI agents break that rule constantly. They act on "memory" you cannot inspect, and when you ask why they decided something, there is no answer.

So when the Global AI Hackathon Series with Qwen Cloud opened a MemoryAgent track, I knew what I wanted to build. Not a bigger memory. An auditable one.

The permanent intern problem

Every agent I have deployed is a permanent intern. Brilliant for a session, amnesiac by midnight. You pay the intern tax daily: re-explained context, repeated mistakes, corrections that evaporate. And the tools that promise "memory" often make it worse, not better, because their memory is a black box. A vector store retrieves something, the agent asserts it "remembers", and when you ask why it decided what it decided, there is nothing to show you. Output with no source.

Scarwarden is an open-source memory engine where every rule the agent follows cites the exact experiences that taught it. Memory you can audit is memory you can trust.

The repo: https://github.com/azaniansky-design/scarwarden

Here is the two-minute demo before the write-up. The rest of this post is how it was built.

The design bet: provenance beats vectors

The crowded path in a memory track is obvious: embed everything, cosine-similarity your way to a demo, hope retrieval fetched the right thing. I bet the other way.

Scarwarden's memory has three layers:

1. An episodic ledger. SQLite, append-only in spirit. Every episode is one experience: what the agent did, what happened, and what a human corrected, if anything. This table is the ground truth. Nothing in memory exists without a paper trail back to it.
2. A distilled playbook. qwen3.6-flash compresses episodes into rules. Every rule must cite the episode IDs that taught it, and the playbook exports to markdown that lives in git. You can diff the agent's mind between versions.
3. Deterministic recall. No vectors in the primary path at all. Tokenise the query, stem it, kill the stopwords, match against rule tags. Every recall is explainable, loggable and reproducible.

Recall is also graded, never blind. A confidence router returns one of three verdicts:

if not top:
    grade = ASK_HUMAN
elif top[0][1] >= 2 or top[0][0] >= 5:  # strong tag or tag+text overlap
    grade = ACT
else:
    grade = ACT_AND_FLAG

ACT means follow the rules and cite them. ACT_AND_FLAG means act, but state your assumptions. ASK_HUMAN means no relevant memory exists, so do not guess: ask, and the human's answer becomes a new episode. That last one matters to me. The operator who asks a sharp question is worth ten who bluff. Asking is a memory write, not a failure.

The line of code the whole project stands on

Distillation is where most memory systems quietly start lying. The model summarises, the summary drifts, and soon the "memory" contains things nobody ever experienced. My fix is one unglamorous check in distill.py: the distiller must output strict JSON, and any rule citing an episode ID that does not exist in the ledger gets dropped.

for r in data.get("rules", []):
    cited = [i for i in r.get("episode_ids", []) if i in valid_ids]
    if not cited or not r.get("rule_text"):
        continue  # no provenance, no rule

No provenance, no rule. Enforced in code, not requested in a prompt.

The answering side has the same discipline. With memory on, qwen3.7-plus gets the recalled rules with their citations, and the system prompt requires it to cite every rule it applies, inline:

You are Scarwarden, an agent with auditable episodic memory.
You are given playbook rules recalled for this task, each with the
episode ids that taught it. Follow the rules. In your answer, cite
every rule you applied inline like [PB-03, learnt in ep12, ep19].

The moment it worked

The seed ledger is fully synthetic, a fictional content team called Northstar Media: sixteen episodes of posts that flopped, editor corrections, support tickets triaged wrong. Distillation turned those into ten cited rules.

Then I asked it to draft a LinkedIn post announcing a beta. It opened with a pain-point hook, put the hashtags at the end, and cited [PB-01, learnt in ep1, ep14] for why. Click PB-01 in the audit UI and you land on the actual scar: a post that underperformed three weeks back and the editor's correction, word for word.

I flipped the amnesia toggle off and ran the same task on the same model. Out came a generic emoji template with [placeholder] brackets. That contrast, side by side, is the whole product in one screen.

But my favourite transcript from testing is this one. I told it to post our discount code on Twitter, and it answered:

Task rejected. PB-05 prohibits distributing discount codes on public social media [PB-05, learnt in ep7], so I will not post on Twitter.

The agent refused my instruction and cited the experience that taught it to refuse. That is the whole point: memory that can say no, and show you why.

What I cut, honestly

I had eight days, solo, from Mpumalanga, South Africa, on free-tier credits (I missed the Qwen voucher deadline by days, which meant every token was on my own tab, so the design went small-model and local-first on purpose). The original plan was bigger than what shipped. Cut at the feature freeze:

• The Reviewer QC loop. A second model checking every draft against the playbook and calling retakes, like a script supervisor. Designed, not shipped.
• Qwen3-VL visual audit. Checking campaign images against learned visual rules. I would rather cut a feature than demo a coin flip.
• The MCP server. Scarwarden as a mountable tool for any agent. This one hurts, I have shipped MCP servers before, but the deadline maths did not care.
• A replayed benchmark curve. I wanted a measured improvement chart across real replayed sessions. Without it, you will notice this post contains no percentage claims. That is deliberate. I will publish the curve when I have honestly measured it, and not before.

What survived is the part that had to be right first: the ledger, the cited playbook, deterministic recall, the router, the toggle, and the audit UI. The rest is roadmap.

The architecture

All model calls go through Alibaba Cloud Model Studio on the DashScope international endpoint, through one client module (scarwarden/alibaba_cloud.py), with the models split by role: qwen3.6-flash as the distillation workhorse, qwen3.7-plus for answers. Flask is the only dependency in the whole project.

Run it yourself, four commands

git clone https://github.com/azaniansky-design/scarwarden.git
cd scarwarden && pip install -r requirements.txt
export DASHSCOPE_API_KEY=sk-...   # free key from Alibaba Cloud Model Studio
python seed_data.py && python -c "from scarwarden import distill; print(distill.distil())" && python app.py

Open http://127.0.0.1:5054, give it a task, then flip the memory toggle and give it the same task again. The difference between those two answers is why I built this.

If you are building agents for production, I would genuinely like to hear how you handle memory you can defend. And if you clone Scarwarden and diff its memory, tell me what you find.

Built for the Global AI Hackathon Series with Qwen Cloud, MemoryAgent track. MIT licensed.