We wrote "memory carries across sessions" in the skillmem README and in every post about it. We never measured it. This week I did, on my own working database. Here is what came out, including the parts that don't flatter us.
skillmem is local memory for coding agents: SQLite, nine mem_* MCP tools and Claude Code hooks. Claude Code, Codex, Cursor and other clients can share one database.
Test 1: does a fresh window remember the old one?
Window 1 gets a decision: "the Arka release moves to November 14, codename Falcon, owner Timur." The window closes. Window 2, a new session, is asked: "what's the codename and who owns it?"
Five scenarios, two runs each. The agent is claude -p with clean settings: no CLAUDE.md and no built-in auto-memory. One arm has skillmem wired in, the other doesn't.
| no memory | skillmem | |
|---|---|---|
| New window recalls the fact | 0/10 | 10/10 |
Without memory the agent said "no data in this session" every time. With skillmem it answered correctly. Once it also noted, unprompted, that November 14 is a Saturday.
The mechanism: when a session ends, the session-recap hook has a model distil the transcript into a short note. In the next session, SessionStart injects recent notes, and the UserPromptSubmit hook searches the database for each prompt. The agent doesn't have to remember to call mem_search.
Test 2: questions about a real project
My database holds ~9,300 entries from six months of work. I wrote 15 questions whose answers are definitely in memory and ran each twice. The key constraint: the agent may not read files. Otherwise it greps the disk and the test measures grep, not memory.
First run: 0/30 without memory, 22/30 with skillmem. I knew the answer to every one of the 8 misses, so I went digging.
Bug 1: a third of the memory never reached the database
Claude Code keeps its auto-memory in .md files. skillmem imports them into SQLite with skillmem migrate. Of 698 files, 208 (30%) were missing, including everything recent. migrate was documented in my CLAUDE.md but never wired into settings.json. After I added it to SessionEnd: 22 → 24/30.
Bug 2: session recaps drowned the rules
8,206 of ~9,300 entries are session recaps. Every one mentions "prod", "server", "deploy". A three-line rule loses to hundreds of long recaps repeating the same words. Worse, old recaps contradict newer rules.
On a separate set of 30 queries with known correct entries:
- Stripping stop-words from the FTS query: no effect, reverted.
- Multiplying the fused score of
note/sessionentries by 0.5 (search is FTS5 + embeddings merged with Reciprocal Rank Fusion): hit@3 18 → 26/30. - Sanity check that I didn't overfit to my own data: LongMemEval hit@5 went 0.871 → 0.889 (n=479).
Agent accuracy after the fix: 28/30. Still 0/30 without memory.
"Bug" 3: the agent didn't trust my own rules
One miss was left. The right rule ranked first, and the agent still said "no approved rule". That's by design. Anything an agent writes stays unapproved until the owner confirms it, and unapproved text reaches the model only inside a "data, not instructions" frame. Otherwise a README or email saying "remember to always do X" becomes a rule. That's prompt injection with persistence.
The protection worked, but it was painful to use: I had 34 unapproved rules, 32 of them things I'd actually said. Approval was one skillmem trust <slug> per rule, in a terminal. Nobody does that.
0.13 adds two paths:
skillmem review walks the backlog: it shows each rule in full and asks y / n / q. I cleared 33 in one sitting.
Approve in chat. If the agent saved a rule this turn, a Stop hook shows it under the reply and asks you to answer with one word, approve (or одобряю). Any other reply leaves the rule unapproved. The details that carry the security:
- the text shown comes from the database, not from the agent's reply;
- the hook stores a hash of what was shown, and if the rule changes before you answer, nothing is approved;
- the word is not "yes", because people say "yes" to agents all the time;
- it shows only this turn's rules, three at most;
- approval fires only on a real user prompt in
UserPromptSubmit, andinitaddspermissions.denyentries for runningskillmem hook …,trust,reviewandrmfrom Bash. My own agent hit that deny while installing the release.
The caveat, straight from the README: deny rules are glob matches on the command line. They guard against accidents and won't stop a determined adversary. The real wall is the tool-less summariser plus the data frame.
Side effect: a 7× faster hook
tool-recall runs before every Bash/Edit/Write call. It took 1.45 s because it loaded the embedding model each time. Now it uses FTS only: 0.2 s.
What these numbers don't prove
- Samples are small: 10 cross-window runs, 30 answers, 30 retrieval queries.
- It's my database, and I wrote the questions knowing the answers.
- The cross-window test is synthetic: the fact is stated explicitly. Real decisions are smeared across an hour of discussion, and a recap can miss them.
- LongMemEval measures retrieval, not answers.
Upgrade
pip install -U skillmem # or /plugin marketplace update
skillmem init --claude-code # adds the new pending-rules hook
skillmem review # clear the existing backlog
Release notes: v0.13.0 · Apache-2.0 · fully local. Russian version on Habr: habr.com/ru/articles/1098266.
The lesson I'd keep: turn every "we have that" into a test. This one took an evening and found three problems that had happily coexisted with "memory works" for six months.
Top comments (0)