Session-Level Memory vs. Fragment Memory: Why AI Coding Agents Hallucinate About the Past

趁早 — Mon, 26 Oct 2026 03:25:56 +0000

Ask your AI coding agent a simple question: "Why did we choose PostgreSQL over MySQL?"

If you settled that decision three weeks ago in a different session, you'll get one of three responses — and all three are failure modes:

"I don't have context on that." Amnesia. Everything you discussed is gone.
A confident, plausible-sounding rationale. Hallucination. It's inventing reasons that were never discussed.
A memory shard that says "the user prefers PostgreSQL." Technically true, functionally useless. The shard lost the why — the constraints, the rejected alternatives, the person who owned the tradeoff.

The industry's answer to this has been "give agents memory" — and the dominant pattern, popularized by tools like claude-mem, mem0, and Letta, is fragment memory: extract salient facts from conversations, compress them into embeddings or summaries, store the shards, and retrieve the top-k at query time.

I built a different thing, and this post explains why. SessionRelay stores complete sessions — every conversation you have with Claude Code, Codex, ZCode, or Qoder, captured passively into one local SQLite database, queryable by any agent through 16 MCP tools, with mandatory provenance on every answer.

The difference isn't a feature list. It's the storage granularity. And I believe fragment memory has a structural problem that no amount of retrieval tuning will fix.

The compression problem

Fragment memory pipelines look like this: conversation → LLM extraction → distilled facts → vector store.

Each arrow is a lossy transform. When the AI "remembers," it's retrieving a summary of a summary, written by a model that decided — months ago, in a context you can no longer inspect — what was worth keeping.

The failure isn't occasional. It's systematic:

Compression distortion. "We chose PostgreSQL because of partitioning needs and the ops team's familiarity" compresses to "PostgreSQL chosen." Later, the agent answers "why PostgreSQL?" with generic Postgres praise — because the constraint that actually drove the decision is gone.
No provenance. A shard says "the user said X." Which session? When? In what context? Was it a firm decision or a 2 a.m. idea that got walked back the next morning? The shard can't say, so the model smooths over the uncertainty with confident guessing.
Context pollution. Most memory systems push — they inject retrieved shards into every session's context. Now your agent's head is full of loosely-related "you once said..." fragments that crowd out the actual task. Half-remembered context is often worse than none.

Session-level memory: three architectural commitments

SessionRelay's unit of storage is the session: title, timeline, source tool, status, and the full original conversation. Nothing is distilled at capture time. The design rests on three commitments:

1. Mandatory provenance. Every answer the agent gives from memory must cite which session, which message numbers, which source tool, which date. When your agent says "PostgreSQL was chosen because of table partitioning requirements (Database Selection, ZCode session, Aug 20, messages 12–18)," you can click through to the raw exchange. Fabrication gets structurally harder when the citation format is enforced by the tool layer, not by model manners.

2. Pull, not push. The AI queries memory only when it needs to — there's no ambient memory injected into context. A clean context window beats a polluted one; a targeted get_decisions() call beats a slow drip of "relevant" shards.

3. Original text, always. The agent retrieves the actual conversation, not a pre-chewed summary. Extraction still happens — decisions, topics, key exchanges get structured for fast lookup — but they're an index over the original, never a replacement for it. The raw session files on disk remain the single source of truth; the entire database is rebuildable with srelay rebuild.

The retrieval tool defaults tell you the philosophy: 20 messages × 1,000 characters, hard-capped at 50KB per response, with a truncated: true flag and hints on how to narrow. Safe by default means not trusting model restraint.

The capture problem nobody talks about

Here's something that surprised me while building this: agents destroy their own history.

ZCode — the agent I use daily — physically deletes messages from its SQLite store when it compacts context. I measured one compaction deleting 3,976 messages from a long session. Claude Code keeps JSONL files, but they're scattered per-project with no unified query layer; other tools vary.

This is why SessionRelay ships with a daemon that syncs every 30 seconds: messages land in the memory database before the source tool can compact them away. Without it, "memory" is really "whatever survived until you happened to export."

Capture is passive — five adapters watch Claude Code, ZCode, Codex, and Qoder (Trae partially: it encrypts assistant replies, so only user prompts are captured). Adding a new tool is a single JS file dropped into .sessionrelay/adapters/, zero core changes.

Retrieval is a language problem, not just an embedding problem

Most memory tools are built and tested on English. My day-to-day work is in Chinese, and Chinese breaks naive retrieval: no whitespace, compound words that must split ("认证方案" → 认证 + 方案), and quoted-phrase queries that must match exactly.

SessionRelay runs jieba tokenization into SQLite FTS5 with dual indexes, gated by six Chinese acceptance cases, session-level AND matching with OR fallback. Semantic search is optional and local (bge-small-zh, CPU-only, zero cloud) — and it follows a rule I think more memory systems should adopt: literal hits always rank first and are never replaced; semantic results only append, flagged viaSemantic, capped at five. Measured effect: synonym-query miss rate dropped from 33% to 0%, with zero regression on exact queries. When semantic search is disabled, behavior is byte-identical to pure literal search.

Retrieval boundaries matter too. A scope.json contract (shared by CLI and MCP) can only narrow what's searchable — intersection semantics, never widening. An automatic 30-day backstop keeps an agent from drowning in two years of history when you only needed this month.

Memory needs a lifecycle, and deletion belongs to humans

Sessions in SessionRelay pass through a two-phase verdict: active → pending_end → confirmed. Nothing gets indexed into the main store until a session has been silent for six hours — because conversations resurrect (you --resume the next morning), and crystallizing an unfinished conversation bakes in incomplete decisions. Even after confirmation, a new message triggers rollback and re-extraction.

Old sessions get archived, not deleted: decision and topic skeletons are kept, message bodies are released (~99.4% of the space).

Actual deletion is a separate, human-only operation:

srelay forget a3f8c2d1        # preview: what gets removed vs kept
srelay forget a3f8c2d1 --yes  # execute, irreversible
srelay forget --history       # audit trail of what was forgotten when

The AI has zero delete power. The MCP tool list has no delete tool, forever. Forget leaves dual anti-resurrection barriers — an ignore rule that survives rebuilds, plus a tombstone table — so the raw file can stay on disk without ever being re-ingested. This is what "right to be forgotten" has to mean when an AI is reading your every conversation.

Humans forget too: handoff between people

The feature that gets the strongest reaction from teammates: .hop handoff packages (open spec, vendor-neutral, MIT). You spent three days with your AI settling an architecture. The person picking up the project inherits — at best — the code.

srelay export produces a package with per-file sha256 integrity, secret redaction by default (cloud keys, private keys, bearer tokens, password assignments, DB connection strings — with a redaction report of exactly what was stripped), and an auto-generated HANDOFF.md: decision table, files involved, open questions, timeline.

The recipient imports it — into any project, any MCP client — and by default the import is quarantined: metadata and summaries land first; message bodies require an explicit release, one by one. Handoff contents are data, not instructions — that trust model is written into the protocol, and the quarantine is its structural enforcement.

Honest limitations

Rule-based decision extraction currently sits at ~60–70% precision — which is exactly why provenance blocks exist (verify item by item; an --ai extraction pass is next). macOS/Linux daemon service registration is still in progress. Trae captures user prompts only. And SessionRelay won't auto-discover sessions from before you renamed a project directory — handoff packages are the migration path. The full list is in the README, because a memory tool that hides its limitations is asking you to trust it blindly, which is the exact failure mode it exists to prevent.

Try it

npx @ewanjasper/sessionrelay init   # zero-install, backfills last 30 days
claude mcp add sessionrelay --scope user -- srelay serve
srelay ui                            # 0.6.0: local read-only dashboard over the same memory

Then open a fresh session and ask: "Why did we decide X?" — if the answer comes back with a date, a source tool, and a session ID, memory is working. If it says "I don't know," run srelay doctor.

Everything runs locally: SQLite on your disk, zero network calls at runtime, MIT licensed. As of 0.6.0 the CLI, MCP, and a browser dashboard are three thin layers over the same core — and the release also merged production hardening from a contributor's real 3-project deployment (a named-pipe lock, a handle-leak fix, and a cross-project contamination bug we only found because someone actually ran it at scale — my favorite kind of issue). I'd genuinely value feedback on the two hardest open questions: extraction precision, and what a third-party HOP reader implementation should look like.

DEV Community: 趁早