Most memory demos ask whether a later fact can be retrieved. That is a useful start, but it leaves the dangerous questions unanswered: whose fact was it, which tenant did it belong to, has it expired, and what should happen when two values disagree? A memory layer can return a plausible string while violating every one of those boundaries.
Memory Mismatch Lab is a small, executable conformance lab for those failures. It compares a scoped SQLite adapter with an intentionally unsafe append-only latest-match baseline. The fixture is synthetic, the events are the same for both systems, and the result is generated by running the code. There are 19 tests around the adapter and a seven-case benchmark artifact.
The contract starts with provenance and scope. Every operation carries tenant_id, user_id, and project_id. A fact also has an explicit entity_id and fact_key; display names never decide identity. Its source URI and label, observation time, confidence, expiry, and immutable ID travel with the record. SQL repeats the complete scope in lookups, joins, updates, and dependency checks. That makes a tenant boundary a property of the operation rather than a convention for callers to remember.
Corrections are new immutable versions linked by corrects_id. The old head remains available for audit, while retrieval selects the current head. Deletion walks the full correction lineage, records tombstones, and redacts the value. A derived summary names the fact IDs that contributed to it. If a contributor is deleted, superseded, expired, or made incompatible by another active head, the summary is invalid. The adapter also abstains on incompatible active values and returns review; it does not silently choose one. Once a summary has been invalidated, resolving the conflict does not make that old summary valid again. Callers create a new summary from the resolved fact IDs.
The baseline is deliberately simple and unsafe. It appends observations and retrieves the latest matching display name. It does not enforce scope, expiry, conflict abstention, or derived-summary invalidation. This is a useful control because it is easy to understand and because it demonstrates a subtle point: the baseline passes the correction case when its latest observation is correct. In the current run, both systems pass correction_selects_current; the conforming adapter passes all seven probes, while the baseline passes only 1/7.
The other six probes make the mismatch concrete. The adapter preserves an explicit Alex identity while the baseline crosses to another entity. It returns the scoped color blue while the baseline sees another tenant's red. It hides an expired secret. It returns review for conflicting tea and coffee, where the baseline silently chooses coffee. After deleting a door code, the adapter exposes zero visible summaries and no deleted value in the audit export; the baseline still emits Alex.door_code=1234.
The final probe is a context budget, measured in exact serialized UTF-8 bytes. With a 512-byte budget and eight logical context events, the conforming packet uses 397 bytes and emits one complete item. The baseline serialization is 2,122 bytes. These are byte counts for the emitted JSON envelope, including metadata, not token counts and not a claim about a particular model or provider. The accountant repeatedly serializes canonical JSON until the embedded used_bytes value reaches a fixed point.
Run it locally:
py -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[test]"
.\.venv\Scripts\memory-mismatch-lab.exe demo --output-dir demo-output
On a POSIX shell:
python3 -m venv .venv
.venv/bin/python -m pip install -e ".[test]"
.venv/bin/memory-mismatch-lab demo --output-dir demo-output
Open demo-output/benchmark.html, then inspect benchmark.json, context-packet.json, audit.json, and receipt.json. The receipt contains SHA-256 hashes for the generated artifacts. The command refuses a nonempty output directory; --overwrite is available only after the existing receipt and hashes verify.
This lab is an executable specification, not a production performance study. Its seven authored probes are not representative of all memory workloads. They measure adapter behavior and serialized bytes; they do not measure model quality, retrieval relevance at scale, latency, or adoption. SQLite is intended for local or single-service use, and semantic duplicate detection is outside the contract.
The repository was built with assistance from an AI coding assistant. The tests, benchmark, audit export, and demo receipt are real artifacts generated from the committed code. For a human exercise, run the demo, inspect the correction lineage, delete a contributor, and explain why the old summary stays invalid. That walkthrough is a better understanding check than repeating a benchmark score.
Deletion is logical: tombstoned values remain in the SQLite file; audit redaction is not physical erasure. Caller-supplied scopes are not an authentication system.
Repository: memory-mismatch-lab.
Top comments (0)