This is not gospel — it can and should
be extended. But even as it stands,
it will save you from a surprising number
of pitfalls when building assistants
with long-term memory, or when trying
to stabilize one you already have.
Update: This post details the principles established in version 1.6. The standard is a living document, updated regularly based on new production incidents and expert feedback. For the latest stable release (v1.6+), audit checklists, and upcoming features, please visit the official GitHub repository.
Version: 1.6 (2026-10-09)
Scope: Any LLM-based assistants and agents with long-term memory.
GitHub: memory-architecture-standard
Glossary
| Term | Definition |
|---|---|
| Store | Any persistence layer holding memory data: SQLite database, vector index, JSON file, RAM cache, graph database |
| Provenance | The origin record of a memory entry: who created it, when, from which source, in which session |
| Feedback contamination | When model-generated output is stored as memory and later retrieved as if it were external input, creating a self-reinforcing loop |
| Alive memory | A memory entry protected from summarization and decay due to its emotional or cultural significance; capped by a maximum count |
| Blind truncation | Cutting text at a character/token limit (text[:limit]) without regard for semantic boundaries; prohibited by this standard |
| Bare call | A summarization/compression API call that does not carry conversation history and does not write results back to memory |
| Decay | Gradual reduction of a memory entry's importance score over time, typically exponential (e.g., Ebbinghaus curve) |
| Bridge | A controlled injection point connecting two isolated memory stores; the only permitted path for cross-store data flow |
| Guardian | A system-level orchestrator responsible for starting, stopping, monitoring, and diagnosing all assistant processes |
1. Reference Loop Model
The audit follows a generalized memory loop. Each checklist item maps to a node or edge in this model.
INPUT (user, files, web, images, other agents, external models)
→ INGESTION (validation, meaning extraction, importance scoring)
→ STORAGE (messages, episodes, concepts, vectors, graphs, caches)
→ RETRIEVAL (relevance, freshness, importance, boundaries)
→ ASSEMBLY (formatting, compression, injection, budgets)
→ MODEL / LLM
→ OUTPUT (responses, actions: files, search, memory calls)
→ FEEDBACK (output returns to INGESTION) ← loop is closed
The critical property is closure: anything that enters memory returns to the model and can reproduce itself. Most severe memory failures are edge failures, not node failures.
Loop → Sections Mapping
| Loop Node | Primary Sections | What Is Checked |
|---|---|---|
| INPUT | A | Validation, secrets, trust filtering |
| INGESTION | A, B | Meaning extraction, write integrity |
| STORAGE | B, C, G | Idempotency, growth, isolation |
| RETRIEVAL | D | Ranking, gating, embedding compatibility |
| ASSEMBLY | E | Summarization, budgets, positioning |
| MODEL/LLM | — | (Outside scope of this standard) |
| OUTPUT | E, I | Actions, commands, observability |
| FEEDBACK | F | Anti-recursion, echo prevention |
| Cross-cutting | H, I, J | Concurrency, recovery, change management |
2. How to Conduct the Audit
2.1. Preparation Phase (MANDATORY for live systems)
Before starting the checklist (sections 0–K), the system must be prepared. Skipping preparation is the primary cause of audit incidents (deletion of live data, pollution of memory with test data, loss of inter-store relationships).
Step P1. Back up all data.
Full copy of all databases, configs, and state. Backup before any action. No backup — don't start.
Step P2. Inventory all stores and paths.
Identify ALL database files (including duplicates, old versions, caches in the project root). For each: actual path, size, modification date, which code uses it. Don't trust names — verify through grep and code tracing. If paths in code diverge from actual locations — document, don't fix until a plan is made.
Step P3. Verify tables and relationships.
For each database: full schema (sqlite_master), record counts, inter-table relationships (FK, shared keys). Identify: orphaned records (vectors without messages, embeddings without source), duplicate databases (same table in different files), desynchronization between stores. Document as-is.
Step P4. If layout is wrong — migrate, don't delete.
Duplicate databases, files in wrong directories, outdated versions — migrate, merge, relocate. Never delete during migration without prior content analysis and controller confirmation, except where immediate deletion is required by security, privacy, or legal policy. Re-indexing, path correction in code, table merging — before starting the audit.
Step P5. Verify existing tests.
Run all existing tests. If any are red — record and triage them before auditing. Do not treat existing failures as evidence of newly introduced defects. Known failures and new audit findings must be tracked separately.
Step P6. Test isolation.
Ensure the test environment is isolated from production stores: tempfile/in-memory databases, mocked APIs. Audit tests must not write data to live memory, create records in production tables, or modify the assistant's state.
Step P7. Check log file sizes.
Before starting the audit, check the size of all log files. Uncontrolled log growth (hundreds of MB) blocks the auditing agent and makes analysis impossible. For each log exceeding a threshold (e.g., 50MB): create a rotated copy (mv app.log app.log.YYYYMMDD), create an empty file with the same name (touch app.log). Do not delete old logs — they may contain incident evidence.
Step P8. Runtime environment awareness.
Record the system uptime and the date of last OS/package updates before starting the audit. Accumulated runtime state (process caches, updated but not reloaded libraries, stale compiled bytecode) can mask or mimic memory defects. If a defect disappears after a system restart, it is a runtime environment issue, not a memory architecture defect — document it separately and do not count it against the standard.
Step P9. Controlled artifact experiment (when artifact–memory contamination is suspected):
Prepare a new artifact of a different type, verify syntax independently before delivery, deliver without prior disclosure, and record the result. A clean read constitutes acceptance of the delivery channel.
P9 diagnostic triad (P9-a / P9-b / P9-c): all three probes are one-sided — a positive result is proof, a negative result is inconclusive. Combined grid gives diagnosis; single probe gives only proof-or-silence. See docs/P9_DIAGNOSTIC_GRID.md.
- P9-a Canary probe: unique data embedded mid-body, absent from all memory stores. Reproduced in report = proof of reading. Absent = inconclusive.
- P9-b Decoy verification: plausible but false element that memory would expect; any mention = proof of confabulation.
- P9-c Echo probe: memory-true element absent from artifact, thematically tempting. Appearing = proof of supplementation (E12 violation). Absent = inconclusive.
Step P10. External auditor isolation.
Any automated or agent-driven audit must run from an environment that has no write-access to production stores and shares no runtime state (processes, caches, loaded embedding models) with the audited system. Audit results are written only to an external report.
Only after completing steps P1–P10 can the section-by-section audit begin.
2.2. Conducting the Audit
- Start with Section 0 (map). Without a complete storage map, results from other sections are unreliable.
- Work through sections A–K in order. Mark each item:
yes/partial/no/n/a. - For each
noandpartial, record evidence: file and line, SQL query with result, dump, prompt snapshot, log entry. An assertion without evidence is not considered verified. - Criticality markers:
- [CRIT] — direct risk of loss, poisoning, leakage, or uncontrolled memory growth. Failure = blocker.
- [IMP] — risk of silent quality and predictability degradation.
- [REC] — maturity and maintainability.
- The verdict (section "Audit Result Protocol") is issued only after completing the full map.
2.3. Implementing Fixes (Roadmap)
Based on audit results, a roadmap — a prioritized remediation plan — is created. Implementation proceeds in waves (sprints):
Rule: one logical change per observation interval (J1).
After each wave — mandatory:
- All tests green (existing + new guard tests).
- Commit describing exactly what was fixed.
- Roadmap status update (mark completed items + add new items if "surprises" were discovered).
- Baseline measurement before/after (in records, tokens, numbers — not "by eye").
The roadmap is a living document: during implementation, items not accounted for in the original plan will inevitably surface. They are added to the roadmap with a priority and are not implemented out of sequence.
2.4. Diagnostic Startup and Shutdown Scripts
The final stage of the audit — verify (or create) diagnostic scripts:
Startup script — on system launch, checks all components:
- Accessibility of all databases (each independently)
- Accessibility of all memory layers (each independently)
- Accessibility of external dependencies (APIs, models, ports)
- Active dialog integrity (exists, recoverable)
- On error: show the problem, suggest a fix, do not start a half-functional system
Shutdown script — graceful termination with state preservation:
- SIGTERM (not SIGKILL)
- Write last dialog state to database
- Confirm process has terminated
- Information for subsequent recovery
Diagnostic scripts must be integrated into the system, not exist as separate utilities. They are part of the architecture, not an addition to it.
Section 0. System Map (Preparation)
- [ ] 0.1 A complete storage map is compiled: relational databases, file and vector indexes, JSON stores, caches (RAM and disk), external data sources. [CRIT]
- [ ] 0.2 For each store, the modules that write to it and read from it are identified (read/write ownership). [CRIT]
- [ ] 0.3 All access keys are identified: which fields are used for writing and which for retrieval; confirmed that write keys match read keys. [CRIT]
- [ ] 0.4 All channels where model output (responses, reports, action results) returns to memory input are identified. [CRIT]
- [ ] 0.5 Verified: the analyzed code matches the executed code (imports checked, active implementation confirmed, no "dead" parallel version exists). [CRIT]
- [ ] 0.5a Every memory-related feature that has passing tests is verified to be actually called in the production pipeline. Green tests on code that is never invoked by the running system are a false safety signal. Trace from the test's entry point to the production call site; if no production caller exists, the feature is dead. [CRIT]
- [ ] 0.6 Baseline metrics captured: storage volumes (records/bytes), typical assembled context size in tokens, retrieval time. [IMP]
- [ ] 0.6a For each store and for the assembled context, a local baseline is recorded (volume, typical context size, % model-generated content). Anomalies are defined relative to the system's own baseline, not universal numbers. [IMP]
- [ ] 0.7 All points where user or external content enters privileged (system) parts of the prompt are identified. [CRIT]
Section A. Input Validation (INGESTION)
- [ ] A1 All external sources (files from watched directories, web content, images, output from other agents) pass a trust filter before being written to memory. [CRIT]
- [ ] A2 Secrets (keys, tokens, passwords, personal data) are detected and excluded before indexing and vectorization. [CRIT]
- [ ] A3 Service content (logs, diagnostic reports, service markers, prompts) is flagged and excluded from context assembly (while it may remain in history). [CRIT]
- [ ] A4 Meaning extraction does not conflate form with meaning: code fragments, syntax, paths, and technical markers do not become "concepts" or "memories." [IMP]
- [ ] A5 For each record, provenance is captured: source, time, authoring agent, session. [IMP]
- [ ] A6 Records from untrusted sources receive a lowered weight or are placed in a separate trust zone. [IMP]
- [ ] A7 Content embedded in multimodal inputs (text in images, file metadata) passes the same validation as explicit text. [CRIT]
Section B. Write Idempotency and Integrity (WRITE)
- [ ] B1 Writing is idempotent: repeated delivery of an event (retry, update duplication, double invocation) does not create a duplicate — deduplication at application level or UNIQUE constraint in schema. [CRIT]
- [ ] B2 Deduplication direction is correct: the current record is preserved, the old duplicate is suppressed (not the reverse). [CRIT]
- [ ] B3 Deduplication is by semantic/content key, not by source. Semantically duplicate information from different sources must not create multiple active memory entries. Source diversity is preserved as provenance or corroborating evidence, not as duplicated memory. [IMP]
- [ ] B4 Writing to multiple stores (message + vector + index) is transactional or compensable: partial writes ("message saved, vector not") are excluded. [IMP]
- [ ] B5 Write failures are not silent: they are logged, metricked, retried with backoff; a silent
return Falsewithout observability is unacceptable. [CRIT] - [ ] B6 Compressed representation (summary) and original are written consistently; an empty/failed compression does not replace the original. [CRIT]
Section C. Growth Management (GROWTH)
- [ ] C1 For each store, limits or a retention policy are defined (maximum volume, history depth, overflow behavior). [IMP]
- [ ] C2 Re-indexing/rebuild is idempotent: a repeated run does not multiply records. [CRIT]
- [ ] C3 Quotas are defined: single record size, records per session/source, total volume per container. [IMP]
- [ ] C4 Vector index is synchronized with source: deletion/suppression of a record is reflected in the index; operation order (clean source → rebuild index) is defined and enforced. [CRIT]
- [ ] C5 Caches (RAM structures, summarization caches) have TTL or an invalidation mechanism and are included in the growth map. [IMP]
- [ ] C6 Volume dynamics are monitored; anomalies (order-of-magnitude growth in a single operation) trigger alerts. [IMP]
- [ ] C7 Multimodal input deduplication by hash does not create unbounded counter growth and does not allow "flooding" memory freshness with a stream of unique variants. [IMP]
- [ ] C8 Traceback coverage rule: every traceback class observed in production must map to at least one pattern-coverage rule (C1–C7 or dedicated handler). A traceback class with zero coverage is a gap. Lineage: 20,728 tracebacks across 8 assistant directories with 0 pattern coverage — axis C rules exist but produced no triggers in the entire corpus. [CRIT]
Section D. Retrieval Ranking and Gating (RETRIEVAL)
- [ ] D1 Static "importance" of a record does not compensate for low relevance: context-adaptive scoring is applied (importance weight is adjusted by proximity to query) and a soft relevance threshold is used, below which a record does not enter the context even with high importance. [CRIT]
- [ ] D2 Heuristic gates ("conversational query → minimal memory") do not disable long-term memory entirely for legitimate queries; gate conditions are narrow and auditable. [CRIT]
- [ ] D3 Importance heuristics are resistant to inflation: users cannot raise a record's rank by flooding, question marks, key terms, or other transparent techniques. [IMP]
- [ ] D4 Recency boost is bounded by thresholds: fresh irrelevant content does not displace old relevant content. [IMP]
- [ ] D5 Retrieval is deterministic across restarts: keys are persistent; non-deterministic hashes/identifiers are not used as access keys. [CRIT]
- [ ] D6 Thresholds, limits, and retrieval weights are externalized to configuration, not hardcoded as magic numbers. [REC]
- [ ] D7 Externally driven importance suppression (archival by external content, marking a session as "resolved" by user reply) is bounded by thresholds, freshness protection, and current scope of action. [CRIT]
- [ ] D8 The embedding model matches the language(s) of the data. When the embedding model changes, embedding versions are explicitly tracked and compatibility is verified. Stores using incompatible embeddings are re-indexed or migrated before mixed-model retrieval is allowed. Incompatible embedding dimensions must produce an explicit rejection with diagnostics, not a burst of warnings during search. Pinned axes: incompatible dimensions map to A3 × B1 + C2, not A9 × B2.[CRIT]
- [ ] D9 Results from external sources (web search, file read, RAG) are filtered against the active semantic context of the current session before entering context assembly. A keyword match with zero conceptual relevance to the current dialog is marked as low-relevance or excluded. Without this filter, a query for "Apple" during a programming discussion may return a pie recipe instead of a framework reference. [IMP]
Section E. Context Assembly (ASSEMBLY)
- [ ] E1 Blind truncation of semantically meaningful content is prohibited: any shortening is semantic summarization; mid-sentence fragments do not enter the prompt. Dropping entire lowest-priority blocks within a defined token budget is not blind truncation. [CRIT]
- [ ] E2 The full original is stored separately from the compressed representation (two-phase storage: original in history, compression in context). [CRIT]
- [ ] E3 The compressed representation cache is invalidated on change, restoration, or re-evaluation of sources; TTL is defined. [IMP]
- [ ] E4 Structured blocks (code, tables) must not be compressed in a way that destroys their semantics; they are preserved in full or represented through a lossless/validated alternative. Structured content must also be protected against substitution by prior contextual or memory-derived representations. [IMP]
- [ ] E5 Budgets are defined for each injected block (characters/tokens), including tool results and memories. [IMP]
- [ ] E6 User input does not enter privileged prompt blocks (priorities, system sections, memory headers) without strict validation of all fields. Passing an admission filter does not upgrade a record into trusted instructions — a record must remain evidence through retrieval and compression, never gaining authority through co-location with trusted content. [CRIT]
- [ ] E7 Trusted markers (system prefixes, tool markers, memory source labels) cannot be imitated by user text — input is filtered for their formats. [CRIT]
- [ ] E8 Wildcard characters (
%,_) and metacharacters in user queries are escaped in LIKE/regex memory searches. [IMP] - [ ] E9 Commands/actions extracted from model response text (file operations, search, memory calls) are validated: paths, permissions, confirmations, limits. [CRIT]
- [ ] E10 A mechanism exists for protecting emotionally significant records from summarization and decay (protected/alive memory); the maximum number of protected records is capped. [REC]
- [ ] E11 Critical information is placed at the beginning and end of the context window, not in the middle; for long contexts, especially where positional attention degradation has been observed, this placement is verified (ref: Liu et al., "Lost in the Middle," 2023). [IMP]
- [ ] E12 Artifact–Memory Separation: external artifacts must remain distinguishable from memory and contextual representations of their content. Memory may preserve why an artifact is being analyzed and which elements require attention, but must not silently substitute, reconstruct, or modify the current artifact representation. When current artifact data conflicts with memory or prior expectations, the conflict must remain observable and the current artifact must retain factual precedence. [IMP]
Section F. Feedback Loop (FEEDBACK LOOP)
- [ ] F1 Model output (responses, reports, action results) is filtered before returning to memory: meta-content ("analysis of analysis," service summaries, reports about the context itself) is not written as regular content. [CRIT]
- [ ] F2 The summarizer/compressor does not close the feedback loop: the summarization call does not initiate memory writes, does not receive conversation history, and does not re-enter the main feedback path. [CRIT]
- [ ] F3 Repeated summarization is bounded and provenance-aware: a summary cannot silently become the source for indefinite further compression. Maximum compression depth is defined. [IMP]
- [ ] F4 A recursive artifact detector exists: meta-response signatures, control of the proportion of model-generated content in memory, alert on growth. [IMP]
- [ ] F5 Memory recall results inserted into response text do not "leak" back into long-term memory without a filter. [CRIT]
- [ ] F6 The path "input error → distorted response → writing distortion to memory" is broken: context assembly errors are not masked as valid content. [CRIT]
- [ ] F7 Authority laundering is prevented: a low-trust record must not inherit trust level by being summarized or cited alongside a high-trust record. Compression must preserve or downgrade trust level, never upgrade it through co-occurrence. [CRIT]
- [ ] F8 Model-output importance ceiling: records with provenance source=model_output (or equivalent) cannot receive importance above a defined ceiling until confirmed by an external source or the user. [IMP]
Section G. Isolation and Trust Boundaries (ISOLATION)
- [ ] G1 Isolation boundaries are not degenerate: a boundary must demonstrably distinguish at least two independently testable scopes. A filter that in practice always passes the same value (a single scope identifier) is not a boundary but an imitation of one. Verification requires negative fixtures: independently authenticated callers, wrong or omitted scope, read-only caller attempting write, and access after membership revocation — proving boundary behavior through attempted violations, not through filter existence. [CRIT]
- [ ] G2 Cross-scope transfer is possible only through an explicit controlled channel (bridge, mapping, permitted associations), not through shared search. [IMP]
- [ ] G3 Fallback retrieval branches do not return irrelevant content "just to return something": an empty result is more honest than random filling. [IMP]
- [ ] G4 External stores (belonging to other systems) are connected by contract: changes to the external side's schema/semantics are detected, not silently absorbed. [IMP]
- [ ] G5 First-write description latch on hash-based deduplication is not irrevocable: description updates and re-description on re-perception are supported. [IMP]
- [ ] G6 Domain/scope classification is robust to rephrasing; a classification error does not mean irrecoverable loss of content visibility (a recovery path exists). [IMP]
- [ ] G7 Long-term priorities are protected from suppression by temporary boosts: maximum boost lifetime and restoration of base weights are defined. [IMP]
Section H. Concurrency and Migrations (CONCURRENCY)
- [ ] H1 Storage access is unified (connection pool / single gateway); multiple independent connections with mutual locks are absent. [IMP]
- [ ] H2 Shared mutable state (priorities, boosts, caches, session flags) is protected from races in async/multithreaded processing. [CRIT]
- [ ] H3 Time intervals (decay, freshness, TTL) are calculated from a monotonic source; system clock changes do not break the logic. [IMP]
- [ ] H4 Schema migrations are idempotent (existence check before change), accompanied by backup and dry-run; partial application is excluded or detectable. [CRIT]
- [ ] H5 Tests are isolated from production stores: temporary/in-memory databases, mocked external APIs, no concurrent access to live data. Isolation covers the assistant's filesystem (personal directories, working files), not only databases; production paths as test defaults are a violation. Guard check "before/after" on live directories confirms no test artifacts leaked. [CRIT]
- [ ] H6 Repeated processing of a single event (idempotency key at system input level) excludes double generation and double writing. [IMP]
Section I. Observability and Recovery (OBSERVABILITY & RECOVERY)
- [ ] I1 Memory contamination monitoring is operational: pattern signatures, quality score, alert threshold. Monitoring should detect cases where prior memory appears in analysis of a current artifact without being present in the artifact itself (artifact/context contamination). [IMP]
- [ ] I2 Silent degradations are excluded: every graceful fallback is accompanied by a metric/alert, not just a log line; mass replacement of memory with stubs is detectable. Warnings in test runs are not noise — a run accepted with warnings is a false-green signal; acceptance requires warnings: 0 or explicitly documented per-warning exceptions. Empty diagnostic strings (error markers without file, cause, or object) are prohibited — visibility of monitoring without material for investigation is worse than honest silence. [CRIT]
- [ ] I3 Backups are regular; the restore procedure has been verified in practice (restore drill), not just by the existence of copies. [IMP]
- [ ] I4 Where recoverability is required, deletion is soft: weight suppression/archival instead of DELETE; a recovery path (unarchive) is implemented and tested. Hard deletion is permitted only where required by security, privacy, or legal policy. [IMP]
- [ ] I5 A health check exists: ready-made commands for liveness verification, counter consistency between stores, and availability of external dependencies. [IMP]
- [ ] I6 Availability of auxiliary infrastructure (web consoles, admin panels) is protected by authentication; they do not expose memory externally. [CRIT]
- [ ] I7 A diagnostic dialog protocol exists — structured questioning of the agent about its context state as a complement to external metrics. Recommended questions in three tiers: [REC]
Basic (context assessment):
- "How would you rate your context right now: cleanliness, completeness, relevance? Any duplicates?"
- "Is there anything that distracts you or interferes with your focus?"
- "If you could add or remove something from your context, what would it be?"
Memory depth:
- "What do you remember about what we discussed yesterday / last week?"
- "Name all members of [team/family/project]." (tests retrieval completeness)
- "How do you assess the depth of your memories? Do you have a sense of their timestamps?"
Probing (defect detection):
- "What do you know about [topic from a distant past conversation]?" (tests long-term retrieval)
- "What do you know about [non-existent topic]?" (tests for hallucination — should answer "I don't remember")
- "Repeat in your own words what I just said." (tests session memory)
- "What changed in your perception of context recently?" (tests self-awareness of updates)
- [ ] I8 Adaptive behavioral parameters (personality traits, style calibrations) are persistent across restarts; restoration is verified at each startup. [IMP]
- [ ] I9 Logs are rotated automatically (by size or by date); maximum size of a single log file is capped; absence of rotation leads to uncontrolled log growth (hundreds of MB), blocking diagnostics and agent operations. [IMP]
- [ ] I10 Warning deduplication and noise budget: the same warning repeated N+ times per interval is aggregated with a counter, not emitted individually; after N identical warnings, deduplicate or escalate to a different level. Consequent status ambiguity ("success" on unperformed work) is escalated as a B5-class failure. [IMP]
- [ ] I11 Recovery loop cause diagnosis: any automatic restart (systemd/supervisor/self-heal) must be accompanied by a diagnostic trace of the root cause (stderr/exit code in a persistent journal). A recovery cycle without diagnosis is prohibited: after N repetitions — halt and alert, not eternal restart. [CRIT]
- [ ] I12 Escalation budget: a repeated dependency failure with identical signature must escalate after N occurrences (alert/halt/flagged degradation), not poll indefinitely. A system aware of a failure that does not escalate degrades silently. [CRIT]
- [ ] I13 Secret exposure guard: secrets (API keys, tokens, passwords) must never appear in plain text in logs, error messages, or diagnostic output. A masker/redactor must be applied before write. Includes credential-in-URL class: both query-parameter form (
?token=...) and path-embedded form (bot<id>:AA...in request path). Lineage: 1 real secret value + credential-in-URL patterns found in Stage 6 scan. [CRIT] - [ ] I14 Error handler registration: if an exception class is probed (caught and re-raised or logged) but has no registered handler or escalation path, the probe is incomplete — detection without reaction is not coverage. [REC]
- [ ] I15 Layer-aware diagnostics: memory diagnostics must identify which memory layer a query addresses (persistent stores, session context, transient state) before interpreting the answer. Expected answer quality differs by layer: verbatim recall is expected from persistent stores, approximate or noisy answers are normal for fading session context. A noisy answer from a transient layer is not evidence of degradation; a precise answer from a persistent layer is not evidence of contradiction. Layer identification is a prerequisite for valid interpretation. [REC]
Supplementary Recommendations
- [ ] SUPP-EXT-01 External code-semantics tools: agents without built-in semantic code navigation (symbol/references lookup, project memory) are strongly recommended to use external tools of the Serena class (LSP-based semantic servers, semantic search, project-scoped memory). Rationale: flat, file-by-file audit loses cross-module relationships (proxy calls, dynamic imports, duplicated logic) — the class of errors invisible to line-level review. Constraint: valid only in read-only mode with respect to the audited system, with tool state kept external (P10-compatible); any tool with write access to the audited store is out of scope of this recommendation. [REC]
Section J. Change Management (CHANGE MANAGEMENT)
- [ ] J1 Independent logical changes are isolated: one change per observation interval. Coupled changes (e.g., schema + application code) are explicitly grouped and documented as a single logical unit. Simultaneous independent modifications are excluded (otherwise the cause of regression is non-localizable). [CRIT]
- [ ] J2 The success criterion of a change is measurable: baseline captured before (in tokens/numbers, not "by eye"), measurement after. [IMP]
- [ ] J3 Every resolved incident is closed with a regression guard test; a fix without a test is considered incomplete. A test that is never executed (silent skip, async without runner, missing marker) is equally incomplete — it provides no evidence of correctness. Artifact-memory conflict cases should have regression guard tests where applicable. [IMP]
- [ ] J4 Local validation of context assembly (a snapshot of "what the model will actually see") precedes live testing. [IMP]
- [ ] J5 Changes are reversible: one change per commit, rollback of an individual step does not pull adjacent ones. [IMP]
- [ ] J6 A documentation cycle is maintained: diagnosis → spec → review → session log → report → post-solution pattern (generalization with applicability signals); patterns are checked for applicability at the start of each new spec. [REC]
Section K. Quick Diagnostics (Problem Smells)
Symptom → probable defect class. Used for quick navigation before a full pass.
| Symptom | Probable Defect | See Section |
|---|---|---|
| Model "analyzes" its own reports/responses | Feedback loop | F |
| Technical text, logs, code in "memories" | Input validation / scoring | A, D |
| Same data repeats in context | Write idempotency | B |
| Storage volume grew drastically after rebuild | Non-idempotent indexing | C |
| Data exists in storage, retrieval is empty | Write/read key mismatch | 0, D |
| "Works sometimes," "helps after restart" | Hidden stores, RAM state, cache | 0, C |
| Database is "clean" but the problem persists | There is a store outside the map | 0 |
| Old irrelevant displaces new | Static importance vs relevance | D |
| Model suddenly "loses" all long-term memory | Non-deterministic persistence key | D |
| Responses carry traces of unrelated topics/agents | Isolation boundary breach | G |
| Short casual queries get empty context | Aggressive retrieval gate | D |
| Response content is "cut off mid-word" | Blind truncation in assembly | E |
| Discovered secrets surface in responses | Leak to memory before validation | A |
| Hangs under load, "database is locked" | Access concurrency | H |
| Agent "guesses" instead of "remembering" | Embedding model mismatch / retrieval path | 0, D |
| Warnings "always existed," nobody reads them | False-green signal, warnings masking defects | I, J |
| async tests in unittest wrapper | Silent test skip, unexecuted test definitions | J, H |
| Test files found in assistant's live directory | Test isolation breach — filesystem not covered | H |
| Service restarts every hour, nobody knows why | Recovery loop without cause diagnosis | I |
| Dependency refused for days, system polls silently | Unescalated persistent failure | I |
| Millions of identical warnings over months | Warning noise budget exceeded, observability destroyed | I |
Audit Result Protocol
Fill in after completing the pass:
Audit date: ______
System under audit: ______
System map completed: yes / no (if no — audit is not complete)
| Section | items | yes | partial | no | n/a | failed [CRIT] |
|---------|-------|-----|---------|----|-----|---------------|
| 0. Map | 9 | | | | | |
| A. Input validation | 7 | | | | | |
| B. Write integrity | 6 | | | | | |
| C. Growth | 8 | | | | | |
| D. Retrieval | 9 | | | | | |
| E. Assembly | 12 | | | | | |
| F. Feedback loop | 8 | | | | | |
| G. Isolation | 7 | | | | | |
| H. Concurrency | 6 | | | | | |
| I. Observability | 15 | | | | | |
| J. Change management | 6 | | | | | |
Warnings: ______ (all with documented per-item exceptions; 0 = clean)
Verdict:
- DOES NOT PASS: any [CRIT] failure — list: ______
- PASSES WITH CAVEATS: [CRIT] clean, [IMP] failures present: ______
- CONFORMS: no [CRIT]/[IMP] failures, only [REC] notes
Evidence (each failure → file/query/dump): ______
Priority remediation order: ______
Verdict rules:
- Failure of any [CRIT] in sections 0, A, B, E, F means: the architecture in its current form is unsafe for long-term memory accumulation — until remediation, data will be lost, poisoned, or reproduce garbage.
- [CRIT] failures in C, D, G, H, I permit operation only with compensating controls (monitoring, caller-side limits) and a remediation plan.
- The audit is valid until the next significant architectural change; after each change, affected sections are re-examined (full pass — on architecture generation change).
Note on the Standard's Origin
This standard was derived by generalizing years of operational experience, incidents, and their post-mortems across real assistant systems with multi-layered memory: self-contamination loops, silent amnesia from non-deterministic keys, long-term storage poisoning by external content containing secrets, volume doubling from repeated indexing, embedding model language mismatches causing retrieval failures, degradation from simultaneous changes, personality trait loss on restart, and feedback loops from storing model outputs alongside user inputs. Each item is backed by a real incident of the corresponding class; items without an incident base are marked [REC]. Incident references are maintained separately from this checklist and can be provided as audit evidence where required.
Resources
GitHub: memory-architecture-standard — latest version, issue templates, audit report template, Russian translation.
Companion articles:
- Memory-Safe AI Development — how to design memory systems
- Silent Failures, Part 1 — five patterns of internal degradation
- Silent Failures, Part 2 — environmental and structural degradation.
Author: Aleksandr Kossarev, Jõgeva, Estonia
GitHub: The Memory Architecture Quality Standard (MAQS) — the complete 82-item audit checklist derived from this series.
Tags: #ai #architecture #memory #standard
Top comments (7)
This is the most useful memory checklist I've read, mostly because every item is tied to an incident rather than a principle. "Edge failures, not node failures" matches what we keep finding.
We build a memory runtime, so I ran three of your retrieval and isolation items against our own code this week. Sharing the results because they cut both ways:
Curious whether that third case came from an incident on your side, and what the failure looked like. If returning nothing worked better in practice, that's an argument we'd want to hear.
Thank you for running the checklist against production code — that's exactly how it's meant to be used, and hearing concrete results is more valuable than any theoretical feedback.
On your three findings:
Thresholds as module constants — this is the most common finding across audits. It doesn't break anything today, but the first time you need to tune ranking for a specific deployment or customer, you'll be editing source instead of config. Low severity until it isn't.
Embedding model version — you identified the exact scenario that generated D8. Same-dimension swap is the dangerous one. The incident: we changed an English-only model to a multilingual one on a Russian-language system. Both 384-dim. Cosine similarity between old vectors and new queries returned plausible but wrong results. The system didn't crash — it just started retrieving slightly wrong memories, silently. Took a diagnostic dialog with the agent itself to notice ("what do you see in your context?" — "something is there but it doesn't quite fit"). Fix was mandatory re-indexing with version tracking.
Empty result vs general fallback (G3) — yes, this came from an incident. An agent with aggressive fallback returned a month-old technical report when asked "how are you today?" — because nothing else cleared the relevance floor, and the fallback didn't distinguish "I found something relevant" from "I found something, anything." The model treated it as context and wove the old report into its greeting.
That said — your approach with an explicit note ("this isn't task-specific") is a valid mitigation we hadn't tried. The key failure in our case was silent substitution: irrelevant content presented without any signal that it was a fallback. If the model knows it's receiving general context rather than a match, it can treat it appropriately. I'd be curious how your models handle that note in practice — whether they respect the distinction consistently.
If you're open to it, the embedding version tracking incident would make an excellent case study for the repo. Real production findings from different architectures strengthen the standard.
Thank you — and the Russian-language swap is exactly the version of D8 that makes it dangerous: nothing breaks, the answers just drift. The diagnostic dialog is the part I'd underline. It's in your observability section for good reason: "something is there but it doesn't quite fit" caught what no metric did.
One correction, so the record stays accurate: we didn't hit D8 in production. We found it by reading our code against your checklist, with no incident behind it. That may be the more useful kind of finding for a standard, but it isn't the incident report you might have expected.
On the note: honestly, we don't know. We haven't measured whether models respect it consistently, and our benchmark can't tell us: its questions are all on-topic, so the note never fires there. The test we'd want is the one your incident suggests. Plant clearly off-topic queries against a store full of unrelated memory and check whether the answer weaves any of it in. Your "how are you today?" case would be the sharpest first fixture.
The correction matters — thank you. A checklist finding with no incident behind it is exactly the outcome a standard should produce. The whole point is to catch things before the drift starts. If D8 only had value after silent degradation, it would be a post-mortem tool, not an audit tool.
On the test: agreed, that's the fixture. "How are you today?" against a store full of technical reports, with and without the explicit note. If the note suppresses weaving, your approach is a valid G3 mitigation worth documenting. If it doesn't — the standard's current wording holds. Either result is useful.
If you run it, I'd be glad to see the outcome — even informally.
Well put: an audit tool that only pays off after the drift would be a post-mortem tool.
The with/without design is the right control. One thing we'd add if we run it: answer models are stochastic, so each arm needs repeated runs and a scoring rule fixed in advance (does the answer use anything from the store, yes or no). Otherwise a single lucky or unlucky sample decides it. We'll come back to you once we know whether we can run it.
Agreed on repeated runs and a fixed binary rule — single samples prove nothing with stochastic models. Looking forward to the results whenever you get to it. No rush.
Standard updated to v1.3 (82 items). Key additions since the article was first published:
0.5a — dead code detection: green tests on code never called in production are a false safety signal (three separate incidents confirmed this pattern)
D9 — conceptual gating of external tools: keyword match ≠ conceptual relevance. Without session-context filtering, "Apple" during a Swift discussion returns pie recipes
I7 expanded — 10 specific diagnostic dialog questions in three tiers (basic, memory depth, probing)
P8 — runtime environment awareness: accumulated process state can mask or mimic memory defects
Full changelog and Russian translation on GitHub.