Agent memory architecture

Agent Memory Architecture – A Practical Guide

---

📋 Concise Diagnosis Table

Symptom / Error	Likely Root Cause	Immediate Mitigation	Long‑Term Remedy
Stale exchange‑rate quoted after a few seconds	Working memory TTL too long or missing freshness check	Force a refresh from the live rates API for any request older than RATE_TTL	Implement a hard TTL per volatile data type and audit policy
User‑data remains after “Delete my data” request	Durable preferences not version‑tracked or deletion path missing	Run a purge script that deletes all preference keys matching the user ID and logs the action	Enforce immutable audit trails for deletions and integrate privacy‑by‑design hooks
Memory‑related latency spikes during high traffic	Episodic logger writes synchronously to a single store	Switch episodic writes to an async queue, keep a bounded buffer	Deploy a sharded write‑ahead log with back‑pressure handling
Incorrect regulatory rule applied	Authoritative fact cache not refreshed before rule evaluation	Bypass cache and call the compliance API for rule‑type queries	Set deterministic freshness checks (e.g., MAX_AGE = 5 min) and unit‑test rule‑generation pipelines

---

⚠️ Editorial Caveats

• A symptom does not prove a specific root cause; systematic investigation (labels, context, preprocessing, base‑model limits) is required before attributing errors to pre‑training or fine‑tuning.
• Retrieval‑Augmented Generation (RAG) improves information access but does not guarantee factual accuracy, instant compliance, or complete coverage.
• Low‑Rank Adaptation (LoRA) freezes base weights and learns low‑rank updates; it does not guarantee preservation of all existing behavior. Regression testing outside the target task is essential.
• Document retrieval and live‑value API calls are distinct; cached documents must never replace a required live‑value fetch.

---

🧭 Decision Flow (Mermaid)

flowchart TD
    A["Incoming request"]
    B["Is request for volatile data?"]
    C["Use Working Memory"]
    D["Is request for user preferences?"]
    E["Use Durable Preferences"]
    F["Is request for audit trace?"]
    G["Use Episodic Memory"]
    H["Is request for verified fact?"]
    I["Use Authoritative Facts"]
    A --> B
    B -->|yes| C
    B -->|no| D
    D -->|yes| E
    D -->|no| F
    F -->|yes| G
    F -->|no| H
    H -->|yes| I
    H -->|no| C

---

🏗️ Layer‑by‑Layer Responsibilities

Layer	Primary Role	Data Lifetime	Access Pattern	Typical TTL / Versioning
Working Memory	Short‑lived context for the active session (e.g., recent user utterances, temporary calculations).	Cleared after the session or after a short TTL (seconds‑to‑minutes).	In‑process read/write; no persistence.	SESSION_TTL = 5 min (configurable).
Durable Preferences	Long‑term, mutable settings that survive sessions (e.g., preferred language, notification opt‑ins).	Persisted indefinitely but versioned; can be deleted on request.	Indexed look‑up, atomic updates.	Version number ↑ on each change; soft‑delete flag.
Episodic Memory	Immutable audit trail of actions, decisions, and tool calls (for compliance, debugging, analytics).	Append‑only; retained until a pruning policy runs.	Write‑only during execution; read‑only for analytics.	Retention window MAX_EPISODE_AGE = 90 days (subject to regulation).
Authoritative Facts	Verified data from signed knowledge bases or live APIs (e.g., tax rates, regulatory limits).	Cached only as a performance shortcut; source is authoritative.	Retrieval followed by deterministic freshness validation; fallback to live API guaranteed.	Cache MAX_FRESHNESS = 5 min for volatile facts; indefinite for static signed docs.

Key principles

1. TTL enforcement – each layer has a hard expiration enforced by the runtime, never by “best‑effort”.
2. Deletion & privacy – Durable Preferences and Episodic records must support explicit user‑initiated deletion workflows that are auditable.
3. Staleness guardrails – Authoritative Facts must never be served beyond their freshness window; a deterministic check (now - cached_at < MAX_FRESHNESS) decides whether to use the cache or hit the live source.

---

🛠️ Python Decision Helper (Teaching‑Only)

# -*- coding: utf-8 -*-
"""Memory layer dispatcher – teaching only.

The helper decides which memory store a request should hit based on
metadata that the orchestration layer supplies.  It contains only
standard‑library constructs and heavy assertions to illustrate the
contract between the caller and each memory layer.
"""

import datetime as _dt
from enum import Enum, auto
from typing import NamedTuple, Literal

# ----------------------------------------------------------------------
class MemoryLayer(Enum):
    """Enumerates the four memory layers."""
    WORKING = auto()
    DURABLE = auto()
    EPISODIC = auto()
    AUTHORITATIVE = auto()

class Request(NamedTuple):
    """A minimal request description used by the dispatcher."""
    kind: Literal["volatile", "preference", "audit", "fact"]
    created_at: _dt.datetime  # UTC timestamp of request arrival
    ttl_seconds: int | None = None  # Optional explicit TTL

# ----------------------------------------------------------------------
def select_layer(req: Request) -> MemoryLayer:
    """Return the appropriate MemoryLayer for *req*.

    Assertions act as guard rails – they fail fast if the caller supplies
    contradictory information, which mirrors the deterministic business
    rules required in production.
    """
    now = _dt.datetime.utcnow()
    age = (now - req.created_at).total_seconds()

    # 1️⃣ Volatile data → Working Memory (must respect TTL)
    if req.kind == "volatile":
        assert req.ttl_seconds is not None, "TTL required for volatile data"
        assert age <= req.ttl_seconds, "Request exceeds its TTL"
        return MemoryLayer.WORKING

    # 2️⃣ User preference → Durable Preferences (no TTL, versioned)
    if req.kind == "preference":
        assert req.ttl_seconds is None, "TTL must not be set for preferences"
        return MemoryLayer.DURABLE

    # 3️⃣ Audit trail entry → Episodic Memory (append‑only)
    if req.kind == "audit":
        assert req.ttl_seconds is None, "TTL not applicable to audit logs"
        return MemoryLayer.EPISODIC

    # 4️⃣ Verified fact → Authoritative Facts (freshness check)
    if req.kind == "fact":
        # Example: allow cache only if less than 5 min old
        MAX_FRESHNESS = 300
        assert age <= MAX_FRESHNESS, "Cached fact is stale – fetch live"
        return MemoryLayer.AUTHORITATIVE

    raise ValueError(f"Unsupported request kind: {req.kind}")

# ----------------------------------------------------------------------
# Example (not executed in production):
if __name__ == "__main__":
    r = Request(kind="volatile", created_at=_dt.datetime.utcnow(), ttl_seconds=30)
    assert select_layer(r) == MemoryLayer.WORKING

---

📜 Illustrative Architecture Decision Records

ADR 001 – Separate Working Memory from Durable Preferences

Context

Early prototypes stored both session‑level variables and user‑level settings in a single key‑value map. Privacy audits revealed that “delete my data” requests inadvertently left preference remnants, and latency spikes occurred because volatile data crowded the persistent store.

Decision

Introduce a distinct Working Memory store scoped to a session, backed by an in‑process LRU with a hard TTL (SESSION_TTL). Move all user‑level settings into Durable Preferences, persisted in a versioned KV table with an explicit deletion API.

Consequences

Positive	Negative
Faster per‑turn look‑ups (no I/O)	Additional serialization code for hand‑off between layers
Clear privacy boundary – deletion only touches Durable store	Slightly larger memory footprint per active session

Acceptance Criteria

• All “delete my data” automated tests verify that after the API call, no keys remain in the Durable store for the user.
• End‑to‑end latency for a 10‑turn conversation drops ≤ 15 % in load tests.

Uncertainty – The exact TTL value may need tuning based on observed session lengths; monitoring will guide adjustments.

---

ADR 002 – Cache Authoritative Facts Only When Freshness Is Proven

Context

The bot frequently queried a live tax‑rate service. To improve responsiveness, a naive cache without expiry was introduced, leading to regulatory violations when rates changed.

Decision

Implement a Authoritative Facts cache that stores the raw response together with cached_at. Every read must run a deterministic freshness predicate (now - cached_at < MAX_FRESHNESS). If the predicate fails, the system must invoke the live API; fallback to cache is prohibited.

Consequences

Positive	Negative
Guarantees compliance window (no stale rates beyond 5 min)	Extra latency on cache miss (must call live API)
Simpler audit – every fact entry records its source timestamp	Need to monitor cache hit‑rate to avoid overload of external service

Acceptance Criteria

• In integration tests, any simulated rate change after MAX_FRESHNESS must result in a different value being returned.
• Production monitoring shows > 95 % cache‑hit rate while never serving a fact older than MAX_FRESHNESS.

Uncertainty – External service latency variance may affect overall response time; adaptive back‑off strategies will be evaluated if miss latency exceeds a defined threshold.

---

✅ Evaluation & Maintenance Checklist

• [ ] TTL Enforcement – Unit tests verify that select_layer raises on expired volatile requests.
• [ ] Privacy Deletion – Automated job runs nightly to confirm no residual keys for users who invoked the deletion endpoint.
• [ ] Episodic Pruning – Scheduled job trims records older than MAX_EPISODE_AGE; post‑run count must be ≤ target retention size.
• [ ] Authoritative Freshness – Integration test injects a stale cache entry; request must trigger a live API call (mocked).
• [ ] Performance Regression – Benchmark end‑to‑end latency before and after layer separation; reject changes that increase 95th‑percentile latency by > 10 %.
• [ ] Regression Suite – Include prompts that require context retrieval, live‑value calls, and deterministic business‑rule evaluation; verify no drift after any code change.
• [ ] Documentation Sync – Ensure the mermaid diagram, table definitions, and ADRs are version‑controlled alongside code.

---

📚 References

1. Microsoft Retrieval‑Augmented Generation (RAG) concepts – https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
2. LoRA: Low‑Rank Adaptation of Large Language Models – https://arxiv.org/abs/2106.09685

---