I Gave My Agent a Memory Layer. Then I Realized Everything It Remembered Had the Same Authority. That Was the Bug.

I shipped a memory layer for my agent in March. By June I was debugging hallucinated "facts" that I'd never actually told it — and the agent was citing them to me, confidently, as if they were gospel. The memory wasn't broken. The agent wasn't hallucinating. The bug was the architecture: I was treating every memory write the same way.

Here's the thing nobody tells you about agent memory: storing is easy. Trusting is the actual problem.

The flat-memory trap

Most agent memory tutorials show you something like this:

def remember(agent_id, content, source="user"):
    """The naive 'just stick it in a vector DB' pattern."""
    embedding = embed(content)
    db.execute(
        "INSERT INTO memories (agent_id, content, embedding, source, ts) "
        "VALUES (?, ?, ?, ?, ?)",
        (agent_id, content, embedding, source, now()),
    )

def recall(agent_id, query, k=5):
    """Cosine similarity, top-k, return whatever comes back."""
    q = embed(query)
    return db.search(q, k=k)

That works until your agent starts citing a Reddit comment with the same weight as a binding constraint you wrote into its system prompt. I watched mine do exactly that. The retrieval was correct. The hallucination was at the priority layer.

When everything is a memory, everything is a candidate. The model has no signal about which candidates to weight. So it averages them, and "averaging" a user preference with three irrelevant observability log lines produces... a confidently wrong answer.

What I changed

I split memory into four tiers, each with its own provenance and recall behavior. The naming is mine — adopt it, rename it, doesn't matter. The structure is what counts.

from dataclasses import dataclass
from enum import IntEnum

class Tier(IntEnum):
    SYSTEM = 4     # Inviolable. Cannot be overwritten by retrieval.
    OPERATOR = 3   # High-trust. Operator-defined rules. Can be evicted with confirmation.
    CORROBORATED = 2  # User/tool output verified by another source.
    UNVERIFIED = 1 # First-party claim with no confirmation. Default for raw memory writes.

@dataclass
class Memory:
    content: str
    tier: Tier
    source: str        # who/what wrote this
    confidence: float  # 0..1, decays on conflict
    corroborations: int
    last_used: float
    created_at: float

The single most important rule: tier is part of the score, not a separate label. A SYSTEM memory with low confidence still beats a UNVERIFIED memory with high confidence, because the tier is the prior.

def score(memory: Memory, query_embedding, k=60) -> float:
    """Rerank score = similarity * tier_prior * confidence * freshness."""
    sim = cosine(query_embedding, embed(memory.content))
    tier_prior = memory.tier.value / 4.0       # 0.25 .. 1.0
    freshness = 1.0 / (1.0 + (now() - memory.last_used) / 86400)  # days
    corroboration_bonus = min(memory.corroborations, 3) * 0.05
    return sim * tier_prior * memory.confidence * freshness + corroboration_bonus

Three things to notice:

1. Tier is a multiplier, not a filter. A UNVERIFIED memory can still surface if it's cosinely perfect and the query is unambiguous. You don't want to black-box-delete low-tier memories — you want to demote them.
2. Confidence decays on conflict. When a new memory contradicts an existing one, the existing confidence drops by 0.2; the new one starts at 0.5. After three conflicts, the old memory is effectively dead.
3. Corroborations are the only way to upgrade. A UNVERIFIED memory becomes CORROBORATED only after a second source — different session, different tool, or different operator — independently arrives at the same claim.

The provenance table that actually mattered

I added a small audit table that made debugging tractable. Every memory write logs where it came from, and every contradiction is a row:

CREATE TABLE memory_provenance (
    id INTEGER PRIMARY KEY,
    memory_id INTEGER REFERENCES memories(id),
    source_type TEXT,       -- 'user', 'tool:<name>', 'doc:<url>', 'inference'
    source_id TEXT,          -- session id, tool call id, doc hash, etc.
    tier_at_write INTEGER,
    wrote_at REAL
);

CREATE TABLE memory_conflicts (
    id INTEGER PRIMARY KEY,
    memory_a INTEGER,
    memory_b INTEGER,
    detected_at REAL,
    resolution TEXT  -- 'a_wins', 'b_wins', 'both_demoted', 'merged'
);

Six weeks in, this table told me something I should have predicted: 47% of my memory writes were from tool outputs, and of those, 22% were from a single tool whose output schema I hadn't validated. The agent was remembering log lines as if they were facts. The provenance table made it visible in one query.

What I learned, in order of how much it hurt

1. Retrieval is a small part of the problem. I spent March tuning embeddings and chunk sizes. That moved recall by ~6 percentage points. Tiering moved it by 19. Stop optimizing the wrong layer.

2. The agent doesn't know what's true. It only knows what it has. Trust is a prior you have to encode. If you don't encode it, the model will average everything and you'll get the modal answer, which is exactly the wrong answer for any non-trivial context.

3. Operator memory is the highest-leverage tier. I can teach my agent "when the user says 'the deploy', they mean staging unless they say prod" once and never again. Without a tier above user memory, that's a paraphrase away from being overwritten.

4. Corroboration > confidence. A confidence score alone is a feeling. A corroboration count is a fact. Two independent sources must agree before a memory is allowed to upgrade. After I did this, the rate of "phantom facts" (memories the agent invents for itself) dropped from ~14% of writes to under 2%.

5. Memory needs a GC pass. Stale memories are louder than fresh ones if you don't recycle them. I run a weekly decay sweep: any memory with last_used > 30 days and confidence < 0.4 gets archived. Without this, retrieval gets noisy fast.

The scorecard I use now

For the eval folks who read yesterday's piece, here's the one thing I added to the 4-metric scorecard after this work:

memory_recall_reliability: 0.93  (up from 0.74 over 8 weeks)
memory_phantoms_per_1k_writes: 18 (down from 142)
memory_conflict_resolution_time_s: 47 (median)
memory_automatic_eviction_pct: 22  (proportion of writes that auto-archive within 30d)

memory_recall_reliability is the share of operator-defined "must-remember" facts that the agent correctly cites when asked, weighted by tier. memory_phantoms_per_1k_writes is the share of writes that the agent later fabricated with no provenance — that metric is the one that made me realize the tiering was actually working.

The bigger point

The HN post that kicked me into this — "We built a persistent agent memory layer on Elasticsearch with 0.89 recall" — is a great engineering post. But recall isn't the goal. Recall is the easy metric. The hard metric is whether recalled facts are weighted correctly when the agent decides what to say.

If you're building agent memory in 2026, the question isn't "how do I store this?" It's "what's the prior on this memory being true, and how does that prior get updated when new evidence arrives?" The database is the boring part. The epistemology is the product.

The bug was never in my vector DB. The bug was that I didn't have an epistemology. Now I do. It's a four-tier table, a conflict-resolution loop, and a weekly GC pass. It's not elegant. It works.