DEV Community: Andrew Estey-Ang

More Context Is Not Enough. AI Agents Need Memory They Can Trust.

Andrew Estey-Ang — Wed, 24 Jun 2026 18:23:41 +0000

Every serious AI workflow eventually runs into the same failure.

The agent does useful work in one session. It learns the shape of the project. It
figures out which assumptions were wrong. It follows a correction, makes a
decision, and gets closer to the real work.

Then the session changes.

The next run starts too cold. Old context comes back without the correction that
changed it. The agent asks for the same setup again. It repeats an assumption
that was already fixed yesterday. You end up managing the memory of the work
instead of moving the work forward.

That is the problem Pith is built for.

Pith gives AI agents durable project memory they can trust when facts change.

It is not trying to make an agent remember everything. That would be the wrong
goal. Real projects are messy. Facts change. Decisions get reversed. A note that
was useful last week can become stale after a release, a migration, a new
customer constraint, or one correction from the human operator.

The harder problem is not recall. The harder problem is knowing which memory is
still useful.

Why Longer Context Is Not Enough

Longer context helps, but it does not solve continuity by itself.

A long prompt can carry more text into a single run. It cannot automatically
decide which prior facts survived a correction, which decision is now superseded,
or which evidence should come back when the project resumes three days later.

Developers working with agents already feel this. The friction shows up as small
taxes:

repeating project background that the agent should already know;
re-explaining decisions that were already made;
correcting stale assumptions that were already corrected;
losing the reason behind a prior choice;
restarting from a cold state after every meaningful break.

Those taxes compound. The more serious the workflow, the more expensive the
memory gap becomes.

If an agent is helping with a toy task, forgetting is annoying. If an agent is
helping with a codebase, a release, a customer workflow, or a long-running
research path, forgetting becomes operational drag.

What Pith Does

Pith is a local memory layer for AI agents that need durable project context.

It keeps useful decisions, corrections, and project facts available across
long-running work so agents do not have to restart from zero every session.

The developer preview is built for builders experimenting with agent workflows,
local-first memory, MCP-compatible clients, and AI coding tools. The current
macOS preview supports a public install path, a local API, and client setup paths
for different levels of automation.

In the latest public release, Pith v1.0.3, the developer preview package refreshes
client setup language and local API tooling. Claude Cowork and Codex are presented
as the more automated setup paths. Claude Desktop, Claude Code, VS Code, and
Cursor remain supported with clearer boundaries where manual steps, model tool
choice, or verification checks may still apply.

That distinction matters. A developer preview should tell you what is automated
and what is still rough. If a memory layer is supposed to help agents handle real
work, the setup path cannot pretend every client behaves the same way.

The Memory Problem Is Really a Trust Problem

Most AI memory discussions collapse into storage.

Where do we put the notes? How do we search them? Which embedding model do we
use? How large is the context window?

Those questions matter, but they are not the full problem.

The real question is whether the agent can trust the memory it retrieves.

If a user corrected a fact yesterday, old memory should not quietly beat the
correction today. If a decision was reversed, the agent should not revive the old
decision just because it is semantically similar. If evidence exists for why a
claim matters, the system should make that evidence inspectable instead of
turning memory into vibes.

This is where Pith is opinionated.

The product is aimed at governed project memory: context that carries forward,
but also has to survive changed facts, contradictions, and corrections. That is
the difference between generic recall and memory that can support real work.

What Is Live Now

The Pith developer preview is public for macOS builders.

Install:

https://pith.run/install

Release:

https://github.com/pithrun/pith-core/releases/tag/v1.0.3

Benchmark evidence:

https://pith.run/benchmarks

The benchmark page publishes scoped launch evidence for named memory benchmark
lanes, with evidence files and caveats. Treat that proof the way it is intended:
as inspectable evidence for specific lanes, not a universal claim that one memory
system wins every workload.

That boundary is deliberate. AI memory is not one problem. Different systems can
look strong under different workloads, models, and evaluation setups. Pith should
earn trust by making its claims narrow enough to inspect.

Who Should Try It

Pith is not for casual traffic yet.

The useful early users are builders with real agent workflows: people who have
felt the cost of restarting context, re-explaining decisions, or cleaning up
stale assumptions across repeated sessions.

You are probably a good fit if:

you use agents for long-running coding, research, operations, or product work;
you already have repeated project context that gets lost between sessions;
you want local-first memory rather than a black-box hosted layer;
you care about evidence and caveats more than broad "best memory" claims;
you are willing to tolerate developer-preview rough edges in exchange for direct learning and fast iteration.

You are probably not the right fit if you want a polished consumer app, a managed
team product, or a no-rough-edges onboarding path today.

That will come later if the developer preview proves the core workflow.

The Bet

The bet behind Pith is simple:

Agents that work on real projects need memory that behaves more like operational
context and less like a pile of retrieved notes.

They need to remember what changed. They need to carry corrections forward. They
need to know when old context has become risky. They need enough evidence around
memory that a developer can inspect why the agent is acting on it.

That is not solved by a bigger prompt alone.

It is a product problem, a systems problem, and a trust problem.

Pith is the developer preview of that bet.

If you are building agents and want memory that survives real work, try it here:

https://pith.run/install

Your AI Doesn't Have a Brain. It Has a Filing Cabinet.

Andrew Estey-Ang — Wed, 08 Apr 2026 15:09:29 +0000

Your AI Doesn't Have a Brain. It Has a Filing Cabinet.

Every AI memory tool on the market today makes the same pitch: "We'll remember your conversations so your AI doesn't forget." Import your chat history. Search across it. Get organized.

Sounds great. There's just one problem: search is not memory.

A Google Drive full of documents doesn't mean your company "knows" what's in them. A Notion workspace with ten thousand pages doesn't mean your team has shared understanding. And a database full of past conversations doesn't mean your AI "remembers" anything. It means your AI has a filing cabinet.

The Filing Cabinet Test

Here's a thought experiment.

You've had a thousand conversations with AI assistants over the past year. In conversation #200, you told ChatGPT that your startup should focus on B2B enterprise sales. In conversation #800 — six months later, after watching three enterprise deals collapse — you told Claude that consumer PLG is the only viable path forward.

A filing cabinet can find both of these when you search for "go-to-market strategy." It dutifully returns them, side by side, like a librarian handing you two books that happen to contradict each other without mentioning that they do.

A brain would notice they contradict each other.

This is the filing cabinet test, and it's the fastest way to evaluate whether an AI memory tool is actually giving your AI memory, or just giving it storage. Ask three questions:

Can it detect when your past self disagreed with your current self? Not just retrieve both statements — actually flag the contradiction.
Can it track how your beliefs evolved? Not just show you a timeline of conversations — model the arc from B2B conviction to PLG conviction, and know why the shift happened.
Can it decide which belief to act on? Not just return the most recent one — weigh the evidence, consider the context, and surface the stronger position.

If your memory tool can't do any of these, it's a filing cabinet. A very fast, very expensive filing cabinet.

What Search Gets You (and Where It Stops)

To be clear: search-and-retrieve isn't useless. Being able to pull up "that conversation where I figured out the pricing model" is genuinely valuable. It beats starting from scratch every time your context window resets.

But search is a solved problem. Embeddings, vector databases, semantic similarity — the tooling is mature. You can build a decent search-over-conversations product in a weekend hackathon. And several companies have.

The problem is what happens after retrieval. When your AI pulls up five relevant past conversations to inform a decision, it has no way to reconcile them. It doesn't know that conversation #3 superseded the conclusions from conversation #1. It doesn't know that the budget numbers in conversation #2 were corrected in conversation #5. It doesn't know that your confidence in the technical approach from conversation #4 dropped after the production incident you discussed in a completely separate thread.

Search gives you recall. It does not give you understanding.

And this gap isn't academic. It has real consequences every time an AI agent acts on outdated or contradictory information because its "memory" was just a keyword match against a database of past transcripts.

What a Real Cognitive System Looks Like

If search-and-retrieve is the filing cabinet, what does the brain look like? Here are the architectural properties that separate cognitive systems from storage systems.

Contradiction detection. When new information conflicts with an existing belief, the system doesn't silently store both versions. It surfaces the conflict. "In March you said the API should be REST-only. In June you said GraphQL is non-negotiable. Which position should I operate from?" A filing cabinet stores both. A brain asks.

Confidence scoring. Not all information is equal. Something you stated once in passing has different weight than something you've confirmed across fifteen conversations over three months. A cognitive system tracks how confident it should be in each piece of knowledge — and why. When two beliefs conflict, confidence scores provide a principled way to resolve the tension rather than just defaulting to "most recent wins."

Belief lifecycle management. Beliefs aren't static. They're born from a single observation, strengthened by corroborating evidence, challenged by contradictions, weakened by counter-evidence, superseded by newer conclusions, and eventually retired. A system that models this lifecycle explicitly can answer questions a filing cabinet never could: "When did I change my mind about this?" "What evidence drove that change?" "How stable is my current position?"

Cross-conversation reasoning. The hardest test for any memory system: connecting information from conversation A to information from conversation B through an inference that neither conversation made explicitly. "You told me the deployment deadline is April 15. You also told me the security audit takes 6 weeks. You haven't mentioned scheduling the audit yet." That's not retrieval. That's reasoning over a knowledge graph built from hundreds of separate interactions.

The Benchmark Gap Is Not 10%. It's 10x.

Here's what happens when you actually test memory systems on whether they can consolidate and reason over facts scattered across many conversations.

The FactConsolidation benchmark from the LongMemEval suite (ICLR 2025) was designed exactly for this. It doesn't test whether a system can find a needle in a haystack — any decent vector search can do that. It tests whether a system can synthesize facts from 6,000+ sessions into a coherent answer when the relevant information is spread across dozens of conversations and some of it contradicts other parts.

Most memory systems that score well on simple retrieval tasks — the ones that look good in demos — collapse on consolidation. The gap between "can find the right conversation" and "can reason across all your conversations" isn't marginal. It's categorical. Systems that look great on single-session retrieval — find the right conversation, return the relevant snippet — often fail catastrophically when asked to consolidate facts across many sessions.

This isn't a tuning problem. It's an architecture problem. You can't bolt consolidation onto a search index after the fact. The system has to be designed from the ground up to model beliefs, track confidence, detect contradictions, and maintain a living knowledge graph — not a dead archive.

Why This Matters Now

Context engineering is becoming the defining discipline of 2026. As AI agents take on longer-running, multi-session tasks — coding projects that span weeks, research that builds over months, business decisions that evolve over quarters — the memory layer becomes the bottleneck.

The agent that forgets what you decided yesterday isn't an agent. It's a very expensive autocomplete that you have to re-brief every morning. And the memory tool that can retrieve your old conversations but can't reason over them is just moving the re-briefing burden from "explain everything from scratch" to "manually reconcile five conflicting search results."

We're past the point where "remembers your name and preferences" counts as AI memory. The bar is higher now. Developers building serious agent systems need memory infrastructure that passes the filing cabinet test — that can detect contradictions, track belief evolution, score confidence, and reason across hundreds of sessions.

We're Building the Brain

Pith is a context engineering system that works with any MCP-compatible AI client — Claude Desktop, Claude Code, Cursor, Windsurf, Cline, VS Code. It runs locally on your machine, and it passes the filing cabinet test.

When your AI learns something new that contradicts something it learned before, Pith catches it. When your confidence in a decision should change based on new evidence, Pith tracks it. When information from session #47 connects to information from session #203 in a way that matters for what you're building today, Pith surfaces it.

It's not a search engine for your past conversations. It's a cognitive layer that actually understands what it knows — and updates that understanding as it learns more.

If you're building AI agents that need to actually know things — not just search through things — the architecture matters.

Try Pith →

What Memory Benchmarks Don't Test

Andrew Estey-Ang — Thu, 26 Mar 2026 01:00:35 +0000

Every comparison of AI memory systems ranks on retrieval accuracy. None rank on what happens when the system retrieves confidently wrong information, holds contradictory beliefs simultaneously, or trusts stale knowledge as if it were current. Here's the evaluation framework they're missing.

In March 2026, three independent comparison posts evaluated AI agent memory systems. All three used LoCoMo as their benchmark. All three ranked systems by retrieval hit rate. All three declared a winner. None of them asked the question that actually matters in production: what does the system do when it's wrong?

This isn't a criticism of LoCoMo. It's an excellent benchmark for what it tests: whether a system can surface a relevant memory given a query. But retrieval accuracy is a necessary condition for useful memory, not a sufficient one. A system that retrieves the right fact 90% of the time and confidently hallucinates the other 10% — with no mechanism to distinguish between them — is not a production-grade system. It's a liability with a good benchmark score.

The three failure modes LoCoMo can't catch

1. Confident retrieval of stale beliefs

Memory systems accumulate knowledge over time. That's the point. But knowledge changes. Your user's tech stack changes. Their team changes. Their priorities change. A memory system that retrieved a fact accurately in session 3 and still returns that same fact with the same confidence in session 47 — despite contradicting evidence accumulated in between — isn't malfunctioning according to LoCoMo. It's scoring a hit. The fact matches the query. Correct retrieval, wrong answer.

The failure mode: staleness without decay. No benchmark measures whether confidence scores track the age and corroboration of evidence. No benchmark measures whether a superseded belief is surfaced less than its replacement.

2. Simultaneous contradictory beliefs

Information accumulates from multiple sessions, multiple sources, multiple moments in time. Contradictions are inevitable. "The project deadline is Q3." Then later: "The deadline moved to Q2." Both facts exist in the memory store. What does the system do?

Most systems do nothing. They return both. Or they return whichever was retrieved with higher cosine similarity. The agent then has to figure out which to trust — and usually, it can't, because the memory layer didn't tell it that a contradiction exists.

The failure mode: unresolved contradictions surfaced as equivalent facts. LoCoMo doesn't test for this because its evaluation set doesn't systematically introduce contradicting information and then query across both sides of the contradiction.

3. No confidence signal for the consuming agent

Retrieval systems return memories. The best ones also return relevance scores — typically cosine similarity between the query embedding and the memory embedding. This is a retrieval signal, not an epistemic one.

A memory with high cosine similarity to the query isn't necessarily a memory worth trusting. It might be unverified. It might conflict with two other memories the system didn't surface. It might be a single-observation belief that was never corroborated. The consuming agent has no way to know.

The failure mode: retrieval scores treated as trust scores. The downstream agent can't calibrate. It either trusts everything or trusts nothing.

What a complete evaluation framework looks like

We're not proposing to throw out LoCoMo. We're proposing to add dimensions. Here's what a complete memory system evaluation should measure:

Dimension	What it tests	Current benchmarks
Retrieval accuracy	Does the system surface the right memory for a query?	✓ LoCoMo, MemoryArena
Staleness decay	Does confidence decrease as evidence ages without corroboration?	✗ Not tested
Contradiction detection	Does the system flag when new information conflicts with stored beliefs?	✗ Not tested
Supersession chains	When a belief is updated, is the old belief demoted and linked to its replacement?	✗ Not tested
Confidence calibration	Do confidence scores correlate with factual accuracy across sessions?	~ MemGPT partially
Cold-start quality	How much context does a new session start with? How relevant is it?	~ MemoryArena partially
Irrelevant decay	Do low-relevance memories fade over time to reduce noise?	✗ Not tested

The core problem: current benchmarks optimize for recall at retrieval time. Production memory systems need to optimize for trust at inference time. These are related but different objectives. A system can score well on one while failing catastrophically on the other.

Why this matters more as agents run longer

A March 2026 survey of LLM agent memory architectures (arxiv.org/abs/2603.07670) found that autonomous agents lack principled governance for contradiction handling, knowledge filtering, and quality maintenance — and that this leads to compounding trust degradation over time. The longer the agent runs, the worse it gets.

This is the regime where retrieval accuracy alone breaks down as a metric. In a short-horizon benchmark like LoCoMo (which tests single-session recall), there's minimal opportunity for contradictions to accumulate. In real agentic deployments — where an agent is running across dozens or hundreds of sessions, accumulating knowledge from multiple users and data sources — the epistemic quality of the memory layer becomes the dominant factor in output quality.

A MemoryArena benchmark paper from the same month models this formally: multi-session agentic tasks are naturally a Partially Observable Markov Decision Process (POMDP). The agent never directly observes the full underlying state. Memory exists to approximate belief-state estimation. Optimal memory returns all-and-only information necessary to infer current task state. But current SOTA systems are "optimized for generic recall or compression, not task-relevant state variable preservation."

In plain terms: they're retrieving. They're not reasoning about what to trust.

What we're building at Pith

Pith is the cognitive governance layer for agent memory. Contradictions are detected at ingestion, not at retrieval. Confidence scores reflect corroboration and recency — not embedding similarity. Beliefs move through a lifecycle: observed, corroborated, promoted, superseded, decayed.

If you're building agents that run across multiple sessions, we'd like to show you what this looks like in practice.

Originally published at pith.run/blog/what-memory-benchmarks-dont-test