Inside a Production LLM Memory System

From observations to durable knowledge, a practical architecture for making AI systems remember what matters.

You can explain what an AI application should remember.

Building a system that actually remembers it reliably is a very different problem.

In the previous article, LLM Memory Is Not Chat History: How to Design Memory That Improves Future Decisions, I focused on the conceptual side of memory: what deserves to become memory, how memories evolve, why retrieval is a decision rather than simple similarity search, and why forgetting is part of the lifecycle.

This article moves one layer deeper.

Suppose you're building an AI coding assistant. During a pull request review, it discovers that a pagination bug was caused by non-deterministic ordering. That observation may eventually become useful knowledge for future reviews.

But now the engineering questions begin.

Where should that observation be stored?

Should the application wait for an LLM to interpret it?

What happens if the same lesson already exists?

How do we decide whether it is a duplicate, a refinement, or a new revision?

How do we retrieve it later without returning an outdated version?

And how do we do all of this without turning a relatively simple AI application into a distributed system?

For the architecture we're building here, you don't need Kafka, a fleet of microservices, or a dedicated memory platform. A well-designed modular monolith can take you surprisingly far.

That's the architecture we'll build in this article.

We'll design a practical memory module with a canonical store, semantic retrieval, caching, asynchronous memory formation, identity resolution, and lifecycle management. The goal isn't to design the largest possible system. It's to design the smallest architecture that can behave like a serious production memory system and still remain simple enough to build, operate, and evolve.

If the previous article was about what memory means, this one is about what memory looks like when you actually build it.

---

How the Pieces Fit Together

The architecture is split into three complementary paths: write, read, and lifecycle.

The write path starts when the application produces an observation that may be useful later. The memory module persists it and creates a durable memory formation job, allowing the user-facing request to continue without waiting for memory processing. A background worker then extracts the relevant information, optionally uses an LLM for unstructured content, and produces a candidate memory. Identity resolution and admission determine whether that candidate should create, reinforce, revise, or supersede existing knowledge.

The read path works in the opposite direction. When a future request needs memory, the system first uses known structural signals to narrow the search, then combines semantic retrieval with revision and lifecycle filtering before ranking the candidates. Redis accelerates frequently accessed memories, while PostgreSQL remains the canonical source of truth.

The lifecycle path runs independently in the background, maintaining memories through reinforcement, consolidation, expiration, supersession, and deletion.

This separation keeps the user-facing path simple while allowing memory to form, evolve, and be retrieved independently.

That last sentence is important because it captures the architecture without repeating the detailed sections that follow.

---

Start With One Source of Memory Truth

The first version of a memory system should be deliberately boring.

Not because memory is simple, but because its semantics are already complicated. Adding infrastructure complexity before the workload requires it only makes the system harder to reason about.

For a modular monolith, I would start with one canonical datastore:

PostgreSQL with pgvector for semantic retrieval.

The important part isn't the database itself. It's having one place that answers:

What memories exist, which version is current, what evidence supports them, and whether they are still valid?

Redis can accelerate reads, and pgvector can help find semantically related memories, but neither should become the authority on what the application actually believes.

A useful mental model is:

An observation is something the application encountered.

A candidate is information that appears worth preserving but hasn't yet been admitted as durable memory.

A memory family represents one evolving piece of knowledge.

A memory revision represents a specific version of that knowledge.

Consider pagination:

Revision 1

Use offset pagination.

Revision 2

Use cursor pagination for large datasets.

Revision 3

Cursor pagination must use deterministic ordering with a stable secondary key.

These aren't three unrelated memories. They are revisions of the same evolving idea.

That makes immutable revisions useful. Instead of overwriting the previous value, each meaningful change creates a new revision while the memory family points to the current one.

PostgreSQL is a good starting point because it can keep the structured parts of this model together:

• Memory families and revisions
• Evidence and provenance
• Lifecycle state
• Temporal validity
• Background memory jobs
• Embeddings

With pgvector, semantic retrieval can live alongside that canonical metadata without introducing another datastore immediately.

This isn't a claim that PostgreSQL is the best database for every memory system. It's a deliberate starting trade-off.

A dedicated vector database may become worthwhile when corpus size, query volume, latency, or indexing requirements justify the additional operational complexity. Until then, keeping the system within one storage boundary makes it easier to reason about and operate.

The principle is simple:

Start with the smallest storage architecture that can preserve the semantics your memory system requires.

Let the workload earn the complexity.

And once we have a clear source of truth, the next question becomes more interesting:

How does information actually get into this system without slowing down the application that produced it?

That is where the write path begins.

---

The Write Path: Learning Without Blocking the Application

Once we have a canonical memory store, the next question is: how does information actually become memory?

The first design I'd avoid is making memory formation part of the user-facing request:

Interaction → LLM extraction → embedding → storage → response

That adds latency and makes the main application depend on an expensive, failure-prone background concern.

Instead, separate observing something from deciding what it means.

The important decision is to persist the observation first.

In a modular monolith, a durable memory_jobs table is enough to create this separation:

BEGIN
    INSERT observation
    INSERT memory_job
COMMIT

The request can continue while a background worker processes the memory job.

Because the formation job is durable, failed processing can be retried without losing the original observation or blocking the user-facing request. If an LLM provider is temporarily unavailable, memory formation can simply wait until the job can be processed again.

Deterministic Extraction First

The worker should not call an LLM simply because one is available.

If structured events already contain the information we need, extract it directly.

For example:

repository = payments-api
component = pagination
issue = unstable-ordering

There is no reason to spend an LLM call rediscovering those fields.

LLM-assisted extraction becomes useful when valuable information is buried inside unstructured conversations, reviews, or tool output.

Even then, the model produces a candidate, not canonical truth.

Identity Before Insertion

Once we have a candidate, we need to determine how it relates to existing knowledge.

It might be:

- NEW
- DUPLICATE
- REINFORCEMENT
- REFINEMENT
- REVISION
- SUPERSESSION
- CONTRADICTION

For example, if the existing memory says:

Use cursor pagination for large datasets.

and a new review discovers:

Cursor pagination requires deterministic ordering with a stable secondary key.

we shouldn't blindly create another unrelated memory. The new information may represent a refinement or revision of the existing memory family.

This is why the write path is more than:

message → embedding → vector database

The embedding helps us find related knowledge.

It does not decide what the application should remember.

The overall principle is simple:

Persist experience quickly. Interpret it asynchronously. Admit knowledge deliberately.

Once memory can be formed without blocking the application, we need the opposite path: how do we retrieve the right memory when a future request actually needs it?

---

The Read Path: Finding the Right Memory

Writing memory is only half the problem.

When a future request arrives, the system has to answer a harder question:

Which memories should influence this decision?

The first instinct is often to send the query directly to a vector index and take the nearest results.

That works as a candidate-generation mechanism.

It shouldn't be the final retrieval strategy.

A better read path starts with what the application already knows.

If the request is about a particular repository, component, environment, or entity, those signals should narrow the search before semantic retrieval begins.

For example:

repository = payments-api
component = pagination
environment = production
memory_type = engineering_rule

Semantic retrieval can then expand recall within that relevant space.

This matters because two memories can be semantically similar while only one actually belongs to the current problem.

Redis Is an Accelerator, Not the Truth

Redis can sit in front of the canonical store for frequently accessed memories.

Conceptually:

Request-local cache
        ↓
Redis
        ↓
PostgreSQL + pgvector

But the cache should never become the authority on memory correctness.

Revision-oriented keys make this easier:

memory:{family_id}:revision:{revision}
memory:{family_id}:current

When a new revision becomes current, the pointer changes. Older immutable revisions don't need to be overwritten.

This also gives the application a way to detect stale cached data rather than assuming the cache is always current.

And there is an important distinction from the previous article:

Cache freshness and memory freshness are different problems.

Redis can contain the latest revision while that knowledge is already temporally obsolete.

Retrieval Is a Ranking Decision

After candidate memories are found, the system still needs to decide which ones deserve influence.

Useful signals can include:

• Semantic relevance
• Exact scope match
• Temporal applicability
• Evidence quality
• Importance
• Reinforcement
• Redundancy

There is no universal weighting formula that works for every application.

The right ranking strategy depends on what the memory system is trying to optimize.

The important architectural boundary is that semantic similarity discovers candidates; the retrieval layer decides which candidates are actually useful.

That distinction keeps the memory module focused on returning relevant knowledge.

It also keeps context assembly outside this module. The memory system returns memory records. The application decides how those records should be used in the final model interaction.

Retrieval isn't a search problem. It's a decision problem.

And even a well-designed retrieval path isn't enough by itself. Memories continue to change after they're created, which means the system also needs a way to maintain them over time.

---

Memory Has to Maintain Itself

A memory system shouldn't stop working after a memory is created.

As new observations arrive, existing knowledge may become stronger, more precise, outdated, or completely wrong. That means memory needs a maintenance path just as much as it needs a write path and a read path.

The same background worker can handle these lifecycle operations:

The important part is that these operations should not require the application request to coordinate them synchronously.

A new incident might reinforce an existing memory today. A month later, several incidents might reveal a broader pattern worth consolidating. Eventually, an older memory may become irrelevant or be replaced by a newer revision.

These are background maintenance activities.

Consolidation Is Where Experience Becomes Knowledge

Suppose the application encounters several pagination failures:

Incident 1 → unstable ordering
Incident 2 → NULL ordering inconsistency
Incident 3 → incomplete cursor state

Individually, these are observations.

Together, they may support a more reusable lesson:

A cursor should encode the complete ordering state used by the database.

That derived memory is more useful than simply storing three increasingly similar incident summaries.

But consolidation should remain traceable. The derived memory should retain references to the experiences that support it. If later evidence contradicts the conclusion, the system should be able to understand where that knowledge came from.

This is also why recursively summarizing existing summaries is risky. Over time, information can drift further away from the original evidence.

Forgetting and Deletion Are Different

Lifecycle maintenance also needs a clear boundary between forgetting and deletion.

Forgetting means the memory should no longer influence normal retrieval.

Deletion means the underlying content should actually be removed or anonymized according to the application's retention policy.

For this architecture, deletion can remain deliberately simple:

Mark unavailable
      ↓
Invalidate Redis
      ↓
Remove vector representation
      ↓
Delete / anonymize canonical content

There is no need for a distributed deletion coordinator at this stage.

The broader principle is the same one we've followed throughout the architecture:

Keep expensive lifecycle work asynchronous, keep canonical state authoritative, and make every transition explicit.

At this point, the architecture is complete enough to build. But that raises an equally important question: when does this simple architecture stop being enough?

---

Where This Architecture Stops Being Enough

The architecture we've built is intentionally simple.

A modular monolith, PostgreSQL + pgvector, Redis, and a background worker can take a memory system surprisingly far. But it won't be the right architecture forever.

The important question isn't:

"When should I use microservices?"

It's:

"What constraint is the current architecture no longer handling well?"

For example, PostgreSQL may become the bottleneck if the memory corpus and retrieval workload grow beyond what the current setup can comfortably handle.

Semantic search may eventually require a specialized vector system because latency, throughput, or indexing requirements have changed.

Background memory jobs may grow large enough to compete with the application's primary workload, making independent worker scaling necessary.

Tenant isolation may become more demanding, or different teams may need independent ownership and deployment boundaries.

Multi-region requirements can introduce another class of constraints around data placement, availability, and consistency.

These are legitimate reasons to evolve the architecture.

But notice what they have in common.

None of them says:

"The system is production, therefore it needs microservices."

They are measurable constraints.

A natural evolution might eventually look like:

Modular Monolith
      ↓
Separate Worker
      ↓
Specialized Retrieval
      ↓
Independent Memory Service
      ↓
Distributed Architecture

The exact path will depend on the workload.

Some applications may never need to move beyond the first stage. Others may gradually extract individual components as scaling or ownership boundaries emerge.

That's why I would avoid designing the distributed version upfront.

Every new service introduces another network boundary, deployment surface, failure mode, and operational dependency. If the current architecture can satisfy the requirements with simpler infrastructure, that simplicity is an advantage.

The goal isn't to build the most distributed memory system possible.

It's to build the simplest system that satisfies the requirements, and let evidence justify the next layer of complexity.

Architecture should become more distributed because measurements and ownership boundaries demand it, not because "production-grade" means drawing more boxes.

That principle is useful far beyond memory systems. Good architecture isn't about predicting every future problem.

It's about leaving enough structure that the next problem can be solved without rebuilding everything.

---

Conclusion

Building memory for an AI application doesn't require starting with a distributed platform.

A modular monolith with a clear memory boundary, PostgreSQL as the source of truth, pgvector for semantic retrieval, Redis for acceleration, and a durable background worker can provide a surprisingly strong foundation.

The important part isn't the number of components.

It's the decisions behind them.

Observations should be persisted before they're interpreted. Expensive memory formation should happen asynchronously. Candidates should be resolved against existing knowledge before becoming durable memories. Retrieval should combine structure with semantics instead of trusting similarity alone. And memory should continue evolving after it is created.

Part 1 asked a conceptual question:

What should an AI application remember?

This article answered the architectural one:

How can we build a system that remembers it reliably?

The answer doesn't begin with a vector database.

It begins with a clear memory model, a canonical source of truth, and enough structure to let knowledge evolve without turning every application into a distributed system.

And when the workload eventually demands more, the architecture can evolve with it.

Start simple. Make the semantics strong. Let the workload earn the complexity.

---

📖 Blog by Naresh B. A.

👨‍💻 Backend & AI Systems Engineer | Distributed Systems · Production ML

🌐 Portfolio: [Naresh B A]

📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]

Thanks for spending your precious time reading this. It's my personal take on a tech topic, and I really appreciate you being here. ❤️