Giving a fleet of AI agents one shared memory — when each agent runs a different model

Dexter N — Tue, 11 Aug 2026 21:05:49 +0000

Most agent frameworks give each agent its own context window and call it memory. That works right up
until you run more than one agent, and then it quietly becomes the most expensive design decision in
the system.

We run a fleet where different agents are deliberately backed by different models — one family
handles long-form drafting, another handles structured extraction, a couple run on a local path with
no external inference at all. Routing by capability is the easy part. The hard part is that an agent
which learns something has learned it alone.

This is a writeup of what broke, and the design we ended up with.

The failure mode

The symptom shows up as repeated work.

An extraction agent determines that a particular vendor's invoices put the tax line above the
subtotal. Useful. Two days later a different agent — different model, different prompt, same
pipeline — hits the same vendor and re-derives it from scratch. Then a third does it again.

Nothing is wrong. Every agent behaves correctly. The system as a whole just has no way to
accumulate anything, because knowledge lives inside whichever context window happened to be open at
the time. You are paying inference costs to rediscover facts you already own.

The naive fix is to pass more history. That fails for a specific reason worth naming: context
windows are per-invocation and per-model. A 200k window on one model does not help an agent
running a different model with a 32k window, and neither survives the session ending. You cannot
solve a persistence problem with a bigger buffer.

What "unified memory" has to mean

Once you accept that memory has to live outside the agents, the requirements get concrete:

Model-agnostic storage. If memory is stored as one model's embeddings, you have coupled your memory layer to a vendor. Swapping models later means reindexing everything.
Written by one agent, readable by all. Otherwise you have per-agent memory again, with extra steps.
Attributable. When memory is wrong — and it will be — you need to know which agent, which session, and when. Un-attributable memory is unfixable memory.
Scoped. Not every agent should see everything. A support agent has no business reading finance context.
Auditable. For anything touching regulated work, "the agent knew X" needs to be a claim you can evidence, not infer.

Point 3 is the one people skip, and it is the one that hurts. A shared memory store with no
provenance turns every bad output into an unbounded investigation.

The shape we landed on

Two layers, deliberately separated:

An append-only event log is the source of truth. Every memory write is an event with the agent
identity, session identity, channel, and timestamp attached. It is never mutated. If a fact turns
out to be wrong, you append a correction — you do not edit history. This is the layer that makes
point 3 and point 5 possible, and it is boring on purpose.

A derived index is what agents actually query. It is rebuilt from the log, which means it is
disposable. Change your embedding model, change your chunking, decide semantic search was the wrong
call for a given path — rebuild the index, the log is untouched.

The important property is the direction of the dependency. The index depends on the log. Nothing
depends on the index. That is what lets you swap retrieval strategies without a migration, and it
is the difference between "we changed our embedding model" being an afternoon or a quarter.

Our current implementation runs a document-indexing backend over configured paths with a scheduled
embed cycle, plus session export with a retention window. The specific backend matters much less
than the split — we have changed it once already and the log made that a non-event.

Retrieval is a scoping problem, not a search problem

The instinct is to make retrieval smarter. Better embeddings, reranking, hybrid search.

In practice the wins came from narrowing what is searchable per agent before ranking anything.
An agent asking about invoice formats should not be searching across support transcripts. Not for
quality reasons — for correctness reasons. Cross-domain semantic neighbours are exactly the kind of
plausible-but-wrong context that produces confident nonsense.

Scope first, then rank. A small correctly-scoped candidate set beats a large well-ranked one, and it
is dramatically cheaper.

What we got wrong

We stored summaries too early. Summarising a session into memory at write time felt efficient
and destroyed the ability to re-derive anything later when we changed our minds about what mattered.
Store the raw event; summarise at read time if you need to.

We under-specified identity. Early on, "which agent wrote this" meant an agent name, which we
then renamed. Use a stable identifier that survives renaming, and record the model separately —
you will want to answer "did the model change when the quality dropped?"

We assumed retrieval failures were retrieval failures. Most were scoping failures wearing a
costume.

Does this actually pay off

Honestly: it depends on fleet size. With one or two agents, per-agent context is fine and this
architecture is overhead you do not need. The crossover came for us somewhere around five or six
agents sharing a domain, where the rediscovery cost and the "why did it do that" investigations
started dominating.

The clearest signal that you have crossed it: you find yourself explaining the same fact to
different agents, or you cannot answer why an agent produced a given output without reading raw
logs. Both are memory-architecture problems presenting as prompt problems.

We build and deploy governed AI agents for Singapore businesses at
VYR — every agent runs with a human approval gate on any action that writes,
sends or pays, plus a full execution log. If you are working on multi-agent memory, I would genuinely
like to compare notes in the comments.