DEV Community: Shrawan Saproo

We Scoped Down Our Agent Skill Library. You Might Not Need One at All.

Shrawan Saproo — Tue, 22 Sep 2026 09:47:15 +0000

Part 3 of a three-part series on the memory architecture behind our multi-agent harness. Part 1 covered why agents should share memory instead of passing context. Part 2 covered how the semantic tier handles users who change their minds.

"A library of skills the agents learn." It is a seductive phrase, and it was our original plan for the third tier of our memory architecture: an agent solves something, the system captures the skill it used, and the skill is stored for every agent to reuse. The library fills itself by watching agents work.

That plan does not survive contact with how agents are actually deployed. This post is the honest version of what the procedural tier (L3 in our shorthand) can and cannot do, why we deliberately dropped skill-learning-by-observation as a design goal rather than deferring it, and the one question that decides whether an organization should build this tier at all. By the end, the entire decision compresses into a single yes-or-no question. Spoiler on the answer: sometimes it is no, and we think saying so is a feature of the architecture rather than a gap in it.

First, what a skill actually is

Most of the confusion around skill libraries traces back to definitions, so here is the line we hold.

A tool is a raw capability: calling a PDF library, hitting an API. Tools are wired into the agent. They do not live in memory.

A fact is about a user or the world: "this client wants a house in five years." Facts belong in the semantic tier.

A skill is a procedure: "to extract totals from our vendor's invoices, the total is in the footer, dates are DD-MM-YY, validate the sum against the line items." Learned, reusable, and specific to how one organization operates.

The test for whether something belongs in the procedural tier is that it is procedural, non-trivial, reused, and specific to how you operate. If it is a raw ability, it is a tool. If it is about someone, it is a fact. And if the base model already knows it, it is noise. Keeping that line sharp is what stops the tier from becoming a dumping ground.

Where a skill library genuinely pays off

Four situations, and it is worth noticing they are not four kinds of content. They are four situations in which the same stored artifact gets used, differing only in who benefits.

Avoiding re-derivation. An agent that re-reasons the same procedure on every run can retrieve the proven version and follow it instead. Cheaper, and it cannot get the logic wrong this time, because it is not reinventing it.

Cross-agent transfer. A procedure worked out by one team's agent becomes available to a different agent that never solved it. This is the only case that truly needs organization-level scope; within one agent, it is just memory.

Consistency. Shared conventions, output formats, phrasing contracts, live in one place, so every agent behaves the same way instead of each improvising and breaking whatever consumes its output.

Onboarding. A newly deployed agent inherits the accumulated procedures immediately and starts competent rather than naive.

All four share one mechanic: an agent describes a need, and the library returns a procedure that was authored once and reused. Hold onto the word "authored." It is about to matter.

The wall: you cannot capture skills from a hosted API call

Here is where deployment reality bites. Most production agents today are calls to hosted models. The call returns output, and nothing else. There is no reasoning trace, and there is not even a discrete "skill" inside a completion to capture. The model produced tokens.

You cannot store a skill you never observed.

So any attempt to extract "the skill the agent used" from a response is reverse-engineering a plausible-looking procedure from the answer alone, with no way to tell whether it reflects what actually produced the result. Our own episodic log does not rescue this either: it records agent outputs, not reasoning traces, so mining it for procedures hits exactly the same wall.

In our assessment, this failure mode is worse than storing nothing. A confident, clean-looking skill that does not reflect reality will get retrieved, trusted, and reused, and then it will fail somewhere downstream where nobody is looking for it. An empty library is honest. A library of plausible guesses is a liability.

The conclusion we reached is worth stating as bluntly internally as we did: populating a skill library by observing running agents is a dead end for hosted models, and we dropped it as a design goal rather than deferring it. It becomes worth revisiting only if two things become true at once: faithful action traces are captured, and we own prompt construction for that agent. Neither is on the horizon for the deployments we see.

The reframe that keeps the tier usable

The black-box call only kills the tier if you think the tier's job is to observe skills from runs. It is not. There are two directions of traffic, and only one of them is affected.

Writing to the library happens by authoring. A person writes a skill file. A vetted skill is imported from a public library. An agent is explicitly asked to write one up after a success, as a deliberate act with review. In every case the skill is created on purpose as an artifact, never harvested passively from a run. Authored, not observed.

Reading from the library works fine for a black-box agent. The orchestration layer retrieves the relevant skill and injects it into the prompt before the API call. You do not need to see inside the model. You control the input to it.

That last clause is the real precondition, and it is the question we now ask before anything else in an integration. Not "can we see inside the model," because nobody can. The question is: **do you control the prompt that goes into it? **If the agent is custom and your platform assembles its prompt, the skill tier works. If it is a sealed third-party agent product where you only hit an endpoint, the tier has nothing to attach to for that agent, and no amount of library curation changes that.

The harder question: store skills, or just download them?

Even granting all of the above, there is a fair challenge that deserves a straight answer: why curate a library speculatively when an agent could fetch a skill from the public ecosystem at the moment it needs one?

One distraction to clear first: storage cost is a red herring. Skills are small text files; a few hundred of them amount to megabytes. The real cost of pre-storing is human effort, the selecting, reviewing, security-scanning, and maintaining of the library. So the honest comparison is curation effort versus fetching on demand.

Download on demand wins when skill needs are unpredictable and long-tailed (you would curate five hundred and use twelve), when the skills are generic (read a PDF, scrape a page, parse a CSV) and the public ecosystem already maintains them better than you will, and when nothing the agent does is specific to your organization. For a team running hosted-model agents on generic tasks, that is usually the whole picture, and the right recommendation is to skip the library and fetch.

The one question that decides it

Per deployment, everything above compresses into a single question: does this agent rely on procedures that are either unique to the organization or too sensitive to run unreviewed?

If yes, that slice justifies a small, curated procedural tier, populated by authoring and read by prompt injection.

If no, download on demand and do not build the tier. And if that answer holds for a given deployment, we would rather say so than sell a tier that sits half-empty. An architecture that tells you when one of its own components is unnecessary is, we would argue, more trustworthy for it.

That closes the series. The episodic tier is the append-only truth. The semantic tier is one belief per fact, with the history to prove it. And the procedural tier is smaller than the hype suggested, on purpose, and everything in it is authored, not observed.

Memory Architecture Series Part 1: Agents Shouldn't Pass Context. They Should Share Memory. Part 2: One Belief Per Fact: Building Agent Memory for Users Who Change Their Minds Part 3: We Scoped Down Our Agent Skill Library. You Might Not Need One at All. (this post)

One Belief Per Fact: Building Agent Memory for Users Who Change Their Minds

Shrawan Saproo — Tue, 22 Sep 2026 09:37:59 +0000

Here is a scenario that breaks most agent memory systems.

In session one, a user tells your financial advisory agents she is cautious with money and dislikes individual stocks. The system remembers it. In session two, she has changed her mind: she wants to go aggressive, individual tech stocks. The system remembers that too.

Now session three begins, and an agent asks memory: what is her risk appetite?

If your memory layer stores a row per session, it returns both answers, side by side, with no way to choose. We know because that is exactly what our first version did. This post is about how the semantic tier (L2 in our shorthand) actually works: what gets embedded, how beliefs update, and what happens on a contradiction. We will use one running example throughout, a user named Priya, drawn from the synthetic test scenario we ran the design against.

How we broke it first

The first cut of the semantic tier keyed facts by session. Every session inserted new rows, so after a handful of sessions there were six copies of Priya's risk tolerance, some of them contradicting each other, and no notion of "current." Retrieval surfaced whichever ones ranked highest, which meant a downstream agent could receive "cautious" and "aggressive" in the same context window.

Two other bugs from that first version are worth confessing, because they shaped the fixes. The transaction analysis agent invented income growth it was never told about, which led to a hard grounding rule: only figures explicitly stated in memory, otherwise answer "unknown," never estimate. And our "semantic" search was not semantic at all: the store had been created without an embedding index, so nothing was ever vectorized, and we were doing keyword matching while believing otherwise. The fix included a runtime check that the vector table is populated and that a paraphrased query actually retrieves the right belief.

The structural fix for the main bug is the subject of this post: one belief per fact, current, with the versioned history to prove it. That phrase is the whole design; everything below is what it takes to make it true.

What actually gets embedded

There are two embedding moments, and keeping them separate clears up most confusion about how retrieval works.

The write side. After a session, an asynchronous projector reads the raw episodic log and asks an LLM to break each entry into typed, self-contained fact sentences. From Priya's intake record it extracts something like:

Only one field is embedded: the text sentence. The store is configured to embed that field alone. The sentence goes to an embedding model and the resulting vector is stored in a separate vector table. Everything else, the key, the version, the flags, the provenance, is plain metadata for identity, filtering, and audit. It is never embedded.

The read side. When an agent asks "what is her appetite for investment risk?", the query sentence is embedded with the same model, and the nearest stored vectors win. The query and the stored fact share almost no vocabulary and still match, because both are compared by meaning.

One design point matters more than it first appears: identity never depends on the embedding. Which belief is which is decided by the stable key. The embedding only decides ranking at retrieval time. A weak embedding model can degrade what surfaces first; it can never corrupt the record.

The division of labor compresses to three words each: embeddings retrieve, keys identify, metadata governs. If you take one thing from this section, take that.

One current belief, a version chain behind it

The stable key is the fact type: risk_tolerance, house_goal, city. It is independent of session. That single change inverts the growth behavior of the tier: it grows with the number of distinct facts about the user, not with usage. Ten sessions restating "cautious" still produce one belief.

The update procedure is search-then-decide. When the projector extracts a new fact, it first fetches the existing belief under that key, then an LLM compares old against new and returns exactly one of three verdicts:

SAME: no new information. Append this session to the provenance list. No new version.
UPDATE: a compatible refinement or change. Archive the old version, write the new one.
CONFLICT: a direct contradiction. Archive and supersede, and set the contradiction flag.

Here is Priya's flip, before and after. After session one:

Session 1
risk_tolerance -> {
  text: "cautious, dislikes stocks",
  version: 1,
  current: true,
  flag: false
}

Session 2
risk_tolerance -> {
  text: "aggressive, individual tech stocks",
  version: 2,
  current: true,
  flag: true,
  provenance: [S1, S2]
}

risk_tolerance__v1 -> {
  text: "cautious, dislikes stocks",
  version: 1,
  current: false
}

There is a history, but it is a version chain hung off one stable key, not a pile of equal rows. The current belief is the single row at the bare key. Past versions are archived with current: false and are not read by default.

Retrieval then needs two mechanisms working together: vector search finds the relevant fact by meaning, and a current: true filter picks the live version. The agent sees one clean answer, the newest belief, instead of the version-one situation where both came back with no way to choose.

The contradiction policy: flip, preserve, flag

When a CONFLICT is detected, three things happen at once, and the design is deliberately not "silently overwrite."

First, the current belief flips to the new value. Agents act on the user's most recent intent; the default read is most-recent.

Second, the old belief is preserved as history with full provenance. "What did she used to want, and when did it change?" stays answerable. Nothing is destroyed.

Third, the contradiction flag is set, which tells the system this belief flipped rather than smoothly refined. Downstream agents can key off it. In our chain, the compliance agent does exactly that: "note: your current aggressive strategy conflicts with your earlier caution and your house-fund goal."

Why resolve to most-recent instead of reconciling the two? Because deciding which of two contradictory beliefs is really true requires judgment the memory layer should not be making on its own. It would need a stance classifier and context it does not have at write time. So the layer's job is kept narrow and safe: surface the newest, preserve the old, flag the conflict. The decision about whether to honor the shift is handed to an agent or a human. In our view, a memory layer that quietly adjudicates its user's changes of heart is a memory layer you cannot trust. One belief per fact does not mean one truth per fact; it means one answer per question, with the disagreement on record.

The limitation we have not fixed yet

Honesty requires one more section. The projector treats each fact type as a single evolving slot, and superseding is correct for single-valued facts: risk tolerance, income, city. A new value should replace the old.

It is not automatically correct for multi-valued facts. "Buy a house" and "save for a baby" are both true at once; the second must not supersede the first. The current build handles this by having the extraction step assign distinct fact types (house_goal, baby_goal) so goals coexist, but that relies on the extraction LLM choosing distinct keys, which is a soft guarantee. If two goals collapse onto the same key, one wrongly overwrites the other.

The clean fix, documented but not yet built, is to mark certain fact types as multi-valued so they append as a set while single-valued beliefs keep the supersede behavior. The test is straightforward: add a second goal and assert both survive, while a risk-tolerance flip still supersedes.

Recap

What is embedded: only the distilled fact sentence, never the metadata. What is matched at read time: the embedded query sentence against embedded fact sentences, same model on both sides. How the tier grows: with distinct facts, not with sessions. One belief per fact, a version chain behind it, and a flag when the story changed. Embeddings retrieve, keys identify, metadata governs, and the judgment calls are deferred to something with more context than a database.

Part 3 turns to the tier we have said the least about: the procedural tier, the "library of skills the agents learn." That phrase turned out to promise a great deal more than the tier can deliver, and the reason why changed what we built.

Agents Shouldn't Pass Context. They Should Share Memory.

Shrawan Saproo — Wed, 02 Sep 2026 08:00:41 +0000

Every additional agent in a chain is supposed to make the system smarter. Too often, it just makes the prompt bigger.

This is the failure mode almost every multi-agent system runs into, and it has nothing to do with model quality. It is token accumulation. Agent A finishes its work and hands everything to Agent B. Agent B adds its own output and hands all of it to Agent C. By the sixth agent in the chain, the prompt is carrying five predecessors' worth of context, needed or not. Then the same user comes back tomorrow, and the whole pile becomes the starting point for session two.

Costs climb. Latency climbs. And attention dilutes: the model is asked to find the three relevant facts inside forty thousand tokens of transcript.

We hit this building a proof of concept for a six-agent financial advisory chain: intake, transaction analysis, goal planning, market context, recommendation, and compliance, running in sequence, with the same user returning across many sessions and expecting continuity. This post describes the memory architecture we built to solve it, and the design decision everything else rests on.

The core idea: pass by reference, not by value

The instinctive way to connect agents is to pass context by value: each agent sends the next one a full copy of everything it knows. That is what makes tokens accumulate at every handoff.

Our design passes by reference. Agents do not hand anything to each other directly. Each agent writes its output to a shared memory layer, and the next agent retrieves only the slice it needs, capped to a token budget. If Agent A produced forty thousand tokens of investigation and Agent B needs the three thousand tokens of conclusion, B receives three thousand.

The consequence is worth stating plainly: token reduction becomes a property of how communication happens, not a feature bolted on afterwards and demonstrated with a benchmark. The budget bounds every prompt no matter how long the chain runs or how many sessions accumulate. In our team's experience, this reframing, from "how do we compress context" to "why are we copying context at all," was the single most consequential decision in the design.

In operational terms, passing by reference is the difference between a system whose cost grows with every conversation and one whose cost stays predictable months into a deployment. We think this belongs to a broader engineering discipline: TokenOps. Not prompt engineering, not model optimization, but designing systems whose token consumption is structurally bounded. The rest of this post is what that looks like in practice.

Three tiers, three jobs

Once we stopped copying context, the next question became obvious: where does the context live instead?

The memory layer has three persistent tiers. We refer to them as L1, L2, and L3 internally, but their real names describe what they hold.

The episodic tier (L1) is an append-only log of everything that happens in a session: every agent output, tagged with an identity tuple that records which tenant, which session, which step in the chain, which agent, and the sequence of the write. Nothing here is distilled, deduplicated, or embedded. It is the raw record and the source of truth. Writes are synchronous: an agent's turn is not done until its row is in the log.

The semantic tier (L2) holds distilled facts that persist across sessions: "this user is cautious with money," "this user wants a house in five years." Where the episodic log is keyed by when something was said, the semantic tier is keyed by what the fact is about. It holds one current belief per fact, with versioned history, and it is vector indexed so agents can retrieve facts by meaning rather than by keyword. Part 2 of this series is entirely about how this tier behaves.

The procedural tier (L3) holds reusable skills: organization-specific procedures for how to do something well, authored deliberately rather than learned by observation. It is the tier we scoped down the most, for reasons that deserve their own post. That is Part 3.

One rule ties the tiers together: facts flow upward, never back down. The episodic log is raw truth. The semantic tier is a projection distilled from it. The procedural tier is a library read alongside them. Because L2 and L3 are projections, they can always be rebuilt from L1. If an extraction bug corrupts your beliefs, you replay the log. Nothing downstream is ever the only copy of anything.

The write path: fast now, thorough later

Distilling raw output into clean facts requires an LLM call, and putting an LLM call on the hot path of every agent turn is exactly the latency this design exists to avoid. So the write path is split in two.

During the session, agents append to the episodic log synchronously. The write acknowledges as soon as the row lands. That is the whole cost an agent ever pays to write.

After the session, an asynchronous projector reads the new log entries, extracts typed fact sentences with an LLM, and merges them into the semantic tier using a search-then-decide procedure (Part 2 covers this in detail). Because this runs off the hot path, it can be slow, retried, or re-run entirely without touching agent latency. The semantic tier is a projection that is allowed to lag.

The read path: hybrid retrieval closes the gap

If the semantic tier is allowed to lag, a reader could see stale data. The fix is that no agent ever reads only one tier. Every retrieval queries the episodic log (fresh, raw, this session) and the semantic tier (distilled, cross-session) together, merges and deduplicates the results, and packs them into a fixed token budget before anything enters the prompt.

This has a property that took us a while to appreciate: the projector's lag is invisible. Anything too fresh to have been distilled into the semantic tier is, by definition, still in this session's episodic log, so the reader sees it from there. The agent is never blind, and the distillation never blocks anyone.

The budget allocator defaults to an even split between the two sources, on the logic that a typical turn needs both "what just happened" and "what do we know about this person."

Reads stay real-time throughout: log lookups are indexed table reads, and vector search over distilled beliefs is fast because the tier is small by construction.

Why the reduction is structural

Four mechanisms, all architectural rather than tuned:

Communication is by reference. No agent's prompt ever contains another agent's full output, only the retrieved slice.
History is queried, not accumulated. The agent at turn ten does not carry turns one through nine verbatim. It queries recent raw events and older distilled facts and receives a budget-capped slice.
Distillation happens once, at write time. Turning a thousand-token observation into a fifty-token fact costs one asynchronous LLM call. Every later read pays only the small cost. Naive context passing pays the large cost on every turn.
Large blobs are stored by reference too. Tool outputs and files live in object storage; the log keeps a pointer, and the allocator inlines a blob only when it is relevant.

Informally: the naive cost per round is the sum of every agent's full history, and it grows without bound. This architecture's cost per round is the sum of each agent's budget-capped slice, and the budget is fixed. The gap between the two is the reduction, and it widens as sessions and agent counts grow.

What this looks like in a chain

In the six-agent chain, the "turn" is an agent. Each agent reads a capped slice of memory that already contains everything agents before it wrote, does its work, and appends its own output back to the log. The shared context of the chain is the log itself, not a payload passed hand to hand. When the user returns for session two, the semantic tier's cross-session beliefs feed the reads from the first turn, so the chain starts already knowing the user without replaying session one.

The same primitive covers single-agent multi-turn work, sequential chains, and parallel multi-agent rounds. That universality was a design goal: one session model, with a round counter and an agent identifier, rather than a special case per workload.

What's next

Replacing context passing with shared memory solved the scaling problem. It immediately exposed a harder one: what happens when memory itself changes? What exactly gets embedded, what happens when a user contradicts something they said last month, and how does the system avoid returning two conflicting beliefs with no way to choose? That is Part 2.

And then there is the tier we have not really discussed. The procedural tier sounds, on first hearing, like "a library of skills the agents learn." We started there too. Part 3 is about why that phrase promises more than the tier can deliver, and what we built instead.