Your on-call agent doesn't need memory. It needs receipts.

At 02:47 my agent said: "We already tried increasing the pool size. It didn't help."

I believed it for about ninety seconds. Then I checked. Nobody had tried that. Someone had suggested it in a thread six weeks earlier, and the agent had filed the suggestion away as a fact, then handed it back to me as institutional knowledge.

That is the failure mode nobody demos.

Sixty-nine posts about on-call agents with memory landed on DEV in the last week and a surprising number of them share a title. I read a stack. The architecture is usually the same: embed the postmortems, retrieve the top five chunks, paste them into the prompt, call it hindsight. It works in the demo because the demo has three documents and no history of being wrong.

In production you get a retrieval system that cannot tell the difference between something that happened and something someone once said. That is not memory. That is a rumor mill with embeddings, and it is worse than a blank slate, because a blank slate doesn't get cited.

What memory actually requires

An agent that remembers usefully needs five properties, and vector similarity gives you none of them:

1. A structured record of what was tried, on what, and what happened.
2. A pointer to evidence a human can open in one click.
3. A scope narrow enough that the fact is still true where you're applying it.
4. A way to be superseded.
5. A way to expire.

The record is the easy part. Everyone stops there. The other four are where the work is.

Here is the table I use. It is deliberately boring, and every column exists because a specific page went badly.

create table episode (
  id            bigserial primary key,
  service       text not null,
  env           text not null,
  symptom       text not null,        -- "p99 on /checkout above 4s"
  action        text not null,        -- "pool 20 -> 40"
  outcome       text not null check (outcome in ('fixed','failed','inconclusive')),
  evidence_url  text not null,        -- incident, PR, or log permalink
  version_from  text,
  version_to    text,
  observed_at   timestamptz not null,
  superseded_by bigint references episode(id),
  expires_at    timestamptz
);

evidence_url is not optional and there is no default value. If the agent cannot point at an incident, a pull request, or a permalinked log line, the fact does not get written. I would rather lose a real lesson than store an unattributable one, because the unattributable ones are exactly the ones that come back wrong at 3 a.m. sounding confident.

version_from and version_to are there because most operational facts have a half-life shorter than the documentation about them. "Raising the pool size helped" was true on Postgres 14 with the old pooler. It stopped being true when we swapped the pooler and the fix became "stop opening a connection per request." An agent that stores the first sentence and not the version range is storing a landmine.

Retrieval is a filter, not a search

The part I got wrong first was ranking. Semantic search over incidents will happily return the most similar-sounding incident from eighteen months ago on a service that has since been rewritten. Similarity is not relevance.

Retrieval should filter hard before it ranks:

select action, outcome, evidence_url
from episode
where service = $1
  and env     = $2
  and superseded_by is null
  and (expires_at is null or expires_at > now())
  and (version_to is null or version_to = $3)
  and outcome <> 'inconclusive'
order by observed_at desc
limit 5;

Five rows, not fifty. If the agent needs more than five past episodes to form a hypothesis, it does not have a hypothesis, it has a vibe.

Negative memory needs an expiry date

The most useful thing my agent learned to do was say "the rollback failed last time." The most dangerous thing it learned to do was say "rollbacks don't work."

Both sentences came out of the same table. The difference is scope. A failed action is evidence about one version, one environment, one failure mode. The moment you let the agent generalize it into a law, you have taught it to refuse the fix that would have worked.

So negative episodes get the tightest scope you can write and a mandatory expires_at, default ninety days. If the underlying cause is still true after ninety days, a human will re-record it, and that re-recording is a signal that the problem is structural rather than incidental. If it isn't re-recorded, the memory quietly stops influencing answers. Decay by default.

Humans write, agents propose

This is the part that gets skipped in every architecture diagram. Writing to memory is a privileged operation, and the agent does not have that privilege.

The flow I use: the agent proposes an episode with outcome = 'inconclusive' and the evidence it has. The human who closed the incident converts it to fixed or failed and confirms the scope. No human, no promotion. The agent's only write is a draft it cannot read back as fact.

That single rule killed most of my false recalls. It also produced a useful side effect: the promote step is where people noticed that half their "fixes" were actually coincidences, and started writing postmortems that said so.

What I measure now

Three numbers, all cheap:

Citation rate. Of the episodes the agent cited in the last thirty days, how many were opened by a human? Anything never opened is either irrelevant or wrong, and I'd like to know which.

Contradiction rate. How often two retrieved episodes disagree on the same service and action. A high rate means my scope columns are too loose.

Time to first correct hypothesis. From page start to the first theory a human says "yes, that's plausible" out loud. This is the only metric that matters, and it is the one nobody puts in the blog post.

The uncomfortable part

Memory is a liability with a maintenance cost. Every row you store can be retrieved, cited, and acted on by something with cluster write access at a moment when nobody is awake to check it. That is a different risk class from a search box over your docs.

If you are building one of these agents this month, and the trend suggests a lot of you are, spend the first day on the schema and the write path, not the embedding model. The model was never the hard part. Knowing whether the thing you just remembered is true is the hard part, and the only durable answer is a link to evidence and a date after which you stop believing it.