Your dedup marker expires. The fact it created does not.

Here is a bug that raises no error anywhere and shows up as a memory store slowly filling with the same fact twice.

You process an event, write a dedup marker, and set a 24-hour TTL on it so the marker table does not grow forever. Every guide tells you to do this, and for delivery-level idempotency it is correct advice. Then a backfill re-reads a ticket from last year. The marker expired months ago, so the ingest layer has no memory of it. But the fact derived from that event is still in the store, because derived facts do not expire on a schedule.

Now you have it twice. Nothing failed.

I went through the papers, talks and practitioner write-ups we had collected on this, and the word "idempotent" carried four incompatible meanings across them. Every source recommending a dedup marker also recommended a TTL on it. None discussed the interaction between an expiring marker and a non-expiring derived fact.

Four definitions sharing one word

Request-level. Repeating the same request leaves state unchanged after the first. The HTTP version is familiar, and RFC 9110 defines it in section 9.2.2: GET and DELETE are naturally idempotent, POST is not, and PATCH depends entirely on what it does. A PATCH setting a name to "DJ" is idempotent. A PATCH incrementing a counter is not. Same verb, opposite guarantee.

Delivery-level. The same event arriving twice gets acted on once. Most event-driven systems mean this, and enforce it with a marker.

Merge-level. The merge function is itself idempotent, plus associative and commutative, so replicas converge regardless of arrival order or repetition. This is the CRDT property.

Derivation-level. Re-running a derivation over the same inputs produces no new derived records. This is the one that matters for agent memory, and the one almost nobody names.

Satisfying any of those tells you nothing about the other three. A system can be fully delivery-idempotent and still accumulate duplicate facts, by exactly the mechanism above.

Why memory pipelines re-read on purpose

Retries are the obvious source. Even a broker promising exactly-once delivery cannot stop an upstream service publishing the same logical event twice under two different event IDs, which is why ID checks usually get paired with a business-key check on something like an order number.

What is specific to memory systems is that re-syncs are not failures. A connector pulling GitHub issues, Slack threads or support tickets runs on a schedule and re-reads overlapping windows every time, by design. A backfill re-reads everything. Neither is an error condition, so treating duplicate suppression as error handling puts the check in the wrong place.

Derive the key, never expire it

Stop generating keys and start deriving them from the event's logical identity. A derived key costs nothing to keep forever, because it is a property of the record rather than a row in a growing side table.

github:acme:api-server:issue:4471:opened

Source, org, repo, record type, issue number, event kind. Re-sync that issue a thousand times and you get one episode. No UUID, no timestamp, no TTL.

One file over in the same connector suite, the IDE companion uses the opposite rule: idempotency is content-addressable, so re-running an unchanged workspace scan maps to the same key and a changed workspace yields a new memory.

Two opposite policies in one codebase, and both are right. A GitHub issue has a stable identity independent of its text, so keying on identity is correct and an edited title should not create a second episode. A workspace scan has no identity apart from its contents, so keying on a content hash is correct and a changed workspace should produce a new one.

Your key encodes a definition of "the same event," and that definition is a per-source decision. Wrong in one direction floods the store. Wrong in the other silently drops real updates.

Compilation has to set, not append

Ingest dedup only covers ingest. The compile step needs its own guarantee, and it comes from one design choice: a compile pass that derives current state from the full episode log behaves like a set. A pass that appends whatever it found this run behaves like an increment, and increments are never idempotent.

That single property is what makes a compile endpoint safe to call from a retry loop, a cron job and a webhook handler at the same time. It is also what lets you batch compilation off the request path instead of running it inline after every turn.

Test for it in one line:

def test_recompile_is_a_noop(client, subject_id):
    first = client.post("/v1/memories/compile", json={"subject_id": subject_id})
    assert first.json()["memories_created"] > 0

    second = client.post("/v1/memories/compile", json={"subject_id": subject_id})
    assert second.json()["memories_created"] == 0

That whole test is the second number. If it is not zero, you have an append where you need a derivation, and no amount of ingest deduplication will save you.

Duplicates and conflicts are different problems

A duplicate is two records saying the same thing, and the fix is to keep one. A conflict is two records saying different things, both legitimately written, and there is no version of "keep one" that is obviously right.

Idempotency does nothing for the second case. Martin Kleppmann's taxonomy of conflict handling is still the cleanest: let a human resolve it, pick a winner automatically, or merge automatically. On the middle option he is precise about the cost, noting that some systems choose one version as the winner and throw away the others.

Throwing away the loser is the default in most agent memory stacks, and it is where the damage lands. You do not just lose the old value. You lose the evidence that a disagreement existed at all.

Recency-wins is near-universal as the automatic rule, and it is fine for state that genuinely changes, like which database a team runs. It is wrong for a stable attribute that a bad extraction re-asserts incorrectly. And it is wrong for facts that only look contradictory: "I am in Berlin this week" does not contradict "this user lives in Lisbon," and a system that supersedes the second with the first has made the memory worse.

Supersede, because the loser is evidence

Mark the older fact superseded, filter it out at read time, keep it in the record with provenance links to both sides. Three states cover it: active, superseded, tombstoned.

A worked example I can point at: three agents read three source documents concurrently and write to one shared subject. One commits Stripe's pre-reversal processing rate, another commits the corrected rate. Where the two memories carry a registered single-valued claim key, the compiler compares claims directly. Otherwise it falls back to Jaccard word overlap between the compiled memories, and at 0.6 or above it marks the older one superseded and records the decision with links to both source episodes.

Downstream, the synthesis agent's bundle contains only the winner. The audit trail contains both, plus the similarity score that triggered the call, 0.72 in the shipped run.

Treat 0.6 as a dial rather than a constant. Set it lower and unrelated facts collide, producing false-positive supersession that deletes true information from the read path. Set it higher and genuine contradictions both survive into the bundle, so you pay tokens to hand the model two incompatible answers.

Where this still fails

Word overlap is a fallback, not the mechanism. Jaccard catches "Stripe charges 3.5% + 35c" against "Stripe charges 2.9% + 30c." It will occasionally fire on unrelated facts sharing common words, and it misses contradictions expressed in entirely different vocabulary.

Idempotent is not deterministic. Run an LLM compiler and the extraction step can produce different facts from the same episodes on different runs. Idempotent compilation means a second pass over already-processed episodes adds nothing. It does not mean the first pass is reproducible. A regex-based heuristic compiler is.

Some conflicts should not be resolved. Two high-confidence claims that contradict each other may mean the boundary conditions differ. If being wrong is expensive in your domain, flag them and abstain instead of auto-resolving.

None of this fixes bad extraction. Deduplicating, superseding and auditing low-quality facts gives you a clean, well-audited store of low-quality facts.

The ten-minute check

Run your compile step twice in a row and look at the second number. If it created anything, that is the bug to fix before any of the rest.

If you would rather not build the key derivation, compile markers and supersession logic, the runtime behind the examples here is Apache-2.0 on GitHub and runs on your own Postgres. The full version of this post covers the per-kind conflict policy table and the replay boundary I skipped.