A follow-up to Part 1: the self-recall thesis — the series runs through Part 6. Code: RE-call — everything below is measured and reproducible (make eval), full study in docs/ENTAILMENT_SUPERSESSION_STUDY.md.
I published a thesis post about agent memory and got five comments that were better than the post.
Two of them didn't just critique the design — they described, precisely, why it would fail and what would fix it. So I did the only reasonable thing: I turned both into experiments, ran them on the same eval harness the series is built on, and shipped what survived. That's RE-call v0.3, and this post is the receipt.
I want to be explicit about why I'm writing it this way. The point of publishing this series was never broadcast — it was error-correction. A design you keep in a drawer accumulates conviction; a design you publish accumulates objections, and objections are the cheapest high-quality signal you will ever get. The comment section of Part 1 did more for this codebase than any week of solo iteration. This post exists to pay that back with the thing commenters almost never receive: evidence that someone listened, measured, and changed the code.
Comment 1: "A similarity score is not a confidence score"
Vinicius Pereira put it in one line I've been quoting since:
Proximity is a candidate; entailment is the evidence.
His argument: the near-misses that hurt most are high-similarity and wrong — memos semantically adjacent to the query that don't answer it. A threshold-based gap_warning (Part 3, Part 5) waves them straight through by construction, because their similarity clears any threshold you could calibrate. The abstention signal cannot be the retriever's own score. You need a separate check that the retrieved memo actually entails an answer.
He was right, and measurably so. I built a held-out challenge set of 10 near-miss queries — each names a strongly on-topic memo that does not contain the asked-for fact ("how much did the cache reduce memory usage" against a memo that measures latency). Baseline, with the calibrated threshold from Part 5 doing its best:
| Embedder | Near-miss FCR @ calibrated threshold |
|---|---|
hashing-64 |
1.00 |
bge-small |
0.80 |
voyage-3 |
0.40 |
The threshold that scores a perfect 0.00 on far-gap queries passes 40–100% of near-misses. There is no threshold to fix. The distractor's cosine is genuinely high — that's what makes it a near-miss.
So v0.3 adds an opt-in entailment stage: a small QNLI cross-encoder ("does this sentence answer this question?") judges the trusted hits, and a hit that doesn't entail the query is demoted to a new verdict, not_entailed. The key property is exactly the one Vinicius predicted: it emits a decision at the judge's own trained boundary, not another score — so there is no per-embedder constant left to recalibrate. And that transfer claim held: the identical judge, zero tuning, on every embedder:
| Embedder | Near-miss FCR: threshold → +entailment |
|---|---|
hashing-64 |
1.00 → 0.60 |
bge-small |
0.80 → 0.50 |
voyage-3 |
0.40 → 0.40 |
Where the comment needed a refinement — which is the point of measuring
The ablation was the honest surprise. Running the judge alone, without the threshold, degrades far-gap detection (gap FCR 0.00 → 0.40 on both semantic embedders): fed nearest-noise from a topic the corpus doesn't cover, the QNLI model sometimes calls it an answer. So entailment does not replace the calibrated threshold — the two guard different failure classes and must be stacked. Threshold catches far gaps; judge catches near-misses.
And the costs are real, and published: ~0.1–1.0 s of judge time per query on CPU, one legitimately answerable query wrongly rejected on both semantic embedders (its gold memo answers by negation — "do we retry on 4xx?" → "we do not retry" — and the judge reads that as not-answering), MRR on answerable queries dips 1.000 → 0.929. The residual near-miss FCR (0.40–0.60) is the judge's own quality bound — Part 5's law, one layer up: gap detection is bounded by the embedder, and abstention-by-entailment is bounded by the judge. Ships OFF by default for exactly these reasons; you opt in with your eyes open.
Comment 2: "Supersession is a relation, not a property"
The same comment carried a second thesis, on the guard I'd already confessed was weakest (freshness):
You are trying to infer a relation between two memos at read time, when both look valid in isolation. That inference is a losing game. Bind the truth when it is created.
And Mateo Ruiz had independently named the target shape:
Retrieval should return confidence + provenance + validity, not just relevance.
That sentence is now, almost verbatim, how RE-call's trust layer describes itself. Every hit returns a verdict (ok / superseded / expired / …), a calibrated confidence, and provenance; a memo declares supersedes: old-memo.md in its frontmatter at write time, and retrieval returns the current head of the chain instead of a resolved-but-still-embedded old decision.
For v0.3 I added the experiment that closes the "why not just timestamps?" question — against the steelman, not a strawman: "among the confidently-relevant hits, trust the newest", with the stale docs re-touched after their successors, the way any living corpus re-syncs constantly. Superseded-trust rate (how often the stale memo is handed back as the answer — lower is better):
| Embedder | Plain search | Recency (steelman) | Declared supersedes:
|
|---|---|---|---|
hashing-64 |
1.00 | 0.83 | 0.00 |
bge-small |
0.83 | 1.00 | 0.00 |
voyage-3 |
1.00 | 1.00 | 0.00 |
Look at the bge-small row: the timestamp heuristic is worse than plain relevance ranking — the tie-break actively promotes the freshly-re-synced stale memo in the one case where ranking had preferred the successor. A per-document timestamp cannot see a two-document relation, and making the timestamp "smarter" makes it more confidently wrong. The declared relation holds at 0.00 in the same runs.
Vinicius also called the residual failure mode in advance: write-time binding is only as good as the author's discipline — a forgotten link is an orphan memo that looks valid forever. But, as I replied then: impossible to infer becomes possible to enforce. So v0.3 ships recall lint — dangling supersedes: references, cycles, ambiguous successors, versioned siblings with no declared edge, closures declared only in prose. No DB, exit 1 on errors, drops into CI in one line. (It paid for itself before it shipped: writing its tests uncovered a real parser bug where a scalar [[wikilink]] was read as a YAML list, producing an edge that silently never resolved.)
The experiment I still owe
Nazar Boyko asked, before Part 5 was even published, whether the gap threshold should be relative — top hit versus the rest of the batch — rather than an absolute cutoff re-tuned per embedder. It's a good idea with a suspected hole (a spread-based check is blind to the single confident distractor — which is precisely the near-miss class above), but suspicion is not measurement. It's on the list, and the harness is now shaped to answer it.
And Tae Kim's point — a typed coverage_check slot so the "no real match" signal can't be silently dropped — sharpened a design rule that now runs through the whole stack: the retriever computes the signal, the schema carries it. Computed, it's a measurement; self-reported by the model, it's a declaration. Those fail very differently.
What I'm actually arguing for
Five people I've never met read a post about a niche RAG problem and, between them, produced: a falsifiable critique of my abstention mechanism, the correct architecture for supersession, a proposed alternative worth benchmarking, and a schema-design principle. Total cost to me: publishing something concrete enough to be wrong about, and taking the replies seriously enough to run them.
That's the whole model. Not "content", not reach — working in public as a form of peer review. The asymmetry is absurdly favorable: you contribute one design and get back the failure modes it would have taken you months to hit alone. The only price is that you have to be willing to write "I was wrong, here's the measurement" — which, in a series whose thesis is calibrated honesty, is not a price at all. It's the product.
So: thank you Vinicius, Mateo, Nazar, Tae — and Amin, whose memory-compaction angle (keep the gist graph, not every turn) is a different axis of the same problem and deserves its own experiment. v0.3 has your fingerprints on it.
If you're reading this and see the next hole — the negation-blind judge, the owed relative-threshold benchmark, a stronger entailment model, something I haven't imagined — the comment section is open and the harness is public. Evidently, it works.
Code: RE-call (MIT). The full v0.3 study with every table: docs/ENTAILMENT_SUPERSESSION_STUDY.md. Series index: Part 1.
Top comments (0)