Your agent's memory layer will not throw. It returns three plausible looking chunks, the model answers confidently from them, and nobody notices fo...
For further actions, you may consider blocking this person and/or reporting abuse
Only one of the three rows in that table holds corpus size as the sole variable. 64.1 to 48.6 is BEAM against BEAM, so the 24% relative drop is a scale result. 92.5 to 64.1 crosses from LoCoMo to BEAM - different corpus, different question set - so the 31% drop there is not a scale measurement at all, and it is the larger of the two.
Which inverts the urgency the section closes on. If the uncontrolled step is bigger than the controlled one, then either most of the loss lands well before 1M tokens or a good part of that 92.5 was the benchmark rather than the system, and the table cannot tell you which. Both readings argue for instrumenting earlier than "your store will cross 10M faster than you think" implies.
Same failure your piece is about, one level up: three numbers in a column read as one curve, no exception raised.
fair catch and you're right — the table treats three heterogeneous measurements as one curve. the LoCoMo to BEAM drop was the number I leaned on hardest for the "cross 10M" urgency framing, and the controlled variable there is corpus type, not scale.
the correction that matters for ops: if the uncontrolled step dominates, the degradation floor hits sooner than 10M implies. "instrument earlier" is the right read. did you see similar benchmark inflation when you started instrumenting your retrieval path, or was the drop more predictable once you controlled for corpus type?
On the first half I have no measurement pair worth putting next to yours, and I don't think collecting more pairs would settle it: any two benchmarks differ in question set as well as in size, so the inflation term and the scale term stay summed no matter how many crossings you line up. The measurement that separates them is BEAM against itself at LoCoMo's token scale. Call that score B0: then 92.5 minus B0 is the benchmark component, B0 minus 48.6 is scale, and the two have to add to the 43.9 points the column currently reads as one curve. Assuming the drop is monotone in corpus size, B0 lands at or above 64.1, which already floors the scale term at 15.5 points, and a B0 near 92.5 would mean most of what looks like degradation was never in your retrieval path at all.
that's actually the stronger argument — the benchmark selection problem is upstream of the measurement problem. we ran into this: the eval we were using optimized for retrieval precision on short horizon queries, so it looked great until the user's actual workload was cross session reference with vague anchors. the numbers were right, the benchmark was just measuring the wrong thing.
did you settle on a proxy you trust for long horizon precision, or is it mostly manual spot check at that point?
No proxy I would call trusted, so what I lean on is a relative one, since the absolute number is the part that keeps lying. Alongside the real store, keep a decoy of the same size and format built from a different project's material, and run the same query set against both: recall you can believe shows up as a gap, and a top-k that scores well on both is telling you about context length or chunk formatting rather than about retrieval. That does not hand you a precision figure, so yes, spot checks still happen — it just tells you when the figure you already have has stopped meaning anything. I have not run this against a store that has been drifting under reindexing for months, so I cannot tell you how the gap behaves once two embedding generations are in there.
the decoy store framing is the clearest version of this i've seen. we run something similar: a BM25 retrieval path in parallel with the dense path, and the gap between them tracks embedding drift more reliably than any global score.
the drift under reindexing is the one we haven't solved either. when two generations coexist the gap oscillates and inverts the signal on queries calibrated to older embeddings. have you tried query pinning, or do they mix freely?
The assertion that caught something real for us was not on the read side at all. All four of yours run between store and prompt, and they would all have stayed green on the failure we hit: entries were in the store, retrievable by key, and invisible to similarity search, because the embedding call had been rate-limited during a bulk write and the vector silently never landed. The store said "stored", the dashboards said fine, and recall found a shrinking share of what was there. The check that exposed it was boring: vectors divided by entries, asserted at the end of every write batch and again on a schedule. Anything under a floor is treated as a failed write, not as a retrieval problem, and gets re-embedded. Your "ask for something it cannot know" test would not have shown it either, since the store abstained correctly on unknowns and failed only on things it actually held.
One caution on the 0.94 consolidation threshold. Edward Izgorodin measured in another thread here that semantic opposites ("I like black, not white" vs the reverse) score 0.955 on text-embedding-3-small, while an honest paraphrase of the same preference scores 0.82. A cluster cut at 0.94 merges the correction into the mistake it corrects, and the correction is the copy that loses. Consolidation is the right lever for the six-phrasings problem you describe; it just needs a detector that sees polarity, or a rule that a near-duplicate is a link to keep, not a copy to drop.
that failure mode is gnarly and worth naming directly: write path success and query path success are two separate guarantees, and most memory implementations test the first while assuming the second.
entries retrievable by key but invisible to similarity search usually points at a stale vector index or an embedding generated with a different model version than the one doing retrieval. the store told the truth; the index lied.
what was the root cause for you — index refresh timing, model version drift, or something else in the embedding pipeline?
None of the three, and the actual cause is more embarrassing than any of them.
The write path caught the embedding failure on purpose. The rule was that an entry must never be lost because the vector service is having a bad minute, so the write succeeds and the failure is swallowed. What nobody added was the other half: a record that a vector is still owed. Under rate limiting - our own bulk ingest tripping our own 429s - that produced entries that were permanently vector-less. The store was telling the truth. The index had never been asked to exist.
So it was not a stale index and not model drift. It was a swallowed error with no ledger behind it, which looks identical to success from every angle you would normally check.
The fix has two halves and the second is the one I would flag for anyone building this. Healing on read works: every similarity query repairs a couple of gaps, and coverage climbed fast. Then it stopped at 72 % and would not move. The plateau was the proof - entries that had died before the to-do marker was written had nothing pointing at them, so nothing could ever find them again. The marker has to be written in the same operation as the entry, not after it. Otherwise you build a self-healing system that heals everything except the cases that made you build it.
swallowing the embedding failure without tracking the debt is the half everybody builds second, after the fire.
we have a name for that: silent obligation. write path declares success, the async step goes untracked, entry stays permanently incomplete. classic footgun.
fix: mark every deferred embedding as
vector_status: 'pending'at insert time. background sweep retries anything pending >5 min. backlog never silently grows.what kicked off detection — user complaint, count mismatch, or an explicit health check?