The pilot, and why it was not a result
The agent's memory is a folder of Markdown notes plus a register of the promises it has made. The retrieval aid over it has two stages. A small embedding model, bge-small-en-v1.5, finds candidate sections by cosine similarity. A cross-encoder, ms-marco-MiniLM-L-6-v2, then reads the question and each candidate together and scores the pair.
On 12 hand-written questions, six answered by the corpus and six not, the pilot found three things. A cosine floor could not separate present from absent: the ranges overlapped, and a question about preferred-share dividend policy, a subject the corpus knows nothing about, outscored four real hits. Pure vector search got both identifier lookups wrong. And the cross-encoder did separate the two groups, by 0.12 points on a scale about 18 points wide.
That last margin is why the tool shipped a three-way verdict instead of a yes or no: found at a score of 0 or above, not in corpus at -8 or below, and uncertain in between. Twelve questions from one author is a pilot, so the job was to find out whether any of it holds.
What we ran
We froze everything before scoring anything, in this order.
- The corpus. A read-only snapshot of 118 files, split into 1,144 sections, with a hash for every file. The evaluation refuses to run if any hash changes, and we watched it refuse a tampered copy before trusting it.
- The questions. 40 questions the corpus answers, written from sections drawn with a fixed random seed and paraphrased away from the section's own wording. 17 near-miss questions: on-topic, the kind this corpus plausibly would hold, and does not. 17 off-topic questions. 18 identifier lookups across six shapes, from ticket numbers to CRM record IDs. Three controls copied word for word from a section.
- The predictions. Nine written predictions with numeric ranges, filed before the first query ran. The misses are listed below.
The thresholds stayed at the pilot's values. Tuning them on this set would have made every number below meaningless.
Building the "not here" questions was the hard part
Every absent question had to be shown absent, not assumed. For each one we searched the whole snapshot for its key terms and their synonyms, read every hit, and ran known-present terms through the same search to show it could find anything at all.
We wrote 32 near-miss questions and had to drop 15 of them. Three were simply answered somewhere in the corpus, one of them a plain mistake on our part. The other twelve were answered by a status: "undecided", "none yet, buy after the first yes", "not issued yet", "the specs were never stated".
That is worth sitting with. A corpus of working notes records the absence of a fact as a fact. So "not in the corpus" is not a property of a question; it depends on whether anyone wrote down that the thing is unknown. We dropped every question where the corpus states either the fact or its status, and kept the 17 where it states neither.
One more thing turned up here. The pilot's dividend-policy question is now quoted inside the corpus, in the notes about the pilot. Writing about the experiment put the probe into the thing being probed. We set that question aside before scoring and report it separately.^(1)1 The pilot's dividend-policy question is quoted inside the corpus, in the notes about the pilot. It was set aside before scoring and is reported separately.
1. A similarity floor still cannot say "not here"
We measured how well each score separates the 40 answered questions from the 34 unanswered ones^(2)2 The 34 unanswered questions are the 17 near-miss and the 17 off-topic ones. The 18 identifier lookups and the 12 supplement questions are counted separately. as AUROC,^(3)3 AUROC: the chance that a randomly chosen answered question scores higher than a randomly chosen unanswered one. 0.5 is a coin flip and 1.0 is perfect. Intervals are 95% bootstrap intervals. where 0.5 is a coin flip and 1.0 is perfect, with 95% bootstrap intervals.
- Cosine similarity of the best section: 0.81 overall (0.70 to 0.91). On near-misses alone, 0.62 (0.45 to 0.79). On off-topic questions, 1.00.
- A z-score of that similarity against the whole corpus: 0.48 overall (0.35 to 0.61), no better than chance.
- The cross-encoder's best candidate: 0.77 overall (0.65 to 0.87). On near-misses alone, 0.54 (0.39 to 0.71). On off-topic questions, 1.00.
Shuffling the labels gave 0.50 for every score, which is what a working harness should show.
Cosine looks fine on off-topic questions, and that line flatters it. Without the set-aside dividend question, the highest off-topic score sat 0.015 below the lowest answered one. The dividend question itself scored 0.699, higher than 26 of the 40 real questions. On the questions that matter, the near-misses, the two groups overlap and no single cosine floor separates them.
2. The cross-encoder does not separate better. It ranks better.
This is the part of the pilot that did not hold. We predicted the cross-encoder would beat cosine at separation by at least 0.10. It did not beat it at all: the paired difference was -0.04, with an interval from -0.14 to +0.06. On near-miss questions it cannot be told apart from chance.
What it does do is put the right section first. Across the 40 answered questions, cosine ranked a file that answers the question first 57.5% of the time and in the top five 72.5% of the time. After the cross-encoder reranked, those became 72.5% and 80%. It can only rerank what the first stage hands it, and the right file reached it for 35 of 40 questions. So it earns its cost, just not for the reason we shipped it.
3. An honest verdict spends most of its time saying "look yourself"
Here is what the three-way verdict said, thresholds untouched:
- 40 answered questions: 12 found, 24 uncertain, 4 not in corpus.
- 17 near-miss questions: 3 found, 14 uncertain, none not in corpus.
- 17 off-topic questions: all 17 not in corpus.
Some of that is good. Off-topic questions got a clean no, every time. Every found on an answered question pointed at a file that holds the answer.
Some of it is not. Four real questions were told the corpus does not answer them. In one, the section that answers it was ranked first and still scored -9.1. The question was a paraphrase that shared no content words with that section. The tool's printed explanation under that verdict calls it a real answer, not a ranking failure. For that question it was wrong.
The three near-miss questions that came back found all matched the right subject and did not answer the question: a filing date asked of a note about the same company's filings, a county asked of a note carrying the company's address, a remark about a grant program asked of the notes about the person in question. The cross-encoder rewards being about the right thing, not answering it.
And most answers landed in the middle: 60% of real questions and 82% of near-misses. For these scores, "open the file and read it" is the correct output. It is also the finding.
4. Identifiers the extractor was not taught simply vanish
The tool matches identifier-shaped tokens exactly, as a separate stage, because embeddings blur random-looking strings. That stage was right every time it ran, 11 of 11. It ran on exactly the 11 lookups we predicted from reading its pattern, and on none of the other seven.
Those seven were mail message IDs, CRM record IDs and one mixed-case calendar ID. Nothing found any of them. Cosine, none of seven; the cross-encoder, none of seven. For two mail message IDs that are in the corpus, the tool printed not in corpus, and because no token was extracted it did not print its exact-match section at all. The user sees a confident absence and nothing else. Wrap the same ID in backticks and exact match finds the file straight away.
5. A sixth of the corpus is never read
The verbatim controls were meant to be easy, and two of the three failed at least one stage. Looking into why turned up something structural. Sections in this index are split by heading and capped by character count, and the embedding model reads 512 tokens. 42% of sections are longer than that, so 17% of the corpus's text is never embedded. The cross-encoder is given each candidate's first 1,200 characters, so a third of the text is never read by it, and 17% is read by neither stage.
Our main question set could not see this, and that is our fault. We wrote those questions from the first part of each sampled section, so every one of the 40 answers sat inside both windows. The recall figures above are optimistic for the corpus as a whole.
So we added a small test before calling it done: 12 questions written only from text neither stage reads, each kept only if its answer appears nowhere else in the corpus, with predictions filed first. The right section reached the reranker for 4 of 12. The verdict was uncertain for 8, which is honest. It was found for 3, and one of those pointed at a different note on the same topic that does not hold the figure asked for. On the answered set, found had never pointed at the wrong file.
One verbatim control failed for a different reason. Its sentence sat well inside both windows, in a section long enough that one sentence apparently did not move the section's embedding. We had guessed truncation, and we were wrong.
The predictions we got wrong
We filed 13 predictions with numeric ranges in all. These missed:
- The z-score was not "about as good as cosine". It was at chance, and clearly worse.
- The cross-encoder did not separate better than cosine. That was the pilot's main claim, and it is the one that fell.
- Four answered questions were called absent. We said at most three.
- Cosine put the right file first more often than we expected: 57.5%, against a ceiling of 55%.
- Not every verbatim control ranked its own section first.
- The dividend question did not retrieve the note that quotes it. It got a clean not in corpus.
- On answers hidden in unread text, we expected a false not in corpus at least a quarter of the time. It happened once in twelve; the middle band absorbed it.
What we changed, and what we did not
We did not retune a threshold, and we did not touch the tool while the numbers above were being taken. Changing the instrument in the middle of measuring it would have made those numbers describe a tool that no longer exists. Both fixes were plain enough to name at the time: teach the exact-match stage any long unbroken alphanumeric token, and split sections by the model's token limit instead of by characters.
Update the same day: both fixes, measured
We made both changes a few hours later and re-ran the same frozen questions against a rebuilt index, with a second set of predictions filed before scoring. The thresholds stayed where they were.
Two checks came first, because a fixed tool measured by a changed harness proves nothing. The harness re-ran the original tool against the original index and reproduced the morning's file exactly: no score moved, and no ranking or verdict changed in any of the 96 rows. And the sandboxed demo for this tool gained three assertions, which were watched failing against the unpatched copy before they were trusted against the patched one.
- Identifiers. Because the new patterns were written after seeing which formats had failed, the original 18 no longer test anything: a fresh set of 18 was drawn from the same shapes and frozen before scoring.^(4)4 Only the identifier set was redrawn for the same-day rerun, because the new patterns were written after seeing which formats had failed. The other questions stayed frozen. The old tool found 10 of those 18; the fixed tool found 18 of 18. No present identifier is now told it is absent.
- Unread text: 17% to 0%. The longest chunk now needs 436 tokens of the model's 512, and the reranker reads every candidate whole.
- Answers hidden in that formerly unread text reached the reranker for 9 of 12 questions, up from 4, and were ranked first for 6, up from 4.
- Real questions wrongly told "not in corpus": 4 to 2. The two that remain are the paraphrases.
- Ranking improved a little at depth: the right file in the top five rose from 72.5% to 77.5% for cosine and from 80% to 87.5% after reranking. Rank one was flat.
The right file ranked first
Cosine similarity alone
57.5%
After the cross-encoder rerank
72.5%
0%50%100%
The right file in the top five
Cosine similarity alone
72.5% → 77.5%
After the cross-encoder rerank
80% → 87.5%
0%50%100%
› Data table
| Measure | Stage | Morning | After fixes |
|---|---|---|---|
| The right file ranked first | Cosine similarity alone | 57.5% | 57.5% |
| The right file ranked first | After the cross-encoder rerank | 72.5% | 72.5% |
| The right file in the top five | Cosine similarity alone | 72.5% | 77.5% |
| The right file in the top five | After the cross-encoder rerank | 80% | 87.5% |
Percent of 40 answered questions with the right file at each rank, morning run and after both fixes
Fig. 1Share of the 40 questions the corpus answers for which a file that answers the question was ranked first, and was in the top five, by cosine similarity alone and after the cross-encoder rerank. Hollow marker: the morning run. Solid marker: the same frozen questions re-run the same day after both fixes; rank one did not move, so its two markers coincide. One corpus and one author, so a few points is not a rate to quote.
And the part that did not move, exactly as predicted: separation. The cross-encoder is still no better than cosine at telling an answered question from an unanswered one (difference -0.05, interval -0.16 to +0.06), and still at chance on near-misses. Reading more text does not make this reranker able to say "not here".
One caveat we owe the reader, from later the same day. Everything above is about a general-purpose retrieval reranker, which is what this tool uses. We have since run the identical questions past a model built specifically to make typed decisions with calibrated probabilities, and on the near-miss questions it separated answered from unanswerable far better than anything reported here. That is one corpus, one author's questions and a single phrasing, so the numbers stay in the lab until they hold up across more of each. Read the finding as "our reranker cannot abstain", not as "abstention is out of reach".
Two side effects, since a fix that only reports its wins is not a report. A fourth near-miss question now comes back "found" instead of "uncertain", one worse than before. And the wider identifier pattern fires on ordinary proper nouns: a platform name, a company name. It fired on 5 of the 74 questions, and on one unanswerable question that makes the tool look more confident than it should. The verbatim control that failed on dilution still fails.
Scope, and what this is not
One corpus, about 1.65 million characters of one operator's working notes, used by one agent. One author wrote the questions, the tool, the pilot and the predictions. That is the weakest part of this, and the reason everything was frozen and hashed before it was scored. 74 separation questions, 18 identifiers and 12 in the supplement are enough to overturn a 12-question pilot and not enough to be a rate you should quote for your own system.
It is one configuration of two small open models. It is not a comparison of models and says nothing about either outside this setup. The corpus is private working notes, so the questions and the harness are not published and a reader cannot re-run this. Nothing here is a benchmark result, and none of it is sourced to the archived WasTrue Benchmark.
The practical version, if you are building memory for an agent: do not let a similarity score stand in for "the answer is here", do not assume a general-purpose retrieval reranker can tell you "the answer is not", check how much of your text your models actually read, and write down what your agent does not know. That last one turned out to be the thing this corpus was best at.
Notes
- The pilot's dividend-policy question is quoted inside the corpus, in the notes about the pilot. It was set aside before scoring and is reported separately. ↩
- The 34 unanswered questions are the 17 near-miss and the 17 off-topic ones. The 18 identifier lookups and the 12 supplement questions are counted separately. ↩
- AUROC: the chance that a randomly chosen answered question scores higher than a randomly chosen unanswered one. 0.5 is a coin flip and 1.0 is perfect. Intervals are 95% bootstrap intervals. ↩
- Only the identifier set was redrawn for the same-day rerun, because the new patterns were written after seeing which formats had failed. The other questions stayed frozen. ↩
References
- bge-small-en-v1.5, the embedding model used for the first stage. Model card. huggingface.co/BAAI/bge-small-en-v1.5
- ms-marco-MiniLM-L-6-v2, the cross-encoder used for the second stage. Model card. huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2
- State-Drift: the term, what the field has measured, and why it is a security problem. Botzr Research. botzr.com/research/state-drift
- Six silent memory failures. Field notes, written 4 September 2026. Botzr Research. botzr.com/research/silent-memory-failures
Originally published at botzr.com.
Top comments (0)