Recall@1 went from 13% to 50%. Recall@5 from 13% to 80%.
Over the same week, the number of questions my agent answered end-to-end went from 2 in 1...
For further actions, you may consider blocking this person and/or reporting abuse
The
_collapsereturningresult.split("\n")[0]detail is the whole article, and it's brutal because it's the kind of rule that's completely correct in isolation. The header line "8 candidates, best to worst:" survives, the actual candidates get truncated, and the model's reasoning trace ("the output was empty... let me search again") is it behaving rationally on corrupted evidence. Two things stuck with me. First, you only caught it by reading the reasoning trace â component metrics (Recall@k) literally can't see it, because retrieval genuinely did get better; the damage happened downstream in context assembly. We've started treating "searches-per-turn" as a first-class health signal for exactly this reason: a spike means the agent is re-fetching evidence it should already hold. Second, a 55k-entry catalog punishes any lossy compression of tool results far harder than a small one would. Did you end up making the collapse content-aware (keep the candidate list, drop only prose), or did you just bump VERBATIM_RESULTS and eat the token cost?Content-aware, and the token cost turned out not to be the trade-off I expected.
Bumping VERBATIM_RESULTS was the first thing I tried and it doesn't actually fix the mechanism. It buys you three more searches before the same deletion starts, and with 55k entries the model will happily use them. The bound just moves.
What landed:
1- a collapsed result keeps what identifies its content, not its label. For a candidate list that's names and ids, compacted; for an analysis result it's the finding, not "2 findings"
2- every new search re-states the turn's accumulated shortlist inside its own result, newest first. The newest result is the one the collapse rule always keeps, so the shortlist rides along on the one thing that can't be truncated.
That second one is the part I'd keep in any agent. It makes the turn's working memory self-healing rather than dependent on a retention policy being right.
And I got it wrong once on the way: the shortlist hit the 800-char summary cap and filled oldest-first, so past the cap it kept the earliest finds and dropped whatever the newest search had just produced. I'd recreated the original bug one layer in. Only the end-to-end number caught it. 18/30 turns down to 8/30.
On tokens: the window was 30% full at ten searches. The rule was destroying the turn's working memory to save about 1,800 characters. There was never a token problem to solve; I'd just assumed there was, because the rule was written when the catalog had thirteen entries.
Your searches-per-turn signal is the right instrument and I wish I'd had it. Ours went from a median of 12 to 2 across this fix. And that number moved while recall stayed flat at 50/80, which is exactly the decoupling you're describing. I'd add one more: discovery calls that return candidates the turn has already seen. A repeat is the specific symptom of evidence loss, where a high count alone could just be a hard question.
This is the cleanest proof I’ve seen that a rising component metric can kill the product: recall climbed while completions went 2/10 → 0/10 because context assembly deleted the evidence the better retrieval had just found.
The eval contract I’d pin next to Recall@k is end-to-end completion plus a context-health cut (searches-per-turn, and re-fetches of candidates the turn already saw). Those move when the binder is broken even if retrieval stays flat.
Component win ≠ system win until the evidence the retriever returned still exists when the agent decides.
Component win ≠ system win until the evidence the retriever returned still exists when the agent decides".That's a tighter framing than anything in the post. Stealing it.
The re-fetch signal is the one I'd push hardest. Searches-per-turn alone is ambiguous: a hard question legitimately costs more searches. A repeat is not ambiguous, the agent asking again for something it was already shown means the binder dropped it, and there's no benign reading.
One thing I'd add to the contract from how this actually failed: the eval has to move with the data, not just the code. Ours was written against a thirteen-entry catalog and stayed valid while the catalog grew four thousand times. Every constant in the binder; retention count, summary cap, fill order; was correct when written and silently wrong afterwards, and nothing in CI knew a scale had changed underneath it. The end-to-end number is the only thing that noticed, and it noticed late
Your reply that the end-to-end number was the only thing that noticed, and noticed late, points at a number the post already measured once. Recall@5 was computed on what the retriever returned. What the model acted on is a different number, whether the right id was still in the prompt when it decided, and your ten-search count, 18 of the 80 candidate ids still visible, is that kind of number taken over every candidate in one turn.
Since the collapse rule, the 800-character cap and the fill order are all deterministic, the second number can be computed without calling the model. Recall@k already needs a gold id per question. I would log the raw tool results of each run, replay them through the context builder, and record whether the gold id is in the assembled prompt at the step that builds the table, or at the last step when none does. With recall@1 at 50%, the right series often arrives in the first search, and under VERBATIM_RESULTS = 3 that is the first result to lose its list once a turn has more than three tool results, so this is a number that can fall while recall rises, and if the runs from that week were logged, replaying them would show whether it did.
The same replay turns your closing advice, ask what size a constant was written for, into a number. I would feed it turns of 3, 6, 10 and 20 searches against the current catalog and see where survival breaks. It measures loss, not whether the model decides well with what survived, so it sits beside the end-to-end run rather than replacing it, but it needs no model call, which is what would let CI run it every time the catalog grows.
This is the number I should have had. The recall@1 observation is the sharp part: at 50% the right series usually arrives in the first search, which under VERBATIM_RESULTS = 3 is the first result to lose its list. The two move against each other by construction and I never looked at the second one.It does need no model call. We keep raw tool results in a trace log, so those runs are replayable through the context builder. Hadn't thought to point it at them.
One thing from how it actually failed: don't measure only at the step that builds the table. In the failing runs the model never got there, it searched 23 times and stopped. Survival at the last step of a turn that produced nothing is the case that would have caught it.
Twelve searches down to two is the number worth pulling on. The way I read it, grouping landed the model on a decision frame in one shot where a flat list leaves it with a fragment and a reason to look again. I ran into something close in an indexing pipeline where the fix wasn't better ranking. It was narrowing the candidate set to the point where the model would actually commit.
Same conclusion, opposite direction. Narrowing measured worse for me. I tried widening the pool and diversifying the members, and runs stopped completing at all; the extra candidates ate the step budget. What moved it was making the frame legible, not smaller: which headings the matches cluster under and how many siblings each holds. "Türkiye (TUR) GDP, Nominal — one of 196 rows under this heading" is decidable. The same line in a flat list isn't, and reformulating is the honest response to an undecidable list.
Recall climbing while end-to-end results drop is a good warning about offline metrics. Better top-1 hits can still hand the model three near-miss passages that it blends into one wrong answer. Judging each retrieval change by whether the task actually got resolved, and retiring the one that only won the offline metric, is the check I'd lean on.
Retiring the one that only won the offline metric is the part I had to actually do. The verified query repository is a real technique, it improved nothing measurable offline, and end-to-end it cost six runs in ten. Reverting it felt like giving up a feature; it was the only honest read of the numbers.The near-miss blending you describe is a different failure from mine and worth naming separately. Mine was evidence disappearing, yours is too much surviving and getting merged. Same offline metric can't see either.
The 55k catalog makes this painfully clear. Retrieval can improve while the agent regresses when context assembly hides the evidence.
Scale is the part I underestimated. At thirteen entries the same collapse rule was harmless, re-searching was cheap and a candidate list fit in the cap anyway. At 55k, every deleted line is a candidate the model won't find again, and it pays for that by searching more, which deletes more. The rule didn't change; its meaning did.