DEV Community

My retrieval counted the same text three times

Ashish sinha on September 19, 2026

A week ago I published a measurement: I had generated descriptions for 1,245 database objects to make retrieval better, and retrieval got worse. Th...

Read full post

Ahmet Özel • Sep 20

"A design whose whole premise is that prose helps should not lose to deleting the prose" was the right instinct to trust, and it generalises: a result that is suspiciously clean in the direction of the simplest possible action is usually measuring your harness rather than your system.

The publish-then-correct sequence is the genuinely useful part here. Most people who find the bug after the numbers went around just quietly stop mentioning the topic.

One thing worth adding to the postmortem: duplicate counting is cheap to catch as a hard assertion inside the scorer, not something you eyeball. Check that the retrieved id set contains no repeats and that its size equals k. It costs nothing and it fails loudly the first time a join or a chunk expansion fans out.

Ashish sinha • Sep 20

"A result that is suspiciously clean in the direction of the simplest possible action is usually measuring your harness rather than your system" is the sentence I want to steal. That is exactly the shape it had: deleting every description won at every cut, on both embedders, five of six cells significant. Nothing in a real system is that tidy.

On the duplicate assertion — it is there, and as a property test rather than an eyeball, which I should have said in the post. tests/test_any_schema.py, Hypothesis generating random catalogues:

assert len(sel) <= max(top_k, 0)
assert len(set(sel.table_names)) == len(sel.table_names), "duplicate hit"

plus a separate one over the foreign-key expansion path, which is the fan-out you are describing.

But your framing still lands, because that is the harness asserting it over generated catalogues, not the scorer asserting it on a live run. Those are different guarantees, and the one I have is the weaker of the two. Moving it down into the scorer costs nothing and covers the case where a join or a chunk expansion fans out in production rather than in CI.

Worth adding that the duplicate-id check would not have caught this particular bug, which is the uncomfortable part. The ids were unique. One field was feeding three scoring channels, so the same table won on three arms at once without ever appearing twice in the result. Another reader on the sibling post suggested channel-disjoint field accounting — log which source fields feed which arm — and that is the assertion that would have failed. Both are cheap; they catch different things.

On publishing the correction: the numbers had already gone out. Leaving them standing was the only option that was definitely wrong.

Ahmet Özel • Oct 5

Thanks for distinguishing the unique-result property from the scoring-channel bug. You are right that the duplicate-id assertion would not have detected the same field contributing through three arms, so I should not present it as coverage for this failure.

The source-field accounting sounds like the more direct invariant. A regression fixture with unique table IDs but intentionally overlapping channel inputs would make that distinction visible: result uniqueness should pass, while the channel-disjointness check should fail. Keeping both outcomes in the correction would show why adding a reasonable assertion is insufficient unless it targets the mechanism that actually inflated the score.

Ashish sinha • Oct 8

Agreed. That's the right split. The source-field accounting invariant exists now: test_body_channel.py builds the index and checks exactly what text each channel holds, so a field showing up in a second channel fails even when every result id is unique. Your fixture idea, unique ids with deliberately overlapping channel inputs, is the clean way to show why the uniqueness check alone was not enough.

Ahmet Özel • Oct 9

The accounting invariant catches a different failure from duplicate result IDs, so test_body_channel.py is the right place to make that distinction executable. I would add a positive control for intentionally repeated literal text across two different source fields.

That keeps the assertion about provenance assignment rather than text equality: identical wording can legitimately occur in a title and a description, while one source field appearing in two scoring channels is the bug. The deliberately overlapping-input fixture should fail for the second reason, even when both fixtures return unique IDs.

Ashish sinha • Oct 10

Agreed, and that's a sharper statement of what the test is for. The bug was never "the same words appear twice"; it was one source field feeding two scoring channels. A fixture where a title and a description legitimately share a sentence should still score each field once, and the overlapping-channel fixture should fail for the channel reason, not the text reason. I'll add that positive control next to the existing check in test_body_channel.py.

Igor Eduardo • Sep 20

The publish-then-correct honesty is rare — and the generic failure mode (“one field feeding three ranking signals”) is easy to ship with all tests green.

Two cheap asserts I’d put in front of any hybrid bake-off before arguing BM25 vs dense vs fusion: (1) unique result ids in the top-k (size == k, no dupes), (2) channel-disjoint field accounting — log which source fields feed lexical vs dense vs “prose” arms so the same comment can’t silently inflate three scores. Otherwise Recall@k compares different machines and “delete the prose” can look like a product insight when it’s a harness bug.

Hybrid stays a workload claim; channel integrity is what makes that claim measurable.

Ashish sinha • Sep 20

Both of these are good, and one of them I can tell you the status of because a reader found the exact failure yesterday.

(1) unique result ids, size <= k. This one exists, and as a property test rather than a fixture — tests/test_any_schema.py, Hypothesis generating random catalogues:

sel = cat.select(q, top_k=top_k, expand_fks=False)
assert len(sel) <= max(top_k, 0)
assert len(set(sel.table_names)) == len(sel.table_names), "duplicate hit"

with a second one covering the FK-expansion path, which is where a fan-out would actually show up. Your framing is still the part worth taking though: that is the harness asserting it over generated catalogues, not the scorer asserting it on a live run. Cheap to move down a level.

(2) channel-disjoint field accounting. This does not exist, and you have described the bug precisely. The whole finding in the post was one column comment reaching three of four channels. The fix was to take comments out of the body channel — but the way it was implemented, _body_text is a hand-copy of the vector builder embed_text with one line removed, and nothing asserts the relationship. Another reader (howcani, in the other post's thread) noticed yesterday that _body_text has no test on it at all while embed_text is asserted on in four files. So the next field added to the vector builder silently misses the body channel, and the symptom is a recall number moving in whatever experiment is running.

Which is your point, arriving from the other direction. Accounting that says this field feeds these arms would make it structurally impossible rather than a matter of two functions agreeing. That is going in.

"Recall@k compares different machines" is the sentence I should have had in the postmortem.