Started building a reranker CLI on top of Jev for the obvious reason: get the relevant documents to the top. Ended up with three modes in jev-reran...
For further actions, you may consider blocking this person and/or reporting abuse
Splitting “is it related?” from “does it contain answerable evidence?” is the distinction most dashboards flatten into one rerank score.
Practical check: next to nDCG / Recall@k, publish an empty-after-evidence-filter rate (and keep it as its own release gate). A pipeline can look healthy on ranking while silently delivering topic-adjacent, answer-empty context — especially on identifier-heavy or multi-constraint questions where lexical and dense disagree. Workload-honest hybrid eval cares about that membership failure as much as order.
Filter returning
[]should be a first-class outcome the caller can act on (abstain / re-retrieve), not a backfill that hides the miss.Thanks for this suggestion. In a follow-up using synthetic product documentation, I removed the answer-bearing candidates and checked whether the filter returned an empty result. A different product’s description still got through.
A prompt change fixed that case, but let more incomplete fragments through on other inputs. I haven’t found a stable overall improvement yet, and I’d like to keep testing this. Before making it a release gate, I’d want separate checks for unusable context getting through and useful evidence being dropped. And I agree that
[]should reach the caller so it can abstain or retrieve again.Splitting the post-retrieval stage into four separate judgments is the framing I would keep. Is it related, does it contain evidence, which part do I pass on, and does it add anything given what I already selected are genuinely different questions, and collapsing them into a single rerank score is why reranker tuning so often feels like it is not moving anything.
The fourth one is the odd one out because it is the only judgment that is not per-document. It needs the selected set as state, so it cannot be a stateless scorer like the other three, which is probably the real reason dedup did not ship rather than anything about its quality.
Pinning the version on the comparison makes those numbers more useful, not less. Worth saying out loud.
Yes, each selection changes what the next candidate adds, and those comparisons add cost. In these runs, I couldn’t find enough benefit to justify that cost, so dedup is something I’d like to revisit with a workload where it has more room to help. Pinning the model version also keeps model updates from becoming another variable in those comparisons.
I overinterpreted the reason dedup did not ship. Your measured lack of benefit is the stronger explanation; having state-dependent comparisons does not by itself make the cost unjustifiable.
A follow-up could report the benefit and extra latency by workload shape, with a category where several candidates independently contribute useful information and one where they repeatedly cover the same answer. That would locate where the selected-set comparison earns its cost, without treating dedup as a stage every pipeline automatically needs.
The split between related and evidence-bearing is the part I would keep when building a knowledge chatbot on top of retrieval.
A heading that names the policy is related. A sentence that states the retention number is evidence. Collapsing those into one rerank score is why a bot can sound on-topic and still invent the number when the top chunk never had it.
I would treat an empty filter result as a real outcome, not as a prompt to backfill. If nothing survives the evidence check, the bot should say the corpus does not support an answer and stop. That is cheaper and safer than letting a fluent model fill the gap from prior training.
The title and bibliography problem you measured is common in doc corpora too. Distance loves filenames and section titles. A short evidence filter after retrieval catches that without another full search.
Agreed on preserving the empty result. If nothing passes the evidence check, the bot should withhold an answer rather than fill the gap from prior training.
I’d have it say “I couldn’t find supporting evidence in the retrieved material.” Retrieval can miss the relevant passage, and the filter can reject a useful candidate. An empty result tells us what passed that check, but doesn’t establish that the whole corpus lacks the answer.
The title and bibliography cases are where that extra check makes sense: the candidates are already there, and we can ask whether they contain usable evidence without searching again. The part I’m still testing is how reliably it makes that distinction. In the follow-up I mentioned in another reply, a different product’s description passed the filter even after I removed the answer-bearing candidates. So I agree with the placement of the check, while treating its ability to catch those cases as something to measure on the target documents.
Your direction test for dedup has a second axis that only shows up in a corpus with history. Put B next to a later "Logs are retained for 90 days": the pair judge sees two candidates that each carry a retention number, and neither covers the other in your sense, because they contradict rather than overlap. The deciding input is not in the pair at all. It is which one was written later and whether it replaced the first. On arXiv papers and planted paraphrases that case does not arise, so the runs could not have shown it.
So in a store that accumulates edits, dedup and conflict resolution become the same step under different names. Similar candidates are either paraphrases, where your coverage direction decides, or versions, where recency or an explicit replaced-by link decides, and treating a version pair as a paraphrase pair can keep a fact that has already been corrected. If you revisit the mode on reposted content, that is the count I would add to your list: pairs where the item dropped was the newer one.
Thanks for this. The corpora had no edit history, so that case never appeared. My list of what to measure assumed similar candidates overlap, and a contradicting pair isn't on it.
A version pair needs an input the pair judge never gets. Which one is newer, and whether one supersedes the other, is metadata the caller holds while Jev only sees the two texts. When the caller already knows which items are versions of each other, through a replaced-by link or another reliable supersession rule, it can settle that before the candidates reach Jev without another model judgment. With only timestamps, finding the pair is the same similarity judgment dedup makes, and the two collapse into one step the way you describe.
If I revisit dedup on reposted content, I'll include version pairs and count the cases where the dropped item was the newer or explicitly superseding one.
The distinction between rerank and filter as separate decisions is the key insight here. Relevance and evidence are not the same judgment — a sentence can be on topic and contain no answer (your A example), and treating them as one thing produces confusing results because raising a relevance threshold doesn't fix what evidence filtering is actually for.
The dedup finding is the most interesting failure: on 23 queries, dropping duplicates and backfilling slots produced a 0.2% character reduction. The slot-backfill mechanic is what kills the expected gain — you don't get less context, you just get different context, and the context you swapped in may have lost the conditions and exceptions you were trying to preserve. The honest conclusion to ship the mode would have needed data where paraphrases dominate the top of retrieval, not arXiv papers.
The retriever behavior observation — "paper titles, section headings, figure captions and bibliography lines score well because they're lexically close to query terms but carry no information" — is the clearest description of what a reranker is actually fixing. The promoted chunks in your test having worse distances than the ones they displaced makes the mechanism concrete.
Good work on separating what each decision is actually making rather than treating the post-retrieval stage as one monolithic "scoring" step.
Thanks, the backfill observation makes me want to compare dedup with and without filling the vacated slots: how much context we remove, and how much useful evidence we retain in a fixed number of results. Conditions and exceptions would need to survive in either case. The distance comparison was revealing too: some of the closest chunks were titles or citation fragments, while the passages that could support an answer were farther down the list.