Short answer: apply the keyword stage as a generous eligibility filter, run vector similarity across those eligible PDF passages, then rerank a smaller union that still includes lexical-only and semantic-only candidates. Do not let the first stage become an accidental answer engine. For an education support system answering questions over policy manuals, the operational target is the best evidence set that fits a declared latency budget and still contains the page an instructor would cite.
The ordering matters because each stage can discard evidence that a later stage cannot recover. I would start experiments with 200 keyword candidates, 80 vector-scored candidates, and 20 reranked candidates, but those are test parameters, not universal constants. Tune them against labeled questions from the actual PDF folder. The page I care about is the one fired when a cited answer is wrong or empty, and it needs to identify which gate removed the evidence.
What page should fire when the right paragraph disappears?
Consider a course-support folder containing a current attendance policy, an older handbook, accessibility guidance, and instructor notes. A learner asks, "Can an excused lab absence be made up after the seven-day window?" Exact terms can find attendance documents quickly, while semantic similarity can surface a paragraph describing the exception without repeating the learner's wording. A reranker can compare the surviving passages in the context of the whole question.
The incident pattern is bounded and dull: ingestion succeeded, the vector query returned on time, and the final answer sounded plausible, yet the authoritative exception paragraph never reached the reranker because the keyword prefilter required every normalized query term. Nothing downstream was broken. The candidate contract was. The trace also looked healthy if examined one service at a time: the lexical stage had results, vector scoring completed before its deadline, the reranker received valid input, and every component could truthfully report success while the system as a whole had already lost the only passage that answered the question. This is why component availability is a weak proxy for retrieval correctness.
It failed quietly.
This is the invariant: a narrowing stage must expose what it excluded, not merely what it returned. Record counts and stable document identifiers at every boundary. If the relevant passage entered the keyword pool but vanished before vector scoring, inspect filter semantics or index synchronization. If it survived vector scoring but missed reranking, inspect the candidate cap and merge policy. If it reached reranking and still lost, the failure belongs to ranking evaluation.
Do not alert on a single score. Score distributions move when documents, tokenization, embeddings, or question mix change. Alert on user-visible symptoms and pipeline invariants: no eligible passages for a question class that normally has evidence, a sharp change in stage-to-stage retention, a latency objective breach, or a citation that cannot be resolved to an indexed PDF revision. Dashboards can help after the page; they should not decide that fluent prose is healthy.
How should hybrid retrieval combine a keyword prefilter and vector query?
A strict serial pipeline sounds tidy: keyword filter, vectors, reranking. It is brittle when "filter" means "keep only the highest lexical scores." BM25-style ranking is useful for identifiers, policy phrases, course codes, and rare names. Dense similarity helps when the question paraphrases the source. These signals fail differently, which is why their disagreement should survive.
Use the keyword stage to establish a broad, explainable set, including hard metadata constraints such as tenant, course, document status, and access scope. Within that set, compute vector similarity. Separately retain a small lexical-only tail. Merge by stable passage ID, attach both scores and ranks, and give the reranker the question plus passage text. Reciprocal rank fusion can merge ranked lists whose raw scores are not calibrated; a learned reranker is another stage, not permission to discard provenance.
| Stage | Primary job | Failure to expose | Useful signal |
|---|---|---|---|
| Keyword eligibility | Find exact terms and enforce scope | Relevant passage excluded | eligible count, matched terms |
| Vector scoring | Find paraphrases within scope | Candidate cap truncates a match | rank, similarity, duration |
| Candidate merge | Preserve disagreement | One list silently dominates | source ranks, unique count |
| Reranking | Order evidence for the full question | Correct evidence ranks poorly | final rank, duration |
Keep document revision and page number beside every passage. Text alone is weak lineage for a mutable PDF folder. A result tied to a file, content digest, and page 37 can be investigated; an answer tied only to a generated chunk ID usually cannot.
A preventative candidate contract
This core is independent of index and model providers. Adapters return ranks through narrow interfaces, while the merge keeps candidates admitted by either path.
package retrieval
import (
"context"
"sort"
)
type Passage struct {
ID, Document, Revision, Text string
Page int
}
type Ranked struct {
Passage Passage
Rank int
Score float64
}
type KeywordIndex interface {
Search(context.Context, string, int) ([]Ranked, error)
}
type VectorIndex interface {
SearchWithin(context.Context, string, []Passage, int) ([]Ranked, error)
}
type Reranker interface {
Rank(context.Context, string, []Passage, int) ([]Ranked, error)
}
func Merge(a, b []Ranked, offset float64) []Ranked {
hits := make(map[string]Ranked, len(a)+len(b))
scores := make(map[string]float64, len(a)+len(b))
add := func(list []Ranked) {
for i, hit := range list {
hits[hit.Passage.ID] = hit
scores[hit.Passage.ID] += 1 / (offset + float64(i+1))
}
}
add(a)
add(b)
out := make([]Ranked, 0, len(hits))
for id, hit := range hits {
hit.Score = scores[id]
out = append(out, hit)
}
sort.SliceStable(out, func(i, j int) bool {
if out[i].Score == out[j].Score {
return out[i].Passage.ID < out[j].Passage.ID
}
return out[i].Score > out[j].Score
})
return out
}
Return stage counts with results because observability belongs in the interface. Fail closed when lexical eligibility is empty instead of quietly asking a generator to improvise. Whether an empty set becomes a user-visible "I could not find evidence" response or triggers a controlled fallback is a product decision, but it must not masquerade as successful retrieval.
One trap deserves emphasis. If vector search receives only the lexical top 20, it is no longer an independent signal; it merely reorders the keyword engine's opinion. Use a broad pool, then verify with recall curves that the correct passage remains present as caps change. The limitation is compute: widening eligibility increases vector work and can push tail latency past the objective, while a narrow pool is faster but sacrifices recall before reranking can help. That is the explicit trade-off.
No stage gets a free pass.
Test the gates, not the prose
An end-to-end answer score is too coarse for diagnosis. Build a judgment set containing a question, acceptable passage IDs, the PDF revision, and required metadata constraints. Include exact-title questions, paraphrases, acronyms, negation, duplicated policy text, superseded handbooks, OCR noise, and questions whose correct outcome is no answer. Split results by class. A mean can hide the class that wakes somebody up.
For each candidate budget, measure whether an acceptable passage survives keyword eligibility, vector scoring, merging, and reranking. Plot recall against latency at every gate rather than reporting one end-to-end number. Then choose the smallest budgets that meet the documented recall target under the latency objective, including tail latency and timeouts. The decision is a quality-versus-latency envelope, not a favorite algorithm.
A preventative test should replace a source term with a plausible paraphrase and assert that the expected passage still reaches the reranker. Another should include a rare course code and assert that the lexical path preserves it even if semantic rank is weak. A third should mark an old revision inactive and assert that none of its passages enter either path. These tests locate regressions faster than reading generated answers.
Deploy candidate-budget changes behind a sampled comparison path. Run old and new retrieval configurations on the same questions without generating two answers merely to compare retrieval. Log stable passage IDs, ranks, stage durations, index revision, and configuration version; keep learner text out of logs unless policy explicitly permits it. Roll back when judged recall regresses or the latency objective is breached.
When should you skip the keyword gate?
Skip it when the corpus is tiny enough to score every authorized passage within the latency objective, or when lexical eligibility repeatedly loses relevant paraphrases and widening it removes any computational benefit. The same applies when OCR is so poor that exact tokens are unreliable. Vector-first retrieval followed by lexical fusion may then be the more defensible order.
Do not skip metadata filtering for access control or document status. Retrieval relevance and authorization are separate concerns, and a reranker must never be expected to remove material the caller was not allowed to retrieve.
For highly structured questions, the opposite can be true. If learners mostly search exact course identifiers, regulation clauses, or named forms, lexical retrieval may already produce a small, high-recall set; vector scoring can add latency without changing the evidence set. Measure disagreement between stages before paying that cost.
The operational rule is compact: preserve independent paths until evaluation proves one adds no recall, attach lineage before ranking, and page on evidence loss or objective breaches rather than score motion. The correct paragraph cannot be reranked after it has been discarded.
Top comments (0)