Short answer: use hybrid retrieval when exact identifiers and semantic meaning both matter; use vector-only retrieval when your corpus is small, stable, and mostly prose. For a privacy-focused personal knowledge manager, make that choice with an index-cost budget and a relevance test set, not with a model leaderboard.
| Retrieval shape | Pick this when | Main cost you carry |
|---|---|---|
| Vector-only | Notes are mostly narrative and vocabulary varies | Embeddings, vector storage, and re-indexing |
| Lexical-only | Users search IDs, names, or quoted phrases | Synonyms and paraphrases are easy to miss |
| Hybrid (lexical + vector) | A note mixes prose with exact terms | Two indexes and a fusion/tuning step |
The application scenario here is logistics: detecting semantically near-duplicate shipment records inside a private knowledge manager. A duplicate may say “package held at Newark hub” in one record and “NWRK facility placed parcel on hold” in another. The tracking number, however, is not semantic. It is an exact key. Losing either signal creates expensive review work.
What makes relevance tuning difficult in a private knowledge manager?
Retrieval-augmented generation (RAG) is a pipeline, not a single search call. The original RAG paper describes a generator conditioned on retrieved passages, with retrieval supplying knowledge that is separate from the model parameters. That separation is useful for private notes: you can update an index without retraining a model, and you can keep the source documents under your own access controls.
The failure mode is usually upstream of generation. If the first-stage retriever returns five near matches and omits the one record containing the decisive tracking event, a better prompt can't repair the omission. Relevance tuning therefore starts with a labeled set: query, expected record IDs, and a judgment such as exact duplicate, semantic duplicate, related, or irrelevant.
I use three measurements in that set: recall at k for “did we retrieve the right record?”, precision at k for “did we waste review slots?”, and a rank-aware score such as reciprocal rank for “how soon did the right record appear?”. Pick k deliberately. A top-5 review queue has a different cost profile from a top-50 investigation queue.
Keep the labels boring and explicit. A small table with 40 to 100 representative queries can expose a tuning regression faster than a week of anecdotal feedback. In a logistics slice, I would include a pair that shares a tracking number but differs in timestamp, a pair that uses “NWRK” versus “Newark,” and a pair where the only overlap is a carrier prefix. The expected labels should say which pair is mergeable and which pair must remain separate, because a relevance score alone cannot encode that business rule. Your mileage may vary, especially when the notes contain several languages or unusually short fields; record those slices instead of hiding them in one average.
That distinction matters.
Vector-only retrieval: when is its index cost justified?
Vector retrieval turns each chunk into an embedding and finds nearby vectors for a query embedding. It is strong when wording changes while intent stays similar. That is exactly what happens with free-form incident notes, handoff summaries, and explanations of why a delivery was delayed.
The index has a recurring bill in operational terms even when the software is self-hosted. You store a vector per chunk, maintain an approximate-nearest-neighbor structure, and regenerate vectors whenever chunking or the embedding model changes. A 768-dimensional float32 vector is about 3,072 bytes before graph or metadata overhead. That number is simple, but it is not the whole bill: duplicate metadata, replicas, and rebuild windows matter at scale.
The practical tuning levers are chunk size, overlap, embedding normalization, and the candidate count passed to any reranker. Larger chunks reduce vector count but blur the evidence. Tiny chunks sharpen matches but increase index entries and can separate the subject from its date or identifier. For shipment records, keep the tracking number and event timestamp in metadata even if the descriptive text is embedded.
Vector-only is a poor fit when a query is mostly an exact token. “ZX-4817” can be represented in a vector, but nearest-neighbor distance is not an equality constraint. Add a lexical filter or a metadata lookup before trusting the semantic score.
Hybrid retrieval: how should lexical and semantic signals be fused?
Hybrid retrieval runs a lexical method, such as BM25, beside vector search, then combines their candidate lists. The shape is easy to picture: two funnels feed one review queue. The lexical funnel catches “ZX-4817” and “Newark”; the semantic funnel catches “parcel held at NWRK facility.” Fusion decides which candidates survive.
Do not add raw scores from the two systems. Their scales are unrelated. Reciprocal Rank Fusion is a useful baseline because it combines rank positions rather than pretending the scores share units. A simple implementation can keep the top 20 candidates from each side, assign each rank a diminishing contribution, and sum contributions by record ID.
type Hit = { id: string; rank: number };
function reciprocalRankFusion(lists: Hit[][], k = 60): string[] {
const score = new Map<string, number>();
for (const list of lists) {
for (const hit of list) {
score.set(hit.id, (score.get(hit.id) ?? 0) + 1 / (k + hit.rank));
}
}
return [...score.entries()]
.sort((a, b) => b[1] - a[1])
.map(([id]) => id);
}
That function is intentionally small. Production code still needs tenant or notebook authorization filters, deleted-record handling, and deterministic tie breaking. Log the lexical rank, vector rank, fused rank, and final decision. Those fields make a relevance complaint debuggable without exposing note contents to a third party. When an operator asks why record A beat record B, the trace should show both candidate lists, the normalization applied to identifiers, the authorization scope, and the final duplicate policy outcome; otherwise the team ends up changing a threshold blindly and learning about the regression from a user.
Measure first.
For near-duplicate detection, add a second pass that compares normalized fields: whitespace, case, punctuation, and known carrier prefixes. Keep this pass separate from retrieval. Retrieval finds candidates; the duplicate policy decides whether two records can be merged. Mixing the policies makes a threshold change surprisingly destructive.
A field guide for choosing the architecture
Start with vector-only if the private corpus is under a few hundred thousand chunks, mostly prose, and the team can tolerate a rebuild when the embedding model changes. It keeps the first implementation understandable. Add lexical retrieval when users paste identifiers, quote titles, or expect a date and status token to be decisive.
Choose hybrid from day one when the logistics record has both narrative and structured identity, or when false positives trigger manual reconciliation. The extra index is a real trade-off: more storage, more ingestion work, and another set of metrics. The payoff is predictable behavior across paraphrases and exact terms.
The catch is privacy boundaries. A local index can still leak through logs, backups, or an overly broad result cache. Encrypt the index at rest, keep embeddings tied to the same deletion policy as source records, and test that a user cannot retrieve a note after its access grant is revoked. A design that improves recall while widening authorization is not an improvement.
I also keep an escape hatch for small notebooks: lexical search plus a lightweight semantic rerank can beat a permanently maintained vector index when writes are rare. That is a cost decision, not a universal ranking. Measure ingestion latency, index bytes per record, query p95, and reviewer minutes per accepted duplicate before committing to a larger topology.
Limits and a review loop
No retriever understands every domain synonym. Carrier codes change. A near duplicate can be intentionally different because its event timestamp changed by one minute. Set a policy for those fields and include borderline examples in evaluation.
Run the test set on every chunking, embedding, or fusion change. Alert on recall@10 and the rate of unauthorized candidates, then inspect a handful of ranked lists. Short feedback loops keep relevance tuning tied to the index cost that motivated the architecture.
References
- https://arxiv.org/abs/2005.11401
- https://www.elastic.co/guide/en/elasticsearch/reference/current/ Okapi BM25 overview
- https://www.cs.cmu.edu/~./wcohen/10-605/irbook/ Information Retrieval concepts
Top comments (0)