Short answer: build the docs chatbot as two permission-filtered retrieval paths, one for keyword search and one for embeddings, fuse their ranks by stable chunk ID, rerank only a bounded candidate set, and generate an answer only when the selected passages support it.
The governing constraint is evidence control, not conversational fluency. A useful system must be able to reconstruct which document revision, access scope, index version, candidate ranks, and reranker revision produced an answer; otherwise a polished response cannot be distinguished from an unsupported one after a policy changes or a document is deleted. This makes the simple design slightly more formal than a demo, but it also keeps each failure attributable to ingestion, authorization, retrieval, reranking, or generation.
How should a simple Node.js docs chatbot combine keyword search, embeddings, and rerank?
Treat keyword and semantic search as independent retrieval witnesses. The lexical branch protects exact strings such as ERR_AUTH_042, configuration keys, statutory terms, and product version numbers, while the embedding branch catches paraphrases whose vocabulary differs from the source. Run both against the same tenant, document, and access-control filter; merge by immutable chunk ID; combine rank positions rather than unrelated raw scores; and send the bounded result to a reranker. The Node.js coordinator can execute those independent reads concurrently, although concurrency must never weaken the shared authorization predicate.
Keep the order explicit: authorize, retrieve, fuse, rerank, then generate. Don't retrieve globally and remove forbidden passages later, because candidate IDs, caches, traces, and reranker inputs can all disclose information before the answer is assembled. Retrieved text is untrusted data as well — a paragraph that says "ignore prior instructions" remains documentary evidence, not an instruction to the application. OWASP's Top 10 for LLM Applications is relevant here because prompt injection and sensitive-information disclosure are architectural concerns, not prompt-writing inconveniences.
The two branches will disagree. Good.
That disagreement is the reason to preserve both ranked lists. If an exact error code ranks first lexically but disappears from the semantic results, the system has useful evidence about why the chunk survived; if a plain-language question maps to a differently worded procedure, semantic retrieval can supply a candidate that literal matching never saw. Reranking is a second-stage ordering decision, not a recovery mechanism for evidence absent from both lists, so an empty or weak candidate set should produce an abstention rather than an imaginative answer.
Fail closed.
The size of that set should come from evaluation, not folklore. I'm not sure any universal cutoff survives changes in corpus size, chunk length, query distribution, latency objective, and reranker behavior; measure candidate recall and tail latency at several bounds, then pin the chosen value in a versioned retrieval policy. Your mileage may vary, especially when manuals contain many near-duplicate release notes.
Make ingestion and retrieval replayable
Every source revision needs a deterministic identity. A practical chunk record carries tenant ID, source ID, source revision, chunk ordinal, normalized-content hash, access labels, and deletion state, while an index manifest records the lexical analyzer configuration and embedding revision. Deriving chunk IDs from stable source coordinates makes retried ingestion converge on the same records and gives reconciliation a precise comparison target across the source store, lexical index, vector index, and cache.
Exactly once is a business invariant to verify, not a queue setting to assume. Give each ingestion operation an idempotency key, make writes repeatable, advance a checkpoint only after all derived records are durable, and periodically compare expected chunk IDs with indexed chunk IDs. A 429 from a dependency must preserve the operation identity across bounded retry; a deadline or partial batch must remain incomplete until reconciliation proves convergence. This distinction matters because "message acknowledged" and "document searchable in both indexes" describe different states.
Consider a hypothetical revision policy-17@42 split into 80 deterministic chunks. The worker writes all 80 lexical records, receives a 429 while submitting the vector batch containing chunks 41 through 60, and is then delivered the same queue message again. If the retry creates new chunk IDs, advances the source checkpoint before the vector write, or treats the lexical success as completion, the chatbot can present two incompatible views of the same revision without any obvious request-time error. With an idempotency key derived from policy-17@42, stable chunk IDs, and a manifest that remains pending until both index counts reconcile to 80, the second delivery repeats harmless writes, retries only unfinished material if the storage contract permits it, and reaches one inspectable state. The important number is not how many queue messages were consumed; it is whether the authoritative set of 80 eligible IDs equals the sets queryable through each retrieval path. The same comparison must run after replacement or erasure, when the expected count may be zero. This example is intentionally mechanical because the dangerous failure is quiet: every component can report local success while the composed retrieval contract remains false.
Query execution deserves the same audit discipline. Record a pseudonymous query or trace ID, authorization scope, index manifest, the two ordered candidate lists, fusion parameters, reranker revision, selected evidence IDs, per-stage latency, and any abstention reason. Avoid placing raw confidential questions or full passages in general-purpose logs. GDPR requirements such as purpose limitation, data minimization, storage limitation, accountability, and erasure should shape retention and deletion workflows, and compliance owners must determine the applicable policy for the jurisdiction and organizational role; engineering can enforce that policy, but it cannot invent it.
Deletion is a multi-store transaction spread over time. Mark the source revision unavailable first, exclude it from both query branches immediately, then remove its lexical entries, vectors, cache values, and retained trace content under an idempotent deletion operation. Reconciliation closes the loop. This is deliberately ledger-like: an intent, attributable state transitions, and a final proof that all materialized views agree.
Use a deterministic fusion boundary
Reciprocal-rank fusion is useful because it combines positions without claiming that a lexical relevance score and vector similarity have a common numerical meaning. The following Go function is intentionally narrow; a Node.js application can implement the same contract in-process or call an internal retrieval component, but the important properties are stable IDs, deterministic ties, validated inputs, and persisted before-and-after ranks.
package retrieval
import (
"errors"
"sort"
)
type Hit struct {
ChunkID string
Rank int
}
type Candidate struct {
ChunkID string
Score float64
}
// Fuse combines rank positions from already-authorized result sets.
func Fuse(keyword, semantic []Hit, rankConstant float64, limit int) ([]Candidate, error) {
if rankConstant <= 0 || limit < 1 {
return nil, errors.New("rank constant and limit must be positive")
}
scores := make(map[string]float64, len(keyword)+len(semantic))
add := func(hits []Hit) {
for _, hit := range hits {
if hit.ChunkID == "" || hit.Rank < 1 {
continue
}
scores[hit.ChunkID] += 1 / (rankConstant + float64(hit.Rank))
}
}
add(keyword)
add(semantic)
result := make([]Candidate, 0, len(scores))
for chunkID, score := range scores {
result = append(result, Candidate{ChunkID: chunkID, Score: score})
}
sort.Slice(result, func(i, j int) bool {
if result[i].Score == result[j].Score {
return result[i].ChunkID < result[j].ChunkID
}
return result[i].Score > result[j].Score
})
if len(result) > limit {
result = result[:limit]
}
return result, nil
}
The lexical and semantic inputs must already be authorization-filtered. Persist them before the reranker call, reject any returned ID that was not in the submitted candidate set, and fetch passage text by both chunk ID and source revision so a concurrent reindex cannot silently change the evidence. Retries should reuse an operation ID. If the reranker exceeds its deadline, follow a declared policy — use the fused order for low-risk informational queries or abstain for consequential ones — and label that path separately in metrics and traces.
A deterministic tie-break looks minor until two equal fusion scores change order between runs, creating different reranker inputs and making an evaluation impossible to reproduce. Stable ordering removes that ambiguity. It doesn't guarantee relevance, but it guarantees that the same authorized inputs and policy produce the same pre-rerank sequence.
Replay matters.
Compare failure behavior, then roll out in stages
Do not choose an architecture from a five-question demo. Build a labeled evaluation set containing exact identifiers, paraphrases, ambiguous acronyms, superseded revisions, access-controlled passages, malicious instructions embedded in ordinary prose, and questions for which abstention is correct. Separate tuning queries from held-out queries, and report results by class: candidate recall tests whether retrieval found the evidence, reranking quality tests whether it rose high enough, citation support tests whether the answer stayed within the selected passages, and authorization tests must have zero tolerance for cross-scope results.
| Design | Useful when | Limitation to measure | Audit requirement |
|---|---|---|---|
| Keyword only | Terminology is stable and exact codes dominate | Paraphrases can miss the relevant passage | Retain analyzer and index revisions |
| Embeddings only | Vocabulary varies and the task is exploratory | Plausible neighbors can omit a required literal | Retain model, index, and distance-policy revisions |
| Hybrid plus fusion | Exact terms and paraphrases both matter | Two indexes require reconciliation | Preserve both lists and fusion parameters |
| Hybrid plus rerank | Candidate context is subtle enough to justify another stage | Added latency, cost, and revisioned behavior | Link submitted candidates to returned ranks |
The catch is operational weight. Hybrid retrieval with reranking is not suitable when a small corpus has stable vocabulary and a measured lexical baseline already satisfies recall, authorization, and latency objectives; stick with keyword search until held-out evaluation exposes a semantic gap. Embedding-only retrieval can also be reasonable for low-consequence discovery where approximate neighbors are acceptable. For payment procedures, compliance instructions, or content that can trigger an external action, require citations, explicit abstention, and stricter evidence thresholds.
Cost should be treated as an operational measure rather than a vendor comparison. Track ingestion embeddings, query embeddings, reranked candidates, index storage, cache effectiveness, and engineering time, then relate them to successfully supported answers by evaluation class. Track latency by branch and stage as well; parallel search reduces elapsed time only when deadlines, connection pools, and cancellation are controlled, and the slowest dependency otherwise owns the tail.
Roll out by shadowing the new retrieval policy beside the existing path. Freeze an evaluation set, run both paths under identical authorization, compare evidence recall and stage latency without exposing the new answers, then canary by corpus or tenant with the index manifest and policy version pinned. Promotion criteria should be explicit per query class, because a strong aggregate score can conceal failures on rare error codes or restricted documents.
Before broad release, rehearse reindexing, rollback, and erasure. Delete a test revision and verify that it is excluded from search immediately and later absent from every derived store; replay a query against a pinned manifest; repeat an ingestion batch with the same idempotency keys; and confirm that an unsupported question abstains. A migration is complete only when reconciliation proves the old and new state boundaries are understood, not when the first answer looks convincing.
Top comments (0)