Short answer: use keyword and semantic retrieval in parallel, merge their candidates by rank, rerank a small shared pool, and allow the chatbot to return no answer when the evidence is weak.
That is the simplest hybrid search shape I would move from a Python notebook into a Node.js docs chatbot. It preserves exact identifiers, catches paraphrases, and gives the final scoring pass a bounded workload. The important decision isn't which library owns each box. It's whether the same document IDs, access rules, and evaluation cases survive every box.
Prove each handoff.
The data flow is plain: normalize a question without deleting meaningful punctuation, send it to lexical and embedding indexes, merge the returned chunk IDs, load the authorized text for those IDs, score each question-passage pair, and pass only the accepted passages to generation. Keep generation downstream. Retrieval should be testable without spending a single output token.
How should a Node.js docs chatbot combine keyword search, embeddings, and rerank?
Run keyword search and semantic search as separate retrievers. The keyword branch is responsible for literal evidence such as ERR_MODULE_NOT_FOUND, cache_ttl, version strings, and command flags. The embedding branch is responsible for meaning when the reader's wording differs from the documentation. Neither branch gets to declare a winner by its raw score because lexical and vector scores have unrelated scales.
Merge by rank instead. Reciprocal rank fusion is a compact option: for each result at position rank, add 1 / (k + rank) to that chunk's fused score. A chunk returned by both branches rises naturally, while a strong specialist result can still enter the candidate set. The constant controls how quickly position matters; it isn't an accuracy guarantee, so treat it as an eval parameter rather than folklore.
Then rerank only the merged pool. In a hosted architecture, the Node.js request handler can call this retrieval component over an internal interface; the example is Python because that makes the scoring pipeline easy to run in a notebook and under an eval harness. The boundary is just structured data: a question in, ordered evidence records out.
Here is a complete, dependency-free example. The tiny embedding function is deliberately a local stand-in so the program runs as pasted; replace it with the same embedding implementation used to build your production index. The reranker is also intentionally transparent. Its job here is to expose the control flow and evidence threshold, not to pretend token overlap is a trained relevance model.
import math
import re
from collections import Counter, defaultdict
DOCUMENTS = [
{
"id": "errors/module-resolution",
"text": "ERR_MODULE_NOT_FOUND means the runtime could not resolve an imported module.",
"embedding": [1.0, 0.0, 0.0],
"allowed_groups": {"public"},
},
{
"id": "guides/retry-policy",
"text": "Configure exponential backoff and a maximum retry count for transient failures.",
"embedding": [0.0, 1.0, 0.0],
"allowed_groups": {"public"},
},
{
"id": "reference/cache",
"text": "Set cache_ttl in seconds to control how long cached entries remain valid.",
"embedding": [0.0, 0.0, 1.0],
"allowed_groups": {"staff"},
},
]
def tokens(text):
return re.findall(r"[a-z0-9_]+", text.lower())
def embed_query(question):
"""Runnable stand-in for the production query-embedding function."""
words = set(tokens(question))
return [
float(bool(words & {"module", "import", "resolve", "err_module_not_found"})),
float(bool(words & {"retry", "backoff", "transient"})),
float(bool(words & {"cache", "ttl", "expiry", "expire"})),
]
def cosine(left, right):
numerator = sum(a * b for a, b in zip(left, right))
left_norm = math.sqrt(sum(value * value for value in left))
right_norm = math.sqrt(sum(value * value for value in right))
return numerator / (left_norm * right_norm) if left_norm and right_norm else 0.0
def keyword_search(question, documents):
query_counts = Counter(tokens(question))
scored = []
for document in documents:
body_counts = Counter(tokens(document["text"]))
score = sum(query_counts[word] * body_counts[word] for word in query_counts)
if score:
scored.append((document["id"], float(score)))
return sorted(scored, key=lambda item: item[1], reverse=True)
def semantic_search(question, documents):
vector = embed_query(question)
scored = [
(document["id"], cosine(vector, document["embedding"]))
for document in documents
]
return sorted(
(item for item in scored if item[1] > 0.0),
key=lambda item: item[1],
reverse=True,
)
def reciprocal_rank_fusion(result_lists, k=60):
fused = defaultdict(float)
for results in result_lists:
for rank, (document_id, _raw_score) in enumerate(results, start=1):
fused[document_id] += 1.0 / (k + rank)
return sorted(fused, key=fused.get, reverse=True)
def rerank(question, documents):
question_terms = set(tokens(question))
ranked = []
for document in documents:
passage_terms = set(tokens(document["text"]))
coverage = len(question_terms & passage_terms) / max(len(question_terms), 1)
semantic = cosine(embed_query(question), document["embedding"])
ranked.append((0.6 * semantic + 0.4 * coverage, document))
return sorted(ranked, key=lambda item: item[0], reverse=True)
def retrieve(question, group="public", candidate_limit=8, final_limit=3, minimum=0.25):
visible = [doc for doc in DOCUMENTS if group in doc["allowed_groups"]]
lexical = keyword_search(question, visible)
semantic = semantic_search(question, visible)
candidate_ids = reciprocal_rank_fusion([lexical, semantic])[:candidate_limit]
by_id = {document["id"]: document for document in visible}
candidates = [by_id[document_id] for document_id in candidate_ids]
ranked = rerank(question, candidates)
return [
{"id": document["id"], "text": document["text"], "score": round(score, 3)}
for score, document in ranked[:final_limit]
if score >= minimum
]
if __name__ == "__main__":
print(retrieve("Why does my import fail with ERR_MODULE_NOT_FOUND?"))
Notice where authorization happens: before either search branch. Filtering after retrieval can leak restricted titles, IDs, or score patterns into traces, and it can waste the candidate budget on passages the caller cannot use. In systems whose index cannot enforce access filters, retrieve from a partition already scoped to the caller rather than trusting the generation prompt to ignore forbidden text.
One caveat matters. This demonstration uses one small record shape and synchronized indexes. Production ingestion needs a stable chunk ID shared by the lexical index, vector index, metadata store, and source document; without it, fusion can quietly join different revisions of what appears to be the same passage. That's a data-contract problem, not a model problem.
Make the eval set decide the budget
Start with questions and expected evidence, not answers written by the generator. A useful row contains the query, the IDs of acceptable chunks, the caller's access group, and a category such as exact identifier, paraphrase, multi-part question, or unanswerable. This makes failures attributable. If an expected chunk never enters the fused pool, tune retrieval or chunking. If it enters and falls after reranking, inspect the second-pass scorer. If good evidence reaches generation and the answer still fails, retrieval isn't the current bottleneck.
I don't assume a universal candidate count. I'm not sure one exists. Your mileage may vary with corpus duplication, chunk length, and how often queries contain exact technical strings. Sweep the lexical depth, semantic depth, fused pool size, final evidence count, and refusal threshold against a frozen eval set; record retrieval quality, latency, and calls to embedding or reranking models for every configuration. This is where prompt-cost awareness pays off: retrieval experiments can run without generation, and only the finalists need end-to-end answer grading.
Keep at least three measurements separate. Candidate recall asks whether any acceptable evidence survived retrieval. A rank-sensitive metric asks whether useful evidence reached the first few positions. The refusal test asks whether unsupported questions produce an empty result. Combining them into one dashboard number makes a cheap, bad system and an expensive, marginally better system surprisingly hard to distinguish.
No vibes.
A notebook is excellent for plotting those trade-offs, but the saved artifact must be boring enough for CI: versioned query fixtures, index revision, embedding revision, reranker revision, configuration, and per-query results. Re-run it when documents are rechunked, when preprocessing changes, or when either scoring model changes. A green answer-quality snapshot from the old index doesn't certify the new one.
Failure modes worth designing out
Hybrid retrieval can still fail in mundane ways. Normalization may split foo_bar in one index but preserve it in the other. A document deletion may reach the metadata store and miss the vector index. Near-duplicate chunks can occupy every top position. Long passages can bury the sentence that actually answers the question. A reranker can receive text in a different order from the IDs it scores. Each failure looks like “the model answered badly” unless the trace records each stage.
Preserve a compact evidence trail under one request ID: normalized query, accessible corpus revision, ranked IDs from each retriever, fused IDs, reranked IDs and scores, final accepted IDs, and refusal reason. Don't log raw queries by default merely because debugging is easier. A docs question can contain a person's name, a private ticket, or copied credentials. Under GDPR, data handling needs a lawful basis and should follow principles including purpose limitation, data minimization, storage limitation, and security. Decide retention and deletion behavior before query logs become an accidental archive.
Retrieved documents are untrusted input too. OWASP's guidance for LLM applications identifies prompt injection and sensitive-information disclosure among the major risks. Treat retrieved prose as quoted evidence, keep system instructions outside it, constrain any tools independently of the prompt, and carry source identifiers into the answer so a user can inspect the basis. A sentence inside a document must never grant itself permission to call a tool.
Keep the blast radius small.
The catch is that a reranker adds latency, model lifecycle work, and another place where text may cross a trust boundary. It is not suitable when plain keyword retrieval already meets the evaluated recall and ranking target, when requests cannot leave a controlled environment and no acceptable local scorer exists, or when the latency budget cannot absorb the second pass. Stick with lexical search for a corpus dominated by exact codes and identifiers. Use semantic retrieval alone when paraphrase is common and the eval set shows the lexical branch contributes no unique relevant chunks. Hybrid plus rerank is a measured choice, not a maturity badge.
What should ship with the retrieval code?
Ship the index contract and eval fixtures with it. The contract should define chunk IDs, source revision, text field, embedding revision, authorization labels, and deletion semantics. Validate those fields during ingestion, build a new index revision away from live traffic, run the frozen retrieval suite against it, and switch traffic only after the checks pass.
At request time, set independent deadlines for lexical retrieval, semantic retrieval, and reranking. Define the degradation policy explicitly: if one retriever misses its deadline, either continue with the other and mark the trace, or refuse the request when the remaining evidence cannot meet the threshold. Bound candidate counts before the reranker. Bound passage length before generation. A retry must carry the same corpus revision so one user request doesn't mix index generations.
The operational checklist is short in spirit, even if the implementation isn't: make every stage observable, reject malformed chunks at ingestion, evaluate refusals alongside successful queries, test authorization as part of retrieval, cap token and candidate budgets, retain only the diagnostic data you can justify, and rehearse index rollback. Once those properties are automated, moving the orchestration from a notebook into a Node.js endpoint is mostly interface work. The hard part is preserving evidence and invariants while the corpus changes underneath it.
References
- OWASP Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/
- GDPR full text: https://gdpr-info.eu
Top comments (0)