Short answer: for a simple ask-your-docs feature, keep Node.js at the web boundary, begin with keyword retrieval, and add semantic embeddings only when an eval set shows that paraphrased questions are being missed. If both methods earn their keep, retrieve with both and merge ranks before generation. This architecture is deliberately boring: one document pipeline, one shared chunk identifier, two optional retrieval paths, and evidence attached to every answer.
The important choice isn't “old search or AI search.” It is what kind of miss your SaaS help center can tolerate. Keyword search is a strong first pass for literal strings such as setting names, API fields, and copied error codes. Embedding retrieval is aimed at meaning expressed with different words. Neither result should be trusted because it sounds plausible; retrieval has to win on the questions users actually ask.
Keep the flow visible. Published help-center pages enter an ingestion job, become addressable passages, and receive stable IDs. The same passages feed a lexical index and, when justified, an embedding index. A Node.js request handler sends a normalized question to the retriever, receives ranked passages, builds a grounded prompt, then streams the generated answer and citations to the browser. Server-sent events fit that final one-way stream: MDN documents the EventSource interface for receiving events over an HTTP connection with the text/event-stream media type.
No mystery layer is required.
How should a simple Node.js SaaS help center combine semantic search, embeddings, and keyword search?
Treat retrieval modes as replaceable lanes behind one contract. The Node.js application shouldn't know how cosine similarity is computed or which lexical index is in use. It should send a query plus filters and receive ordered passage records with chunk_id, document_id, text, source URL, and retrieval metadata. That boundary lets a notebook experiment become a small Python service without moving prompt assembly, authentication, or browser streaming out of the main application.
Start with a single lane. A lexical baseline is easy to inspect: if the query contains an exact product term and the matching passage is absent, the tokens, filters, or index contents can be examined directly. Add an embedding lane after the eval set contains meaningful paraphrases that the baseline misses. Running two systems from day one creates two indexes to refresh, two score distributions to observe, and a harder answer to the most useful debugging question: why did this passage appear?
When both lanes pass evaluation, merge their ranks rather than averaging their raw scores. A keyword score and a vector similarity are outputs of different ranking systems; rank fusion avoids pretending they share a calibrated scale. Keep the candidate pool larger than the final context, deduplicate by stable chunk ID, and return enough metadata to reproduce the decision. The generator should never see a citation-free blob assembled from anonymous text.
Here is a compact retrieval core for the Python side of that boundary. It reads pre-chunked records from chunks.jsonl, calls a configurable embedding endpoint, calculates lexical overlap and cosine similarity, then combines rank positions. It is intentionally an exact scan: useful for validating behavior in a notebook and small test corpus, but not a claim that an in-memory scan belongs in every production deployment.
from __future__ import annotations
import json
import math
import os
import re
import urllib.request
from collections import Counter
from dataclasses import dataclass
TOKEN = re.compile(r"[a-z0-9_./-]+")
@dataclass(frozen=True)
class Chunk:
chunk_id: str
document_id: str
text: str
source_url: str
embedding: tuple[float, ...]
def load_chunks(path: str) -> list[Chunk]:
with open(path, encoding="utf-8") as handle:
return [Chunk(**json.loads(line)) for line in handle if line.strip()]
def tokenize(text: str) -> Counter[str]:
return Counter(TOKEN.findall(text.lower()))
def lexical_score(query: str, chunk: Chunk) -> float:
query_terms = tokenize(query)
chunk_terms = tokenize(chunk.text)
return sum(min(count, chunk_terms[term]) for term, count in query_terms.items())
def embed_query(query: str) -> tuple[float, ...]:
payload = json.dumps(
{"model": os.environ["EMBEDDING_MODEL"], "input": [query]}
).encode("utf-8")
request = urllib.request.Request(
os.environ["EMBEDDING_URL"],
data=payload,
headers={
"Authorization": f"Bearer {os.environ['EMBEDDING_KEY']}",
"Content-Type": "application/json",
},
method="POST",
)
with urllib.request.urlopen(request, timeout=20) as response:
body = json.load(response)
return tuple(body["data"][0]["embedding"])
def cosine(left: tuple[float, ...], right: tuple[float, ...]) -> float:
numerator = sum(a * b for a, b in zip(left, right))
left_norm = math.sqrt(sum(value * value for value in left))
right_norm = math.sqrt(sum(value * value for value in right))
return numerator / (left_norm * right_norm) if left_norm and right_norm else 0.0
def ranked_ids(scores: list[tuple[str, float]], limit: int) -> list[str]:
return [chunk_id for chunk_id, _ in sorted(scores, key=lambda row: row[1], reverse=True)[:limit]]
def reciprocal_rank_fusion(rankings: list[list[str]], offset: int = 60) -> list[str]:
fused: dict[str, float] = {}
for ranking in rankings:
for position, chunk_id in enumerate(ranking, start=1):
fused[chunk_id] = fused.get(chunk_id, 0.0) + 1.0 / (offset + position)
return sorted(fused, key=fused.get, reverse=True)
def retrieve(query: str, chunks: list[Chunk], limit: int = 6) -> list[Chunk]:
query_vector = embed_query(query)
lexical = ranked_ids(
[(chunk.chunk_id, lexical_score(query, chunk)) for chunk in chunks],
limit=20,
)
semantic = ranked_ids(
[(chunk.chunk_id, cosine(query_vector, chunk.embedding)) for chunk in chunks],
limit=20,
)
by_id = {chunk.chunk_id: chunk for chunk in chunks}
return [by_id[chunk_id] for chunk_id in reciprocal_rank_fusion([lexical, semantic])[:limit]]
if __name__ == "__main__":
for hit in retrieve("How do I change the invoice contact?", load_chunks("chunks.jsonl")):
print(hit.chunk_id, hit.source_url)
This is the notebook-to-production checkpoint. Replace the in-memory scan with an index only after corpus size and latency measurements demand it; preserve the input and output contract so the eval harness doesn't care. The lexical scorer in the sample is a transparent baseline, not a sophisticated production ranker. Its value is that a team can run it, inspect it, and establish whether extra machinery improves the result.
There is also one quiet invariant: embeddings stored for passages and embeddings created for questions must come from the same model configuration. Record that configuration with the index generation, and rebuild the affected vectors when it changes. Don't let a deployment point queries at an index whose provenance is unknown.
Make retrieval earn its place with evaluation
An eval set should precede the semantic index. Pull representative questions from permitted support data, redact sensitive content, and label each question with acceptable source documents or passages. Include exact UI terms, terse fragments, conversational paraphrases, ambiguous account questions, and questions the help center cannot answer. That last group matters because an ask-your-docs system needs a measured refusal path, not a forced citation.
Score retrieval before scoring prose. Recall at a fixed candidate depth answers whether at least one acceptable passage reached the generator. Rank-sensitive measures help when context space is tight. Then evaluate the final response for citation support, correctness against the cited passage, refusal when evidence is absent, and useful presentation. A polished answer can hide a retrieval miss; separating the stages keeps the diagnosis honest.
I prefer a small, reviewed eval set that runs on every retrieval change over a large pile of synthetic questions nobody reads. I'm not sure synthetic expansion improves a particular corpus until its outputs are sampled and compared with real query language. The resolving evidence is straightforward: measure performance separately on real questions and generated variants, then inspect where the rankings disagree.
The comparison needs slices, not one aggregate score. Exact identifiers can make a lexical system look excellent while paraphrased policy questions fail. A semantic system can lift the paraphrase slice while moving a precise identifier down the list. Hybrid retrieval is justified when it improves the weak slices without unacceptable regressions elsewhere; it isn't justified merely because two retrievers feel more advanced than one.
Consider a concrete eval pair without turning it into a made-up benchmark. One question copies a field name from the product UI: “invoice contact.” Another asks, “Where will billing notices go?” The expected source passage may contain both ideas but only the first phrase. Inspect the lexical and semantic ranks for each question, then inspect the fused result and the cited answer. If lexical retrieval finds the copied label but misses the paraphrase, the miss has a clear shape. If semantic retrieval recovers the paraphrase but demotes the exact label, that trade-off is visible too. Now add an unanswerable variation about changing a payment method when the indexed passage discusses only notification recipients. A system that retrieves the loosely related billing page and invents steps has failed even if its prose looks helpful. This single cluster exercises literal matching, paraphrase retrieval, fusion, and refusal while keeping the expected evidence reviewable. Expand the cluster with actual query language only after support data shows which variations matter. The point isn't to manufacture a flattering score; it is to expose how each stage behaves under a controlled change in wording and answerability, then carry the losing cases into the next regression run.
| Observed eval result | Smallest justified architecture change |
|---|---|
| Exact terms and paraphrases both rank well | Keep the lexical path |
| Exact terms rank well; reviewed paraphrases miss | Test an embedding lane and rank fusion |
| Relevant passages rank, but answers lack support | Fix prompt and citation evaluation before retrieval |
| Unanswerable questions receive confident answers | Add and test an explicit insufficient-evidence path |
| Both lanes regress after publication | Audit corpus revision, chunking, and index freshness |
Stop there for a moment.
Generation settings belong in the same experiment record because retrieval depth changes prompt size and answer behavior. Log the corpus revision, chunking rule, retriever configuration, embedding model identifier, candidate depth, prompt revision, and generator model identifier. This is prompt-cost awareness in practical form: each extra passage consumes context and can add distraction, so “retrieve more” is a hypothesis to test rather than a default safety margin.
The failure modes are mostly around the model
Chunking is the first trap. Splitting solely by a fixed character count can detach a procedure from its prerequisites or separate a table row from its header. Preserve headings and source URLs, prefer coherent sections, and let the eval failures reveal where a long section needs finer boundaries. Duplicated navigation, footers, and repeated legal text should be removed before either index sees them, because repeated boilerplate can occupy several candidate positions while adding no new evidence.
Freshness is next. Publication, update, and deletion events must drive both retrieval lanes from the same source-of-truth revision. Use idempotent jobs and stable content hashes so an unchanged passage doesn't need new work. A partial refresh shouldn't expose lexical results from one revision and semantic results from another. The operational goal is not clever indexing; it is being able to answer which document revision produced a cited passage.
Permissions cannot be repaired after retrieval. If the corpus contains tenant-specific or role-restricted material, apply access filters before candidates enter the prompt, and test those filters independently from relevance. A highly relevant forbidden passage is still forbidden. For a public help center, this is simpler, but draft articles and retired pages can create the same class of leak if the ingestion source is broader than the published site.
Empty or weak evidence needs explicit behavior. Define a minimum evidence policy, return a stable “not found in the help center” state, and give the UI a route to ordinary support. Don't ask the language model to decide, from vibes, whether the retrieved context is trustworthy. Thresholds vary with the ranker and corpus, so tune them against labeled answerable and unanswerable questions instead of copying a similarity number from an unrelated implementation.
Streaming comes last. It improves perceived responsiveness, but it doesn't improve retrieval quality. A Node.js endpoint can expose a one-way event stream to the browser while the retrieval service remains a normal request-response dependency. The browser client should distinguish answer tokens, citations, completion, and application-level error events. Automatic reconnection is part of the browser EventSource behavior described by MDN, so assign event IDs only when replay semantics are intentional; blindly reconnecting to a non-replayable generation can duplicate text.
The catch is operational load. Two retrieval lanes add index refresh work and a query embedding call, while generation already dominates the user's wait in many designs. Measure each phase separately. If keyword retrieval meets the quality target, semantic search is not suitable merely as an architectural ornament. Stick with the simpler index when exact terminology dominates, and consider semantic retrieval when genuine paraphrase misses remain visible in the eval slices. For high-volume or latency-sensitive workloads, an exact in-process vector scan like the example is also the wrong production choice; use an index designed for the measured corpus and service target.
Ship the smallest architecture you can explain
Before release, walk one document through publication, chunking, both indexes, retrieval, prompt assembly, citation rendering, update, and deletion. Then walk one unanswerable question through the refusal path. The logs should connect those steps with request IDs and stable passage IDs without recording sensitive prompt content by default. Dashboards should separate retrieval latency, generation latency, empty-evidence rate, citation coverage, and eval regressions; a single end-to-end latency chart can't tell the team what to fix.
A gateway can keep model-provider details out of application code. For teams that choose that boundary, an open-source project such as LiteLLM documents a centralized proxy approach, but the architectural requirement is the stable interface, not that specific implementation. Pin model identifiers in configuration, capture usage returned by the chosen model interface, and make retries bounded. Cost controls should focus on changed-chunk embedding, query volume, prompt size, and accidental re-indexing rather than on an assumed universal price comparison.
My release criterion is plain: the retrieval method must beat the lexical baseline on reviewed queries, citations must resolve to the exact published passages, access and freshness tests must pass, and the team must be able to explain a bad ranking from logs. If embeddings don't clear that bar, leave them out. If they do, keep the hybrid boundary small enough that the next evaluation can replace either lane without rewriting the Node.js application.
That is a simple architecture in the useful sense. It has fewer hidden decisions, not fewer quality checks.
Sources
- MDN, “Using server-sent events”: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
- LiteLLM, open-source LLM gateway repository: https://github.com/BerriAI/litellm
Top comments (0)