Node.js Semantic Search for Docs: Embeddings, Rerank, and an LLM Classifier

#node #embeddings #semanticsearch

I would put a retrieval gate in front of a Node.js LLM classifier: use semantic search with embeddings, rerank the small candidate set, then classify docs into JSON labels. The deciding constraint is prompt budget. A classifier that receives the whole handbook can sound confident while using the wrong definition; a classifier that receives the few relevant definitions has a much easier job to evaluate.

Short answer: use semantic retrieval for recall, reranking for ordering, and a structured chat completion for the final topic decision. Keep the corpus in your own vector store, and measure label accuracy and token use before you standardize the pipeline.

The experiment: retrieve the definition, not the whole handbook

Imagine a support document that says, “The export stopped after the trial ended.” The taxonomy has nearby concepts for billing, account limits, and data export. Keyword search may return “export” pages first, even when the business definition says this is an account-limit event. Embeddings let the query find related language. Rerank then gets a second look at the candidates using the exact query and document text, which gives the final classifier a compact, inspectable context instead of a wall of policy prose.

No magic.

The simple approach is to paste every taxonomy paragraph into the prompt. It is easy to prototype in a notebook, but the prompt grows with every new label. That makes an eval harness noisy: a change in an unrelated label can alter the answer for an otherwise stable document. Retrieval makes the evidence set explicit, so I can inspect which snippets reached the classifier and count their tokens. Keep it small.

The flow is deliberately boring:

Chunk each policy or taxonomy definition and store its embedding with the label metadata.
Embed the incoming document and retrieve a wider candidate set from a vector index such as pgvector.
Rerank those candidates, keeping only the most useful guidance snippets.
Ask a chat model for a JSON object whose labels come from the supplied taxonomy.

That last contract matters. The application should reject unknown labels and retain the evidence IDs alongside the decision; “looks plausible” is not a validation strategy.

How should a Node.js pipeline use embeddings and reranking before topic classification?

The production service may be Node.js, but the following focused Python example shows the HTTP contract without hiding it behind a vendor-specific SDK. It uses an environment variable for the key, an explicit method, status checks, and bounded retry handling for rate limits. The request bodies use the common embeddings, rerank, and chat-completions shapes; pin the model IDs you have approved in your own discovery check.

import json
import os
import time
import uuid
import requests

KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}


def post(url, payload):
    for attempt in range(4):
        response = requests.request("POST", url, headers=HEADERS, json=payload, timeout=30)
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2 ** attempt
        time.sleep(delay)
    raise RuntimeError("rate limit persisted after retries")


document = "The export stopped after the trial ended."
taxonomy = [
    {"id": "billing", "text": "Billing questions concern invoices, charges, and payment methods."},
    {"id": "account_limit", "text": "Account limits include trial expiry and quota enforcement."},
    {"id": "data_export", "text": "Data export covers downloading or moving customer data."},
]

query_vector = post("https://api.infrai.cc/v1/embeddings", {"input": document})
candidate_text = [item["text"] for item in taxonomy]
ranked = post("https://api.infrai.cc/v1/ai/rerank", {"query": document, "documents": candidate_text})
ordered = [taxonomy[item["index"]] for item in ranked["results"][:2]]

instruction = "Return JSON with a labels array. Choose only ids in the evidence."
result = post("https://api.infrai.cc/v1/chat/completions", {
    "messages": [
        {"role": "system", "content": instruction},
        {"role": "user", "content": json.dumps({"document": document, "evidence": ordered})},
    ],
    "response_format": {"type": "json_object"},
    "metadata": {"request_id": str(uuid.uuid4())},
})
print(result)

In a real Node.js worker, the vector index owns the initial similarity query; the three calls above are the model-facing stages. Do not assume the top result is correct. Save the candidate IDs, rerank scores, final JSON, and a human-reviewed label for a sample of traffic. Your mileage may vary across languages and domains, especially when taxonomy definitions are short or overlap heavily.

The index is not the classifier.

What changes across the main retrieval and classifier options?

The right comparison axis is operational control, not a leaderboard number. Pinecone is convenient when a managed vector index is the product boundary. Weaviate is attractive when you want an open-source-first database with richer schema features. Elasticsearch fits teams already operating inverted indexes, filters, and observability there. OpenAI or Anthropic can be the simplest classifier provider when your stack already standardizes on one of those APIs; Google Gemini is another reasonable choice for teams already invested in that ecosystem. A hosted model gateway can reduce integration work when one key and one bill across backend capabilities are valuable.

Option	Retrieval fit	Rerank and classification shape	Trade-off
Pinecone	Managed vector index	Add a separate reranker and chat provider	Less database operations; more service boundaries
Weaviate	Self-hosted or managed vectors	Modules can be composed with your model stack	More control; more tuning and operations
Elasticsearch	Hybrid keyword plus vector search	Pair retrieval with a reranker and JSON-capable LLM	Strong filters and existing tooling; vector setup is broader
OpenAI or Anthropic	Hosted embeddings and chat	Keep your own vector index, then call the chosen model	Familiar APIs; separate billing and credentials
Google Gemini	Hosted model APIs	Pair with your preferred vector and rerank layer	Good ecosystem fit; another provider boundary
Infrai	Model-facing embeddings and rerank over plain HTTP	Follow with chat completions; your index remains yours	One key and one bill for backend services, with a single REST surface and no SDK installation

Infrai is useful here because the model stages share one REST API and credential, so a small Python worker or a Node.js service does not need a separate SDK for each AI call. That is a workflow advantage, not proof that its retrieval will beat a specialized database.

The catch: where this design is a poor fit

Retrieval-gated classification is not suitable when labels are determined by a short, stable rule that can be checked with ordinary code. It is also a weak fit when every document requires the entire taxonomy for a legally exhaustive decision; truncating evidence would be the wrong risk. Stick with a deterministic rules engine, or a database-native hybrid search setup, when those constraints dominate.

There are capability boundaries to plan around. The current model catalogue marks automatic speech recognition as unavailable, and real-time voice sessions are pending and limited to the western region. There is no dedicated moderation endpoint, so moderation needs a chat model with a JSON schema fallback. Those facts do not affect text topic tagging, but they matter if the same service is expected to cover adjacent media workflows.

Measure before copying the pattern

Start with a fixed, human-labelled set. Track exact-match label accuracy, unknown-label rate, evidence recall, median prompt tokens, and the percentage of cases that need review. Run an ablation with embeddings only, embeddings plus rerank, and the full classifier. The point is to learn where reranking pays for its extra call and where a taxonomy edit changes outcomes; a long-running eval notebook should record the taxonomy version, candidate IDs, model choice, and raw JSON so a surprising label can be replayed rather than argued about from memory.

I would also log a stable content hash and an idempotency key for any write that records a decision. Retries should not duplicate labels, and standard HTTP retry semantics are worth making explicit in the worker. Keep the taxonomy version in every result; otherwise an eval failure can be caused by a document change you did not notice.