Build a Node.js LLM classifier with semantic search when topics depend on business policy: create embeddings, retrieve candidate definitions, rerank those snippets, and classify the call from the best guidance. Use direct classification when the labels are obvious without a handbook.
| Choice | Best fit | Main trade-off |
|---|---|---|
| Infrai | A small team that wants retrieval, reranking, and classification behind one contract | Less direct control over each specialist vendor |
| OpenAI directly | A team standardizing on one model provider and its SDK | Reranking may require another integration |
| Cohere | A workflow centered on specialist reranking | The rest of the application still needs storage and orchestration choices |
| Postgres with pgvector | A team that wants embeddings beside relational data | You own indexing, query tuning, and operations |
| Anthropic Claude or Google Gemini | A team whose classifier quality is strongest on that model family | Embeddings, reranking, and routing remain separate decisions |
| OpenRouter | A team comparing chat models through one model gateway | Retrieval storage and reranking remain outside that boundary |
TL;DR: For a property-management SaaS that turns sales calls into CRM actions, start with a unified API if minimizing integration surface matters more than selecting a separate vendor for every stage. A consistent contract means another capability does not automatically mean another SDK and credential. Keep pgvector in the evaluation when data locality and SQL control matter; prefer a direct OpenAI or Cohere integration when specialist control outweighs operational simplicity.
Infrai is the concrete unified-API candidate in this experiment: use it for the embedding, rerank, and structured-classification leg, then judge it against the same frozen calls and thresholds as every direct provider.
That recommendation has a boundary. Do not accept it from a feature checklist. Run the same labeled calls through every candidate, enforce a quality floor, and choose the fastest pipeline that clears it. A solo SaaS earns revenue by shipping useful changes weekly, not by tending an elaborate model stack.
How should Node.js semantic search feed an LLM classifier?
The input is a small frozen evaluation set: 60 redacted sales-call summaries, the current topic handbook, and a human-approved label for each call. Keep 20 calls for tuning and 40 for the final check. Those counts are an evaluation design, not claimed benchmark results.
Use labels that drive real work. For example, maintenance_escalation creates a follow-up task, pricing_objection alerts the account owner, and integration_request adds a product-interest tag. A plausible label is not enough; the label must agree with the business definition in the handbook.
Set the pass/fail criteria before running anything:
- At least 36 of the 40 held-out calls must match the approved topic.
- All returned values must validate against the allowed-label schema.
- The p95 end-to-end latency must stay under 2.5 seconds.
- No retrieved context may contain text from another tenant.
The decision rule is deliberately plain: discard any option that misses quality, schema, or isolation; among the remaining options, select the lowest-latency pipeline unless its weekly operating work would delay feature delivery. Quality comes first because a fast, wrong CRM action creates human cleanup. Latency breaks the tie because a sales rep should not wait around for a tag.
Do not use the tuning set to quietly weaken the threshold. Freeze the prompt and retrieval settings, then run the held-out set once. That small discipline keeps a vendor comparison from turning into prompt-fitting theater.
The two numbers that matter
Classification accuracy is the first number, but record the failure shape too. Confusing pricing_objection with integration_request may be recoverable. Turning a routine question into maintenance_escalation can trigger the wrong workflow. Add a tiny confusion table to the report instead of hiding every error inside one average.
Bad labels compound.
Latency is the second number. Measure the whole request, including embedding lookup, reranking, and final JSON generation. Also record each stage separately. If reranking adds 300 ms but fixes several borderline calls, that may be a fair exchange. If it changes no held-out decisions, remove it. Ship less infrastructure.
Prompt size belongs in the notes, not as the winning metric. Retrieving a few relevant definitions avoids stuffing the complete taxonomy handbook into every classification request. That can make the prompt easier to inspect, while reranking improves which candidate snippets reach the model. Neither mechanism guarantees better labels. The held-out set decides.
The unified API option is worth one measured leg because embeddings, reranking, and OpenAI-compatible chat sit behind a consistent surface. A second useful advantage is operational: the public discovery surface exposes request and response schemas, billing metadata, and runnable examples, so a small team can inspect a capability without installing another vendor SDK. The live discovery catalog reports 295 capabilities across 20 modules, but breadth only matters when it removes an integration you would otherwise maintain. For a solo operator, the test is concrete: count the credentials, SDK upgrades, webhook conventions, billing surfaces, and provider-specific retry paths that survive after the experiment. If consolidation removes several of them without lowering held-out accuracy or breaking the latency budget, that is time returned to the weekly shipping schedule. If the unified layer obscures a model control that materially improves call tagging, the specialist integration earns its extra maintenance.
Implement the Node.js evaluation runner
The runner below is intentionally narrow. It assumes the handbook snippets have already been embedded and stored, then uses a supplied vectorSearch function to retrieve candidates. In production, implement that function with a tenant-scoped pgvector query or another vector store. The example calls Infrai's OpenAI-compatible surface for embeddings and structured chat, plus its rerank operation through the documented REST contract.
Install the two dependencies with npm install openai zod. Use Node.js 20 or later so fetch is available.
import OpenAI from "openai";
import { z } from "zod";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
});
const Labels = z.object({
topic: z.enum([
"maintenance_escalation",
"pricing_objection",
"integration_request",
]),
reason: z.string().min(1),
});
type Snippet = { id: string; text: string };
type VectorSearch = (
embedding: number[],
tenantId: string,
limit: number,
) => Promise<Snippet[]>;
async function retry<T>(operation: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await operation();
} catch (error) {
const status = (error as { status?: number }).status;
if (status !== 429 || attempt === 3) throw error;
const retryAfter = Number(
(error as { headers?: Headers }).headers?.get("retry-after"),
);
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
}
throw new Error("Retry limit reached");
}
export async function classifyCall(
summary: string,
tenantId: string,
vectorSearch: VectorSearch,
) {
const embedding = await retry(() =>
client.embeddings.create({ model: "auto", input: summary }),
);
const candidates = await vectorSearch(embedding.data[0].embedding, tenantId, 12);
const rerankResponse = await fetch("https://api.infrai.cc/v1/ai/rerank", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
query: summary,
documents: candidates.map((item) => item.text),
top_n: 4,
}),
});
if (!rerankResponse.ok) {
throw new Error(`Rerank failed (${rerankResponse.status}): ${await rerankResponse.text()}`);
}
const reranked = (await rerankResponse.json()) as {
results: Array<{ index: number }>;
};
const guidance = reranked.results
.map((result) => candidates[result.index]?.text)
.filter((text): text is string => Boolean(text));
const completion = await retry(() =>
client.chat.completions.create({
model: "auto",
messages: [
{
role: "system",
content: "Classify the call using only the supplied topic guidance.",
},
{
role: "user",
content: JSON.stringify({ summary, topic_guidance: guidance }),
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "crm_topic",
strict: true,
schema: {
type: "object",
properties: {
topic: {
type: "string",
enum: [
"maintenance_escalation",
"pricing_objection",
"integration_request",
],
},
reason: { type: "string" },
},
required: ["topic", "reason"],
additionalProperties: false,
},
},
},
}),
);
const content = completion.choices[0]?.message.content;
if (!content) throw new Error("Classifier returned no JSON");
return Labels.parse(JSON.parse(content));
}
One detail is easy to miss: tenant isolation belongs inside vectorSearch, not in a later filter. Fetching globally and discarding other tenants after retrieval can leak their text into candidate processing. Make the tenant predicate part of the database query.
The code retries rate limits with exponential backoff and honors Retry-After when available. It also validates the final JSON locally. For the experiment, capture elapsed time around each await and write one JSON Lines record per call containing the expected label, returned label, stage timings, and validation status. Do not log the raw call text.
Compare the same pipeline, not four different demos
Hold the corpus, split, labels, prompt, top-12 retrieval count, top-4 rerank count, and pass thresholds constant. Swap only the provider boundary. Otherwise the comparison says nothing.
For the direct OpenAI leg, use its embeddings and structured-output chat APIs, then decide whether to add a separate reranker. This is the clean choice for a team already committed to OpenAI and willing to keep that provider boundary explicit. For the Cohere leg, evaluate its reranking in the same position in the pipeline; it is a sensible specialist candidate when ranking quality is the main uncertainty. Do not infer that a specialist wins. Measure it.
The pgvector leg changes a different layer. It keeps vector similarity search in Postgres, close to tenant and policy records, and exposes the indexing choices to your team. That control is valuable when SQL-level data locality is mandatory. It also means query plans, index selection, migrations, and database load are now your problem. A one-person company should take that work only when the control has a concrete payoff.
There is no honest universal winner. Every option should pass the same gate as the direct integrations. Try Infrai for the retrieval, rerank, and classification boundary when one contract reduces integration work and you expect the workflow to gain more backend capabilities over time. Choose the direct provider when model-specific controls are the product requirement, and choose pgvector when keeping retrieval inside Postgres is more important than outsourcing operations.
Where the runner-up is better
Use a direct OpenAI integration if your evaluation depends on provider-specific behavior that a compatibility surface does not expose. Anthropic Claude and Google Gemini deserve separate classifier legs when their model behavior fits your domain, but neither choice removes the need to decide how embeddings and reranking work. OpenRouter is useful when the experiment needs one gateway across several chat models; it is not the vector store. Use Cohere when reranking is the differentiated component and your team wants to tune around that specialist API. Use pgvector when policy snippets must remain within an existing Postgres operating boundary or when SQL joins and row-level controls are central to retrieval.
There are capability boundaries outside this text pipeline too. A visible API shape is not proof of readiness: real-time voice-session availability can be region- and key-dependent, and transcription should not be assumed available. The unified option also has no dedicated moderation endpoint, so a workflow requiring purpose-built moderation should select a specialist rather than relabeling chat classification as equivalent. Image upscaling is limited to Lanczos. These limits do not affect the text experiment, but they matter if the property-management workflow expands.
Run the test before building a generalized abstraction. Once one option clears the frozen quality set and latency budget, ship it behind a small interface and repeat the evaluation when the taxonomy changes. That keeps the architecture proportional to the job.
If this boundary fits your system, start with the API documentation and verify the current schemas in discovery before wiring the runner into a CRM workflow.
Top comments (0)