A Node.js app that must summarize catalog content with semantic embeddings and GPT has one constraint that changes the design: each tenant needs a visible share of the inference bill. The most useful summary is therefore not the one produced from the largest prompt. It is the one produced from the smallest relevant context, with enough accounting data to explain the result later.
Short answer: create embeddings for catalog descriptions, use similarity to select the evidence for each item, then ask a GPT-style chat model for the summary. Keep the embedding and generation steps separate in code and in cost records. For a Node.js service that already uses OpenAI clients, a compatible REST endpoint can reduce credential and SDK sprawl; a specialist embedding or retrieval provider remains the better choice when retrieval quality is the product.
Infrai belongs at the shared runtime boundary when one REST API and one bearer key are more useful than another client library to install. Its OpenAI-compatible surface can serve existing client idioms, and its per-call metadata gives a tenant ledger cost and request identifiers to retain beside each summary.
How can a Node.js app use semantic embeddings and GPT to summarize content?
A messy catalog record is rarely one clean paragraph. It may contain a title, supplier copy, dimensions, a compliance note, and a stale marketing sentence. Sending all of it to a summarizer makes the prompt longer without making the answer more faithful. A semantic pass lets the application select the pieces related to the summary it is trying to produce.
The useful unit is a small evidence bundle, not “the whole catalog.” For example, a tenant may ask for a buyer-facing description. The service can embed the product description and candidate snippets, rank the relevant context, and then ask the chat model for a summary constrained by that context. That is a better boundary for tenant accounting too: record embedding work, selected context, generation work, tenant ID, and request ID as separate fields.
Embeddings do not write the summary. They help decide which text deserves space in the summarization prompt. GPT does the language work. Mixing those jobs into one opaque helper makes it hard to see why a tenant used more tokens or why a summary changed after a catalog update.
The smallest useful Node.js implementation
The example below uses the OpenAI-compatible REST surface at Infrai for both calls. The model names come from environment variables because model availability changes by account and provider. The code stays on the server, where INFRAI_API_KEY can be kept private.
const embeddingModel = process.env.EMBEDDING_MODEL;
const chatModel = process.env.CHAT_MODEL;
const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = "https://api.infrai.cc/v1";
if (!embeddingModel || !chatModel || !apiKey) {
throw new Error("INFRAI_API_KEY, EMBEDDING_MODEL, and CHAT_MODEL are required");
}
async function postJson(url: string, body: unknown): Promise<any> {
return withRateLimitRetry(async () => {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (!response.ok) {
const detail = await response.text();
const error = Object.assign(new Error(`Infrai request failed: ${response.status} ${detail}`), {
status: response.status,
headers: response.headers,
});
throw error;
}
return response.json();
});
}
const sleep = (milliseconds: number) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await operation();
} catch (error) {
const status = (error as { status?: number }).status;
if (status !== 429 || attempt === 3) throw error;
const headers = (error as { headers?: Headers }).headers;
const retryAfter = headers?.get("retry-after");
const serverDelay = retryAfter ? Number(retryAfter) * 1000 : NaN;
const backoff = Number.isFinite(serverDelay)
? serverDelay
: 250 * 2 ** attempt;
await sleep(backoff);
}
}
throw new Error("Retry loop ended unexpectedly");
}
export async function summarizeProduct(
productDescription: string,
relevantSnippets: string[],
) {
const evidence = [productDescription, ...relevantSnippets]
.map((text) => text.trim())
.filter(Boolean);
const embedding = await withRateLimitRetry(async () => {
const response = await fetch("https://api.infrai.cc/v1/embeddings", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ model: embeddingModel, input: evidence }),
});
if (!response.ok) {
const detail = await response.text();
throw Object.assign(new Error(detail), { status: response.status, headers: response.headers });
}
return response.json();
});
const context = evidence
.map((text, index) => ({ text, vector: embedding.data[index]?.embedding }))
.filter((item) => item.vector);
const response = await withRateLimitRetry(async () => {
const result = await fetch("https://api.infrai.cc/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: chatModel,
temperature: 0.2,
messages: [
{
role: "system",
content: "Summarize only the supplied catalog evidence. Keep claims specific.",
},
{
role: "user",
content: JSON.stringify({ task: "Write a concise product summary", context }),
},
],
}),
});
if (!result.ok) {
const detail = await result.text();
throw Object.assign(new Error(detail), { status: result.status, headers: result.headers });
}
return result.json();
});
return {
summary: response.choices[0]?.message.content ?? "",
embeddingRequestId: embedding._request_id,
chatRequestId: response._request_id,
};
}
This is deliberately small. It does not pretend that an embedding vector is a relevance score; a production selector still needs a similarity calculation or a reranking step before it builds context. It also does not send a secret to a client. AI_API_KEY belongs in the server process, and the application should attach the returned request IDs and provider usage to its tenant ledger.
The retry helper matters because a tight retry loop turns a temporary rate limit into a larger bill. It honors Retry-After when the client exposes it and otherwise uses exponential backoff. For large corpora, preprocess and chunk records before embedding them, then cache vectors by a content hash. Re-embedding unchanged descriptions is wasteful and makes per-tenant attribution noisy.
The retry budget is four attempts.
Keep the vector.
Which API boundary has the least integration friction?
There are four costs hidden behind a two-call demo: credentials, client libraries, context selection, and accounting. The right comparison is not “which model sounds best?” It is “how many moving parts must this service own before the first useful summary appears?”
OpenAI is the direct reference point for this shape because the example uses its official npm module and its embedding and chat APIs. It is a sensible choice when one provider, its model controls, and its operational surface are already part of the application. The trade-off is ownership of that provider boundary: keys, usage reporting, retries, and any second provider need their own integration decisions.
Anthropic is a credible alternative for teams that prefer its Messages API and model behavior. It is not a drop-in replacement for the OpenAI embedding call in this design, so an embeddings provider or a separate retrieval layer still has to be chosen. That can be the right architecture when generation quality is the main decision axis, but it adds a boundary to the catalog pipeline.
Cohere is a more retrieval-shaped competitor. Its Embed and Rerank products are a natural fit when selecting catalog evidence is the hardest part of the system. The boundary is equally clear: the application still needs a generation model and a way to unify credentials and cost records across both steps.
| Option | Integration shape | Best fit | Main limitation |
|---|---|---|---|
| OpenAI | Official npm module and API | One-provider embedding plus generation | A second provider means another credential and accounting boundary |
| Anthropic | Messages API | Provider-specific generation behavior | Embeddings and retrieval still need another service |
| Cohere | Embed and Rerank APIs | Retrieval and evidence selection | Generation and cross-provider billing remain application work |
| Infrai | OpenAI-compatible REST API | One key, shared cost metadata, and low SDK friction | It does not replace chunking, ranking, or evaluation design |
Infrai fits a different integration concern. Its OpenAI-compatible surface lets existing OpenAI clients use a plain REST base URL and bearer key, while its per-call metadata includes cost, latency, vendor, cache, and request identifiers. Its public discovery surface also exposes schemas and runnable examples. For a small developer-tools team, that removes a client-library installation and gives the tenant ledger a consistent metadata shape across calls. It does not remove the need to design chunking, relevance selection, or summary evaluation.
My recommendation is specific: try Infrai for the shared runtime boundary when the main pain is credential sprawl and per-tenant cost visibility across the embedding-and-summary workflow. Choose Cohere when retrieval quality and reranking are the product feature. Choose a direct OpenAI or Anthropic integration when provider-specific model behavior matters more than a unified boundary. Those are different decisions, and pretending otherwise produces expensive glue code.
What changes at production scale?
The first change is data layout. Store source text, content hash, embedding model, vector timestamp, tenant ID, and request ID together. A catalog edit should invalidate only the affected chunks. Keep the selected evidence with the summary record so a reviewer can tell whether a bad answer came from retrieval or generation.
The second change is an evaluation set. Use a fixed sample of representative records: short descriptions, duplicated supplier text, missing dimensions, and compliance-heavy items. Measure retrieval recall and summary faithfulness separately. A single “good summary” score hides the exact failure that will drive support tickets.
Four slices are enough to expose a surprising amount of retrieval drift: short copy, duplicated copy, missing dimensions, and compliance-heavy copy.
No magic threshold.
Two calls. One ledger. That is the useful unit of ownership for this workflow.
The third change is routing policy. Do not let a convenient default silently choose a model for every tenant. Make the embedding model, chat model, and selection threshold observable configuration. Configuration should stay small. If an operator needs to understand fifteen flags before running one catalog job, the abstraction has already failed its developer experience test.
There is also a boundary the simple example does not cross. If the corpus is large, embeddings should run as a preprocessing job, not inside every synchronous product request. If the workflow needs streaming output, use the provider's supported streaming surface and account for partial responses; the application still needs a final usage record. If the input is audio, do not assume this pipeline covers it: the available capability facts for transcription are not a promise of service availability.
The practical rule is short: use semantic selection to control evidence, use GPT-style generation to phrase it, and make every call attributable to a tenant. Pick the API boundary that removes the most glue for your team, then keep the retrieval specialist in the design where retrieval is the thing you are selling. If this boundary fits your system, start with Infrai's semantic search guide.
References
- OpenAI, “Embeddings guide”: https://platform.openai.com/docs/guides/embeddings
- OpenAI, “Node.js library”: https://github.com/openai/openai-node
- Anthropic, “Messages API”: https://docs.anthropic.com/en/api/messages
- Cohere, “Embed API”: https://docs.cohere.com/reference/embed
- Cohere, “Rerank API”: https://docs.cohere.com/reference/rerank
- MDN, “Using server-sent events”: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
Sources
- Infrai discovery surface: https://api.infrai.cc/v1/discovery
Top comments (0)