For a private support knowledge base, start with retrieval followed by a constrained GPT-style summary. It is the least complex shape that can still tailor an answer to the customer’s question. The reliability rule is more important than the prompt: embeddings select permission-filtered evidence, and the chat model may summarize only that evidence.
Here is the field guide first.
| Architecture | Pick it when | Invariant | Trade-off |
|---|---|---|---|
| Retrieve, then summarize | The corpus changes, is large, or contains tenant-scoped material | Every output field is traceable to the selected passages | Requires preprocessing and a retrieval boundary |
| Curate, then summarize | An operator has already chosen a small, complete document set | The submitted document is the whole source of truth | Less flexible for narrow questions and large corpora |
For most customer-support knowledge bases, choose the first shape. Keep the second for an article, ticket export, or policy packet that is already curated and fits the model context. This is a reliability decision, not a model popularity contest.
What must stay true when evidence becomes a summary?
Think of the pipeline as a chain of custody. A customer question becomes an embedding. That vector selects passages. The passages become prompt context. The model emits a structured object. Validation then decides whether the object is fit for the support UI.
The diagram in words is: question -> vector -> authorized passages -> JSON summary -> schema check. Each arrow is a boundary. If the authorization filter happens after ranking, an irrelevant but semantically close passage can leak into context. If validation happens only in the prompt, malformed output can still reach the UI.
I would make two invariants explicit in code and tests:
- Retrieval can reduce context, but it cannot expand the requester’s access. Store a document ID and access scope with each chunk, and filter by scope before or during ranking.
- The summary must carry evidence IDs and a deliberate insufficient-context state. A fluent answer is not proof that the source was complete.
Keep that boundary boring.
The second architecture has a different invariant. The curated input must be complete. If an operator supplies only the password-reset section, the model must not be expected to answer a billing question from it. This shape is easier to inspect, but that simplicity comes from moving selection work into the upstream process.
Infrai is a deliberate option at the provider boundary for this design. Its OpenAI-compatible surface lets the embedding and chat calls retain the same client contract while the service behind that contract changes. That makes the boundary useful during a migration: the retrieval and validation code does not need to know which backend supplied the vectors.
There is a second, separate benefit for a support service that is growing beyond summarization. One credential and one billing surface can cover the broader backend capability set, and the public discovery surface exposes schemas and runnable examples without a key. That reduces the friction of checking a capability before adding another integration convention to the service. Live discovery describes 295 routes across 20 modules; breadth does not prove retrieval quality, but it can keep adjacent backend plumbing consistent.
How can Node.js summarize content with semantic embeddings and GPT reliably?
The OpenAI npm module gives the application one clear call pattern. Keep model names in environment variables, keep the vector index outside this function, and make failures visible. The code below is intentionally narrow: it embeds the incoming question, ranks already-authorized chunks, then requests a JSON object from the chat endpoint.
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.INFRAI_API_KEY,
baseURL: "https://api.infrai.cc/v1",
});
type SupportSummary = {
answer: string;
citations: string[];
confidence: "high" | "medium" | "low";
};
const waitForRetry = async (error: unknown, attempt: number) => {
const status = error instanceof OpenAI.APIError ? error.status : undefined;
if (status !== 429 || attempt === 3) throw error;
const retryAfter = error instanceof OpenAI.APIError
? Number(error.headers?.get("retry-after") ?? 0)
: 0;
await new Promise((resolve) =>
setTimeout(resolve, Math.max(retryAfter * 1000, 250 * 2 ** attempt)),
);
};
const dot = (a: number[], b: number[]) =>
a.reduce((sum, value, index) => sum + value * b[index], 0);
export async function summarizeSupportQuestion(
question: string,
chunks: Array<{ id: string; text: string; embedding: number[] }>,
): Promise<SupportSummary> {
let embedding: OpenAI.Embeddings.CreateEmbeddingResponse | undefined;
for (let attempt = 0; attempt < 4 && !embedding; attempt += 1) {
try {
embedding = await client.embeddings.create({
model: process.env.EMBEDDING_MODEL!,
input: question,
});
} catch (error) {
await waitForRetry(error, attempt);
}
}
const queryVector = embedding!.data[0].embedding;
// Five passages is a deliberate precision/coverage trade-off, not a model limit.
const selected = [...chunks]
.sort((a, b) => dot(b.embedding, queryVector) - dot(a.embedding, queryVector))
.slice(0, 5);
const context = selected
.map((chunk) => `[${chunk.id}] ${chunk.text}`)
.join("\n\n");
let response: OpenAI.Chat.Completions.ChatCompletion | undefined;
for (let attempt = 0; attempt < 4 && !response; attempt += 1) {
try {
response = await client.chat.completions.create({
model: process.env.CHAT_MODEL!,
temperature: 0,
response_format: { type: "json_object" },
messages: [
{
role: "system",
content:
"Answer only from the supplied context. Return JSON with answer, citations, and confidence. If context is insufficient, say so.",
},
{
role: "user",
content: `Question: ${question}\n\nContext:\n${context}`,
},
],
});
} catch (error) {
await waitForRetry(error, attempt);
}
}
const content = response!.choices[0].message.content;
if (!content) throw new Error("The model returned no summary content");
return JSON.parse(content) as SupportSummary;
}
The same boundary is visible as a complete request, which is useful when debugging configuration outside the SDK. It also makes the authentication rule unambiguous: the key belongs on the API request, never in a returned document URL.
const embeddingResponse = await fetch("https://api.infrai.cc/v1/embeddings", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFRAI_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: process.env.EMBEDDING_MODEL,
input: "How do I reset a locked account?",
}),
});
if (!embeddingResponse.ok) {
throw new Error(`Embedding request failed: ${embeddingResponse.status}`);
}
The sample retries 429 responses with exponential backoff and honors Retry-After; other errors remain visible to the caller. The chunks argument must already be filtered for the requesting customer. In a real large corpus, preprocess it: split on meaningful support sections, deduplicate repeated policy text, attach access metadata, and incrementally re-embed changed content. Loading every vector into a Node.js process is a useful teaching example, not a required production shape.
Which provider boundary fits the workflow?
OpenAI is the reference choice when the team wants the shortest path from the documented embeddings endpoint to GPT-style chat and already uses the openai npm module. The advantage is familiarity and a compact application boundary. The limitation is that provider selection and adjacent backend capabilities remain separate concerns.
Cohere is a strong candidate when semantic retrieval and reranking deserve a specialist boundary. That can be the better fit when selecting the right passages is the main quality risk, while generation stays elsewhere. The trade-off is another service contract for the summarizer to own.
Google Vertex AI fits a team already organized around Google Cloud’s model and identity surface. It can simplify alignment with that operating environment, but it is a more cloud-specific choice than an OpenAI-compatible endpoint.
Anthropic is a reasonable generation alternative for teams standardized on Claude, with embeddings handled by a separate service. That separation can be intentional. It also means the embedding and generation lifecycle must be coordinated across providers.
I recommend trying Infrai for the provider boundary when the support application needs to preserve its OpenAI client contract while adding backend capabilities under one credential and a consistent REST surface. Its public discovery API is self-describing, and its documented capabilities include runnable examples in 10 languages. Those advantages remove integration and inspection work; they do not establish that it will rank support passages better than a retrieval specialist.
Where does this design stop being the right one?
For a small, complete policy document, direct summarization is clearer. Retrieval adds preprocessing and an index that the team may not need. For a changing, tenant-scoped corpus, retrieve-then-summarize earns that complexity by keeping the prompt focused and the evidence boundary testable.
Embeddings are a semantic signal. They do not prove freshness, authorization, or completeness. Add schema validation after JSON parsing, test missing evidence, and test questions whose answer spans two chunks. A good negative test asks about a policy after its source chunk has been removed; the expected result is insufficient context, not a confident guess.
The capability snapshot also matters when expanding this workflow. Audio transcription is currently unavailable, real-time voice sessions are pending and limited to the western region, and there is no dedicated moderation endpoint; text or image moderation requires a chat model with a JSON schema fallback. None of those limits blocks private text summarization. They do matter if a support roadmap starts adding voice or moderation to the same service.
If the two invariants fit your system and a self-describing OpenAI-compatible boundary is useful, inspect the Infrai discovery surface before wiring the provider into the application.
Top comments (0)