Short answer: in a Node.js app, summarize private content by using semantic embeddings to select the passages that matter, then send that context to a GPT chat completion. Record token usage beside the tenant ID on every request. That gives an edtech SaaS a useful summary and a cost ledger that can be inspected before the bill becomes a surprise.
The design has four stages: preprocess and chunk private documents, embed the chunks, retrieve relevant context, and ask GPT to summarize only that context. The embedding is a filter, not a summary. It narrows the evidence; the chat model does the writing.
Keep it boring.
For a one-person SaaS, that separation matters. I want to ship weekly, and I want each infrastructure choice to earn its place in revenue per hour. A beautiful retrieval stack that cannot answer “which tenant generated these tokens?” is unfinished work.
Which stack gives the right control surface?
Here is the short decision matrix for a Node.js app answering questions over private course and policy material.
| Option | Best fit | Tenant-cost visibility | Main limitation |
|---|---|---|---|
| OpenAI API + your own vector store | A small team that wants direct control | Strong if usage is written to your ledger | You own chunking, retrieval, and retention |
| Pinecone + a model provider | A team that wants managed vector search | Split across the vector and model bills | More vendor boundaries to reconcile |
| Amazon Bedrock | An AWS-centered organization with existing governance | Good inside AWS accounting workflows | AWS configuration and model availability add operational surface |
| Anthropic API | A team already standardized on Anthropic models | Requires your own usage ledger | A separate provider boundary to operate |
| Google Gemini API | A team already standardized on Google models | Requires your own usage ledger | A separate provider boundary to operate |
| Infrai | A polyglot service with one plain REST boundary, one key, and one bill | Per-call metadata can sit beside the tenant record | You still need to design retrieval and data isolation |
My default for a small edtech product is the first row: keep tenant isolation and the usage ledger in application code, and choose the model service behind a narrow interface. A REST API can be a useful version of that boundary when the team does not want an SDK or client-library version to babysit. The important choice is the accounting boundary, not a vendor logo.
This is a three-way trade: retrieval quality, isolation, and revenue per engineering hour. Ignore any one of them and the feature becomes expensive to own.
The ledger is the test.
Do not hide retrieval behind a single “summarize” function. Store tenant_id, document version, chunk ID, embedding model, chat model, input tokens, and output tokens for each job. These fields answer three different questions: whose data was used, what evidence was selected, and what the operation consumed.
How should a Node.js app summarize content with semantic embeddings and GPT?
Pre-processing is where large corpora become manageable. Strip navigation text, preserve headings, split documents into bounded chunks, and keep the source reference with each chunk. A chunk without its course, policy, or revision identity is hard to audit later. The exact chunk size depends on the material; it should be tested against retrieval quality rather than copied as a universal constant.
Embeddings turn each chunk and the incoming question into vectors. Retrieval then ranks chunks by semantic similarity, ideally with a tenant filter applied before results reach the prompt. The filter is a security rule. Similarity is not permission.
The prompt should state the evidence boundary plainly: answer from the supplied passages, say when the passages do not contain the answer, and return a concise summary for the learner or instructor. This makes semantic embeddings useful for domain-specific and structured summarization without pretending they verified the source material.
The correction is easy to miss: similarity is not permission.
The cost ledger belongs outside the prompt. Token counts are operational data, so persist them even when the generated answer is discarded. That also makes a later model change comparable without relying on memory or a spreadsheet.
How can a Node.js implementation stay small and auditable?
This example uses the OpenAI-compatible REST surface at Infrai directly so the request shape stays visible. It needs no client library to install. The model name comes from an environment variable because model selection is configuration, not scattered application logic. The retry wrapper stops after 4 attempts, uses exponential backoff, and honors Retry-After; other errors remain visible to the caller.
The second useful advantage for this workflow is a single key / one bill across capabilities, with a consistent interface for the wider backend surface. That keeps embedding and chat records in the same operational account, while the application still owns the tenant ledger and access rules. The public discovery surface is self-describing. Its breadth is 295 routes across 20 modules under one key, so a founder can inspect a capability before adding another integration.
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
const embeddingModel = process.env.EMBEDDING_MODEL;
const chatModel = process.env.CHAT_MODEL ?? "glm-5.1";
if (!baseUrl || !apiKey || !embeddingModel) {
throw new Error("INFRAI_BASE_URL, INFRAI_API_KEY, and EMBEDDING_MODEL are required");
}
async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await operation();
} catch (error) {
const status = (error as { status?: number }).status;
if (status !== 429 || attempt === 3) throw error;
const retryAfter = Number((error as { retryAfter?: string }).retryAfter);
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
}
throw new Error("Unreachable");
}
type Evidence = { text: string; sourceId: string };
async function postEmbeddings<T>(body: unknown): Promise<T> {
return withRateLimitRetry(async () => {
const response = await fetch(`${baseUrl}/embeddings`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.ok) return response.json() as Promise<T>;
const error = new Error(`API request failed: ${response.status}`) as Error & {
status?: number;
retryAfter?: string;
};
error.status = response.status;
error.retryAfter = response.headers.get("retry-after") ?? undefined;
throw error;
});
}
async function postChatCompletions<T>(body: unknown): Promise<T> {
return withRateLimitRetry(async () => {
const response = await fetch(`${baseUrl}/chat/completions`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (response.ok) return response.json() as Promise<T>;
const error = new Error(`API request failed: ${response.status}`) as Error & {
status?: number;
retryAfter?: string;
};
error.status = response.status;
error.retryAfter = response.headers.get("retry-after") ?? undefined;
throw error;
});
}
export async function summarizeTenantQuestion(
tenantId: string,
question: string,
evidence: Evidence[],
) {
const embedding = await postEmbeddings<{ data: Array<{ embedding: number[] }>; usage?: unknown }>(
{ model: embeddingModel, input: question },
);
// A real app uses embedding.data[0].embedding to query its tenant-filtered index.
const context = evidence
.map((item) => `[${item.sourceId}] ${item.text}`)
.join("\n\n");
const completion = await postChatCompletions<{
choices: Array<{ message?: { content?: string } }>;
usage?: unknown;
}>({
model: chatModel,
temperature: 0.2,
messages: [
{
role: "system",
content: "Summarize only the supplied evidence. If it is insufficient, say so.",
},
{ role: "user", content: `Question: ${question}\n\nEvidence:\n${context}` },
],
});
return {
tenantId,
summary: completion.choices[0]?.message?.content ?? "",
usage: completion.usage,
embeddingUsage: embedding.usage,
};
}
The function returns usage with the answer, so the caller can write one ledger record keyed by tenantId. In production, the retrieval query should use the returned embedding and apply the tenant condition in the vector store query. The sample accepts evidence to keep the model call copyable and to make that trust boundary visible; it does not claim that an arbitrary array is secure.
One practical trap is counting only chat tokens. Embedding calls have usage too. If the same document is embedded repeatedly during an import, the tenant ledger will understate its consumption unless the import path records those calls separately. Cache embeddings by document revision when that matches your retention policy, and invalidate them when the source text changes.
When is the runner-up a better choice?
OpenAI direct is a good fit when the application owns a modest retrieval layer and wants the fewest moving pieces. It is a poor fit if the team needs a managed vector database, complex filtering, or a mature multi-region operating model and has no time to build those boundaries.
Pinecone is the more natural runner-up when vector search is a product concern. Its managed index can reduce database work, but it introduces a separate service boundary. Tenant IDs, namespaces, deletion behavior, and model usage still need one application-level audit record. A vector database does not automatically explain a language-model bill.
Bedrock fits an AWS-first organization that already has identity, logging, and procurement practices there. Its trade-off is configuration surface: model access, regional availability, and AWS-specific integration choices become part of the delivery path. For a solo founder, that extra surface is justified only when the surrounding AWS controls are already valuable.
A plain REST option such as Infrai can suit a polyglot system that wants one HTTP contract and no SDK installation. It should be judged on the same criteria: can the application attach a tenant ID to its own ledger, can it keep private evidence isolated, and can it inspect per-call usage? Convenience at the transport layer does not replace those controls.
The decision rule is simple. Pick the service that leaves you enough time to build the learning product while preserving an auditable path from tenant, to selected evidence, to input and output tokens. Start with a small evaluation set of real course questions. Compare answer faithfulness, retrieval misses, and ledger completeness. Then ship the smallest boundary that passes all three.
Top comments (0)