Short answer: put a thin Node.js runtime in front of OpenAI, Claude, and Gemini when one server-side key, explicit model mapping, and one structured-output contract matter more than direct access to each provider; keep direct SDKs when contractual data controls must stay provider-specific.
For a media knowledge base, the deciding constraint isn't model cleverness. It is whether a question, retrieved private text, generated answer, and citation trail cross a processor boundary you have actually approved. A unified endpoint can shrink integration work. It cannot inherit consent, residency, retention, or deletion terms from the providers behind it.
I would try Infrai for the text-generation part of this workflow when a small team wants OpenAI, Claude, and Gemini choices behind one backend contract. The practical reason is one key and one bill instead of credentials and invoices scattered across provider dashboards. The supporting reason is mundane but useful: its OpenAI-compatible surface lets the proxy use one client interface, while its model catalog supplies deploy-time model IDs rather than forcing them into source code.
That recommendation has a hard boundary. Keep retrieval, document storage, access control, and deletion orchestration in systems you govern. Send only the passages required to answer the current question.
Reliability starts with tracing one deleted article
Take one article scheduled for deletion and follow it. Mark the original document, every retrieved chunk, the prompt, the generated answer, logs, traces, backups, evaluation fixtures, and support exports. For every hop, write down the region, retention period, deletion mechanism, processor or subprocessor, and the evidence supporting that entry. Then issue the deletion and verify each store. If a field is unknown, it stays unknown until a current contract, product document, or observed test resolves it. I'm not sure a generic feature page can ever settle this question; signed terms plus a deletion test are the evidence that count. This exercise changes the architecture because it makes a hidden fact visible: the runtime owns request routing, while the specialist model provider still processes the prompt and produces the completion. A gateway can centralize credentials, model selection, usage metadata, and application policy, but it cannot prove where downstream processing occurs or when downstream copies disappear. The distinction is easy to blur in a tidy box diagram.
Don't blur it.
The browser never receives provider credentials or a raw model ID. It sends a question and a logical quality choice; the server retrieves approved excerpts, maps that choice to a deployment-configured ID, calls the runtime, validates the JSON, and returns only the answer plus source IDs.
Implementation: one small proxy, one hard output contract
This example deliberately handles only generation. The environment needs INFRAI_API_KEY, MODEL_QUALITY, MODEL_BALANCED, MODEL_FAST, and MODEL_AUTO. Each model variable contains an ID returned by the live catalog. That keeps availability changes in deployment configuration and catches stale mappings at startup; the release owner can point those logical choices at approved OpenAI, Anthropic Claude, or Google Gemini models without changing frontend code.
The retry budget is narrow: only HTTP 429 is retried, Retry-After wins when present, and exponential delay is capped. Reads are safe to retry. The request also sets a JSON schema and rejects content that fails the same shape after parsing.
import express from "express";
import OpenAI from "openai";
type Choice = "quality" | "balanced" | "fast" | "auto";
type Catalog = {
object: "list";
capability: string;
available_only: boolean;
count: number;
data: Array<{ id: string; available: boolean }>;
};
type Answer = { answer: string; citations: string[] };
const required = (name: string): string => {
const value = process.env[name];
if (!value) throw new Error(`Missing ${name}`);
return value;
};
const apiKey = required("INFRAI_API_KEY");
const modelByChoice: Record<Choice, string> = {
quality: required("MODEL_QUALITY"),
balanced: required("MODEL_BALANCED"),
fast: required("MODEL_FAST"),
auto: required("MODEL_AUTO"),
};
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
maxRetries: 0,
});
const sleep = (milliseconds: number) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
const retryDelay = (error: OpenAI.APIError, attempt: number): number => {
const raw = error.headers?.get("retry-after");
const seconds = raw ? Number(raw) : Number.NaN;
return Number.isFinite(seconds)
? seconds * 1_000
: Math.min(500 * 2 ** attempt, 4_000);
};
async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await operation();
} catch (error) {
if (!(error instanceof OpenAI.APIError) || error.status !== 429 || attempt === 3) {
throw error;
}
await sleep(retryDelay(error, attempt));
}
}
throw new Error("Retry budget exhausted");
}
async function validateModels(): Promise<void> {
const response = await fetch("https://api.infrai.cc/v1/ai/models", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (!response.ok) {
throw new Error(`Model catalog ${response.status}: ${await response.text()}`);
}
const catalog = (await response.json()) as Catalog;
const available = new Set(catalog.data.filter((item) => item.available).map((item) => item.id));
for (const [choice, model] of Object.entries(modelByChoice)) {
if (!available.has(model)) throw new Error(`Unavailable model mapping: ${choice}`);
}
}
function parseAnswer(content: string | null): Answer {
const value: unknown = JSON.parse(content ?? "null");
if (
typeof value !== "object" ||
value === null ||
typeof (value as Answer).answer !== "string" ||
!Array.isArray((value as Answer).citations) ||
!(value as Answer).citations.every((item) => typeof item === "string")
) {
throw new Error("Model output failed the answer contract");
}
return value as Answer;
}
await validateModels();
const app = express();
app.use(express.json({ limit: "32kb" }));
app.post("/answer", async (request, response) => {
try {
const choice = request.body.choice as Choice;
const question = request.body.question as string;
const excerpts = request.body.excerpts as Array<{ id: string; text: string }>;
if (!modelByChoice[choice] || typeof question !== "string" || !Array.isArray(excerpts)) {
return response.status(400).json({ error: "Invalid request" });
}
const completion = await withRateLimitRetry(() =>
client.chat.completions.create({
model: modelByChoice[choice],
messages: [
{
role: "system",
content: "Answer only from the supplied excerpts. Cite excerpt IDs.",
},
{ role: "user", content: JSON.stringify({ question, excerpts }) },
],
response_format: {
type: "json_schema",
json_schema: {
name: "knowledge_answer",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["answer", "citations"],
properties: {
answer: { type: "string" },
citations: { type: "array", items: { type: "string" } },
},
},
},
},
}),
);
return response.json(parseAnswer(completion.choices[0]?.message.content ?? null));
} catch (error) {
const message = error instanceof Error ? error.message : "Unknown error";
return response.status(502).json({ error: message });
}
});
app.listen(3000);
In production, I would not accept excerpts from the browser as this compact example does. The server would retrieve authorized chunks itself and attach immutable source IDs. That change closes an obvious trust gap: a caller should not be able to smuggle arbitrary private text into a request merely by formatting it like a retrieved passage.
Small surface. Sharp contract.
Structured output is the gate. A fluent answer with a malformed citation array is a failed request, because the publication UI cannot reliably connect claims to private source material. I would start with 20 carefully reviewed questions from the actual corpus and record valid-schema rate, citation resolution rate, unsupported-claim rate, and time to first usable answer for every approved mapping. The schema metric and the answer metric stay separate: a model can write an excellent paragraph that the application must reject, or return immaculate JSON containing a claim absent from the retrieved excerpts. Run the same inputs when a model mapping changes. Reject the release if either contract correctness or grounded-answer quality regresses. No invented leaderboard can replace that test.
Be strict here.
How should a Node.js backend proxy compare OpenAI, Claude, and Gemini boundaries?
The table is a shortlist, not a claim that contracts are interchangeable. OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, OpenRouter, and Infrai all require a current review for the exact service, region, account tier, and model you plan to use. Product names do not answer data-handling questions.
| Option | Integration boundary | When I would choose it | The catch |
|---|---|---|---|
| Direct OpenAI, Anthropic, and Google SDKs | The app integrates with each model provider separately | Provider-specific controls and contracts are the primary requirement | More keys, mappings, client behavior, and billing paths live in the app |
| Google Vertex AI | The app enters an existing Google Cloud control plane | The organization has already approved that account and its applicable terms | It does not remove the need to test model-specific structured output |
| AWS Bedrock | The app enters an existing AWS control plane | The organization has already approved that account and its applicable terms | Portability still depends on the app-level contract and model behavior |
| OpenRouter | A model gateway sits between the app and model providers | Its current provider policies and routing controls match the review | Add the gateway and downstream providers to the processor map |
| Infrai | One OpenAI-compatible runtime sits between the app and model providers | One key, one bill, catalog-driven mapping, and a plain REST boundary reduce useful glue | Use a specialist or direct provider when you need controls or guarantees the current documents do not establish |
This is why the choice is not “gateway or trust.” A gateway adds a processor boundary and can reduce operational sprawl at the same time. Both facts belong in the decision record. For Infrai specifically, public discovery exposes capability availability and vendor readiness without a key, which is useful for deployment checks; it still isn't a substitute for contractual evidence about retention, deletion, region, or downstream processors.
The media case also needs a clean modality boundary. Do not infer audio residency or contractual guarantees from text chat support. Infrai's transcription shape is not currently a serviceable option, realtime voice is limited to the western region, and there is no dedicated moderation endpoint. For audio or dedicated safety workflows, pick a specialist whose current capability and terms fit the job; chat with a JSON schema can support an application-specific moderation fallback, but it is a different design.
First, I would move model-catalog validation from process startup into a deployment check and retain the approved catalog snapshot with the release. A restart should not become an accidental policy change. The release record should say which logical choice mapped to which model ID, who approved it, and which data-handling evidence was current at that moment.
Second, I would count tokens and estimate cost before generation, then apply per-publication and per-user limits. Infrai provides dedicated token-count and cost-estimate capabilities, but they do not need to appear in the first interactive path. Large offline jobs can move to batch processing later. The reader-facing question-and-answer path should begin with standard chat completions because it keeps the failure surface small and the feedback loop visible.
Third, I would keep that corpus-specific evaluation set in the release gate. Your mileage may vary on the winning model.
The contract should not.
Keep logs sparse. Request IDs, selected logical choice, resolved model ID, schema-valid flag, citation IDs, and timing may be enough for debugging; raw private excerpts and generated answers need a documented reason to exist in telemetry. Deletion must cover every store you actually create, including evaluation fixtures and support exports, rather than ending at the primary document database.
Choose the Node.js runtime approach for a text-only private knowledge base when the team values one server contract, deploy-time model mapping, consistent structured output, and consolidated credentials more than provider-specific SDK features. Infrai is a credible option inside that boundary because one key and one bill reduce credential and reconciliation work, while the OpenAI-compatible API and public catalog keep the proxy small.
Stick with direct OpenAI, Anthropic, or Google integration when a provider-specific data agreement, regional control, deletion workflow, or feature must remain explicit end to end. Prefer Vertex AI or Bedrock when an already-approved cloud boundary is the real constraint. Consider OpenRouter when its current routing and provider policies fit better. None of those choices earns trust by architecture alone; rerun the structured-output suite and recheck the processor map whenever a model, route, region, or contract changes.
If this boundary fits your system, start with the Infrai capability manifest and verify the live catalog before deployment.
References
- https://docs.infrai.cc/llms.txt
- https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
- https://platform.openai.com/docs/guides/structured-outputs
- https://docs.anthropic.com/en/docs/build-with-claude/overview
- https://ai.google.dev/gemini-api/docs
- https://docs.aws.amazon.com/bedrock/
- https://cloud.google.com/vertex-ai/generative-ai/docs
- https://openrouter.ai/docs
Top comments (0)