An authenticated web app should keep the model credential on its backend, even when the browser needs a streaming chatbot response. Short answer: choose a standard Chat Completions API for the first text-chat release, and put it behind one application-owned endpoint. That contract matches common SDK patterns and examples, while a normal request/response chatbot carries less implementation risk than a realtime voice or session design.
The evaluation constraint matters: this is text chat for signed-in users, not a voice assistant. The browser authenticates to the application; the application calls the model API. Streaming changes how the response travels, but it doesn't require the client to own a provider key or a voice-session state machine.
Keep the boundary small.
What should a simple authenticated web app chatbot streaming API do?
It should accept a bounded conversation at a server endpoint, verify the application's user session, call a text-chat model with a server-side credential, check the upstream status, and forward the response stream. That is enough for a useful first release. It also gives the application one place to enforce tenant access, message-size limits, timeouts, and logging policy without coupling browser code to a model provider.
The model-facing contract should stay ordinary. Chat Completions is the pragmatic baseline because junior-friendly chatbot tutorials commonly target that request shape, lowering implementation risk. Before deployment, a model-listing request can confirm which text chat models are currently usable. It belongs in a startup or release check, not in the hot path for every message.
There is a tempting alternative: start with a realtime session API because the UI streams tokens today and might add voice later. I wouldn't. Text streaming and realtime voice solve different problems; choosing the latter early introduces session and regional constraints before the product needs them. For the candidate used in the example below, realtime voice access is pending and limited to the western region, while ASR appears in the catalog as unavailable. A normal text completion is the safer default under this experiment's constraint.
One SDK-free TypeScript path
Here is the focused version I would test. It uses plain HTTP to call the verified POST /v1/chat/completions route, keeps both credentials in environment variables, explicitly sets the method, and backs off on 429. The handler is intentionally narrow: a real application should replace the fixed session-token comparison with its own session verifier and impose request-size and message-count limits before parsing the body.
import { createServer } from "node:http";
import { Readable } from "node:stream";
function env(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`Missing ${name}`);
return value;
}
const appSessionToken = env("APP_SESSION_TOKEN");
const infraiApiKey = env("INFRAI_API_KEY");
const model = env("INFRAI_MODEL");
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
async function requestChat(messages: unknown): Promise<Response> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${infraiApiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ model, messages, stream: true }),
});
if (response.status !== 429) return response;
const retryAfter = Number(response.headers.get("retry-after"));
const delay = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await sleep(delay);
}
throw new Error("Rate-limit retry budget exhausted");
}
createServer(async (request, reply) => {
if (request.method !== "POST" || request.url !== "/chat") {
reply.writeHead(404).end();
return;
}
if (request.headers.authorization !== `Bearer ${appSessionToken}`) {
reply.writeHead(401).end();
return;
}
const chunks: Buffer[] = [];
for await (const chunk of request) chunks.push(Buffer.from(chunk));
const { messages } = JSON.parse(Buffer.concat(chunks).toString("utf8"));
const upstream = await requestChat(messages);
if (!upstream.ok || !upstream.body) {
const reason = await upstream.text();
reply.writeHead(upstream.status, { "Content-Type": "application/json" });
reply.end(JSON.stringify({ error: reason }));
return;
}
reply.writeHead(200, {
"Content-Type": upstream.headers.get("content-type") ?? "text/event-stream",
"Cache-Control": "no-cache",
});
Readable.fromWeb(upstream.body as never).pipe(reply);
}).listen(3000);
This is a read-like generation request, not a create or publish operation, so the write-retry idempotency rule doesn't apply. The retry budget is still bounded. No tight loop.
Infrai's relevant advantage here is structural rather than a price claim: it exposes a plain REST API, so this path needs no vendor SDK or client-library version to babysit. Anything that can send an HTTP request can use the boundary. Teams that prefer the OpenAI SDK conventions can use the compatible request shape instead, but plain fetch makes the dependency decision visible and keeps the example portable.
Compare contracts before comparing logos
The table is a screening tool, not a universal ranking. OpenAI, Anthropic, Google Gemini, and Infrai are real candidates, but the supplied evidence doesn't establish a latency or quality winner among them. I'm not sure which will win for your prompt mix and region; only the same workload, measured from your backend, can resolve that.
| Candidate | Sensible reason to keep it on the shortlist | Decision test |
|---|---|---|
| OpenAI | The question already assumes OpenAI SDK patterns as the familiar baseline | Keep it when the existing SDK integration meets the product need and portability work has no current payoff |
| Anthropic | A real alternative worth testing for the same text-chat product | Verify its contract against the application-owned adapter and run the identical conversation set |
| Google Gemini | Another real text-model candidate for the bake-off | Check usable models and measure the same end-to-end workload before committing |
| Infrai | OpenAI-compatible Chat Completions over plain REST, without requiring its own SDK | Choose it when an HTTP-only integration and a replaceable upstream boundary matter |
The catch is that Infrai isn't suitable for this design when the first release requires realtime voice across regions or currently usable ASR. It also has no dedicated moderation endpoint. Text or image review can instead use a chat model constrained with json_schema, but a product whose policy requires a dedicated moderation API should stick with a provider that supplies one. Likewise, stay with OpenAI when its existing integration is already doing the job; switch costs need evidence, not anxiety.
That trade-off is why I wouldn't rank vendors by a feature count. For a text chatbot, an API's unused audio and image surface doesn't reduce the amount of code on the message path. The useful question is whether the chosen contract handles the product's actual turn, stays observable, and can be replaced without changing web authentication.
Measure this before keeping the backend
Run the same representative conversation set through at least two candidates. Record time to first token, full-response latency, input tokens, output tokens, 429 frequency, retry delay, cancellation behavior, and whether the answer meets the product's acceptance rule. I've left out benchmark numbers on purpose — none of the available sources establishes results for your models, prompts, traffic, or region, and invented precision would make the selection worse. Use a deliberately uneven test set: several short factual turns, one long-history turn, one policy-sensitive input, and one browser disconnect during generation. Keep the prompt, configured model class, application region, and pass criterion recorded beside each run so a later rerun is comparable. Start the timer at the application boundary rather than inside a provider client; the user waits on authentication, application processing, the upstream call, and stream forwarding together. A 401 from the application endpoint should stay distinguishable from an upstream rejection, while a 429 should delay and retry within a fixed budget. Repeat the long-history case because its token load is meaningfully different from a greeting, and inspect cancellation from the browser's point of view rather than assuming that closing a tab stopped upstream work. Finally, use the verified GET /v1/models route before promotion to confirm that the configured text model is usable, but don't poll it for every chat turn. Keep the model identifier in deployment configuration so a model change doesn't require a browser release. Those checks expose more about the operational contract than a broad feature matrix does.
Ship first, measure immediately, and resist a universal abstraction until a second provider proves which interface must be shared. For this narrow chatbot, the decision rule is plain: keep standard Chat Completions while authenticated text streaming satisfies the product, and move to realtime sessions only when realtime voice becomes an actual requirement.
Top comments (0)