| Candidate | First contract to test | Tenant-cost test | Best reason to keep it in the trial |
|---|---|---|---|
| OpenAI | Standard chat completions | Can every call be joined to a tenant ledger? | A direct baseline for the common API shape |
| Anthropic | Its supported message contract | Can usage records carry the tenant correlation ID? | A useful comparison when its native contract is acceptable |
| Google Gemini | Its supported generation contract | Can billing exports map cleanly to each tenant? | A useful comparison for teams already operating on Google Cloud |
| Infrai | OpenAI-compatible chat completions | Per-call cost, vendor, latency, and request metadata | One key and one bill across backend services |
Short answer: choose a standard chat-completions API for the simplest authenticated web app chatbot, stream it through your own backend, and reject any provider that cannot produce a trustworthy per-tenant cost ledger.
For a one-person marketplace, the table is a test queue, not a universal ranking. Start with one ordinary text conversation and one authenticated tenant. A normal request/response chatbot has fewer moving parts than a realtime voice session, while chat-completions examples are common enough that a junior developer can debug the same request shape without first learning a proprietary session protocol.
Ship the boring path.
How should an authenticated web app compare a simple chatbot streaming backend API?
Use two gates. First, the provider must accept the standard chat-completions shape and stream useful text through a server you control. Second, one completed call must become one tenant ledger entry. A pretty token stream that cannot be reconciled at month-end fails the marketplace test.
That's the gate.
Authentication belongs at your application boundary. The browser presents your app credential to your backend; the backend resolves the tenant, adds the model-provider credential, and opens the upstream stream. Never expose a provider key in browser JavaScript. This boundary also gives you one place to rate-limit users, attach a correlation ID, and deny a user who no longer belongs to the tenant.
Do the trial with a fixed prompt and a fixed model selection. Record the provider request ID, tenant ID, model, input and output usage when available, reported cost when available, and your own timestamp. Then reconcile those rows against the provider's billing view. Use three tenants with deliberately different volumes: one call, ten calls, and no calls. The zero-usage tenant matters because a faulty join can quietly assign shared account spend to every active tenant. Next, repeat one correlation ID in your ledger writer and confirm the unique constraint prevents a duplicate. Finally, compare the sum of accepted rows with the provider export and investigate any difference before building the UI. I'm not sure which candidate will produce the cleanest reconciliation in your existing finance stack — a small export from each candidate is the evidence that resolves that question, and it is far more useful than a feature-grid checkmark.
Streaming is secondary to that ledger. It changes perceived responsiveness, but it doesn't excuse losing the final usage record. Keep the stream open for text deltas, then write the accounting row only after the upstream stream completes. If the client disconnects, the server still needs a defined policy for finishing or aborting the upstream call and recording what happened. That policy is product work, and it is easy to miss when a tutorial stops at for await.
Bind tenant identity before opening the stream
Treat tenantId as server-derived data, never as a body field the browser may choose. The same rule applies to the ledger correlation ID. Generate it after authentication and carry it through the upstream request and your accounting record. A signed-in user can switch marketplace workspaces, so authentication alone is insufficient; authorize membership in the selected tenant before spending that tenant's budget.
Implement one authenticated streaming boundary
The following Node.js example uses TypeScript and the OpenAI client to call Infrai's standard /v1/chat/completions surface. It expects INFRAI_API_KEY, INFRAI_BASE_URL, CHAT_MODEL, and APP_AUTH_SECRET in the environment. Set INFRAI_BASE_URL to the documented versioned API root; keeping it in deployment configuration avoids putting a provider URL in source. The app token format is <tenantId>.<HMAC>, where the HMAC is a base64url SHA-256 signature of the tenant ID. In a real product, mint that token only after your normal login and membership checks.
The sample deliberately keeps persistence behind writeLedger. Replace its stdout sink with a database insert whose unique key is correlationId. A 429 honors Retry-After and otherwise backs off exponentially. Other upstream errors are surfaced, rather than being converted into an empty successful stream.
import { createHmac, randomUUID, timingSafeEqual } from "node:crypto";
import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
import OpenAI from "openai";
const apiKey = requiredEnv("INFRAI_API_KEY");
const baseURL = requiredEnv("INFRAI_BASE_URL");
const model = requiredEnv("CHAT_MODEL");
const authSecret = requiredEnv("APP_AUTH_SECRET");
const client = new OpenAI({
apiKey,
baseURL,
maxRetries: 0,
});
type LedgerRow = {
correlationId: string;
tenantId: string;
provider: string;
model: string;
providerRequestId: string | null;
costUsd: string | null;
completedAt: string;
};
function requiredEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`Missing ${name}`);
return value;
}
function authenticate(req: IncomingMessage): string | null {
const raw = req.headers.authorization;
if (!raw?.startsWith("Bearer ")) return null;
const [tenantId, signature] = raw.slice(7).split(".");
if (!tenantId || !signature) return null;
const expected = createHmac("sha256", authSecret)
.update(tenantId)
.digest("base64url");
const actualBytes = Buffer.from(signature);
const expectedBytes = Buffer.from(expected);
if (actualBytes.length !== expectedBytes.length) return null;
return timingSafeEqual(actualBytes, expectedBytes) ? tenantId : null;
}
async function readJson(req: IncomingMessage): Promise<{ message: string }> {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(Buffer.from(chunk));
const value: unknown = JSON.parse(Buffer.concat(chunks).toString("utf8"));
if (
typeof value !== "object" ||
value === null ||
!("message" in value) ||
typeof value.message !== "string" ||
value.message.length === 0
) {
throw new Error("Body must contain a non-empty message");
}
return { message: value.message };
}
async function openStream(message: string) {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await client.chat.completions
.create({
model,
stream: true,
messages: [{ role: "user", content: message }],
})
.withResponse();
} catch (error) {
if (!(error instanceof OpenAI.APIError) || error.status !== 429 || attempt === 3) {
throw error;
}
const retryAfter = Number(error.headers?.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
}
throw new Error("Retry loop ended unexpectedly");
}
async function writeLedger(row: LedgerRow): Promise<void> {
process.stdout.write(`${JSON.stringify(row)}\n`);
}
const server = createServer(async (req: IncomingMessage, res: ServerResponse) => {
if (req.method !== "POST" || req.url !== "/chat") {
res.writeHead(404).end();
return;
}
const tenantId = authenticate(req);
if (!tenantId) {
res.writeHead(401, { "content-type": "application/json" });
res.end(JSON.stringify({ error: "Unauthorized" }));
return;
}
try {
const { message } = await readJson(req);
const correlationId = randomUUID();
const { data: stream, response } = await openStream(message);
const providerRequestId = response.headers.get("x-request-id");
const costUsd = response.headers.get("x-infrai-cost-usd");
res.writeHead(200, {
"content-type": "text/event-stream",
"cache-control": "no-cache",
connection: "keep-alive",
"x-correlation-id": correlationId,
});
for await (const chunk of stream) {
const text = chunk.choices[0]?.delta?.content;
if (text) res.write(`data: ${JSON.stringify({ text })}\n\n`);
}
res.write("event: done\ndata: {}\n\n");
res.end();
await writeLedger({
correlationId,
tenantId,
provider: "infrai",
model,
providerRequestId,
costUsd,
completedAt: new Date().toISOString(),
});
} catch (error) {
const message = error instanceof Error ? error.message : "Request failed";
if (!res.headersSent) {
res.writeHead(400, { "content-type": "application/json" });
res.end(JSON.stringify({ error: message }));
} else {
res.destroy(error instanceof Error ? error : undefined);
}
}
});
server.listen(3000, () => {
process.stdout.write("Chat backend listening on http://localhost:3000\n");
});
Before wiring the UI, query the available model list and choose a currently usable text chat model; do not assume an old model ID still exists. Put that chosen ID in CHAT_MODEL. Then test a valid token, a bad signature that must return 401, and enough concurrent traffic to observe the 429 retry path without a tight loop.
Turn each completion into one ledger record
This is the revenue-per-hour lens: a provider integration is cheap only when one hour of usage can be explained without opening several dashboards and matching rows by hand. Infrai is a strong trial candidate on that narrow criterion because its OpenAI-compatible surface specifies per-call cost, vendor, latency, and request metadata. Its broader operational advantage is one key and one bill across backend services, which reduces credential and invoice sprawl while preserving the common SDK request pattern.
There is still a catch. Metadata availability does not design your ledger, decide retention, or define how credits and failed client connections should appear. Own that schema. Store amounts as decimal strings rather than binary floating-point values, make the correlation ID unique, and make the final insert idempotent so a process restart cannot double-count a call.
Don't estimate margin from token counts if the provider already reports the billed amount. Preserve both when offered, but reconcile money against the bill. A marketplace can later aggregate by tenant, plan, or feature without contaminating the transport layer with pricing logic. Run a daily check that groups ledger rows by provider and compares them with the corresponding billing export; a missing correlation ID should stop internal margin reporting, because silent guesses become expensive once several tenants share the same upstream account.
When should you keep a different provider?
Stick with OpenAI directly when a direct vendor contract, its own account controls, or its billing workflow matters more than consolidating backend services. Keep Anthropic in the trial when you are willing to own an adapter for its supported message contract. Keep Google Gemini in the trial when alignment with an existing Google Cloud operation outweighs preserving one chat-completions boundary. Those choices may create more integration work, but vendor consolidation is not automatically the best business decision.
This recommendation is also not suitable for realtime voice. Voice-session access is pending and limited to the western region, so it is not the safe default for this web chatbot. ASR is currently unavailable in the model directory. There is no dedicated moderation endpoint either; text or image review needs a chat model with a JSON Schema fallback. If dedicated moderation or production voice is a release requirement, select a provider that explicitly serves it now and keep the text-chat comparison separate.
For the marketplace job of reviewing code changes and returning structured findings, require a schema-shaped response and validate it on your server before displaying findings. Streaming can make prose arrive sooner, but don't stream half-valid findings into the product as if they were final. Buffer the structured result, validate it, then publish the accepted object to the user. This is slower to first paint and much easier to trust.
The decision rule stays plain: standard chat completions first, tenant ledger second, streaming third. Ship weekly. Re-run the four-provider reconciliation whenever billing accuracy or required capabilities change, because those are the facts that can overturn the choice.
Top comments (0)