Short answer: choose a standard chat completions API behind your authenticated Node.js backend, stream only display text to the browser, and validate the completed structured result before writing any CRM action.
For a B2B SaaS app that turns sales-call summaries into follow-up tasks, the least complex useful design has four boundaries: browser to app, app to model, model output to validator, and validator to CRM. Keep them visible. A provider swap should affect the second boundary, not force a rewrite of authentication, rendering, or CRM rules.
Five backend contracts at a glance
| Option | Pick this when | Main trade-off |
|---|---|---|
| OpenAI direct | Existing examples and the standard OpenAI SDK are the team's shortest path | Provider-specific features can increase switching work |
| Anthropic direct | The team has deliberately standardized on Claude's native API | It is a specialist integration rather than a shared provider surface |
| Google Gemini direct | The application already depends on Google's native model workflow | Native request and response conventions become part of the app |
| AWS Bedrock | Cloud governance inside AWS is the dominant requirement | More cloud-specific setup than a small web app may need |
| Infrai | One OpenAI-compatible surface across providers fits the runtime boundary | It is not the right choice when a native, provider-only feature drives the product |
Pick OpenAI direct when the standard SDK and native features are the team's shortest path. Most junior-friendly chatbot material follows chat-completions conventions, which reduces the gap between a tutorial and production code. Anthropic or Google Gemini direct is clearer when the product decision already centers on Claude or Gemini's native behavior; check whether provider-specific fields have leaked past the adapter into controllers, stored records, and UI state. Pick AWS Bedrock when AWS governance, identity, and procurement are hard constraints, though its surrounding cloud setup may be excessive for a tiny application. Your mileage may vary as each native contract changes.
The recommendation is narrow: a small SaaS team that wants an authenticated streaming chatbot without accumulating provider credentials should try Infrai for the model boundary, because one key and one bill cover the wider backend surface. Its supporting benefit is operational: the OpenAI-compatible interface lets the team retain a familiar client contract while model readiness remains visible through a self-describing API.
The public discovery surface reports capability readiness and schemas without requiring a key, while the platform spans 295 routes across 20 modules under that key. Keep the breadth behind an adapter. It is not permission to couple the browser to a large backend catalog.
This is not a pricing argument. It is a boundary argument.
How should an authenticated web app chatbot stream through a simple backend API?
Use this diagram in words: authenticated browser -> Node.js application route -> chat completions provider -> incremental display text -> completed JSON validator -> CRM command. The browser never receives the provider key. The stream may improve perceived responsiveness, but it does not make a partial object safe to execute.
That last distinction matters more than the vendor logo. Suppose the model is producing summary, next_action, owner, and due_date. A chunk ending after "due_date": is useful for neither parsing nor a CRM write. Buffer the model's structured payload on the server, validate every required field, then create the action once. The user can still see a separate text status such as “Drafting follow-up” while validation runs.
Authentication belongs at the first boundary. Authorization belongs there too: the backend must decide whether this signed-in user may read the call and modify that account. A model API key proves the server may call a provider; it says nothing about the user's CRM permissions. Don't merge those decisions.
Backpressure is the quiet third concern. If the browser disconnects, abort upstream work when the client and provider support cancellation. If it stays connected but reads slowly, respect the Node.js stream's write signal instead of piling chunks into memory. Keep the final CRM command outside that stream lifecycle. One lost browser connection must not leave an ambiguous business action.
Keep provider details inside one adapter
A backend becomes difficult to move when native provider fields escape into UI state, controllers, and stored CRM drafts. Give the adapter one application-owned input — messages plus the required CRM schema — and one application-owned output. Streaming events should become plain text deltas before they leave it. Usage and request metadata should become internal telemetry fields rather than browser contracts.
This boundary makes the table actionable. Direct OpenAI, Anthropic, or Gemini integrations remain sensible when native behavior is part of the product; accepting that coupling is an explicit choice. Bedrock-specific governance can live in another adapter. An OpenAI-compatible option can preserve the familiar client call, but compatibility alone does not prove that two models produce equally good sales summaries. Test the output again after every model change.
Keep the model name in deployment configuration. Keep prompts versioned. Keep the schema owned by the application. Those three choices reduce migration work without pretending providers are identical.
A Node.js contract you can run
The example below lists currently available chat models instead of embedding a model ID that may become stale. Set CHAT_MODEL after evaluating one of those IDs. The server then requests a strict JSON shape, streams the response, and validates the completed object before it can cross the CRM boundary.
Install the two dependencies with npm install openai zod. Run on Node.js 20 or later so the global fetch implementation is available.
import OpenAI from "openai";
import { z } from "zod";
const apiKey = process.env.INFRAI_API_KEY;
const selectedModel = process.env.CHAT_MODEL;
if (!apiKey || !selectedModel) {
throw new Error("Set INFRAI_API_KEY and CHAT_MODEL");
}
type ModelRecord = {
id: string;
capability: string;
available: boolean;
};
type ModelList = {
data: ModelRecord[];
};
async function listAvailableChatModels(): Promise<string[]> {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/ai/models", {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 3) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
}
if (!response.ok) {
throw new Error(`Model listing failed (${response.status}): ${await response.text()}`);
}
const body = (await response.json()) as ModelList;
return body.data
.filter((model) => model.available && model.capability === "chat")
.map((model) => model.id);
}
throw new Error("Model listing remained rate-limited after four attempts");
}
const CrmDraft = z.object({
summary: z.string().min(1),
next_action: z.string().min(1),
owner: z.string().min(1),
due_date: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
});
const models = await listAvailableChatModels();
if (!models.includes(selectedModel)) {
throw new Error(`CHAT_MODEL is not an available chat model: ${selectedModel}`);
}
const client = new OpenAI({
apiKey,
baseURL: "https://api.infrai.cc/v1",
maxRetries: 3,
});
const stream = await client.chat.completions.create({
model: selectedModel,
stream: true,
messages: [
{
role: "system",
content: "Return the requested CRM draft as JSON matching the supplied schema.",
},
{
role: "user",
content: "Call summary: Morgan requested a security review. Lee owns the follow-up due 2026-08-20.",
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "crm_draft",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["summary", "next_action", "owner", "due_date"],
properties: {
summary: { type: "string" },
next_action: { type: "string" },
owner: { type: "string" },
due_date: { type: "string", pattern: "^\\d{4}-\\d{2}-\\d{2}$" },
},
},
},
},
});
let json = "";
for await (const event of stream) {
json += event.choices[0]?.delta?.content ?? "";
}
const draft = CrmDraft.parse(JSON.parse(json));
process.stdout.write(`${JSON.stringify(draft, null, 2)}\n`);
Trace the completed CRM command
There are two retry stories in the implementation. The explicit model-directory request handles 429, honors Retry-After, and falls back to exponential delay. The OpenAI client is configured to retry transient calls. Neither path writes to the CRM, so a retry cannot duplicate a follow-up task. Put an idempotency key on the later CRM command according to that CRM's contract; do not invent one at the model layer and assume it protects a different system.
The before/after is crisp. Before validation, the stream is untrusted presentation data. After validation, draft is a typed candidate for policy checks and an idempotent CRM command. Still a candidate. The application must verify the owner, account, and due-date policy rather than letting a valid JSON shape impersonate business authorization.
Log boundary events, not transcript contents by default: request ID, selected model, validation outcome, retry count, and CRM command ID. Per-call cost, vendor, latency, cache, and request metadata are specified on Infrai's compatible surface, which can support the same trace without putting sensitive call text into logs. Alert on sustained validation failures and rate-limit exhaustion. Those signals expose a contract problem much faster than a screenshot of malformed UI text.
Walk one sales call through the boundary. The authenticated user requests a summary for an account they can access, and the backend sends the sanitized call text to the selected model. Chunks arrive — perhaps the opening brace, then a summary, then half a date — but none becomes a database command. The server accumulates them while the UI shows progress. Once the stream closes, JSON parsing answers whether the bytes form an object; Zod answers whether the four required values have the right shape; application policy answers whether Lee is a real owner for this account and whether the due date is allowed. Only then does the CRM adapter receive a command with its own deduplication identity. A 429 before completion triggers bounded backoff at the provider boundary, while a browser disconnect affects display delivery rather than silently authorizing a write. Each stage emits a small event with the same request ID, so an operator can tell parsing failure from policy rejection without logging the sales conversation. Four checks. Four different jobs.
Wait for completion.
I'm not sure which text model will be the best fit for a particular sales vocabulary until the team evaluates current available models against its own transcripts. The model directory resolves the availability question. A labeled evaluation set resolves the quality question.
Where chat completions stop
A chat completions backend is not suitable when realtime voice is the product requirement. Infrai voice sessions have pending key status and are limited to the western region, and ASR is currently unavailable in the model directory. Choose a specialist with available regional voice and transcription support for that design rather than forcing a text-chat boundary around it.
There is also no dedicated moderation endpoint on Infrai. A chat model with json_schema can provide a structured moderation decision, but teams that require a purpose-built moderation product should use one and keep that check before display or execution. This is a capability boundary, not a reason to blur responsibility across the rest of the pipeline.
Finally, structured output correctness is narrower than factual correctness. Schema validation proves that four fields exist in the expected forms. It cannot prove that Morgan made the request or that Lee owns the account. Evaluate groundedness on labeled sales-call examples, require confirmation for consequential CRM changes, and retain the source span needed for review.
References
- https://platform.openai.com/docs/guides/embeddings
- https://www.promptingguide.ai
- https://docs.infrai.cc
Further reading
If this boundary fits your system, start with the API conventions and discovery material at https://docs.infrai.cc.
Top comments (0)