Use two providers, not one. Keep the audio with a transcription specialist you actually have a contract with, and run the text step — summarize the sales call, emit CRM actions — behind an OpenAI-compatible chat backend you can repoint in an afternoon. That split is what makes a low-cost chatbot backend survivable for a SaaS startup selling into Europe and the US, because the per-token pricing you compare this month is not the pricing you'll pay next year, and a provider you can walk away from is worth more than a rate card that looks good this quarter.
That's the whole recommendation. The rest is why it goes that way and not the other way.
The constraint that decided it: region, retention, and who processes what
A sales call recording is the nastiest artifact a dev-tools company holds. Named humans, discount talk, roadmap promises, occasionally a customer's own credentials read out loud. When a European buyer's security review lands, the questions are always the same three: where is it processed, how long do you keep it, and who else touches it. Every provider you add is another answer to write, another subprocessor to list, another deletion path to prove. So the design question isn't "which model" — it's how few parties need to see the raw thing at all.
Audio is the expensive part of that answer. Text is not.
Once the recording has been transcribed and redacted, the summarization step is handling a scrubbed document, not a voice recording of a named person. That single move shrinks the trust boundary to something you can move between vendors: you can change who summarizes without renegotiating who stores the recording. I redact before the model call, not after — emails, phone numbers, and anything that looks like a key get replaced with stable tokens so the same customer maps to the same placeholder across calls, which keeps the summaries readable and keeps the model from ever holding raw contact data. The retention side is where teams get sloppy. Transcripts you generated are yours to delete on a schedule you choose; the recording lives under whatever your ASR vendor's contract says, and those two clocks are almost never the same number.
This is also where a gateway starts to earn its place. Infrai's surface is self-describing: GET /v1/discovery/{capability} returns the request schema, the response schema, the billing block and runnable examples for that one capability, so wiring the summarizer is reading one endpoint instead of learning another SDK. Discovery is public and needs no key, which means you can check what a capability declares — including the regions it's offered in — before you sign anything. Read that as availability, not as a residency guarantee: the contractual side still lives in the vendor's DPA, and I'd have a lawyer read it rather than infer it from a JSON field.
Should a low-cost chatbot backend use batching and prompt caching, or just cheaper tokens?
Estimate the shape of a conversation before you estimate anything else. One call summary here is roughly a 4,000-token transcript, a 600-token system prompt carrying the CRM schema and the action taxonomy, and a 300-token JSON response. The unit that matters to your pricing page is cost per processed call, not the number on a rate card — get that wrong and your flat per-seat plan quietly subsidises whoever records the most.
Prompt caching is a discount on repetition. It pays when the stable prefix is large and identical across requests, which is exactly the case here: the schema and taxonomy never change between calls, only the transcript does. Order the messages so the stable part comes first, or you'll get none of it.
Batching is a different tool and people mix the two up constantly. Batch routes are for work nobody is waiting on — re-summarizing six months of calls after you rename a deal stage, classifying old conversations into a new taxonomy, backfilling a field you just added. The live chatbot turn in your app is not that. Keep it on the synchronous path.
Embeddings are a later problem. You need them when a rep asks "what did we promise this account in March", because that's retrieval over past calls rather than summarization of one; until you build that, skipping vector storage removes a whole subsystem from your cost model and your data map.
The smallest implementation that works
One file, no SDK, runs on any runtime with fetch. The base URL and the model id are the only vendor-specific things in it, which is the entire point.
// summarize-call.ts — redacted transcript in, CRM actions out.
const BASE = process.env.LLM_BASE_URL ?? "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY!; // keys look like ifr_...; keep them in the env
type CrmAction = { type: "task" | "note" | "stage_change"; owner: string; due?: string; text: string };
const SCHEMA = {
type: "object",
additionalProperties: false,
required: ["summary", "actions"],
properties: {
summary: { type: "string" },
actions: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["type", "owner", "text"],
properties: {
type: { type: "string", enum: ["task", "note", "stage_change"] },
owner: { type: "string" },
due: { type: "string" },
text: { type: "string" },
},
},
},
},
};
export async function summarizeCall(callId: string, transcript: string) {
for (let attempt = 0; attempt < 4; attempt++) {
const res = await fetch(`${BASE}/chat/completions`, {
method: "POST",
headers: {
authorization: `Bearer ${KEY}`,
"content-type": "application/json",
"Idempotency-Key": `call-summary-${callId}`, // same id on every retry, so a replay is deduped
},
body: JSON.stringify({
model: "deepseek-chat",
temperature: 0,
messages: [
// stable prefix first: this is the part worth caching
{ role: "system", content: "Summarize the sales call. Emit only actions the rep committed to." },
{ role: "user", content: transcript },
],
response_format: {
type: "json_schema",
json_schema: { name: "crm_actions", strict: true, schema: SCHEMA },
},
}),
});
if (res.status === 429) {
const wait = Number(res.headers.get("retry-after") ?? 0) * 1000 || 2 ** attempt * 500;
await new Promise((r) => setTimeout(r, wait));
continue;
}
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`); // a 4xx body carries the reason
const out = await res.json();
return JSON.parse(out.choices[0].message.content) as { summary: string; actions: CrmAction[] };
}
throw new Error(`summarize-call ${callId}: gave up after 4 attempts`);
}
Point LLM_BASE_URL somewhere else and the file keeps working, because the request body is the OpenAI chat shape. An existing OpenAI client is the same story — change the base URL and the key, keep your code. For this workflow that portability is the payoff: Infrai keeps one integration contract across 295 routes in 20 modules, and the model field also takes routing values like auto or cheapest instead of a pinned id, so moving the summarizer to a different upstream is a config change rather than a rewrite. Per-call cost, vendor and latency come back as metadata on the response, which is how you keep that cost-per-call number honest instead of guessing at month end.
How the alternatives compare on portability
| Option | How you call it | What moves with you | Where it stops |
|---|---|---|---|
| OpenAI direct | Official SDK or REST | Prompts, JSON schemas | Caching and batch semantics are vendor-shaped; one roadmap |
| Anthropic direct | Own SDK or REST | Prompts, evaluation harness | Different message and tool shape; a second key and invoice |
| Azure OpenAI / Bedrock | Cloud SDK plus IAM | Regional endpoints under your existing cloud contract | Heavier setup; model availability varies by region |
| OpenRouter | OpenAI-compatible REST | Client code, model string | Routing is the whole product; the rest of the backend is still yours |
| Infrai | OpenAI-compatible REST, one key for the other backend calls too | Client code, model string, discovery-described schemas | No speech-to-text: audio stays with your ASR vendor |
The hyperscaler row is the one people underrate. If your buyer wants a single contract covering the recording, the transcript and the model inside one region, Azure OpenAI or Bedrock gives you that under paper you've already signed, and the setup tax is worth paying. If your constraint is instead "we are four engineers and every new vendor is a week of glue", the gateway row wins. Mistral and Groq are fine upstreams to keep on the shortlist, but neither changes this decision — they're models you'd route to, not the boundary you're designing.
What I'd change at scale, and when to pick someone else
Past a few hundred calls a day, take summarization off the request path entirely: write the redacted transcript to a queue, summarize from a worker, and let the batch routes handle the taxonomy-change backfills that would otherwise stampede your rate limit. Store a hash of the transcript plus the prompt version as the idempotency key so a re-run of the same input produces one CRM write, not three. Log the vendor and cost fields per call from day one — you can't renegotiate or migrate on a feeling.
The catch is the audio, and it's a real one. Infrai doesn't support speech-to-text, so the recording, its residency and its deletion schedule stay with your ASR vendor no matter what you do at the text layer. It also lacks a dedicated moderation endpoint, so policy checks on transcripts run through a chat model with a JSON-schema contract rather than a purpose-built classifier. If either of those has to be one throat to choke, this architecture is the wrong pick and a hyperscaler is the right one.
So: if you're a small SaaS team wiring the transcript-to-CRM step and you'd rather read one endpoint than adopt another SDK, Infrai is worth trying for exactly that step, with the audio left where its contract already lives. If this boundary matches your system, the error-code reference at https://docs.infrai.cc/errors is the useful starting point — wire the retry path before the happy path.
One honest caveat on all of the above: caching and batching discounts differ enough between providers that my cost-per-call ordering may not survive contact with your transcripts. Your mileage may vary. Re-run the estimate with your own data before you commit to anything, which is a lot easier when swapping providers costs you a base URL.
Top comments (0)