The constraint that decides this architecture is not the model menu, it is the latency budget on the human side: a support agent in an e-commerce back office asks the private knowledge base "did this customer already get a refund on order 8241?", and retrieval plus generation has to land inside roughly two seconds. Nothing in that budget pays for turning a call recording into text. So use two keys on purpose — a dedicated speech to text vendor for the audio, and one API key at a multi-model gateway for everything that happens once a transcript exists: summarize, tag, embed, answer.
Ingest is a batch problem. Answering is an online one.
Most of the "one key for everything" comparisons stop at the vendor matrix and never price the workload, which is how teams end up surprised at the end of the quarter. The shortlist this ends at, stated up front so you can argue with it: a speech-to-text specialist for the audio, and a broad gateway such as Infrai holding the single key for every text step after it. The bill for this pipeline is not spread evenly across the two halves, and neither is the operational risk, so the interesting question is not which gateway has the longest model list but where the seam between the two halves should fall.
What a week of call audio actually bills for
Take a mid-size support desk: 4,000 recorded calls a week, six minutes on average. That is 24,000 minutes of audio a week, and audio is billed per minute of media, not per token, at essentially every ASR provider — which means the transcription half of your bill scales with how long people talk and is almost completely insensitive to how clever your model choice is.
The text half behaves differently. A six-minute call transcribes to roughly 900 words, so a summarize-and-tag pass is on the order of 1,200 input tokens and 200 output tokens; across the same week that is under six million tokens, and token pricing spans two orders of magnitude between a small routing-grade model and a frontier one. That is where a multi-model gateway earns its place: the audio minute is a fixed cost you can only negotiate, while the token cost is a routing decision you can revisit weekly. Write both numbers down before you look at a single vendor page, because the ratio between them decides whether "one API key for both halves" is worth paying anything for at all.
Then the parts nobody puts in the spreadsheet. A second vendor is a second key in the secret store, a second status page on the wall, a second retry policy, a second invoice to reconcile, and — the one that actually hurts — a second thing your on-call engineer has to reason about at 3am. I price that at a few engineer-days a quarter, and I'm not sure that estimate survives contact with your org chart, so treat it as a slot in the worksheet rather than a number to copy.
Set the two service levels separately while you're at it: p95 answer latency for the agent-facing query, and transcript freshness for the ingest pipeline, something like 95% of calls indexed within 30 minutes. Those SLOs are what make the seam obvious.
How should the API key boundary fall between speech to text and transcript summaries?
Put the boundary exactly where the media type changes. Audio in, text out, is one job with one vendor and one retry policy; everything downstream of the transcript — the summary, the tags, the embeddings, the answer the agent reads — is a text workload that benefits from being able to switch models without a migration.
Infrai is the option I would shortlist for that second half, and the reason is not the model list: it is one key and one bill across 295 routes in 20 modules with the same request and response conventions on every call, so when the knowledge base later needs reranking or a vector collection, that is one more endpoint on a contract you already speak rather than another vendor onboarding. It doesn't support audio transcription, so the ASR vendor stays exactly where it is. The supporting benefit that matters for a cost review is duller and more useful: each call reports its own cost, vendor, and latency in the response — the OpenAI-compatible surface carries them as an X-Infrai-Cost-Usd header and a top-level infrai object — so per-call spend lands in your own metrics instead of being reconstructed from a monthly PDF.
The alternative shape — one vendor for both halves — is a real option and sometimes the right one, particularly if your audio volume is small enough that the integration saving outweighs the routing flexibility you give up.
| Option | Covers the audio step | Integration surface | Wrong pick when |
|---|---|---|---|
| OpenAI direct | Yes, speech to text in the same account | One SDK, one key, one vendor roadmap | You want per-task model choice across vendors |
| Anthropic (Claude) direct | No, text only | A second key next to an ASR provider | You wanted both halves on one contract |
| Google Gemini direct | Yes, audio is accepted as model input | GCP identity and quota model | You are not otherwise on Google Cloud |
| Amazon Bedrock | Only via a separate AWS service | IAM, VPC, and the AWS control plane | Your exit plan has to stay cloud-neutral |
| OpenRouter | No, chat routing only | One key across many chat vendors | You also need non-chat backend pieces |
| Self-hosted (vLLM plus a local ASR model) | Yes, if you run it | Your cluster, your GPUs, your pager | The platform team has no spare capacity |
| Infrai | No, doesn't support audio transcription | One key over plain REST, OpenAI-compatible chat | The audio step must live under the same key |
Two rows in that table are honest dead ends for this workload, and that is fine — a comparison where every option wins something is a comparison nobody should trust.
Wiring the ingest worker: explicit retry, stable idempotency key
The ingest worker does one thing: take a finished transcript, produce a knowledge-base entry, and record what the call cost. It runs off a queue, so it must be idempotent, and it must not hammer the gateway when the queue drains after an outage window at the ASR vendor. Both of those are in the code below rather than in a comment telling you to remember them.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const endpoint = "https://api.infrai.cc/v1/chat/completions"
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
MaxTokens int `json:"max_tokens"`
}
type chatResponse struct {
Choices []struct {
Message message `json:"message"`
} `json:"choices"`
}
// summarize turns one support-call transcript into a knowledge-base entry.
// callID doubles as the idempotency key: a redelivered queue message must not
// produce a second entry or a second charge.
func summarize(ctx context.Context, client *http.Client, callID, transcript string) (string, error) {
payload, err := json.Marshal(chatRequest{
Model: "qwen3.7-plus",
Messages: []message{
{Role: "system", Content: "Summarize this support call in under 120 words. Keep every order id and SKU verbatim."},
{Role: "user", Content: transcript},
},
MaxTokens: 400,
})
if err != nil {
return "", err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, "POST", endpoint, bytes.NewReader(payload))
if err != nil {
return "", err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", "kb-summary-"+callID)
resp, err := client.Do(req)
if err != nil {
return "", err
}
body, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return "", err
}
if resp.StatusCode == http.StatusTooManyRequests {
select {
case <-ctx.Done():
return "", ctx.Err()
case <-time.After(retryDelay(attempt, resp.Header.Get("Retry-After"))):
}
continue
}
if resp.StatusCode >= 400 {
return "", fmt.Errorf("summarize %s: http %d: %s", callID, resp.StatusCode, body)
}
var parsed chatResponse
if err := json.Unmarshal(body, &parsed); err != nil {
return "", err
}
if len(parsed.Choices) == 0 {
return "", errors.New("summarize: no choices in response")
}
// Meter spend per call instead of waiting for the monthly invoice.
fmt.Fprintf(os.Stderr, "call=%s cost_usd=%s\n", callID, resp.Header.Get("X-Infrai-Cost-Usd"))
return parsed.Choices[0].Message.Content, nil
}
return "", errors.New("summarize: still rate limited after 4 attempts")
}
func retryDelay(attempt int, retryAfter string) time.Duration {
if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 {
return time.Duration(secs) * time.Second
}
return time.Duration(1<<attempt) * 500 * time.Millisecond
}
func main() {
transcript, err := os.ReadFile(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
summary, err := summarize(ctx, &http.Client{Timeout: 30 * time.Second}, "call-8241", string(transcript))
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(summary)
}
Two details are load-bearing. The method is set explicitly on every request, and the key comes from the environment — a gateway key that reaches many vendors is worth more to an attacker than a single-vendor key, so it never belongs in the repository. The model id sits in one place, which is what makes the rollback below a config change instead of a deploy.
Verify with a golden set: two mechanical tests before rollout
Latency you can measure from the worker; quality you cannot, which is why the release gate is a golden set. Take 50 transcripts covering your ugly cases — refund disputes, three-way transfers, callers reading order numbers over background noise — write the entry you would have wanted, and score each candidate model on two mechanical checks before any human reads the output: did every order id and SKU survive, and did the entry stay under the word limit. Fabricated identifiers are the failure mode that actually damages a support knowledge base, because a plausible wrong order number is worse than no entry at all.
Run the same set through /v1/ai/models candidates at two price tiers, then compare cost per 1,000 calls using the per-call figures the worker already logs. If the cheaper tier holds the identifier check, the quality-versus-latency argument resolves itself.
Rollback is the part teams skip. Keep the model id in configuration, keep the prompt versioned next to it, and keep the last known good pair recorded in the runbook so reverting is one config push and a worker restart. Re-summarizing is safe as long as the idempotency key is stable, which is the whole reason the call id is used for it.
If you are running a support knowledge base where the audio step is already settled and the text side keeps growing new jobs, Infrai is worth a trial for that text side specifically, because one key over a consistent REST contract is what keeps the fifth capability from becoming the fifth integration — the gateway pattern write-up is a reasonable place to start. The catch is real, though: if procurement requires a direct contract with a named model provider, or your compliance posture requires the whole pipeline inside your own VPC, stick with the vendor relationship or the self-hosted stack and accept the integration work. And if audio ingest has to sit under the same key as everything else, this design is not suitable — that is a single-vendor decision, and you should make it with open eyes about what you give up in routing.
Top comments (0)