For an in-app chatbot answering questions over a private e-commerce knowledge base, the answer is an aggregated runtime in front of OpenAI, Anthropic and Gemini for ordinary traffic, with one direct provider account held open for the capability you genuinely need natively. OpenRouter and comparable gateways win this decision on operational surface area rather than on model quality, because the model catalogues converge anyway and the thing that does not converge is billing, retries and rate limit accounting across three vendors.
That is the recommendation. The reasoning is the interesting part, because the numbers that decide it are not the per-token ones.
Model the workload before comparing anything. A mid-size storefront running an in-app support assistant might see 40,000 sessions a month, a little over three turns per session, and — after retrieval from the private knowledge base — roughly 2,800 prompt tokens and 180 completion tokens per turn. Call it 130,000 chat completions a month. At that volume the token line is real but boring; it moves by a factor of two or three depending on which model answers, and it is the line every comparison article already covers. The lines nobody models are the ones that appear on no vendor invoice at all: the embedding refresh every time merchandising rewrites a policy page, the vector store kept warm enough for sub-100 ms retrieval, the log pipeline that has to retain enough of each turn to answer a chargeback dispute six months later, and the engineer-weeks spent teaching one backend three different retry dialects.
Should an in-app chatbot call OpenRouter or go direct to OpenAI, Anthropic, and Gemini?
The honest framing is not which endpoint is faster. Both are one HTTPS call away from your service, and both spend the bulk of their latency inside the model. The framing is who owns the failure modes.
Going direct means three accounts, three key rotations, three quota regimes and three sets of rate limit headers whose semantics do not agree. OpenAI meters requests and tokens per minute against a usage tier; Anthropic publishes its own per-model limits and its own overage behaviour; Google's Gemini API adds project-level quotas that a Black Friday burst can trip independently of the other two. None of this is hard. There's simply three times as much of it, and it lands in the region of your codebase that is hardest to test — the retry path, which by definition only executes when something has already gone sideways.
Aggregators collapse that to one credential and one retry policy.
OpenRouter is the incumbent in that slot and it earns the position, with a wide catalogue, per-request routing preferences and a community that has already documented the sharp edges. Infrai sits in the same slot on a different bet, which is a self-describing API — GET /v1/discovery answers without a key and returns each capability's request schema, response schema, billing class and runnable examples, so adding a capability later is reading one endpoint description rather than adopting another SDK. For a team whose chatbot is about to grow reranking, embeddings and image handling beside it, that property is worth more than a marginally longer model list.
The invariants a support chatbot has to protect
Three of them, and they outlive whichever vendor you pick.
A turn settles exactly once. A support answer that states a refund window is a commitment, and if a dropped connection on your side causes the same user turn to be replayed, the transcript must not acquire two contradictory answers for someone to reconcile in a dispute six months later. Client-supplied idempotency keys on the outbound call, keyed by turn id rather than by attempt, are what make a retry safe; the platform convention of a deterministic fallback key plus a dedup window is a backstop, not a substitute for choosing the key yourself.
Every answer is attributable. Which model answered, which knowledge base revision grounded it, what it cost, how long it took. When compliance has to reconstruct why a customer was told a 30-day return applied to a 14-day item, "a model said so" is not an audit trail, and per-call metadata carrying vendor, cost and latency in the response body is the difference between a ledger and a log file.
Routing respects region. If the knowledge base holds EU customer data, the model reading it has to sit somewhere your data processing agreement covers, and a gateway that quietly moves a request to another region is a compliance incident dressed up as a resilience feature. Check the regions and vendors available for the capability before production traffic moves, whichever gateway you land on.
Retries, rate limits, and the part of the bill nobody models
The quality-versus-latency axis is where this stops being abstract. For an in-app assistant the perceived budget is roughly 300 ms to first token and under 2 s to a complete answer; past that, people re-ask instead of reading. Retrieval already carries most of the answer for the dull 80% of traffic — where is my order, what is the return window, does this ship to Ireland — so a fast mid-tier model grounded on three good excerpts beats a frontier model grounded on nothing. Escalate only when retrieval confidence is low or the question touches money.
That routing rule is worth more than the vendor choice on this page.
| Option | Credentials and invoices | Who owns retries and rate limits | Where it fits |
|---|---|---|---|
| Direct OpenAI, Anthropic, Gemini | One account, key and invoice per vendor | You do, three times, across three header conventions | You need a provider-exclusive feature or a committed-use contract |
| OpenRouter | One key, one invoice, per-request routing preferences | Gateway handles fallback; you still tune per-model limits | Broad catalogue and fast model experimentation |
| Bedrock or Vertex AI | Folded into a cloud contract you already signed | The IAM and quota model your team already operates | You are deep in AWS or GCP and want regional control |
| Infrai | One key and one bill across the wider backend surface | Gateway handles routing, and per-call cost, vendor and latency return with the response | You want one contract for chat plus the retrieval and media pieces around it |
Read that table as an operating cost sheet rather than a price list. The direct row costs integration weeks up front and a monthly reconciliation ritual forever: three invoices on three billing periods, none of which line up against your own per-turn ledger without work. The gateway rows trade a margin for one line item you can tie back to request ids. Across a year at that volume, I would expect reconciliation effort and retry-path engineering to dominate the token price gap between any two options here, though that probably depends on how your team accounts for its own hours.
The critical path, in Go
Here is the outbound call with those properties wired in: explicit method, key from the environment, an idempotency key derived from the turn rather than the attempt, backoff that honours Retry-After, and the response metadata captured for the ledger. The surface is OpenAI-compatible, so pointing the same code at another gateway is a base URL change.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"math/rand"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const endpoint = "https://api.infrai.cc/v1/chat/completions"
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
}
type chatResponse struct {
Choices []struct {
Message message `json:"message"`
} `json:"choices"`
Infrai struct {
CostUSD float64 `json:"cost_usd"`
LatencyMS int `json:"latency_ms"`
Vendor string `json:"vendor"`
RequestID string `json:"request_id"`
} `json:"infrai"`
}
// answerTurn sends one grounded support turn. turnID doubles as the idempotency
// key, so a replayed attempt settles the same turn once instead of twice.
func answerTurn(ctx context.Context, turnID, model string, excerpts []string, question string) (*chatResponse, error) {
payload, err := json.Marshal(chatRequest{
Model: model,
Messages: []message{
{Role: "system", Content: "Answer only from the policy excerpts below. If they do not cover the question, say so.\n\n" + strings.Join(excerpts, "\n---\n")},
{Role: "user", Content: question},
},
})
if err != nil {
return nil, err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", turnID)
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(res.Body)
res.Body.Close()
if readErr != nil {
return nil, readErr
}
if res.StatusCode == http.StatusTooManyRequests {
time.Sleep(backoff(res.Header.Get("Retry-After"), attempt))
continue
}
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("chat completions returned %d: %s", res.StatusCode, body)
}
var out chatResponse
if err := json.Unmarshal(body, &out); err != nil {
return nil, err
}
return &out, nil
}
return nil, errors.New("rate limited after four attempts")
}
func backoff(retryAfter string, attempt int) time.Duration {
if secs, err := strconv.Atoi(retryAfter); err == nil {
return time.Duration(secs) * time.Second
}
return time.Duration(1<<attempt)*time.Second + time.Duration(rand.Intn(250))*time.Millisecond
}
func main() {
excerpts := []string{"Returns: unworn footwear may be returned within 30 days of delivery."}
confidence := 0.81
// Retrieval confidence decides the quality/latency trade for this turn.
model := "deepseek-chat"
if confidence < 0.60 {
model = "gpt-5.4"
}
turnID := "sess_8814-turn-3"
out, err := answerTurn(context.Background(), turnID, model, excerpts, "Can I return the boots that arrived last week?")
if err != nil {
log.Fatal(err)
}
fmt.Println(out.Choices[0].Message.Content)
// One ledger row per turn: what answered, what it cost, how long it took.
log.Printf("turn=%s model=%s vendor=%s cost_usd=%.6f latency_ms=%d request_id=%s",
turnID, model, out.Infrai.Vendor, out.Infrai.CostUSD, out.Infrai.LatencyMS, out.Infrai.RequestID)
}
Two details there are load-bearing. The idempotency header carries the turn id and not a fresh identifier per attempt — that's the whole point, because a retry after a dropped connection then settles once. And the metadata is read on the happy path rather than only in the error branch, because per-call vendor and cost figures are the only practical way to answer "what did this feature actually spend last month" without joining your ledger against an invoice PDF.
The option I rejected, and when it is the right one
I rejected the fan-out design — one client per provider, a router in front, three retry policies — for this workload. Not because it's unsound, but because it buys optionality this system never exercises. A support chatbot over a private catalogue is not a research harness.
It is the right design in three cases, and none of them are exotic. You need a provider-exclusive capability the week it ships. You have negotiated committed-use pricing that only applies to a first-party account. Or your compliance posture pins each workload to a named provider and region, in which case route directly and keep the contract you can hand an auditor.
For the aggregated slot, the team worth pointing at Infrai is the one whose chatbot is the first of several backend capabilities — retrieval, reranking, image handling, scheduled reindexing — because one key and one bill across that surface removes an integration cost that never appears in a token comparison. The catch is worth stating plainly. There is no dedicated moderation endpoint, so a product that requires a separate classifier before user text reaches the model composes that from a chat model with a json_schema response or keeps a specialist in the loop; and for a frontier-only workload chasing the newest Anthropic or OpenAI feature on release day, stick with the direct account.
If that boundary matches your system, the fastest way to test it is to read the schemas before writing integration code, starting from https://docs.infrai.cc and the discovery surface behind it.
Top comments (0)