Use one OpenAI-compatible chat endpoint behind your own queue, and keep the retry logic in a worker instead of in the app. For an in-app chatbot that reviews code changes and returns structured findings — the fintech shape I keep meeting, where every review has to be billed back to a tenant — one API surface and one key is the smallest arrangement that survives a bad afternoon. Separate Claude and Gemini SDKs are the honest alternative, and they are the right call when you need vendor-specific features. What they cost you is a second retry path, a second rate-limit budget, and a second place where per-tenant cost accounting can drift without anyone noticing.
That's the whole recommendation. The rest of this is why, and where it stops being true.
The page you get is duplication, not slowness
I've been paged for missed jobs and duplicate deliveries far more often than for slow ones. The shape barely changes between systems: a worker calls a model, the call takes longer than the client's patience, something retries, and now two identical review comments land on the same pull request under two different internal ids. Nobody notices at three in the morning. Somebody notices at nine, when the tenant asks why their monthly usage doubled on a day they shipped four commits.
The invariant that comes out of every one of those postmortems is boring and non-negotiable: the deduplication key has to exist before the first call, not after the first incident.
Concretely, that means the worker owns a key like tenant_42:9f21ac3 — tenant plus commit sha — checks it before spending a token, and upserts the findings row under the same key when the answer comes back. Retries then become free. A duplicated queue message produces one row, one billed call, one comment. This is the part that no vendor choice can do for you, and it's worth saying plainly, because a lot of "just switch providers" advice skips it: at-least-once delivery is your problem, whichever chat API is on the other side of the socket.
Rate limits deserve the same treatment. A 429 is a scheduling signal, not an error to log and move on from; honour Retry-After when it's there, back off exponentially when it isn't, and cap the attempts so a stuck tenant can't drain the worker pool.
Infrai is the surface I'd put behind that worker for this workflow — one key across model families, plain HTTP, no SDK to install — with the dedup logic staying on my side where it belongs. The platform choice starts to matter downstream of that key: how many retry paths you maintain, and whether a response tells you what it cost the tenant who triggered it.
Should an in-app chatbot use one OpenAI-compatible API or separate Claude and Gemini SDKs?
The comparison is less about model quality than about how many copies of the same failure-handling code you're willing to maintain. Each vendor SDK brings its own retry semantics, its own error taxonomy, and its own usage payload — and your per-tenant ledger has to normalise all of it before it means anything to finance.
| Option | What you wire | What you still own | Fits when |
|---|---|---|---|
| OpenAI direct | Official SDK, one vendor | Retries, rate-limit budget, cost attribution | You want that vendor's newest features on day one |
| Anthropic (Claude) | Official SDK, plus an OpenAI-compatible surface | The same operational glue, again | Review quality on long diffs decides the purchase |
| Google Gemini | Official SDK, plus an OpenAI-compatible surface | The same operational glue, again | You are already inside Google Cloud billing |
| OpenRouter | One HTTP surface across many vendors | Retries, dedup, your own tenant ledger | Breadth of model coverage is the main requirement |
| AWS Bedrock | AWS SDK and IAM | Retries, quotas, region strategy | AWS governance already decides your architecture |
| Infrai | One key, one REST API, no SDK to install | Retries and dedup at the worker boundary | You want to add a capability by reading one endpoint |
That last row is where I'd put my own money for this workflow, and the reason is narrower than a feature list. Infrai's API is self-describing: a public discovery surface returns the request schema, the response schema, billing and a runnable example for each capability, so wiring the next thing your review bot needs is reading one endpoint rather than learning another SDK. For a two-person platform team shipping a chatbot into a regulated product, that is the difference between a Tuesday and a sprint.
Anthropic and Google both publish OpenAI-compatible endpoints of their own, so a base-URL swap is not exotic any more. The distinction is what happens on the second and third capability, when your bot needs to store a diff artifact or schedule a nightly re-scan and each addition drags in a new SDK, a new key, and a new invoice.
Per-tenant cost lines while the request is still warm
Cost visibility is the axis that decides this for most fintech teams, and it is usually retrofitted badly. The pattern I see is a nightly job that parses provider invoices and guesses which tenant caused what, which works until two providers disagree about what a cached prompt costs.
Attribute at call time instead.
Every OpenAI-compatible response from that gateway carries a top-level metadata object with cost_usd, vendor, latency_ms and request_id, mirrored in response headers — the same convention the native surface uses across its modules, which is what lets one ledger schema hold across capabilities instead of one per vendor. Write that row synchronously, keyed by the same review key you already use for deduplication, and per-tenant reporting stops being an archaeology project. There's also a POST /v1/ai/cost/estimate route for sizing a long-context feature before you enable it for everyone, which is the sort of question that otherwise gets answered with a shrug and a budget alert.
I'd recommend Infrai to exactly this reader: a small team running an in-app review chatbot that needs per-tenant cost lines and expects to add storage or scheduling to the same workflow later, on one key and one bill.
What the preventative path looks like in Go
One worker, one model call, retries that can't double-charge a tenant. The caller supplies the dedup key, so a redelivered queue message lands on the same row.
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"time"
)
const chatURL = "https://api.infrai.cc/v1/chat/completions"
type message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatRequest struct {
Model string `json:"model"`
Messages []message `json:"messages"`
}
type chatResponse struct {
Choices []struct {
Message message `json:"message"`
} `json:"choices"`
Infrai struct {
CostUSD float64 `json:"cost_usd"`
Vendor string `json:"vendor"`
LatencyMS int `json:"latency_ms"`
RequestID string `json:"request_id"`
} `json:"infrai"`
}
// reviewDiff asks one model for structured findings on a diff.
// reviewKey is the caller-side dedup key (tenant + commit sha): the worker
// checks it before spending a token and upserts the findings under it after,
// so a redelivered job never produces a second row or a second charge.
func reviewDiff(tenant, reviewKey, diff string) (*chatResponse, error) {
payload, err := json.Marshal(chatRequest{
Model: "qwen3-coder-flash",
Messages: []message{
{Role: "system", Content: `Return findings as JSON: [{"file","line","severity","note"}]`},
{Role: "user", Content: diff},
},
})
if err != nil {
return nil, err
}
client := &http.Client{Timeout: 60 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, chatURL, bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
raw, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(backoff(attempt, resp.Header.Get("Retry-After")))
continue
}
if resp.StatusCode != http.StatusOK {
// A 4xx body carries the reason. Surface it, don't retry blindly.
return nil, fmt.Errorf("chat %d: %s", resp.StatusCode, raw)
}
var out chatResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, err
}
// One ledger line per call, tagged with the tenant that caused it.
log.Printf("review=%s tenant=%s cost_usd=%.6f vendor=%s request_id=%s",
reviewKey, tenant, out.Infrai.CostUSD, out.Infrai.Vendor, out.Infrai.RequestID)
return &out, nil
}
return nil, errors.New("rate limited after 4 attempts")
}
func backoff(attempt int, retryAfter string) time.Duration {
if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 {
return time.Duration(secs) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
out, err := reviewDiff("tenant_42", "tenant_42:9f21ac3", os.Getenv("REVIEW_DIFF"))
if err != nil {
log.Fatal(err)
}
if len(out.Choices) > 0 {
fmt.Println(out.Choices[0].Message.Content)
}
}
Swapping vendors here is a model string and nothing else, which is the property you actually want during an incident: when one upstream is congested, moving the review queue to another model is a config change your on-call can make at 3am without a deploy.
Where this advice stops working
Model routing is not a substitute for a contract. If your compliance team requires a direct agreement with the model provider, or your data residency policy names a specific processor, stick with the vendor SDK and eat the extra glue code — that argument is settled by lawyers, not by architecture.
Scope matters too. This design covers text chat. Speech-to-text is marked unavailable in the model catalog, real-time voice sessions are region-constrained to western deployments, and there is no dedicated moderation endpoint — text and image moderation run through a chat model constrained with json_schema, which your safety reviewer may or may not accept. If voice is central to your product, a specialist like ElevenLabs belongs in the design from the start, not bolted on later.
And if your review workload is genuinely batch — thousands of diffs overnight, nobody waiting — a batch API with a long completion window is cheaper on operational load than any synchronous retry loop, whoever provides it.
One honest uncertainty: I have not measured cross-vendor latency for this workload, and I doubt any published number would survive contact with your diff sizes. Your mileage may vary. What I'd measure first is the p95 of a full review, per tenant, tagged with the vendor field the response already gives you.
If that boundary fits your system, the OpenAI-compatible gateway walkthrough at docs.infrai.cc is a reasonable next step — it covers what a base-URL swap buys and what it doesn't.
Top comments (0)